Title: Iterative Foundation Model Fine-Tuning on Multiple Rewards
Venue: NeurIPS 2025
Accepted: yes
Keywords: Language Model Fine-Tuning

RLHF

Drug Discovery
Abstract: Fine-tuning foundation models has emerged as a powerful approach for generating objects with specific desired properties. Reinforcement learning (RL) provides an effective framework for this purpose, enabling models to generate outputs that maximize a given reward function. However, in many applications such as text generation and drug discovery, it can be suboptimal to optimize using a single reward signal, as multiple evaluation criteria are often necessary. This paper proposes a novel reinforcement learning-based method for fine-tuning foundation models using multiple reward signals. By employing an iterative fine-tuning strategy across these rewards, our approach generalizes state-of-the-art RL-based methods. We further provide a theoretical analysis that offers insights into the performance of multi-reward RL fine-tuning. Experimental results across diverse domains including text, biological sequence, and small molecule generation, demonstrate the effectiveness of the proposed algorithm compared to state-of-the-art baselines.

Iterative Foundation Model Fine-Tuning on Multiple Rewards

Abstract

Fine-tuning foundation models has emerged as a powerful approach for generating objects with specific desired properties. Reinforcement learning (RL) provides an effective framework for this purpose, enabling models to generate outputs that maximize a given reward function. However, in many applications such as text generation and drug discovery, it can be suboptimal to optimize using a single reward signal, as multiple evaluation criteria are often necessary. This paper proposes a novel reinforcement learning-based method for fine-tuning foundation models using multiple reward signals. By employing an iterative fine-tuning strategy across these rewards, our approach generalizes state-of-the-art RL-based methods. We further provide a theoretical analysis that offers insights into the performance of multi-reward RL fine-tuning. Experimental results across diverse domains including text, biological sequence, and small molecule generation, demonstrate the effectiveness of the proposed algorithm compared to state-of-the-art baselines.

1 Introduction

Foundation models have emerged as powerful tools capable of performing a wide range of tasks. Trained on large-scale datasets, they acquire broad knowledge that enables their application across diverse domains. To better align a foundation model with the specific preferences of a downstream task, fine-tuning can be applied to improve both performance and task alignment. Given access to a reward model or a preference dataset, reinforcement learning offers an effective framework for finetuning foundation models and large language models (LLMs) to better align with downstream tasks [46, 41, 2]. Preference criteria used to evaluate the quality of responses generated by LLMs can vary, and in some cases, it may not be possible to derive a single reward or preference. Furthermore, these criteria can sometimes conflict with one another, making it difficult to summarize them into a single, unified preference metric. For example, human preferences can be diverse and conflicting with one another, such as the trade-off between harmlessness and helpfulness [3]. As another example, LLMs can be used to generate novel small molecules for drug design [50, 22, 35]. In such applications, candidate molecules are often evaluated based on multiple criteria [24, 44]. In such cases, fine-tuning foundation models on multiple objectives becomes essential.

Multi-objective reinforcement learning can be employed to address diverse rewards and preferences. Existing methods in the literature primarily follow two approaches. The first approach combines all reward signals corresponding to different objectives into a single scalar reward [31], which is then used to fine-tune the foundation model. The second approach involves fine-tuning the foundation model separately and independently for each objective to obtain a set of expert policy networks, each specialized for a specific objective. These expert policies are then merged to form a unified policy [42], effectively acting as an ensemble with the aim of capturing knowledge from all experts. However, combining reward signals into a single objective may prevent the model from learning objective-specific skills. This can result in high performance variance across objectives, particularly

when a minority of objectives conflict with a majority that are more similar. On the other hand, merging expert policies into a single policy may lead to suboptimal performance across some or all objectives, especially when there is significant divergence among the expert policies due to conflicting objectives.

This paper introduces a novel multi-objective reinforcement learning method for fine-tuning foundation models. To enable the model to acquire objective-specific skills, the proposed algorithm fine-tunes the foundation model separately for each objective, resulting in an expert policy network for each one. However, this fine-tuning is not performed independently. To control variance among the expert policies, the algorithm breaks the fine-tuning process into smaller steps and performs it iteratively . After each step, the expert policies are merged into a single policy, which is then used as the starting point for the next round of objective-specific fine-tuning. We show that the proposed method can be interpreted as a generalization of both reward-combining and expert-policymerging approaches. Furthermore, we analyze the convergence properties of the algorithm, providing theoretical insights into its performance. The contributions of this paper are summarized as follows:

We propose a novel and generalized algorithm that offers greater flexibility than rewardcombining and expert-merging baselines, leading to improved performance.

We provide a theoretical analysis of the proposed algorithm, offering insights into its properties.

We conduct experiments across diverse tasks including small molecule design, DNA sequence generation, and text summarization, to demonstrate the effectiveness of the proposed method.

2 Related Works

RLHF. Reinforcement learning with human feedback (RLHF) has been extensively studied in the literature and has demonstrated promising results across various applications [30, 51, 10, 21]. In the context of aligning foundation models with human preferences, RLHF emerges as a compelling approach, as it enables the model to interact with humans feedback to their preferences [26]. Several approaches have been proposed to improve the performance and efficiency of RLHF [13]. The safety of RLHF has been studied by [12]. The alignment of multimodal large language models with human preferences has been investigated by [56, 60]. However, these works typically assume that preferences can be captured using a single feedback signal. In practice, preferences can be diverse, and relying on a single signal may be insufficient to represent this variability.

Multi-Objective Reinforcement Learning. The problem of multi-objective optimization has attracted significant attention in reinforcement learning [20, 24]. Several studies have extended deep reinforcement learning techniques to address multi-objective problems [53, 1, 34]. However, focusing on a single mode of the reward function can limit the ability of multi-objective reinforcement learning methods to learn objective-specific skills and may reduce the diversity of the generated outputs. Moreover, when fine-tuning large foundation models, the scalability of multi-objective reinforcement learning becomes critical, potentially making traditional approaches unsuitable for such large-scale applications. To fine-tune foundation models on multiple objectives, the Rewarded Soups [42] method has been proposed. It follows an expert-merging approach, where a separate model is fine-tuned for each objective and then linearly combined to obtain a unified policy. To improve the performance of expert-merging methods particularly in molecular design applications, a more complex merging algorithm was introduced in [7].

Supervised Fine-Tuning. Multi-dimensional attributes can be used as conditioning signals for supervised fine-tuning of LLMs [14, 43]. This strategy has been applied to the problem of fine-tuning LLMs on multiple objectives in [52]. By appending the rewards associated with the objectives of interest to the prompts, supervised fine-tuning approach in [52] enables the LLM to learn the relationships between prompt-response pairs and the corresponding multi-objective reward space.

3 Preliminaries

This section defines the problem of fine-tuning language models with multiple objectives and reviews relevant approaches.

3.1 Multi-Objective Fine-Tuning Problem

Let π θ denote the policy of a language model parameterized by θ , and let π ref represent the initial (reference) policy of the model. Given a prompt x , the policy π θ generates a response y by sampling from the distribution π θ ( y | x ) . Suppose there are N objectives, R 1 , . . . , R N , where for each objective R i , the goal is to learn a policy π θ that minimizes the corresponding loss function L i ( π θ ) , defined as:

Each R i can represent an objective commonly used in reinforcement learning-based methods such as PPO or DPO. For example, in the context of Reinforcement Learning from Human Feedback (RLHF), assuming access to a reward model r ϕ parameterized by ϕ , the objective R i may be defined as:

where β ≥ 0 is a regularization coefficient that penalizes deviation from the reference policy, ensuring that the learned policy does not diverge excessively from π ref. Assume that the weight 0 < w i < 1 represents the preference for objective R i , where the weights satisfy ∑ N i =1 w i = 1 . In this work, we assume that the preference weights w i are known for each objective R i . Under this assumption, the problem of multi-objective model fine-tuning can be formulated as:

This optimization problem can be addressed using stochastic gradient descent (SGD) techniques. In the remainder of this section, we review two approaches for solving it.

3.2 Reward Combining

One approach to solving the optimization problem in equation 3 is to apply reinforcement learning with combined rewards. We refer to this approach as MORLHF in this paper. Let the policy π θ be optimized over T steps, with θ t denoting the policy parameters at step 1 ≤ t ≤ T . At each step t , MORLHF defines a combined single-objective loss as:

Using the loss function in equation 4, the parameters are updated via gradient descent as:

where η is the learning rate. It is worth noting that multi-objective reinforcement learning can be implemented in various ways through reward combination, with the formulation in equation 4 being just one of them.

3.3 Rewarded Soups

An alternative approach to solving the problem in equation 3 is the Rewarded Soups method. This technique optimizes the policy π θ over T steps with respect to each objective R i , yielding a set of parameters θ i . Specifically, at each step t , the parameters are updated as follows:

After T steps, the parameter θ i = θ i,T is obtained. The final policy π θ RS is formed by merging the set of expert policies { π θ i } N i =1 . Each π θ i is treated as an expert trained on objective R i , and the merged policy acts as an ensemble of these experts. A common merging strategy is to take a weighted linear combination of the parameters:

where each 0 ≤ λ i ≤ 1 is a weight associated with the i -th objective, satisfying ∑ N i =1 λ i = 1 . These weights λ i can be optimized to minimize the loss in equation 3. One approach is to randomly sample candidate weight sets using Monte Carlo methods and select the one yielding the lowest loss. However, this can be computationally expensive. A simpler and more efficient alternative is to set λ i = w i , thereby weighting each expert policy in proportion to its corresponding objective preference.

Comparing MORLHF (Subsection 3.2) with Rewarded Soups (Subsection 3.3), we observe key differences in their approaches. MORLHF optimizes a combined reward signal, aiming to directly learn a policy that balances multiple objectives. In contrast, Rewarded Soups trains separate expert policies for each objective and then constructs the final policy by merging these experts. Because MORLHF does not explicitly specialize in any individual objective, the resulting policy may exhibit high performance variance across different objectives. Conversely, while Rewarded Soups ensures that each expert is well-optimized for its corresponding objective, significant variance among the experts themselves can lead to a merged policy that performs poorly across all objectives.

4 Proposed Iterative Fine-Tuning with Multiple Objectives

As discussed in Section 3, MORLHF may exhibit high performance variance across objectives, while Rewarded Soups may experience significant variance among expert policies. This section introduces the proposed approach for fine-tuning models on multiple objectives. By iteratively training expert policies for individual objectives and merging them, the proposed method offers a principled way to mitigate both performance variance across objectives and variances among expert policies.

4.1 Algorithm

The proposed algorithm learns an expert policy corresponding to each reward. Let θ i,t denote the parameters of the policy associated with the i -th objective at optimization step t . Every m steps where m is an integer hyperparameter the expert policy parameters θ i,t are merged to produce an updated shared parameter vector ρ t . This merged parameter is then assigned to all expert policies, synchronizing them before continuing individual optimization. To reduce computational complexity, a subset of M ≤ N objectives can be selected uniformly at random at each merging step to update only the corresponding expert policy parameters between two merging steps. Let S t denote the set of indices for the selected objectives at step t . The update rule is defined as follows:

̸

̸

where t mod m denotes the remainder of t divided by m . Note that when t mod m = 0 , the subset remains unchanged, i.e., S t = S t -1 . Various strategies can be used to merge the policy parameters θ i,t to compute ρ t . For simplicity, we adopt a linear combination:

Furthermore, if t mod m = 0 , a new subset of objectives S t is selected by uniformly sampling M objectives at random. The weights λ i,t ≥ 0 can be determined using Monte Carlo methods, by sampling different sets of coefficients and selecting the one that minimizes the weighted loss. To reduce computational overhead, a simpler alternative is to fix the weights as

aligning them with predefined objective preferences. Algorithm 1 summarizes the proposed algorithm. Since every m steps involve a merging procedure similar to Rewarded Soups, we refer to the proposed method as IterativeRS , short for Iterative Rewarded Soups.

4.2 Analysis

This section analyzes the performance of IterativeRS. To gain a clearer understanding, we examine its convergence behavior in cases where the loss function L i ( π θ ) is convex with respect to θ . While this

Algorithm 1 IterativeRS: Iterative Multi-Objective Model Fine-Tuning

1: Input: Reference policy π ref, learning rate η , merge frequency m .

2: Initialize π θ i, 1 , ∀ i ∈ { 1 , . . . , N } as π ref ; S 0 by sampling M objectives uniformly.

3: for t = 1 , . . . , T do

4: Set S t = S t -1

5: if t mod m = 0 then

6: Merge policy weights { θ i,t } N i =1 to obtain the shared parameter ρ t as in equation 9.

7: Sample uniformly at random M objectives to update S t .

8: end if

9: For any objective i ∈ S t , update the policy parameter θ i,t as in equation 8.

10: end for

11: Merge all policy weights { θ i,T } N i =1 to obtain the shared parameter ρ T .

12: Output: Policy π ρ T .

convexity assumption may not hold in practical scenarios, the analysis provides valuable insight into the impact of hyperparameters on IterativeRS's performance. It is worth noting that MORLHF and Rewarded Soups can be viewed as special cases of IterativeRS by setting λ i,t = w i and optimizing all objectives at each step. According to Algorithm 1 and Subsections 3.2 and 3.3, setting m = 1 in IterativeRS recovers MORLHF described by equation 4, while setting m = T corresponds to Rewarded Soups.

The following assumptions are made for the analysis:

A 2. Loss functions L i ( · ) , ∀ i ∈ { 1 , . . . , N } are µ -strongly convex such that L i ( π θ 1 ) ≥ L i ( π θ 2 ) + ( θ 1 -θ 2 ) ⊤ ∇L i ( θ 2 ) + µ 2 ∥ θ 1 -θ 2 ∥ 2 , ∀ θ 1 , θ 2 .

A 3. Loss gradients are bounded from above as ∥∇L i ( π θ ) ∥ ≤ G , ∀ θ , ∀ i ∈ { 1 , . . . , N } .

Let the overall loss of a policy π θ be defined as

where L i ( π θ ) is defined in equation 1. Let θ ∗ denote the optimal policy parameters for the multiobjective loss, and let θ ∗ i denote the optimal policy parameters for the objective i , defined as

The following theorem provides a convergence bound for IterativeRS, with the proof presented in the Appendix A. The theorem is proved under the assumption that the merged policy parameter is computed as ρ t = N M ∑ i ∈ S t w i θ i,t where w i = 1 N , ∀ i ∈ { 1 , . . . , N } . The extension to non-uniform weights w i is straightforward and is discussed in Appendix A.

Theorem 1. Let the learning rate at step t is set as η t = 2 µ ( γ + t ) where γ = max { 8 L µ , m } -1 . Furthermore, let θ ref denote the policy parameter of the initial reference policy π ref . Under assumptions A 1-A 3, the performance gap of policy learned by IterativeRS with respect to the optimal policy π θ ∗ is bounded from above as:

where ∆ ∗ be defined as:

In what follows, the effects of the hyperparameters are analyzed using Theorem 1. It is important to note, however, that a tighter performance gap upper bound in equation 13, does not necessarily translate to better performance during deployment. It primarily reflects improved convergence during training and may increase the risk of overfitting. Therefore, while the theoretical analysis helps in understanding the impact of hyperparameters, practical performance should be monitored using a validation set.

Effects of π ref and ∆ ∗ . From equation 13, it can be inferred that decrease in ∥ θ ref -θ ∗ ∥ improves the performance gap upper bound. This suggests that initializing with a stronger reference policy yields a more effective fine-tuned policy. Furthermore, equation 13 shows that smaller ∆ ∗ results in tighter performance gap upper bound. A smaller ∆ ∗ can be achieved when the optimal policies corresponding to individual objectives exhibit less variation. Therefore, Theorem 1 suggests that greater similarity among objectives facilitates learning the optimal policy.

Choice of M and T . Using the bound in equation 13, it can be observed that increasing the number of selected objectives M leads to a tighter upper bound on the performance gap. This is expected, as learning over a larger set of objectives at each time step typically results in a better final policy. However, increasing M also increases the computational complexity. Similarly, equation 13 indicates that increasing the number of steps T tightens the upper bound on the performance gap, but at the cost of greater computational complexity. Thus, a trade-off arises between minimizing the performance gap and managing computational cost.

Choice of m. In order to understand the effect of m on the upper bound in equation 13, let break the upper bound into two terms A 1 and A 2 where

Given that γ = max { 8 L µ , m } , if m ≥ 8 L µ , increasing m can lead to a reduction in the term A 1 . On the other hand, increasing m is more likely to increase the term A 2 . The overall effect of m on the upper bound depends on which term dominates. Therefore, the impact of m is influenced by several factors, including the loss function and even the dataset, which may not be known a priori. As previously discussed, MORLHF and Rewarded Soups represent two extreme cases, where m = 1 (MORLHF) and m = T (Rewarded Soups). However, a moderate choice of m may yield the best trade-off. Therefore, it can be concluded that IterativeRS offers greater flexibility and potential for improvement by allowing arbitrary values of m .

5 Experiments

To evaluate the performance of IterativeRS, we conducted extensive experiments across a diverse set of tasks, including small molecule generation (Subsection 5.1), DNA sequence generation (Subsection 5.2), and text summarization (Subsection 5.3). We compare IterativeRS against stateof-the-art baselines: MORLHF [31], Rewarded Soups (RS) [42], and Rewards-in-Context (RiC) [52]. We assume that all objectives are equally important across all tasks, setting the weights to w 1 = w 2 = w 3 = 1 3 . It should be noted that the implementation of the MORLHF baseline in this section differs from the formulation presented in equation 4 in Subsection 3.2. To fine-tune models using IterativeRS, RS, and MORLHF, we employed PPO [46]. We evaluate the performance of each algorithm using the average rewards of its generated samples, both per objective and across all objectives. In addition, we report an inverse coefficient of variation (ICV) score to quantify performance consistency across objectives. For a given sample, the ICV score is defined as the average reward across all objectives divided by the standard deviation of those rewards. The average ICV score over S samples is computed as

where R j,i denotes the reward obtained by sample j on objective i , and std ( R j, 1 , . . . , R j,N ) represents the standard deviation of rewards across objectives. A higher ICV score indicates lower variability

alpha vs gap

gap vs uo

1.0

MORLHF

RS

1.6

X

MORLHF

RS

1.6

X

MORLHF

RS

0.9

I

+

RiC

Iterativers

1.4

RiC

Iterativers

1.4

RiC

Iterativers

0.8

1.2

1.2

de6

0.7

1.0

1.0

0.6

0.8

0.8

0.5

0.6

0.6

0.4

0.6

0.8

1.0

1.2

1.4

1.6

0.4

0.6

0.8

1.0

1.2

1.4

1.6

0.5

0.6

0.7

0.8

0.9

1.0

alpha

alpha

gap

and more balanced performance across objectives. Codes are available at https://github.com/ pouyamghari/IterativeRS .

5.1 Small Molecule Generation

The goal of this task is to generate small molecules that exhibit specific desirable energy properties. Specifically, the task involves generating molecules that (1) maximize polarizability ( α energy), (2) maintain a moderate HOMO-LUMO gap, and (3) minimize internal energy at 0 K ( U 0 ). To evaluate the properties of molecules generated by IterativeRS and the baseline methods, we use PAMNet [59] as the oracle model. PAMNet is specifically trained to predict molecular properties from the QM9 dataset. A GPT-2 model is pre-trained on SMILES representations of 2 million molecules from the MOSES dataset [40], resulting in a model referred to as MolGPT-2. This pre-trained model is then fine-tuned on the QM9 dataset [6, 45] to optimize for multiple objectives. To fine-tune models using IterativeRS, RS, and MORLHF, we employed PPO [46] with a reward model trained on the QM9 dataset. Rewards for each objective are normalized to the interval [0 , 1] using statistics computed from the training data. For RiC, supervised fine-tuning was performed using the QM9 dataset. To generate molecules, we first sample 10,000 SMILES representations using the fine-tuned model. Then, using RDKit, we construct 3D structures for each generated SMILES. Due to potential randomization in the 3D coordinates produced by RDKit, we generate 10 distinct 3D conformations for each SMILES. The resulting structures are then evaluated using PAMNet. More implementation details can be found in Appendix B.1.

Table 1 presents the performance of IterativeRS and other baseline methods in molecule generation. For IterativeRS, the merging frequency is set to m = 4 . Moreover, for all reinforcement learningbased approaches (IterativeRS, RS, and MORLHF) the number of optimization steps is set to T = 100 . To evaluate each method, we compute the Pareto front of the generated molecules and report the average reward for each objective within that front. If one generated molecule outperforms another (both produced by the same model) across all objectives, the former is said to dominate the latter and is included in the Pareto optimal set, while the dominated sample is considered suboptimal. As shown in Table 1, RL-based methods outperform RiC in terms of reward. This is likely because IterativeRS, RS, and MORLHF allow the pre-trained foundation model to interact with reward models during training, enabling it to explore and learn to generate higher-quality SMILES. In contrast, RiC relies solely on labeled data and lacks the exploration benefits provided by reinforcement learning. Since the distribution of pre-trained data differs from the labeled dataset, RL-based methods are better equipped to discover molecules with higher rewards than those present in the training set. Furthermore, as can be seen from Table 1, IterativeRS outperforms both MORLHF and RS in terms of average reward.

K562 vs HepG2

K562 vs SKNSH

HepG2 vs SKNSH

MORLHF

MORLHF

MORLHF

0.8

X

RS

RiC

0.8

X

RS

RiC

0.8

X

RS

RiC

Iterativers

Iterativers

Iterativers

0.6

0.6

0.6

HepG2

SKNSH

SKNSH

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.4

0.6

0.8

1.0

0.2

0.4

0.6

0.8

1.0

0.2

0.4

0.6

0.8

K562

K562

HepG2

Figure 1 presents scatter plots of the molecules generated by each method. Each subplot depicts the relationship between two objectives, with each point representing a molecule generated by the corresponding model. These plots illustrate how the generated molecules are distributed across the objective space. Notably, Figure 1 shows that the highest-scoring molecules are produced by IterativeRS. This is particularly important for molecule design, where the goal is often to identify a small number of molecules with optimal properties. These results highlight the effectiveness of IterativeRS in the small molecule generation task.

5.2 DNA Sequence Generation

The goal is to generate DNA sequences that exhibit desired regulatory activities in specific cell lines K562, HepG2 and SKNSH. To this end, a GPT-2 model referred to as DNAGPT-2 is pre-trained on approximately 700,000 unlabeled DNA sequences, each 200 base pairs long, from the MPRA dataset [18], comprising over 35 million tokens. The objective is to generate sequences with maximal regulatory activity across three different cell lines. For fine-tuning, we use a labeled subset of 100,000 sequences along with their corresponding activity measurements in the three target cell lines. To assess the quality of the generated sequences, we utilize the Malinois model [18] as an oracle predictor of regulatory activity. Rewards for each objective are normalized to the interval [0 , 1] using statistics computed from the training data. Each method generates 10,000 DNA sequences. More implementation details can be found in Appendix B.2.

Table 2 presents the performance of different algorithms in generating DNA sequences. For IterativeRS, the merging frequency is set to m = 8 , and for all reinforcement learning (RL)-based methods the number of optimization steps is fixed at T = 200 . For each method, the Pareto front of the generated DNA sequences is extracted based on their rewards across the objectives. As shown in Table 2, RiC achieves higher average reward scores than MORLHF and RS. Unlike the small molecule generation task, the distribution of data used to pre-train the foundation model aligns closely with the supervised training data for DNA sequences. As a result, RL-based methods provide less benefit in this setting compared to supervised fine-tuning. While IterativeRS achieves an average reward that is 1% higher than RiC, IterativeRS attains a 35% higher ICV score, indicating significantly greater consistency in performance across objectives. Moreover, IterativeRS outperforms both RS and MORLHF in terms of average reward.

Figure 2 presents scatter plots of DNA sequences generated by each method, with each subplot comparing the rewards of two objectives. The results indicate that sequences generated by RiC exhibit greater variability across objectives compared to those produced by IterativeRS. Notably,

faithful_score_normalized vs summary_score_normalized

faithful_score_normalized vs deberta_score_normalized

summary_score_normalized vs deberta_score_normalized

1.0

1.0

MORLHF

1.0

MORLHF

0.8

0.8

RiC

Iterativers

RS

0.8

RiC

Iterativers

RS

deberta_score_normalized

deberta_score_normalized

0.6

0.6

0.6

0.4

0.4

0.4

MORLHF

0.2

0.2

0.2

RS

RiC

Iterativers

0.0

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.2

0.4

0.6

80

1.0

faithful_score_normalized

faithful_score_normalized

summary_score_normalized

IterativeRS generates fewer DNA sequences with low rewards, demonstrating a more consistent performance compared to RiC.

5.3 Text Summarization

The task is to summarize Reddit posts. To accomplish this, we use Llama-3.2-3B-Instruct as the base model. This foundation model is fine-tuned on the Reddit Summary dataset [49] for the post summarization task. To evaluate the quality of the generated summaries, we employ three different reward models: bart-faithful-summary [9], gpt2-reward-summary 2 , deberta-v3 3 . The rewards assigned by bart-faithful-summary, gpt2-reward-summary, deberta-v3 are referred to as the faithful score, summary score, and deberta score, respectively. All reported rewards are normalized to the range [0 , 1] using statistics computed from the training dataset. The merging steps in RS and IterativeRS are performed using seven different sets of merging weights. For each set, a merged model is obtained, and the model that achieves the highest average reward according to the reward models is selected as the final merged model for the text summarization task. It is worth noting that one of the main differences between IterativeRS and RS is that, according to Algorithm 1, IterativeRS performs merging both during and after training, whereas RS merges the expert policies only once after training. More implementation details can be found in Appendix B.3.

Table 3 presents the performance of the algorithms on the text summarization task for Reddit posts. The merging frequency for IterativeRS is set to m = 40 , while the number of steps for all RL-based methods is T = 160 . For each generated summary, we computed the average of the faithful, summary, deberta, and ROUGE scores as the evaluation metric to incorporate a standard metric such as ROUGE in addition to the scores assigned by the reward models. This average is reported as Avg Score in the table. The ICV score is calculated using the faithful, summary, and deberta reward scores. The results in Table 3 show that IterativeRS outperforms the other baselines across all metrics. These findings indicate that employing IterativeRS can lead to improvements over RL-based approaches such as MORLHF and RS. It is also worth noting that RiC is a supervised fine-tuning (SFT) approach, unlike the RL-based methods. Although IterativeRS achieves higher scores than RiC, the superiority of RL-based approaches over SFT methods such as RiC is not universally generalizable and is influenced by the experimental conditions. Figure 3 shows scatter plots of the summaries generated by each method, with each subplot comparing two objectives. As seen in the figure, IterativeRS is less likely to produce responses with relatively low scores.

6 Conclusion

This paper introduced IterativeRS, an iterative multi-objective reinforcement learning algorithm for fine-tuning foundation models. IterativeRS fine-tunes a separate model for each objective to capture objective-specific knowledge, while mitigating divergence across expert models through an iterative merge-and-fine-tune strategy. The paper presents a theoretical analysis of the convergence properties of IterativeRS, offering deeper insight into its behavior. Furthermore, by formulating the problem as an optimization task, our work can potentially open new directions for improving multi-objective fine-tuning of foundation models. Experimental results across diverse tasks including small molecule generation, DNA sequence generation, and text summarization demonstrate that IterativeRS achieves higher average rewards compared to both MORLHF and Rewarded Soups.

References

Abbas Abdolmaleki, Sandy Huang, Leonard Hasenclever, Michael Neunert, Francis Song, Martina Zambelli, Murilo Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view on multi-objective policy optimization. In International conference on machine learning , pages 11-22, 2020.

Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740 , 2024.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. In Advances in Neural Information Processing Systems , volume 34, pages 27381-27394, 2021.

Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J. Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. Journal of Machine Learning Research , 24(210):1-55, 2023.

L. C. Blum and J.-L. Reymond. 970 million druglike small molecules for virtual screening in the chemical universe database GDB-13. J. Am. Chem. Soc. , 131:8732, 2009.

Diego Calanzone, Pierluca D'Oro, and Pierre-Luc Bacon. Mol-moe: Training preference-guided routers for molecule generation. arXiv preprint arXiv:2502.05633 , 2025.

Huili Chen, Jie Ding, Eric W Tramel, Shuang Wu, Anit Kumar Sahu, Salman Avestimehr, and Tao Zhang. Self-aware personalized federated learning. Advances in Neural Information Processing Systems , 35:20675-20688, 2022.

Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. Improving faithfulness in abstractive summarization with contrast candidate generation and selection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 5935-5941, June 2021.

Geoffrey Cideron, Sertan Girgin, Mauro Verzetti, Damien Vincent, Matej Kastelic, Zalán Borsos, Brian McWilliams, Victor Ungureanu, Olivier Bachem, Olivier Pietquin, Matthieu Geist, Léonard Hussenot, Neil Zeghidour, and Andrea Agostinelli. Musicrl: aligning music generation to human preferences. In Proceedings of the International Conference on Machine Learning , 2024.

Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In Proceedings of the International Conference on Machine Learning , volume 139, pages 2089-2099, Jul 2021.

Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In International Conference on Learning Representations , 2024.

Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research , 2023.

Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. Steerlm: Attribute conditioned sft as an (user-steerable) alternative to rlhf. arXiv preprint arXiv:2310.05344 , 2023.

Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In Advances in Neural Information Processing Systems , volume 33, pages 3557-3568, Dec 2020.

Pouya M. Ghari and Yanning Shen. Personalized federated learning with mixture of models for adaptive prediction and model fine-tuning. In Advances in Neural Information Processing Systems , 2024.

Pouya M. Ghari, Alex M Tseng, Gökcen Eraslan, Romain Lopez, Tommaso Biancalani, Gabriele Scalia, and Ehsan Hajiramezanali. GFlownet assisted biological sequence editing. In Advances in Neural Information Processing Systems , 2024.

Sager J Gosai, Rodrigo I Castro, Natalia Fuentes, John C Butts, Susan Kales, Ramil R Noche, Kousuke Mouri, Pardis C Sabeti, Steven K Reilly, and Ryan Tewhey. Machine-guided design of synthetic cell type-specific cis-regulatory elements. bioRxiv , 2023.

Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi. Federated learning with compression: Unified analysis and sharp guarantees. In International Conference on Artificial Intelligence and Statistics , pages 2350-2358, 2021.

Conor F Hayes, Roxana R˘ adulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems , 36(1), 2022.

Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without reinforcement learning. In International Conference on Learning Representations , 2024.

Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega-Guerrero, and Berend Smit. Leveraging large language models for predictive chemistry. Nature Machine Intelligence , 6(2):161-169, 2024.

Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garcia, Jarrid Rector-Brooks, Bonaventure F. P. Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, Lena Simine, Payel Das, and Yoshua Bengio. Biological sequence design with GFlowNets. In International Conference on Machine Learning , volume 162, pages 9786-9801, Jul 2022.

Moksh Jain, Sharath Chandra Raparthy, Alex Hernández-Garcıa, Jarrid Rector-Brooks, Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. In International conference on machine learning , pages 14631-14653, 2023.

Haoqiang Kang, Enna Sachdeva, Piyush Gupta, Sangjae Bae, and Kwonjoon Lee. Gflowvlm: Enhancing multi-step reasoning in vision-language models with generative flow networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3815-3825, June 2025.

Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312.14925 , 10, 2023.

Hyeonah Kim, Minsu Kim, Sanghyeok Choi, and Jinkyoo Park. Genetic-guided GFlownets for sample efficient molecular optimization. In Advances in Neural Information Processing Systems , 2024.

Michał Koziarski, Mohammed Abukalam, Vedant Shah, Louis Vaillancourt, Doris Alexandra Schuetz, Moksh Jain, Almer M. van der Sloot, Mathieu Bourgey, Anne Marinier, and Yoshua Bengio. Towards DNA-encoded library generation with GFlownets. In ICLR 2024 Workshop on Generative and Experimental Perspectives for Biomolecular Design , 2024.

Michał Koziarski, Andrei Rekesh, Dmytro Shevchuk, Almer van der Sloot, Piotr Gai´ nski, Yoshua Bengio, Cheng-Hao Liu, Mike Tyers, and Robert A. Batey. Rgfn: synthesizable molecular generation using gflownets. In Advances in Neural Information Processing Systems , 2024.

Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192 , 2023.

Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement learning for multiobjective optimization. IEEE Transactions on Cybernetics , 51(6):3103-3114, 2021.

Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In Proceedings of International Conference on Machine Learning , volume 139, pages 6357-6368, Jul 2021.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations , 2020.

Xi Lin, Zhiyuan Yang, and Qingfu Zhang. Pareto set learning for neural multi-objective combinatorial optimization. arXiv preprint arXiv:2203.15386 , 2022.

Xianggen Liu, Yan Guo, Haoran Li, Jin Liu, Shudong Huang, Bowen Ke, and Jiancheng Lv. Drugllm: Open large language model for few-shot molecule generation. arXiv preprint arXiv:2405.06690 , 2024.

Kanika Madan, Jarrid Rector-Brooks, Maksym Korablyov, Emmanuel Bengio, Moksh Jain, Andrei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability. In International Conference on Machine Learning , pages 23467-23483. PMLR, 2023.

Nikolay Malkin, Salem Lahlou, Tristan Deleu, Xu Ji, Edward J Hu, Katie E Everett, Dinghuai Zhang, and Yoshua Bengio. GFlownets and variational inference. In International Conference on Learning Representations , 2023.

Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal. Federated multi-task learning under a mixture of distributions. In Proceedings of International Conference on Neural Information Processing Systems , volume 34, pages 15434-15447, Dec 2021.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Proceedings of International Conference on Artificial Intelligence and Statistics , volume 54, pages 12731282, Apr 2017.

Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, Artur Kadurin, Simon Johansson, Hongming Chen, Sergey Nikolenko, Alan Aspuru-Guzik, and Alex Zhavoronkov. Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models. Frontiers in Pharmacology , 2020.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36:53728-53741, 2023.

Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems , 36:71095-71134, 2023.

Sahana Ramnath, Brihi Joshi, Skyler Hallinan, Ximing Lu, Liunian Harold Li, Aaron Chan, Jack Hessel, Yejin Choi, and Xiang Ren. Tailoring self-rationalizers with multi-reward distillation. arXiv preprint arXiv:2311.02805 , 2023.

Nian Ran, Yue Wang, and Richard Allmendinger. Mollm: Multi-objective large language model for molecular design-optimizing with experts. arXiv preprint arXiv:2502.12845 , 2025.

M. Rupp, A. Tkatchenko, K.-R. Müller, and O. A. von Lilienfeld. Fast and accurate modeling of molecular atomization energies with machine learning. Physical Review Letters , 108:058301, 2012.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Max W. Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving gflownet training. In International Conference on Machine Learning , 2023.

Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. In Advances in Neural Information Processing Systems , volume 30, pages 4424-4434, Dec 2017.

Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In Advances in Neural Information Processing Systems , 2020.

Ye Wang, Honggang Zhao, Simone Sciabola, and Wenlu Wang. cmolgpt: a conditional generative pre-trained transformer for target-specific de novo molecular generation. Molecules , 28(11):4430, 2023.

Yuanhao Wang, Qinghua Liu, and Chi Jin. Is rlhf more difficult than standard rl? a theoretical perspective. Advances in Neural Information Processing Systems , 36:76006-76032, 2023.

Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards-in-context: multi-objective alignment of foundation models with dynamic preference adjustment. In Proceedings of the International Conference on Machine Learning , 2024.

Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multiobjective reinforcement learning and policy adaptation. Advances in neural information processing systems , 32, 2019.

Adam Younsi, Abdalgader Abubaker, Mohamed El Amine Seddik, Hakim Hacid, and Salem Lahlou. Accurate and diverse llm mathematical reasoning via automated prm-guided gflownets. arXiv preprint arXiv:2504.19981 , 2025.

Fangxu Yu, Lai Jiang, Haoqiang Kang, Shibo Hao, and Lianhui Qin. Flow of reasoning: Training LLMs for divergent reasoning with minimal examples. In International Conference on Machine Learning , 2025.

Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and Tat-Seng Chua. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1380713816, June 2024.

Dun Zeng, Siqi Liang, Xiangjing Hu, Hui Wang, and Zenglin Xu. Fedlab: A flexible federated learning framework. Journal of Machine Learning Research , 24:1-7, 2023.

Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Fedala: Adaptive local aggregation for personalized federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 11237-11244, 2023.

Shuo Zhang, Yang Liu, and Lei Xie. A universal framework for accurate and efficient geometric deep learning of molecular systems. Scientific Reports , 13(1):19171, 2023.

Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, et al. Mm-rlhf: The next step forward in multimodal llm alignment. arXiv preprint arXiv:2502.10391 , 2025.

Yiheng Zhu, Jialu Wu, Chaowen Hu, Jiahuan Yan, Chang-Yu Hsieh, Tingjun Hou, and Jian Wu. Sample-efficient multi-objective molecular optimization with GFlownets. In Advances in Neural Information Processing Systems , 2023.

A Proof of Theorem 1

This section proves Theorem 1. The notations used in this section are summarized in Table 4. For simplicity of analysis, we assume that all data samples are used at each step, although the case where a random subset of data is sampled at each step can also be considered. In that case, Assumption A 3 can be modified to E [ ∥∇L i ( π θ ) ∥ ] ≤ G , where the expectation is taken with respect to the randomness in data sampling. Extending the results to the case with random data sampling is straightforward. Let ψ t be defined as ψ t = ∑ N i =1 w i θ i,t . Furthermore, let the merged policy parameter ρ t be defined as

̸

We can write

To obtain an upper bound on ∥ ψ t +1 -θ ∗ ∥ 2 , consider the Lemma 1. This lemma is taken from [33]. Lemma 1. Suppose Assumptions A 1 and A 2 hold. If η t ≤ 1 4 L , then the following inequality holds:

where ∆ ∗ = L ( π θ ∗ ) -∑ N i =1 w i L i ( π θ ∗ i ) .

Proof. Let g t be defined as g t := ∑ N i =1 w i ∇L i ( π θ i,t ) . It can be inferred that ψ t +1 = ψ t -η t g t . Therefore, we may write

From L -smoothness of L i stated in assumption A 1, it follows that

where θ ∗ i = arg min θ L i ( π θ ) . Using the above inequality and due to the convexity of ∥ · ∥ 2 , we can write

Furthermore, we can rewrite the term -2 η t g ⊤ t ( ψ t -θ ∗ ) in equation 20 as

Using AM-GM inequality, we can obtain

while due to µ -strong convexity of L i , we can write

Taking a weighted average over the objectives and applying Jensen's inequality to the right-hand side of equation 25, we obtain

Furthermore, taking a weighted average and applying the inequality in equation 21 to equation 24, we get

Combining equation 20 with equation 22, equation 23, equation 26 and equation 27, we arrive at

Taking the definition of ∆ ∗ into account, the last two terms in the right hand side of equation 28 can be rewritten as

To bound ∑ N i =1 w i ( L i ( π θ i,t ) -L i ( π θ ∗ ) ) , considering the convexity of L i , we can write

Applying AM-GM inequality to the right hand side of equation 30, we get

Applying the inequality in equation 21 to the right hand side of equation 31, we conclude that

Due the fact that 0 ≤ η t ≤ 1 4 L , it can be concluded that η t ≤ 2 η t (1 -2 η t L ) ≤ 2 η t . Multiplying both sides of equation 32 by -2 η t (1 -2 η t L ) , we obtain

The inequality in equation 33 can be rewritten as

Using the facts that L i ( π ψ t ) -L i ( π θ ∗ ) ≥ 0 , η t L -1 ≤ -3 4 and 1 -2 η t L ≤ 1 , from equation 34 we obtain

Combining equation 28 with equation 29 and equation 35 proves the Lemma.

Since IterativeRS merges every m steps, there exists t ′ such that t -t ′ < m and θ i,t ′ = ψ t ′ , ∀ i ∈ { 1 , . . . , N } . Considering the facts that ψ t ′ is the expected value of { θ i,t ′ } N i =1 , over distribution { w i } N i =1 and E ∥ X -E [ X ] ∥ 2 ≤ ∥ E [ X ] ∥ 2 , we can conclude that

Assume that η t is selected such that it is non-increasing and satisfies η t ≤ 2 η t + m , ∀ t . Taking the assumption A 3 into account, we can infer that

Combining equation 37 with equation 19, we get

Recall that IterativeRS in Algorithm 1 samples M objectives uniformly without replacement. Such a sampling scheme is unbiased and we can write

where E S t [ · ] denote the expectation with respect to sampling randomization. Therefore, taking expectation from equation 18 leads to

Combining equation 40 with equation 38, we get

̸

According to equation 17, if ( t + 1) mod m = 0 , it can concluded that E S t [ ∥ ρ t +1 -ψ t +1 ∥ 2 ] = 0 . If ( t + 1) mod m = 0 , considering the assumption that w 1 = . . . = w N = 1 N , the term E S t [ ∥ ρ t +1 -ψ t +1 ∥ 2 ] can be expressed as:

̸

Considering the facts that Pr[ i ∈ S t +1 ] = M N and Pr[ i, j ∈ S t +1 ] = M ( M -1) N ( N -1) and

we can rewrite equation 42 as

Using equation 36 and equation 37, from equation 44 we arrive at

Combining equation 45 with equation 41, we get

Define B as

With a step size chosen as η t = β t + γ for some β > 1 µ and γ > 0 satisfying η 1 ≤ min { 1 µ , 1 4 L } and η t ≤ 2 η t + m , using induction it can be proved that E S t [ ∥ ρ t -θ ∗ ∥ 2 ] ≤ v γ + t where

Since ρ t is an unbiased estimator of ψ t , we can conclude that E S t [ ∥ ρ t -θ ∗ ∥ 2 ] = ∥ ψ t -θ ∗ ∥ 2 . Definition of v ensures that ∥ ψ t -θ ∗ ∥ 2 ≤ v γ + t for t = 1 . Assume that ∥ ψ t -θ ∗ ∥ 2 ≤ v γ + t holds for t . Using equation 38, we can write

which proves that E S t [ ∥ ρ t +1 -θ ∗ ∥ 2 ] ≤ v γ + t +1 holds. Choosing β = 2 µ and γ = max { 8 L µ , m } -1 , we have η t = 2 µ ( γ + t ) . It can be verified that in this case η t ≤ 2 η t + m . Then, we can write

Combining equation 50 with equation 49 and the fact that E S t [ ρ t ] = ψ t , we get

According to smoothness assumption in A 1, we can write

Combining equation 52 with equation 51, we arrive at

Plugging in ψ 1 = θ ref in equation 53 proves the Theorem. Note that for the ease of notation we drop the expectation in Theorem 1. Furthermore, it should be noted that most of proof steps are taken from [33]. Furthrmore, it is useful to mention that Theorem 1 is proved for the case where w 1 = . . . = w N = 1 N . However, extension to non-uniform cases is straightforward. Define the scaled loss for the objective i as ˜ L i ( π θ ) = w i N L ( π θ ) . Then it can be concluded that L ( π θ ) = 1 N ∑ N i =1 ˜ L i ( π θ ) . Therefore, in that case the proof can be applied to scaled losses.

B Supplementary Experimental Results and Details

This appendix provides supplementary experimental results and implementation details.

B.1 Implementation Details for Small Molecule Generation

To fine-tune the MolGPT-2 model using MORLHF, RS, and IterativeRS, we employed PPO from the TRL library. For each objective, a reward model was trained using the labeled QM9 dataset. Each reward model consists of a MolGPT-2 backbone with a three-layer MLP head; only the MLP head was trained. The dataset was split into 80% training, 10% validation, and 10% test sets. All models were fine-tuned with a learning rate of 1 . 41 × 10 -5 using the Adam optimizer and a batch size of 128. The RiC baseline was configured with the same hyperparameters and settings as MORLHF, RS, and IterativeRS. We set p = 2 for RiC. Model training was conducted using four V100 GPUs. To perform merging for IterativeRS and RS, we average all objective-specific model weights. For IterativeRS the number of selected objectives was 3.

B.2 Implementation Details for DNA Sequence Generation

Similar to the molecule generation task, we fine-tuned the DNAGPT-2 model using MORLHF, RS, and IterativeRS with PPO from the TRL library. A subset of 100,000 samples was uniformly sampled from the MPRA dataset and evaluated using the Malinois model to obtain activity scores across three cell lines. This subset was used as the labeled dataset to train the reward models for PPO. For each objective, a separate reward model was trained using this labeled data. Each reward model consists of a DNAGPT-2 backbone with a three-layer MLP head, where only the MLP head was trained. The dataset was split into 70% training, 10% validation, and 20% test sets. All models were fine-tuned with a learning rate of 1 . 41 × 10 -5 using the Adam optimizer and a batch size of 128. The RiC baseline used the same hyperparameters and settings as MORLHF, RS, and IterativeRS. We set p = 2 for RiC. Model training was performed on four V100 GPUs. For IterativeRS the number of selected objectives was 3.

B.3 Implementation Details for Text Summarization

To fine-tune the Llama-3.2-3B-Instruct model using MORLHF, RS, and IterativeRS, we employed PPO from the TRL library. We first passed all prompt-response pairs from the Reddit dataset through three oracle reward models to construct a multi-labeled dataset. For each objective, a proxy reward model was trained using the dataset and the corresponding objective-specific labels. Each proxy model consists of a Llama-3.2-3B-Instruct backbone with a two-layer MLP head, with only the MLP head being trained. We used the training set from the Reddit dataset for training and randomly split its validation set into two subsets to serve as validation and test sets. The validation set was used both for training the proxy reward models and for supervised fine-tuning in RiC. During inference, prompts from the test set were provided to the fine-tuned models to generate text summaries. The maximum summary length was set to 32.

Before applying PPO fine-tuning, we first trained an SFT model. The PPO fine-tuning was then performed using this SFT model. We observed that initializing PPO with an SFT model leads to improved ROUGE scores in the generated summaries. To construct the SFT training dataset, for each prompt in the training set, we selected the summary that outperformed the alternative in the majority of objectives based on reward scores. SFT was performed for two epochs with a learning rate of 1 . 41 × 10 -6 using this constructed dataset. For fine-tuning with PPO, we selected a random rollout of 1,024 samples per epoch and used a batch size of 128. Each model was fine-tuned for 20 epochs using a learning rate of 1 . 41 × 10 -6 and the Adam optimizer. The RiC baseline was fine-tuned on the entire training set for 2 epochs, using the same learning rate and batch size. We set p = 2 for RiC. All other hyperparameters were kept consistent across MORLHF, RS, and IterativeRS. Training was conducted on four A100 GPUs. For IterativeRS, the number of selected objectives was set to 3. For both RS and IterativeRS, model merging was performed by selecting from seven candidate merged models obtained using seven different sets of merging weights. The model with the highest average reward, as evaluated by the reward models, was selected. The seven sets of merging weights consisted of [1 / 3 , 1 / 3 , 1 / 3] , all permutations of [1 / 6 , 1 / 6 , 2 / 3] , and all permutations of [1 / 6 , 5 / 12 , 5 / 12] . During training, at each merging step, we computed the reward of each merged model on 256 samples from the training data and selected the model with the highest average reward for the next iteration. After training, to obtain the final merged model, we evaluated all seven merged models on 1,024 validation samples and selected the one with the highest average reward. We then assessed its performance on the test set. Note that RS does not perform merging during training.

B.4 Supplementary Results

We performed additional experiments on the DNA sequence generation task to evaluate the performance of RL-based fine-tuning methods MORLHF, RS, and IterativeRS, using RLOO, which does not rely on a value model. The results are presented in Table 5. The results show that IterativeRS achieves a higher average reward than both MORLHF and RS when using RLOO.

To investigate the influence of merging on the performance of IterativeRS, we conducted supplementary experiments. For DNA sequence generation task, we considered ten different sets of merging weights, including [1 / 3 , 1 / 3 , 1 / 3] , permutations of [1 / 6 , 1 / 6 , 2 / 3] , permutations of [1 / 6 , 5 / 12 , 5 / 12] and permutations of [1 / 2 , 1 / 4 , 1 / 4] . To assess the performance of each merged model, we generated 8 , 192 samples per model and identified the Pareto-optimal sequences based on scores from reward

models. The final selection was based on the model that achieved the highest average reward score on its Pareto optimal sequences. This method is referred to as selective merging , whereas merging with uniform weights is referred to as fixed merging . Table 6 presents the results obtained using RLOO. The results suggest that selective merging offers no improvement over fixed merging. One possible reason for the limited effectiveness in the DNA sequence generation task is the evaluation method, which relies on Pareto-optimal samples generated by the model. This makes identifying the bestperforming model more challenging. During training, merging allows experts to transfer cross-task knowledge, which can help the final merged model generate higher-quality sequences. However, selecting the best merged model among candidates is difficult because it depends on evaluating the Pareto-optimality of generated sequences using reward models. These reward models are trained on limited data derived from an oracle model used during evaluation, leading to a performance gap between the reward models and the oracle. As a result, assessing Pareto-optimality using these reward models may not yield reliable outcomes.

To examine the effect of the merging strategy on both RS and IterativeRS in the molecule generation task, we applied MolMoE [7] to each method. MolMoE is an expert merging method designed for molecular applications, whereas IterativeRS focuses on expert training. Therefore, these two methods can be used in conjunction. We applied MolMoE to both IterativeRS and RS, and the results are presented in Table 7. As shown, IterativeRS with MolMoE outperforms RS with MolMoE across all objectives. Comparing Table 1 and Table 7, we observe that incorporating MolMoE improves the performance of RS across all objectives. While IterativeRS with MolMoE achieves nearly the same average reward as IterativeRS without MolMoE, it yields an 11% improvement in ICV. It is worth noting that one of the main advantages of using MolMoE is its ability to handle scenarios where preferences over objectives change dynamically over time, which is outside the scope of this paper's experimental study.

C Supplementary Related Works

Federated Learning. Federated learning involves a group of users, called clients, who collaborate with each other through communication with a central server to train a global model [39, 19, 57]. There is an analogy between federated learning and multi-objective reinforcement learning in the context of foundation model fine-tuning. In federated learning, the clients and the server work together to train a model that performs optimally across all clients' data. However, this can be challenging since the data may be distributed non-i.i.d. among clients, which similar to multiobjective reinforcement learning can lead to conflicting objectives during model training. To address this issue, several personalized federated learning algorithms have been proposed in the literature [48, 15, 11, 32, 38, 8, 58, 16].

GFlowNet. GFlowNets, initially proposed by [4], were introduced as a generative reinforcement learning framework designed to effectively handle scenarios with multiple paths leading to a common state. They have been widely applied to biological sequence [23, 28, 17] and molecule design [61, 29, 27] tasks, where their effectiveness has been well documented. In this paper, we focus on policy gradient-based methods such as PPO, due to their computational efficiency for foundation

model fine-tuning. It is also worth noting that several methods have been proposed in the literature to improve the learning efficiency of GFlowNets [5, 37, 36, 47]. Furthermore, GFlowNets have recently been utilized to enhance the reasoning capabilities of large language models and vision-language models [25, 55, 54].

D Societal Impact

In this paper, we addressed the problem of fine-tuning large language models (LLMs) on multiple objectives-a challenge with significant implications in areas such as small molecule design for drug discovery and biological sequence design. Methods that enable LLMs to generate molecules or biological sequences with desirable functionalities hold great promise for accelerating the discovery of new drugs and therapeutics, potentially benefiting society at large. However, we recognize the dual-use nature of this research. There is also the risk that such technologies could be misused or exacerbate existing health disparities, particularly among marginalized communities. As researchers, we underscore the importance of carefully considering both the societal benefits and the potential unintended consequences of this work. We remain optimistic that the broader impact of our contributions will lean toward positive, equitable outcomes.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly explained the main contributions in both introduction and abstract. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In the experimental section, we reported that some baselines perform better than ours in certain aspects.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In section 4.2, we provide the accurate definition of assumptions. We provide the complete proof for the Theorem in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provided the full implementation details in both the paper and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The datasets are publicly available, and we provide the code used to generate the experimental results in the corresponding GitHub repository of the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In Appendix B, we provided the implementation details including training and test details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the inverse coefficient of variation (ICV) scores in experiments section to provide information about statistical significance.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In Appendix B we explained our computer resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper conform, in every respect, with the NeurIPS Code of Ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discussed societal impact in Appendix D.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We believe the paper pose no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly cited all models and datasets used by the paper in the experimental section.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework for fine-tuning foundation models by leveraging multiple given rewards functions. The proposed approach unifies two existing techniques, Reward Combining and Reward Soups. Furthermore, the authors provide theoretical support for their method, including an upper bound on the error between the optimal policy and the policy trained using their framework.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality**:\nThe paper appears to be technically sound in its formulation. However, the claims made could be more robustly supported by the experimental results.\n\nStrengths: The theoretical derivation of an error bound is a positive aspect. The core idea of unifying Reward Combining and Reward Soups is presented.\n\nWeaknesses: The experimental validation seems limited. There is a need for more extensive experiments to demonstrate the stability and generalizability of the proposed method. For instance, the method shows promise primarily on biological tasks. The results on the text summarization task, however, indicate a marginal improvement only in average reward, while falling behind in other metrics, which is not really representative of the method’s utility. The work could benefit from a more thorough empirical evaluation.\n\n**Clarity**: The paper is generally well-organized, with a coherent structure and logical flow between sections.\n\nStrengths: The overall organization is good.\n\nWeaknesses: At times, the exposition could be clearer, making it somewhat challenging to fully grasp the rationale behind certain choices or implications of some arguments. (See Questions 1 and 4)\n\n**Significance**:\nThe potential impact of this work on the community is an area of concern.\n\nStrengths: The paper attempts to unify existing techniques, which can be valuable. The inclusion of theoretical analysis is commendable.\n\nWeaknesses: The proposed method appears to be a relatively straightforward combination of two prior techniques. For a method that builds upon existing ideas, especially if the combination seems intuitive, a comprehensive suite of experiments demonstrating clear advantages across diverse tasks would be expected. The current set of experiments (two biological tasks and one text-based task where performance is not compelling) does not fully meet this expectation. \n\n**Originality**:\nThe work does present a novel combination of existing methods.\n\nStrengths: To the best of my knowledge, the specific unified framework presented has not been proposed previously.\n\nWeaknesses: While the combination is new, the novelty would be more impactful if accompanied by surprising empirical results or deeper, unexpected theoretical insights.\n\nQUESTIONS:\nBelow are some questions and suggestions for the authors. Addressing these could help clarify certain aspects of the work and potentially strengthen the paper:\n\n1) The proof of Theorem 1 appears to utilize notation that differs from what is used in the main text. This inconsistency can create confusion for the reader. Would the authors consider revising the proof to maintain consistent notation with the rest of the paper? Also, clarity of Theorem 1 proof and Lemma 1: The proof of Theorem 1, especially the transition in notation around lines 432-435 and its connection to Lemma 1, could be made more accessible. The current presentation has some notational overlaps that harm comprehension. To improve clarity and consistency, please consider: 1.1) Adding a table of notations at the beginning of the appendix. 1.2) Providing a detailed proof of Lemma 1 using the same consistent notation as presented in the main text. 1.3) Breaking down the proof of Theorem 1 into more clearly delineated logical steps.\n\n2) The experiments and analysis seem to exclusively use weights $w_i = \\frac{1}{N}$. The paper mentions that other settings for these importance weights are possible. Have the authors investigated alternative parametrizations for $w_i$? Presenting results from such explorations could demonstrate the sensitivity of the method to this choice or some different behaviour.\n\n3) Have the authors considered comparing diversity-specific metrics (e.g., entropy, or other established measures) between their method and baselines? In addition, authors mention some GFlowNets baselines [1]. GFlowNet-based approaches are also good at generating more diverse candidates, have you considered a comparison with GFlowNets?\n\n4) In conclusion authors write: “We showed that both MORLHF and Rewarded Soups can be viewed as special cases of IterativeRS, highlighting its greater flexibility for performance enhancement.” Though, in my humble opinion, it is a little hidden, I would rather make a detailed section pinpointing this relation. Although, it is a minor issue of mine. \n\nReferences: \n\n[1] Jain et al. Multi-Objective GFlowNets. ICML 2023.\n\nLIMITATIONS:\nThe paper does not include a dedicated section discussing the potential limitations of the proposed algorithm. Here are my thoughts:\n\n1) The current experimental results do not conclusively demonstrate that the proposed method offers a significant advantage over standard fine-tuning baselines, particularly for tasks outside the biological domain. Further evidence would be needed to establish its superiority in a broader range of applications.\n\n2) While the paper unifies existing techniques, the extent to which this unification yields fundamentally novel insights beyond the combination itself is not fully clarified.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the fine-tuning problem of foundation models in multi-reward scenarios and proposes the IterativeRS algorithm. By iteratively fine-tuning expert policies for each objective and periodically merging parameters, it balances objective-specific skills and policy consistency. Theoretical analysis proves its convergence superiority over baselines such as MORLHF and Rewarded Soups under convex loss assumptions, providing a more flexible and effective solution for multi-objective reinforcement learning fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper proposes a novel IterativeRS method. Through iterative training and merging of expert policies, it provides a unique alternative to existing methods such as MORLHF, and addresses multi - objective optimization problems in a more principled manner.\n2. It conducts a comprehensive theoretical analysis of the convergence behavior of IterativeRS, which not only explains the characteristics of this method but also provides a theoretical basis for understanding related baseline methods.\n3. Extensive experiments are carried out in multiple tasks such as small - molecule generation, DNA sequence generation, and text summarization. By comparing with a variety of baseline methods, the advantages of IterativeRS in terms of average reward and cross - objective consistency are verified.\nWeakness\n1. The theoretical analysis relies on assumptions such as the convexity of the loss function, which may not hold in real - world scenarios. This leads to limited universality of the theoretical results, and the performance of the method in complex practical applications may deviate from theoretical predictions. \n2. The method involves iterative merging and multi - objective optimization, resulting in a relatively high computational complexity. Although the trade-offs of hyper-parameter tuning are discussed, it may still face challenges in terms of computational resources under complex configurations, restricting its practical application and promotion.\n3. Although the method is applied to the text summarization task, there is a lack of in - depth interpretation of the actual application scenarios of the generated summaries. It fails to analyze the specific impact of different reward model weights on the quality of summaries in combination with domain - specific indicators such as faithfulness and readability. \n4. The number of baseline methods used in the experiments is relatively small. Incorporating more baseline methods widely adopted in relevant sub - fields could lead to a more rigorous and comprehensive evaluation of IterativeRS.\n\nQUESTIONS:\nPlease refer to the above weakness.\n\nLIMITATIONS:\n1. Theoretical analysis relies on assumptions like convexity and smoothness of loss functions (e.g., A1–A3), but loss functions in practical multi-objective optimization are often non-convex, possibly causing deviations between theoretical performance bounds and actual performance when objectives conflict.  \n\n2. The mechanism of iteratively training expert policies and merging parameters leads to superlinear growth of computational cost with the number of objectives N and merging frequency m. For example, in DNA sequence generation experiments, the training time is about 3 times that of MORLHF, restricting large-scale applications.  \n\n3. Tasks such as text summarization do not analyze the impact of weight configurations on results by combining domain-standard metrics (e.g., ROUGE scores), and lack targeted discussions on practical scenarios (e.g., news summarization).  \n\n4. Experiments only compare methods like MORLHF and Rewarded Soups, without including emerging multi-objective optimization algorithms (e.g., Multi-Objective GFlowNets), and lack comparison with professional methods (e.g., Mol-MOE) in biological sequence generation, affecting the universality of conclusions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce an algorithm for performing fine-tuning on foundation models with multi-objective rewards. In particular, they propose a generalization of current methods which either (1) train separate models on each objective and merge parameters at the end or (2) train a single model on all parameters with a weighted reward function. The generalization simply performs an iterative merging procedure where separate models are trained in parallel for $m$ steps and then merged before continuing another round of independent training. They derive a theoretical bound on the performance gap in the training loss between their proposed method and the optimal policy for the combined reward function and offer some insights on hyperparameter selection based on this bound. Finally, they provide experimental justification for their method on three practical and pertinent fine-tuning domains, demonstrating out-performance of their method in terms of average reward across all domains.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n(1) The paper is very clearly written and easy to follow.\n(2) The experiments cover three diverse use cases and hence show a broad range of practical application domains for the proposed algorithm. The practical domains examined are relevant and important domains for the research community.\n(3) The authors provide both theoretical and experimental results for their proposed algorithm\n\nWeaknesses:\n(1) The results are not wholly convincing. From Figs 1-3, the only clear out-performance is versus RiC and it is somewhat unclear why this is a relevant baseline. It would be more interesting to see out-performance versus MORLHF and RS since, as the authors note, these are two extremes of the proposed IterativeRS algorithm. While the results in Tables 1-3 show that IterativeRS consistently outperforms in terms of average reward on the Pareto front, it is not exactly clear how to interpret this result (see questions below). Moreover, it is not clear if the out-performance is statistically significant since no confidence intervals are reported. I am also not clear on how to weigh the average reward results versus the Sharp-like score (and it is not clear what the Sharp-like score is measuring).I would be willing to increase my score if these issues are resolved.\n\n(2) The utility of the insights from the theoretical analysis seem limited. The theoretical results do not indicate any theoretical justification for IterativeRS over MORLHF or RS. The authors claim that the theoretical results provide insights into the behavior of these various algorithms, however, this claim is not clearly demonstrated. The authors themselves note that the results are limited to the training loss and do not apply to generalization capabilities which is of most interest. Moreover, the analysis of the hyperparameter choices, particularly the choice of $m$ do not seem to offer much practical use for selecting the hyperparameters. How was $m$ chosen in each of the three experimental cases? Was the theory provided leveraged to provide any insight into the choice of $m$ or was a hyperparameter sweep conducted?\n\nQUESTIONS:\nQuestions: \n\n(1 How do you compute the Pareto front? Are the results in Table 1-3 only for generated results on the Pareto front? Wouldn’t it be necessary to report how many results generated were on this front for each algorithm?\n\n(2) What exactly is the Sharp-like score? Is it better to be high or low? How should we balance the Sharp-like score versus the average reward?\n\n(3) How can we interpret Figs 1-3. e.g. in Line 265 “Notably, Figure 1 shows that the highest-scoring molecules are produced by IterativeRS” – From the plots, it is hard to varify this claim from the plots – the Iterative RS and MORLHF distributions seem to be mostly overlapping. This is true in most of the Fig 1-3 results.\n\nLIMITATIONS:\nThe authors do not discuss any limitations of their work. They do note a limitation of the theoretical analysis, that it does not apply to generalization results, but this should probably be restated in the conclusion. Other limitations may include the reliance on knowing the $w$ weights to balance the value of various rewards, which is impractical in most scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new multi-objective reinforcement learning method (IterativeRS) that generalizes both multi-objective RLHF (MORLHF) and Rewarded Soups. Existing MORLHF approaches typically optimize a single expert using a scalar-weighted sum of multiple objectives. This approach lacks expert diversity (since there’s only one expert) and is prone to collapsing toward the objective with the dominant reward signal—a well-known challenge in balancing multiple rewards. In contrast, the rewarded soups method trains separate experts for each objective and mixes their parameters post-training. While this allows for expert diversity, it can result in high variance and conflict between experts due to a lack of synchronization. The proposed method introduces **periodic synchronization** using the hyperparameter *m*, effectively balancing objective-level and expert-level variance. By tuning *m*, the method achieves stable performance across three tasks: QM9 molecular generation, DNA sequence design, and Reddit summarization.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The paper proposes an elegant algorithm that unifies two existing approaches to multi-objective RL: MORLHF and rewarded soups.\n- It highlights the importance of balancing trade-offs between expert-level and objective-level variance.\n- The work provides both theoretical insight and empirical validation.\n\n**Weaknesses:**\n\n- The method is only evaluated using PPO. It is unclear how well it generalizes to other RL methods, such as value-based approaches.\n- The scale of the applications is relatively small. It is questionable whether this method is necessary for fine-tuning foundation models, as the tasks could potentially be handled by training from scratch.\n\nQUESTIONS:\n- This method appears to be a general multi-objective RL approach, not limited to fine-tuning. Have you tried applying it to RL training from scratch?\n- In the context of LLM fine-tuning—currently the most common use case for RLHF—what are realistic scenarios where multiple objectives are necessary? Can this approach be applied effectively to tasks like text summarization?\n\nLIMITATIONS:\nthere is no section for the limitations, please include these.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper proposed a multi-objective reinforcement learning algorithm called IterativeRS to fine-tune foundation models on multiple reward signals. Specifically, the algorithm fine-tunes a separate "expert" policy for each objective to learn objective-specific skills , and then mitigates divergence between these experts by iteratively merging their parameters into a single shared policy, which is then used as the starting point for the next round of fine-tuning. The experiments on diverse tasks including small molecule generation, DNA sequence generation, and text summarization show that IterativeRS generally achieves higher average rewards.

STRENGTHS AND WEAKNESSES:
**Strength**

The proposed method generalizes existing methods into a single flexible framework that offers greater potential for performance improvement by controlling the trade-off between specialization and knowledge sharing.

**Weaknesses**

1. Marginal performance gains in text summarization. IterativeRS barely outperforms MORLHF on average reward and is outperformed by RS on the "faithful score" metric. This suggests that the proposed method might have limited applications.
2. The assumption A.3 is too strong and not commonly used in previous literature. The authors should provide a convincing reason on why this is realistic or necessary.
3. Following 2, the theoritical analysis might not be helpful for explaining the practicality of the proposed method.

QUESTIONS:
See weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-04