Iterative Foundation Model Fine-Tuning on Multiple Rewards
Authors: Pouya M. Ghari
simone sciabola
Ye Wang
Venue: NeurIPS 2025 poster ()
Keywords: Language Model Fine-Tuning
RLHF
Drug Discovery
Primary area: deep_learning

Abstract: Fine-tuning foundation models has emerged as a powerful approach for generating objects with specific desired properties. Reinforcement learning (RL) provides an effective framework for this purpose, enabling models to generate outputs that maximize a given reward function. However, in many applications such as text generation and drug discovery, it can be suboptimal to optimize using a single reward signal, as multiple evaluation criteria are often necessary. This paper proposes a novel reinforcement learning-based method for fine-tuning foundation models using multiple reward signals. By employing an iterative fine-tuning strategy across these rewards, our approach generalizes state-of-the-art RL-based methods. We further provide a theoretical analysis that offers insights into the performance of multi-reward RL fine-tuning. Experimental results across diverse domains including text, biological sequence, and small molecule generation, demonstrate the effectiveness of the proposed algorithm compared to state-of-the-art baselines.

Iterative Foundation Model Fine- on Multiple Rewards
Pouya M. Ghari Biogen
Simone Sciabola Biogen
Ye Wang" Biogen
Abstract
Fine-tuning foundation models has emerged as a powerful approach for generating objects with specific desired properties. Reinforcement learning (RL) provides an effective framework for this purpose, enabling models to generate outputs that maximize given reward function. However, in many applications such text generation and drug discovery, it can be suboptimal to optimize using a single re- ward signal, multiple evaluation criteria are often necessary. This paper proposes a novel reinforcement learning-bai based method for fine-tuning foundation models using multiple reward signals. By employing iterative fine-tuning strategy across these rewards, our approach generalizes state-of-the-arl RL-based methods. We further provide theoretical analysis that offers insights into the performance of multi-reward RL fine-tuning. Experimental results across diverse domains includ- ing text. biological sequence, and small molecule generation, demonstrate the effectiveness of the proposed algorithm compared to state-of-the-art baselines.
1 Introduction
Foundation models have emerged as powerful tools capable of performing a wide range of tasks. Trained on large-scale datasets, they acquire broad knowledge that enables their application across diverse domains. To better align a foundation model with the specific preferences of a downstream task, mine-tuning can be applied to improve both performance and task alignment. Given access reward model or a preference dataset, reinforcement learning offers an effective framework for fine- tuning foundation models and large language models (LLMs) better align with downstream tasks 46 412) (2]. Preference criteria used to evaluate the quality of responses generated by LLMs can vary, and in some cases, it may not be possible to derive single reward or preference. Furthermore, these criteria can sometimes conflict with one another, making it difficult to summarize them into a single, unified preference metric. For example, human preferences can be diverse and conflicting with one another, such as the trade-off between harmlessness and helpfulness [3. As another example, LLMss can used to generate novel small molecules for drug design (50
In such applications, In such cases, fine-tuning
candidate molecules are often evaluated based on multiple criteria 24 44
foundation models on multiple objectives becomes essential.
Multi-objective reinforcement learning can be employed to address diverse rewards and preferences. Existing methods in the literature primarily follow two approaches. The first approach combines all reward signals corresponding to different objectives into single scalar reward (3I), which is then used fine-tune the foundation model. The second approach involves tinettunig the foundation model separately and independently for each objective to obtain set of expert policy networks, each specialized for specific objective. These expert policies are then merged form unified policy [42), effectively acting an ensemble with the aim of capturing knowledge from all experts. However, combining reward signals into single objective may prevent the model from learning objective-specific skills. This can result in high performance variance across objectives, particularly
"Corresponding Author: ye. wen.abbnoogn..n com com
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

when a minority of objectives conflict with majority that are more similar. On the other hand, merging expert policies into a single policy may lead to suboptimal performance across some or all objectives, especially when there is significant divergence among the expert policies due to conflicting objectives.
This paper introduces a novel multi-objective reinforcement learning method for fine-tuning foun- dation models. To enable the model acquire objective-specific skills, the proposed algorithm fine-tunes the foundation model separately for each objective, resulting in an expert policy network for each one. However, this fine-tuning is not performed independently. To control variance among the expert policies, the algorithm breaks the fine-tuning process into smaller steps and performs iteratively. After each step, the expert policies are merged into a single policy, which then used as the starting point for the next round of objective-specife fine-tuning. We show that the proposed method can be interpreted as a generalization of both reward-combining and expert-policy- merging approaches. Furthermore, we analyze the convergence properties of the algorithm, providing theoretical insights into its performance. The contributions of this paper are summarized as follows:
We propose novel and generalized algorithm that offers greater flexibility than reward- combining and expert-merging baselines, leading to improved performance.
We provide theoretical analysis of the proposed algorithm, offering insights into its properties.
We conduct experiments across diverse tasks including small molccule design, DNA se- quence generation, and text summarization, to demonstrate the effectiveness of the proposed method.
2 Related Works
RLHE. Reinforcement learning with human feedback (RLHF) has been extensively studied in the
literature and has demonstrated promising results across various applications [30 [1 I0 21].. In the context of aligning foundation models with human preferences, RLHF emerges compelling approach, as it enables the model to interact with humans feedback to their preferences [26] Several
approaches have been proposed to improve the performance and efficiency of RLHF 13. The safety of RLHF has been studied by [12]. The alignment of multimodal large language models with human preferences has been investigated by [56 60]. However, these works typically assume that preferences can be captured using a single feedback signal. In practice, preferences can be diverse, and relying on single signal may be insufficient to represent this variability.
Multi-Objective Reinforcement Learning. The problem of multi-objective optimization has at- tracted significant attention in reinforcement learning (20 (24]. Several studies have extended deep reinforcement learning techniques to address multi-objective problems
[53 34]. However, focusing on a single mode of the reward function can limit the ability of multi-objective reinforcement learning methods to learn objective-spede sppecifcc skills and may reduce the diversity of the generated outputs. Moreover, when fine-tuning large foundation models. the scalability of multi-objective reinforcement learning becomes critical, potentially making traditional approaches unsuitable for such large-scale applications. To fine-tune foundation models on multiple objectives, the Rewarded Soups method has been proposed. It follows an expert-merging approach, where separate model is fine-tuned for each objective and then linearly combined to obtain a unified policy. To improve the performance of sxpert-merging methods particularly in molecular design applications, more complex merging algorithm was introduced in 77.
Supervised Fine- Tune-Tuning Multi-dimensional attributes can be used as conditioning signals for supervised finettnning of LLMs 14 43 This strategy has been applied the problem of fine-tuning
LLMs on multiple objectives in [52] By appending the rewards associated with the objectives
of interest to the prompts, supervised fine-tuning approach in [52] enables the LLM learn the relationships between prompt-response pairs and the corresponding multi-objective reward space.
3 Preliminaries
This section defines the problem of fine-tuning language models with multiple objectives and reviews relevant approaches.

---

3.1 Multi-Objective Fine-Tuning Problem
Let The denote the policy of language model parameterized by 0, and let Thr represent the initial (reference) policy of the model. Given prompt the policy generates response y by sampling from the distribution ( z). Suppose there are N objectives, Ri.... Ry. where for each objective RG, the goal is to learn policy no that minimizes the corresponding loss function Li(To). defined as:
(T( = - Bewponn y-uTT (2, To: Trer)]
Each R₂ can represent an objective commonly used in reinforcement learning caaning-based methods such as PPO or DPO. For example, in the context of Reinforcement Learning from Human Feedback (RLHF), assuming access to reward model rs parameterized by db, the objective R2 may defined as:
Te ( x))
Ri( y, me, Tret) (2 y) 3 log
Tyr a:)
where 3 2 0 is regularization coefficient that penalizes deviation from the reference policy, ensuring that the learned policy does not diverge excessively from Tef Assume that the weight wy < represents the preference for objective R,, where the weights satisfy Di=1 w, In this work, we assume that the preference weights wi are known for each objective Rg Under this assumption, the problem multi-objective model fine-tuning can formulated as:
0" arg min
(..o(
This optimization problem can be addressed using stochastic gradient descent (SGD) techniques. In the remainder this section, we review two approaches for solving
3.2 Reward Combining
One approach to solving the optimization problem in equation 3 is to apply reinforcement learning with combined rewards. We refer to this approach MORLHF in this paper. Let the policy me be optimized over 7 steps, with O, denoting the policy parameters at step < S T. At each step t, MORLHF defines combined single-objective loss as:
(TO.) = -E Paom gou
Σ w; (I no, Tret)
(4)
Using the loss function in equation 4 the parameters are updated via gradient descent as:
0 0++1 0. Vo
where v) is the learning rate. worth noting that multi-objective reinforcement learning can be implemented in various ways through reward combination, with the formulation in equation being just one of them.
3.3 Rewarded Soups
An alternative approach to solving the problem in equation is the Rewarded Soups method. This technique optimizes the policy нe over T steps with respect to each objective R,, yiclding a set of parameters O, Specifically, at each step the parameters are updated as follows:
(6) After T steps, the parameter 0, 0 is obtained. The final policy Thes is formed by merging the set of expert policies {TT, J.V. Each me, is treated as expert trained on objective R., and the merged policy acts as an ensemble of these experts. A common merging strategy is to take a weighted linear combination of the parameters:
0 = 0,,- 1/ v)V
N 0gs
O
(7)
3

---

where each 0 < A I is a weight associated with the i-th objective, satisfying > = = These weights X, can be optimized minimize the loss in equation One approach is to randomly sample candidate weight sets using Monte Carlo methods and select the one yielding the lowest loss. However, this can computationally expensive. A simpler and more efficient alternative is to set A, wy thereby weighting each expert policy in proportion to its corresponding objectíve preference.
Comparing MORLHF (Subsection 3.2 with Rewarded Soups (Subsection 3.3 we observe key differences in their approaches. MORLHF optimizes a combined reward signal, aiming to directly leam policy that balances multiple objectives. In contrast, Rewarded Soups trains separate expert policies for each objective and then constructs the final policy by merging these experts. Because MORLHF does not explicitly specialize in any individual objective, the resulting policy may exhibit high performance variance across different objectives. Conversely, while Rewarded Soups ensures that each expert is well-optimized for its corresponding objective, significant variance among the experts themselves can lead to a merged policy that performs poorly across all objectives.
Proposed Iterative Fine-Tuning with Multiple Objectives
As discussed in Section B MORLHF may exhibit high performance variance across objectives, while Rewarded Soups may experience significant variance among expert policies. This section introduces the proposed approach for fine-tuning models on multiple objectives. By iteratively training expert policies for individual objectives and merging them, the proposed method offers principled way to mitigate both performance variance across objectives and variances among expert policies.
4.1 Algorithm
The proposed algorithm learns an expert policy corresponding to each reward. Let O.., denote the parameters of the policy associated with the i-th objective at optimization step t. Every steps where m is an integer hyperparameter the expert policy parameters 0.,, are merged to produce an updated shared parameter vector Pt. This merged parameter is then assigned to all expert policies, synchronizing them before continuing individual optimization. To reduce computational complexity, subset of M < NN objectives can be selected uniformly at random at each merging step to update only the corresponding expert policy parameters between two merging steps. Let S. denote the set of indices for the selected objectives at step t. The update rule is defined as follows:
00.. Voo. if t mod m + e S,
0,,441 (p: - (T if mod m
(8)
where mod denotes the remainder of divided by m. Note that when t mod m * 0. the subset remains unchanged, i.e., Sy = S,-1- Various strategies can be used to merge the policy parameters 0,, to compute Pt. For simplicity, adopt linear combination:
Pt = Σ Ni.0Or. such that Σ Ni. V. 1 mod m 0.
XESE
Furthermore, if mod m 0, new subset of objectives S, is selected by uniformly sampling M objectives at random. The weights Xi, 2 can be determined using Monte Carlo methods, by sampling different sets of coefficients and selecting the one that minimizes the weighted loss. To reduce computational overhead, a simpler alternative fix the weights as
W/
Xi.t
(10)
Ejes, Wi
aligning them with predefined objective preferences. Algorithm I summarizes the proposed algorithm. Since every m steps involve a merging procedure similar to Rewarded Soups, we refer to the proposed method as IterativeRS, short for Iterative Rewarded Soups.
4.2 Analysis
This section analyzes the performance of IterativeRS. To gain clearer understanding, we examine its convergence behavior in cases where the loss function (iTe) ) is convex with respect to 0. While this

---

Algorithm IterativeRS: Iterative Multi-Objective Model Fine-Tuning
1: Input: Reference policy Tref, learning rate v), merge frequency m..
2: Initialize T Vi E {I,-.., N} as weff Sg by sampling M objectives uniformly.
3: for t = ,,..,,,'' do
4: Set S; S
5: if' mod m = 0 then
6: Merge policy weights {0 to obtain the shared parameter Pt as equation 9
7: Sample uniformly at random M objectives to update Sy
8: end if
9: For any objective i E St, update the policy parameter Og as in equation g
10: end for
11: Merge all policy weights (0,,) to obtain the shared parameter pT.
Output: Policy Rpr.
convexity assumption may not hold in practical scenarios, the analysis provides valuable insight into IterativeRS's performance. It is worth noting that MORLHF and
the impact of hyperparameters Rewarded Soups can be viewed as special cases IterativeRS by setting A w; and optimizing all objectives at each step. According to Algorithm T and Subsections 3.2 and 3.3 setting m 1 in IterativeRS recovers MORLHF described by equation while setting m corresponds to Rewarded Soups.
The following assumptions are made for the analysis:
1. Loss functions C.(-), Vi E (I,...,N) N} are L-smooth such that (.(TT))) S (.(TTe) + 1110
02ll?. Y0 02.
2. Loss functions C.(.). Vi (1, (1...., N) p-strongly convex such that Li( 2 (T(
(0: 02) (0,) + 1 - 02113, V01, O2.
A 3. Loss gradients are bounded from above [[ S G, VO, Vi (1.
Let the overall loss of a policy Te be defined as
N
((TO) = Σ (Ta)(
(11)
where (TT) is defined in equation| Let 0" denote the optimal policy parameters for the multi- objective loss, and 0r denote the optimal policy parameters for the objective defined
0t arg min [(Te),
arg min (TT))
(12)
0
The following theorem provides convergence bound for IterativeRS, with the proof presented in the Appendix A The theorem proved under the assumption that the merged policy parameter computed as Pt N Eses, w.G.., where w, N Vi E f1...., N). The extension non-uniform weights w; is straightforward and discussed in Appendix(A)
Theorem Let the learning rate at step set as 7), p(7+() where 7 max μ m.
1. Furthermore, let Ong denote the policy parameter of the initial reference policy Tef. Under assumptions [ 3 the performance gap of policy learned by IterativeRS with respect to the optimal policy =g- is bounded from above as:
4L
N M m² G2
C(Tor) - L(TT-) <
3LA* + +(22= 1)²
12(7 +T)
N M
7L
+ 2(7 + T)
110 12
(13)
where A be defined as:
A C(TTT)
Σ
wi (To))
(14)
5

---

In what follows, the effects of the hyperparameters are analyzed using Theorem 1 It is important to note, however, that a tighter performance gap upper bound in equation [3, does not necessarily translate to better performance during deployment. It primarily reflects improved convergence during training and may increase the risk of overfitting. Therefore, while the theoretical analysis helps in understanding the impact of hyperparameters, practical performance should be monitored using validation set.
Effects of गeर and A.. From equation 13 it can be inferred that decrease in [[0 0'll improves the performance gap upper bound. This suggests that initializing with a stronger reference policy yields a more effective fine-tuned policy. Furthermore, equation 13 shows that smaller A results in tighter performance gap upper bound. smaller A can be achieved when the optimal policies corresponding to individual objectives exhibit less variation. Therefore, Theorem I suggests that greater similarity among objectives facilitates learning the optimal policy.
Choice of M and T. Using the bound in equation 13. it can be observed that increasing the number tighter upper bound on the performance gap. This expected,
of selected objectives M leads as learning over a larger set of objectives at each time step typically results in a better final policy. However, increasing M also increases the computational complexity. Similarly, equation [3 indicates that increasing the number of steps 7 tightens the upper bound on the performance gap, but at the cost of greater computational complexity. Thus, a trade-off arises between minimizing the performance gap and managing computational cost.
Choice of m. In order understand the effect of m on the upper bound in equation 13 let break the upper bound into two terms A₁ and A2 where
12LA
A
(15a)
12(7+ TT
8L
N - M m² N
7L 2(7 + T) 10 1- (15b)
A2 1, + T)
22m 1)2
M C²
m). if m 2 SL increasing m can lead to a reduction in the term A1. On Given that y = max 26 is the other hand, increasing m is more likely to increase the term A2. The overall effect of m on the upper bound depends on which term dominates. Therefore, the impact of m is influenced by several factors, including the loss function and even the dataset, which may not be known a priori. As previously discussed, MORLHF and Rewarded Soups represent two extreme cases, where (MORLHF) and m = T (Rewarded Soups). However, a moderate choice m may yield the best trade-off. Therefore, it can be concluded that IterativeRS offers greater flexibility and potential for improvement by allowing arbitrary values of m.
1
Experiments
To evaluate the performance of IterativeRS, conducted extensive experiments across diverse set of tasks, including small molecule generation (Subsection [.1)). DNA sequence generation (Subsection (.2)). and text summarization (Subsection 5.3) We compare IterativeRS against state- of-the-art baselines: MORLHF (31), Rewarded Soups (RS) (2). and Rewards-in-Conter (RiC) [52. We assume that all objectives equally important across all tasks, setting the weights to W1 = W02 = W33 3- It should be noted that the implementation of the MORLHF baseline this section differs from the formulation presented in equation 4 in Subsection [322] To fine-tune models
using IterativeRS, RS, and MORLHE, we employed PPO (46). We evaluate the performance of each algorithm using the average rewards of its generated samples, both per objective and across all objectives. In addition, we report an inverse coefficient of variation (ICV) score to quantify performance consistency across objectives. For given sample, the ICV score is defined as the average reward across all objectives divided by the standard deviation of those rewards. The average ICV score over S samples is computed as
1
(R + + Rg,N) std ... R2,N)
ICV
(16)
where Ry,; denotes the reward obtained by sample j on objective i, and std( Ry....... RB,N) represents the standard deviation of rewards across objectives. A higher ICV score indicates lower variability

---

abota 242
apha
16
16
14
1.5
08
Figure 1: Pairwise scatter plots of generated molecules in the reward space for the three objectives.
Table 1: Average performance of Pareto-optimal molecules generated by multi-objective approaches.
at energy
gap
Un energy
Avg Reward
ICV
MORLHF
1.4229
0.9355
1.5146
1.2910
4.1883
RS
1.4134
0,9589
1.5464
1.3062
4.2674
RiC
0.5955
0.6795
0.7544
0.6765
3.7538
IterativRS
1.5893
0.9508
1.6649
1.4017
3.5854
and more balanced performance across objectives. Codes are available at htttts://ggthut: com/ pouyangarrs/eerativeen
5.1 Small Molecule Generation
The goal of this task to generate small molecules that exhibit specific desirable energy properties.
Specifically, the task involves generating molecules that (1) maximize polarizability (a energy), (2) maintain a moderate HOMO-LUMO gap, and (3) minimize internal energy at K (U)). To evaluate PAMNet [59]
the properties of molecules generated by IterativeRS and the bascline methods, as the oracle model. PAMNet is specifically trained to predict molecular properties from the QM9 dataset. A GPT-2 model is pre-trained SMILES representations of 2 million molecules from the MOSES dataset 40 resulting in a model referred to as MolGPT-2. This pre-trained model is then fine-tuned on the QM9 dataset 66 45 to optimize for multiple objectives. To fine-tune models using IterativeRS, RS, and MORLHE, we employed PPO 46 with a reward model trained on the QM9 dataset. Rewards for each objective normalized to the interval [0. 1] using statisties computed from the training data. For RiC, supervised fine-tuning was performed using the QM9 dataset. To generate molecules, we first sample 10,000 SMILES representations using the fine-tuned model. Then, using RDKit, we construct 3D structures for each generated SMILES. Due to potential randomization in the 3D coordinates produced by RDKit, we generate 10 distinct 3D conformations for each SMILES. The resulting structures are then evaluated using PAMNet. More implementation details can be found in Appendix B.
Table presents the performance of IterativeRS and other baseline methods in molecule generation. For IterativeRS, the merging frequency is set m 4. Moreover, for all reinforcement learning- based approaches (IterativeRS, RS,, and MORLHF) the number of optimization steps is set to T = 100. To evaluate each method, we compute the Pareto front of the generated molecules and report the average reward for each objective within that front. If one generated molecule outperforms another (both produced by the same model) across all objectives, the former is said dominate the latter and is included in the Pareto optimal set, while the dominated sample is considered suboptimal. As shown in Table [ RL-based methods outperform RiC in terms of reward. This likely because IterativeRS, RS, and MORLHF allow the pre- pre-trained foundation model to interact with reward models during training, enabling
explore and learn to generate nigher-quality SMILES. contrast, RiC relies solely on labeled data and lacks the exploration benefits provided by reinforcement learning. Since the distribution of pre-trained data differs from the labeled dataset, RL-based methods are better equipped to discover molecules with higher rewards than those present in the training set. Furthermore, as can seen from Table IterativeRS outperforms both MORLHF and RS in terms of average reward.
7

---

1260 ya Hap02
KO5D va SENSH
HepOS SCOSH
26
06
26
0.4
X002
K352
HE:02
Figure 2: Pairwise scatter plots of generated DNA sequences in the reward space for the three objectives.
Table 2: Average performance of Pareto-optimal DNA sequences generated by multi-objective approaches.
K562
HepG2
SKNSH
Avg Reward
ICV
MORLHF
0.2724
0.7096
0.7183
0.5667
3,1356
RS
0.3057
0.6808
0.7131
0.5666
3,8235
RiC
0.4221
0.6615
0.6688
0.5842
2.4672
IterativeRS
0.3032
0.7370
0.7378
0.5927
3.8310
Figure 1 presents scatter plots of the molecules generated by each method. Each subplot depicts the relationship between two objectives, with each point representing molecule generated by the corresponding model. These plots illustrate how the generated molecules are distributed across the objective space. Notably, Figure siowre[[shows shows that the highest-scoring molecules are produced by IterativeRS. This is particularly important for molecule design, where the goal is often to identify a small number of molecules with optimal properties. These results highlight the effectiveness of IterativeRS in the small molecule generation task.
5.2 DNA Sequence Generation
The goal generate DNA sequences that exhibit desired regulatory activities in specific cell lines K562, HepG2 and SKNSH. To this end, GPT-2 model referred to as DNAGPT-2 is pre-trained on approximately 700,000 unlabeled DNA sequences, each 200 base pairs long, from the MPRA dataset [88. comprising over 35 million tokens. The objective is to generate sequences with maximal regulatory activity across three different cell lines. For fine-tuning, we use a labeled subset of 100,000 sequences along with their corresponding activity measurements in the three target cell lines. To assess the quality of the generated sequences, we utilize the Malinois model [88 as an oracle predictor of regulatory activity. Rewards for each objective are normalized to the interval [0, 1] using statistics computed from the training data. Each method generates 10,000 DNA sequences. More implementation details can be found in Appendix B.2
Table presents the performance of different algorithms in generating DNA sequences. For Iter- ativeRS, the merging frequency is set to = 8, and for all reinforcement learning (RL)-based methods the number of optimization steps fixed = 200. For each method, the Pareto front of the generated DNA sequences is extracted based on their rewards across the objectives. As shown in Table 2, RiC achieves higher average reward scores than MORLHF and RS. Unlike the small molecule generation task, the distribution of data used to pre-train the foundation model aligns closely with the supervised training data for DNA sequences. As a result, RL-based methods provide less benefit in this setting compared to supervised fine-tuning. While IterativeRS achieves an average reward that is 1% higher than RiC, IterativeRS attains a 35% higher ICV score, indicating significantly greater consistency in performance across objectives. Moreover, IterativeRS outperforms both RS and MORLHF in terms of average reward.
Figure 2 presents scatter plots of DNA sequences generated by each method, with each subplot comparing the rewards two objectives. The results indicate that sequences generated by Ric exhibit greater variability across objectives compared to those produced by IterativeRS. Notably,

---

rovaired
SL FOVER onherte rOVTRIEN
32
24
05
10
09
54
20
32
64
Figure 3: Pairwise scatter plots of generated summaries in the reward space for the three objectives.
Table 3: Average performance of text summarization by multi-objective approaches.
faithful
summary
deberta
Avg Score
ICV
MORLHF
0.6530
0.5778
0.3857
0).4525
4.5500
RS
0.6732
0.5807
0.4296
0.4732
4.5870
RiC
0.6497
0.5688
0.3455
0.4518
3,9579
IterativeRS
0.6927
0.5854
0.4398
0.4849
4.9134
IterativeRS generates fewer DNA sequences with low rewards, demonstrating a more consistent performance compared to RiC.
5.3 Text Summarization
The task is to summarize Reddit posts. To accomplish this, we use Llama-3.2-3B. -nsstuuttt as the base model. This foundation model is fine-tuned on the Reddit Summary dataset 49 for the post summarization task. To evaluate the quality of the generated summaries, we employ three different reward models: bart-faithful-sune -ummmurry 9 assigned by bart-faitruul-summar
wewaadd-wummaaeee D deberta-v3 | The rewards sp12-rewardssummarr deberta-v3 are referred to as the faithful score, summary score, and deberta score, respectively. All reported rewards are normalized to the range [0, 1] using statistics computed from the training dataset. The merging steps in RS and IterativeRS are performed using seven different sets of merging weights. For each set, a merged model is obtained, and the model that achieves the highest average reward according to the reward models selected as the final merged model for the text summarization task. is worth noting that one of the main differences between IterativeRS and RS is that, according to Algorithm T IterativeRS performs merging both during and after training, whereas RS merges the expert policies only once after training. More implementation details can be found in Appendix B.3
Table 3 presents the performance the algorithms on the text summarization task for Reddit posts. The merging frequency for IterativeRS is set to m 40, while the number of steps for all RL-based methods is T 160. For each generated summary, we computed the average of the faithful, summary, deberta, and ROUGE scores as the evaluation metric incorporate standard metric such ROUGE in addition to the scores assigned by the reward models. This average is reported as Avg Score the table. The ICV score is calculated using the faithful, summary. and deberta reward scores. The results in 3 show that IterativeRS outperforms the other baselines across all metrics. These findings indicate that employing IterativeRS can lead to improvements over RL-based approaches such as MORLHF and RS. It is also worth noting that RiC is a supervised fine-tuning (SFT) approach, unlike the RL-based methods. Although IterativeRS achieves higher scores than RiC, the superiority of RL-based approaches over SFT methods such as RiC is not universally generalizable and influenced by the experimental conditions. Figure shows scatter plots of the summaries generated by each method, with each subplot comparing two objectives. As seen in the figure, IterativeRS is less likely produce responses with relatively low scores.
http://waaaaaaatccoooooossssssse 13-laree-v2

---

6 Conclusion
This paper introduced IterativeRS, an iterative multi-objective reinforcement learning algorithm for fine-tuning foundation models. IterativeRS fine-tunes separate model for each objective to capture objective-specific knowledge, while mitigating divergence across expert models through iterative merge-and-fine-ture fin- strategy. The paper presents a theoretical analysis of the convergence properties of IterativeRS, offering deeper insight into its behavior. Furthermore, by formulating the problem as an optimization task, our work can potentially open new directions for improving multi-objective fine-tuning of foundation models. Experimental results across diverse tasks including small molecule generation, DNA sequence generation, and text summarization demonstrate that IterativeRS achieves higher average rewards compared to both MORLHF and Rewarded Soups.
References
[1] Abbas Abdolmaleki, Sandy Huang, Leonard Hasenclever, Michael Neunert, Francis Song, Martina Zambelli, Murilo Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view multi-objective policy optimization. In International conference machine learning, pages 11 11-22, 2020.
[2] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback 11ms. arXiv preprint arXiv:2402. 14740, 2024.
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862
2022.
[4] Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. In Advances in Neural Information Processing Systems, volume 34, pages 27381-27394 2021.
[5] Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J. Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations. Journal of Machine Learning Research, 24(210):1-55, 2023.
[6] L. C. Blum and J.-L. Reymond. 970 million druglike small molecules for virtual screening in the chemical universe database GDB-13. J. Am. Chem. Soc., 131:8732, 2009.
[7] Diego Calanzone, Pierluca D'Oro, and Pierre-Luue Bacon. Mol-moe: Training iifference-guideen routers for molecule generation. arXiv preprint arXiv:2502.056333 2025.
[8] Huili Chen, Jie Ding, Eric W Tramel, Shuang Wu, Anit Kumar Sahu, Salman Avestimehr, and Tao Zhang. Self-aware personalized federated learning. Advances in Neural Information Processing Systems, 35:20675-20668 2022.
[9] Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. Improving faithfulness in abstractive summarization with contrast candidate generation and selection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5935-5941, June 2021.
[10] Geoffrey Cideron, Sertan Girgin, Mauro Verzetti, Damien Vincent, Matej Kastelic, Zalán Borsos, Brian McWilliams, Victor Ungureanu, Olivier Bachem, Olivier Pietquin, Matthieu Geist, Léonard Hussenot, Neil Zeghidour, and Andrea Agostinelli. Musicrl: aligning music generation to human preferences. In Proceedings of the International Conference on Machine Learning, 2024.
[11] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared rep- resentations for personalized federated learning. In Proceedings of the International Conference on Machine Learning, volume 139, pages 2089- 2099, Jul 2021.
[12] Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback, In International Conference Learning Representations, 2024.
10

---

June [13] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023.
[14] Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. Steerlm: Attribute conditioned sft as (user-steerable) alternative rlhf. arXiv preprint arXiv:2310.05344, 2023.
[15] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In Advances in Neural Information Processing Systems, volume 33, pages 3557-3568, Dec 2020.
[16] Pouya M. Ghari and Yanning Shen. Personalized federated learning with mixture of models for adaptive prediction and model fine-tuning. Advances in Neural Information Processing Systems, 2024.
[17] Pouya M. Ghari, Alex M Tseng, Gökcen Eraslan, Romain Lopez, Tommaso Biancalani, Gabriele Scalia, and Ehsan Hajiramezanali. GFlownet assisted biological sequence editing. In Advances in Neural Information Processing Systems, 2024.
[18] Sager Gosai, Rodrigo I Castro, Natalia Fuentes, John C Butts, Susan Kales, Ramil R Noche, Kousuke Mouri, Pardis C Sabeti, Steven K Reilly, and Ryan Tewhey. Machine-guided design of synthetic cell type-specific cis-regulatory elements. bioRxiv, 2023.
[19] Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi. Federated learning with compression: Unified analysis and sharp guarantees. In International Conference Artificial Intelligence and Statistics, pages 2350-2358, 2021.
[20] Conor F Hayes, Roxana Rādulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36(1), 2022.
[21] Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W. Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without reinforcement learning. In International Conference on Learning Representations, 2024.
[22] Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega-Guerrery and Berend Smit. Leverag- ing large language models for predictive chemistry. Nature Machine Intelligence, 6(2):161-169, 2024.
[23] Moksh Jain, Emmanuel Bengio, Alex Hernandez-Gareia. Jarrid Rector- Rector-Brooks, Bonaventure F. P. Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, Lena Simine, Payel Das, and Yoshua Bengio. Biological sequence design with GFlowNets. In International Conference on Machine Learning, volume 162, pages 7786-9801, Jul 2022.
[24] Moksh Jain, Sharath Chandra Raparthy, Alex Hernández-Gareia, Jarrid Rector-Brooke Yoshua Bengio, Santiago Miret, and Emmanuel Bengio. Multi-objective gflownets. In International conference on machine learning, pages 14631 14631-14653, 2023.
[25] Haoqiang Kang, Enna Sachdeva, Piyush Gupta, Sangjae Bae, and Kwonjoon Lee. Gflowylm: Enhancing multi-step reasoning in vision-language models with generative flow networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2025.
[26] Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312.14925, 10, 2023.
[27] Hyeonah Kim, Minsu Kim, Sanghyeok Choi, and Jinkyoo Park. Genetic-guide GFlownets for sample efficient molecular optimization. In Advances in Neural Information Processing Systems, 2024.
11

---

[28] Michal Koziarski, Mohammed Abukalam, Vedant Shah, Louis Vaillancourt, Doris Alexandra Schuetz, Moksh Jain, Almer M. van der Sloot, Mathieu Bourgey, Anne Marinier, and Yoshua Bengio. Towards DNA-encoded library generation with GFlownets. In ICLR 2024 Workshop on Generative and Experimental Perspectives for Biomolecular Design, 2024.
[29] Michat Koziarski, Andrei Rekesh, Dmytro Shevchuk, Almer van der Sloot, Piotr Gaiński, Yoshua Bengio, Cheng- thong-Hao Liu, Mike Tyers, and Robert A. Batey. Rgfn: synthesizable molecular generation using gflownets. In Advances in Neural Information Processing Systems, 2024.
[30] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint mxXiv2230..12199 2023.
[31] Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement learning for multiobjective opti- mization. IEEE Transactions on Cybernetics, 51(6): 2021.
[32] Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In Proceedings of International Conference on Machine Learning, volume 139, pages 6357-6368, Jul 2021.
[33] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, 2020.
[34] Xi Lin, Zhiyuan Yang, and Qingfu Zhang. Pareto set learning for neural multi-objective combinatorial optimization. arXiv preprint arXiv:2203. 15386, 2022.
[35] Xianggen Liu, Yan Guo, Haoran Li, Jin Liu, Shudong Huang, Bowen Ke, and Jiancheng Lv. Drugllm: Open large language model for few-shot molecule generation. arXiv preprint arXiv:2405.0666 06690, 2024.
[36] Kanika Madan, Jarrid Rector-Brooks Maksym Korablyov, Emmanuel Bengio, Moksh Jain, Andrei Cristian Nica, Tom Bosc, Yoshua Bengio, and Nikolay Malkin. Learning gflownets from partial episodes for improved convergence and stability. In International Conference Machine Learning, pages 23467-23483. PMLR, 2023.
[37] Nikolay Malkin, Salem Lahlou, Tristan Deleu, Xu Ji, Edward J Hu, Katie E Everett, Dinghuai Zhang, and Yoshua Bengio. GFlownets and variational inference. In International Conference on Learning Representations, 2023.
[38] Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal. Federated multi-task learning under mixture of distributions. In Proceedings of International Conference on Neural Information Processing Systems, volume 34, pages 15434-15447, Dec 2021.
[39] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- Efficiicttttnnnffiiettt Learning of Deep Networks from Decentralized Data. In Proceedings of International Conference Artificial Intelligence and Statistics, volume 54, pages 1273- 1282, Apr 2017.
[40] Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-LLengeling Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, Artur Kadurin, Simon Johansson, Hongming Chen, Sergey Nikolenko, Alan Aspuru-Guzik, and Alex Zhavoronkov. Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models. Frontiers in Pharmacology, 2020.
[41] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. 2023.
Advances in Neural Information Processing Systems,
[42] Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances Neural Information Processing Systems, (66771095-71144 2023.
12

---

[43] Sahana Ramnath, Brihi Joshi, Skyler Hallinan, Ximing Lu, Liunian Harold Li, Aaron Chan, Jack Hessel, Yejin Choi, and Xiang Ren. Tailoring saffrrationalierss with multi-reward distillation. arXiv preprini arXiv:231 1.t::33/1.02005.. 2023.
[44] Nian Ran, Yue Wang, and Richard Allmendinger. Mollm: Multi-objective large language model for molecular design-optimizing with experts. arXiv preprini arXiv:2502. 2245, 2025.
[45] M. Rupp, A. Tkatchenko, K.-R. Müller, and O. A. von Lilienfeld. Fast and accurate modeling of molecular atomization energies with machine learning. Physical Review Letters, 108:058301, 2012.
[46] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
[47] Max W. Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and Tommaso Biancalani. Towards understanding and improving gflownet training. In International Conference on Machine Learning. 2023.
[48] Virginia Smith, Chao-Kai Chiang. Maziar Sanjabi, and Ameet S Talwalkar, Federated multi-task learning. In Advances in Neural Information Processing Systems, volume 30, pages 4424-4434, Dec 2017.
[49] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In Advances in Neurul Information Processing Systems, 2020.
[50] Ye Wang, Honggang Zhao, Simone Sciabola, and Wenlu Wang. cmolgpt: a conditional generative tre-trd transformer for target-specific de novo molecular generation. Molecules, 28(11): 2023.
[51] Yuanhao Wang, Qinghua Liu, and Chi Jin. Is rlhf more difficult than standard rI? a theoretical perspective. Advances in Neural Information Processing Systems, 11776006-76032. 2023.
[52] Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong Dong Yu, and Jianshu Chen. -eworet-in-comtext: multi-objective alignment of foundation models with dynamic preference adjustment. In Proceedings of the International Conference on Machine Learning, 2024.
[53] Runzhe Yang. Xingy uan Sun, and Karthik Narasimhan. A generalized algorithm for multi- objective reinforcement learning and policy adaptation. Advances in neural information pro- cessing systems, 32, 2019.
[54] Adam Younsi, Abdalgader Abubaker, Mohamed El Amine Seddik, Hakin Hacid, and Salem Lahlou. Accurate and diverse I1m mathematical reasoning via automated prm-guided gflownets. arXiv preprint arXiv:2504. 19981, 2025.
[55] Fangxu Yu, Lai Jiang, Haoqiang Kang, Shibo Hao, and Lianhui Qin. Flow of reasoning: Training LLMs for divergent reasoning with minimal examples. In International Conference Machine Learning, 2025.
[56] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng. Maosong Sun, and Tat-Seng Chua. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13807- 13816, June 2024.
[57] Dun Zeng, Siqi Liang, Xiangjing Hu, Hui Wang, and Zenglin Xu. Fedlab: A flexible federated learning framework. Journal of Machine Learning Research, 24:1-7, 2023.
[58] Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Fedala: Adaptive local aggregation for personalized federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1 11237-11244, 2023.
[59] Shuo Zhang, Yang Liu, and Lei Xie. A universal framework for accurate and efficient geometric deep learning of molecular systems. Scientific Reports, 1311) 2023.
13

---

[60] Yi-Fan Zhang. Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, al. Mm-rlhf: The next step forward in multimodal Ilm alignment. arXiv preprint arXiv:2502. 10391, 2025.
[61] Yiheng Zhu, Jialu Wu, Chaowen Hu, Jiahuan Yan, Chang- Yu Hsieh, Tingjun Hou, and Jian Wu. Sample-efficient multi oojeti-o molecular optimization with GFlownets. Advances in Neural Information Processing Systems, 2023.
14

---

A Proof of Theorem 1
This section proves Theorem | The notations used in this section are summarized in Table For simplicity of analysis, we assume that all data samples are used at each step, although the case where random subset of data sampled each step can also be considered. In that case, Assumption AB can be modified to ( in data sampling. Extending the results to the case with random data sampling is straightforward. Let bh be defined as w/ = Σ w,Or.. Furthermore, let the merged policy parameter pt be defined as
I G, where the expectation is taken with respect to the randomness
M Eies, w;Oi,, mod m (
p:
if mod m + 0
(17)
We write
|| - 0-1 |Pp+1 4/411 + 1/411 11
PP - -14+111" 12 + 2(p +l+1) (2b++1 - 0*) + 11bb1+11 -².
(18)
To obtain an upper bound on | 10112. consider the Lemmal This lemma is taken from [33
Lemma 1. Suppose Assumptions A and A
hold. If TL. then the following inequality holds:
||bb+1 - -1-12 s -ma)llbb 1- + 6L, A + w. llbb - -0,,12
(19)
where = ((TTO--EEt we (TOZ)
Proof. Let be defined as = E (L.. ). It can be inferred that 4/.+1 4/2
7:9
Therefore, we may write
| 1- l| 7h:: 0-112- = 11 0-1 | 2n.9. (eb. 0²) -n2+ 11
(20)
From L-smoothness of L; stated in assumption A
[ it follows that
[Te, < 2L (C, - (T(r))
(21)
where 0 = arg ming (mo). Using the above inequality and due the convexity of || we can write
N
n 19:1 < 2
wi VL,(TT..))
2 2m
Σ
wi
(C.(TT..)) Lr(To)))
(22)
1=1
Furthermore, we can rewrite the term -2n.g. (xb; - 0") in equation 20 as
N
2n.9. (4b, - -0) -
w. V. (To..) T O,,))
N
2m
w.
W.. ( 00) -0)
(23)
Using AM-GM inequality, we can obtain
-2V L. ( (yhe Oc.) - 11 0c.011 + V 7
(24)
while due to p-strong convexity of L.. write -V.,(TO..)) T (0...
0*) < (C.(TT..) (TTO
11 0,,
0*1
(25)
2
Taking weighted average over the objectives and applying Jensen's inequality to the right-hand side of equation 25 we obtain
N
N
VL.(T...) (0)--0)-- 00) 5- Σ
w;(CC(TTL..) Li( 1/11 || - -0
-0-
(26)
15

---

Table 4: Notation Table.
Symbol
Description
N
Number of objectives
M
Number randomly selected objectives at each step
St
Set of selected objectives at step
w;
Preference weight associated with the i-th objective
Policy of the language model parameterized by 0
Og
Parameters of the policy associated with the i-th objective at optimization step /
Pr
Merged parameters of all policies at step t, defined as in equation 17
2/:
Weighted average of policy parameters, defined as 2/. Di=1 w,O,.c
gt
Weighted average of policy gradients, defined as gr =i=1 w; V (To...)
Optimal policy parameter for the multi-objective loss, defined as in equation
0,
Optimal policy parameter for objective i, defined as in equation 12
Furthermore, taking a weighted average and applying the inequality in equation 21 to equation 24
get
Wi VL, ((ab (bb: 02,))
Wi
11 02,r11
T)t
+ 27p Σ wi (Li(To...) (i(Ta (27)
and Combining equation with equation 22 equation 23 equation and equation 27 arrive
||/1+1 0-112 m 1- +
Σ will 02,el?
N 2m.
+ 4n L Σ wi (C.((Te..) (T(
Σ w. (,
(x(00))) (28)
Taking the definition of A into account, the last two terms in the right hand side of equation 28 can be rewritten as
N Σ
N
4mj L wi (Ci(TT..)) - (i(To))) 2T
Σ
wi (Ci(TTo..) Li (TT-))
1=1
1=1
N
2m (1- 217; L)
Σ
wi (TI(TT)).) - +22
(To (T(oT) ))
i=1
t=1
--2n((1
27, Σ
wi (CI(TT...) (T(TO))) 4n L
w. w,((((TTr)) -
N
-227e(1 - 27, L)
wi (Li(TT...) - Li(TT-)) + 4 LA
(29)
1=1
To bound Ein wi (Ci( (=(mre)). )), considering the convexity of C., can write
N
N
wi (CI(TT...) - L.(TTo)) = Σ
wi (C.(TT...) - -L.(TT))) + Σ w; (Ci(TT,) Ci(To)))
i=1
1=1
N
w; VL.())) (0., -be) + L.(TT)) - L;(TTo).
(30)
16

---

Applying AM-GM inequality to the right hand side of equation 30
we get
Wi (TT()T...) Li(TT.))
no [el )1 +
- T
1
11 -
2b:11?
(T)) (
(31)
Applying the inequality in equation 21 to the right hand side of equation 31
we conclude that
N
N
wi (TT()T...) - (iTTT))) 2 - Σ wi 1hL (Ci( - Li(To;)) + 277:
10,, - 2b.1?
==1
+ (mTgp) Li((To))-
(32)
Due the fact that 0 < 1/t S
4L it can be concluded that n/e S 2np(11 - 21z, I.) S 2mp Multiplying both
1
sides of equation 32 by 277e - 2Tp; L), we obtain
2n (1 2)
wi - Σ
1=1
wi (2n) (1 2n, - (T(O)) + +11 21,,,1))00 - 13)
(1 21x [)) ( ()(
(33)
The inequality in equation 33 can be rewritten
N
- -2 - 21/ L)
Σ
wi -
22n (1 2np: LTL Σ wi(1 (1 - 2m7 1.l(0s) - ab2ll2
1=1
+ (n - 1)2 (1- - 2m (Li(TT)) - -i((To)))
(34)
Using the facts that (,(,,)))) (TT 2 0. M,L 15 and 2n S 1, from equation 34 we obtain
N 2Tp (1- - 2) Σ wi (CI(TTo..) - (i( S LA
Σ w. 10,, - 2.l12.
(35)
Combining equation with equation 29 and equation 35 proves the Lemma.
Since IterativeRS merges every steps, there exists t' such that t-1' < m and O.. th. Vi E (1,... N}. Considering the facts that why is the expected value of (0 J.V. over distribution {w; N and E|x B[x]]|2 < [[[[X]] we can conclude that
N
Σ wy w/llibh 0
Σ
we well 4b. + 0re 0u.
wi O, 0g.112
(36)
Assume that n, is selected such that is non-increasing and satisfies ni S 27 Vt. Taking the assumption A B into account, can infer that
N
Σ
wi 0 =
Σ Well T=1
ne VL, 4(m
(37)
T=0
Combining equation
37 with equation
19
we get
| 1-112 < (1 n1)))bbb
+ 8(m 1)3
(38)
17

---

Recall that IterativeRS in Algorithm samples M objectives uniformly without replacement. Such sampling scheme unbiased and we can write
Es, |Pt+1] 4/4+1
(39)
where Es, [-] denote the expectation with respect to sampling randomization. Therefore, taking expectation from equation 18 leads to
Es, P 0 1-] = Es, ll 2/411 + 1/bb+11 0"11?.
(40)
Combining equation 40
with equation 38 we get
Es, [|]
[[2] < Es, Pt+1 11+1
+ (1 mp) || 00 11 +
6 + 8(m - 1/
(41)
According to equation 17 if (t 1) mod m * 0, can concluded that Eg, |llp 3/4441 1
0. If (t + 1) mod m = 0, considering the assumption that W1 = WN = N. the term
Es, P| - 1/²] can be expressed as:
Es, [|lpr+1 1/++1
EEs
Σ rEB1+1
Oi,011 Vi+1
M
M Es,
Σ
Pr|i E Sc+1](0re+1 - sc+i)
N
Pr|i S411ll0s,ct1 - +1+1112
M
Pr[i.j ( vb++1) (( (0),++1 - abe+1).
(42)
M2 #
Considering the facts that Pr(i Set1] N
M
and Pr[i. E Sc-1]= N(N-1) and
M(M-1)
N
0 11111
14+1112
Σ
0 (11 $1+111
(0i,4+1 (10+)) (0++))(00314)) +b441) 0,
(43)
8.jE8
we can rewrite equation 42 as
M
Es P++1
+ M(N - 1)
1
0 - 1141 12
(44)
Using equation 36 and equation 37 from equation
we arrive at
Es, [| - ahr+ |1
M(N -1) (1
4N
M N
nhe G2
(45)
Combining equation 45 with equation 41 we get
Es, [1le++1 - 0" 11] 5(1 m1)))lbb: - 1011 + 6L N M
2(m 12(m-1)² 10)2
M(N - 1) m² mn²² 11222
(46)
Define B
B = 6L +4 (2 1122 N M
M(N 1) m² G².
(47)
18

---

With step size chosen as 7/.
B_ for some B >
1 and > 0 satisfying S minf 3. _1 and
N S 21/4mm using induction can be proved that Es Ps. 0" 1011 S ytt where
{
B² Bp - 17 1)1l<> 1-1
v = max
(48)
Since Pt is an unbiased estimator of she. we can conclude that Es, P: 1-
1 0 ||
Definition of e ensures that ||4/: - 0-1 v 7tE
fort = 1. Assume that |/4
0-12
S
holds for
t. Using equation 38 we can write
|| 1- (1
0-1 + B
Bp
V + B²B
1
t+
t t+y
(t+
t ++7 -1
BB
3p-1
5 1+
(49)
(L 7)2
(1+
(1 (1+
which proves that Es, P 0* [|2] S y+++T holds. Choosing 3 = and y max 7L
" m)
1,
we have 77, u((+t)
2
It can be verified that in this case ?/e S 2 Then, we can write
BB
B²B
max
Bp 1 (v 1)1l01 0-11]
+ (r + 1)1l 1-1
3p 1
4B
+ (7 + 1)1l - 10
(50)
122
Combining equation 50 with equation 49
and the fact that LEg, [pr] = vr. get
4B
Eg, Pr+1 e*
[|2] = || e-
+ (x+ + 1)1l>b 0-113)
(51)
t t + 1
422
According to smoothness assumption in A
[ we can write
[]]((pp)] - L(TT ) S 2 Es, Pllp-+1 - 10
(52)
Combining equation 52 with equation 51 we arrive at
[[[((T))] [(TT-)
L
AB 112
+
0'113)
(53)
+(1+7)
Plugging in wh = Over equation 53 proves the Theorem. Note that for the ease of notation we drop the expectation in Theorem Furthermore, it should be noted that most of proof steps are taken from 33 Furthrmore, is useful to mention that Theorem Tis proved for the case where wi = WN N. However, extension to non-uniform cases is straightforward. Define the scaled loss for the objective as ( = w,N ((T)). Then it can be concluded that L(TT) = N Σ (TTT). Therefore, in that case the proof can be applied to scaled losses.
B Supplementary Experimental Results and Details
This appendix provides supplementary experimental results and implementation details.
B.1 Implementation Details for Small Molecule Generation
To fine-tune the MoIGPT-2 model using MORLHF, RS, and IterativeRS, we employed PPO from the TRL library. For each objective, a reward model was trained using the labeled QM9 dataset. Each reward model consists of MolGPT-2 backbone with three-layer MLP head; only the MLP head was trained. The dataset was split into 80% training, 10% validation, and 10% test sets. All models were fine-tuned with a learning rate of 1.41 x 10 using the Adam optimizer and a batch size of 128. The RiC baseline was configured with the same hyperparameters and settings as MORLHE, RS, and IterativeRS. We set 2 for RiC. Model training was conducted using four V100 GPUs. To perform merging for IterativeRS and RS, we average objective-spedtt model weights. For IterativeRS the number of selected objectives was 3.
19

---

B.2 Implementation Details for DNA Sequence Generation
Similar to the molecule generation task, we fine-tuned the DNAGPT-2 model using MORLHF, RS. and IterativeRS with PPO from the TRL library. A subset of 100,000 samples uniformly sampled from the MPRA dataset and evaluated using the Malinois model to obtain activity scores across three cell lines. This subset was used as the labeled dataset to train the reward models for PPO. For each objective, a separate reward model was trained using this labeled data. Each reward model consists of DNAGPT-2 backbone with three-layer MLP head, where only the MLP head was trained. The dataset was split into 70% training, 10% validation, and 20% test sets. All models were fine-tuned with a learning rate of 1.41 x 10-5 using the Adam optimizer and batch size of 128. The RiC baseline used the same hyperparameters and settings as MORLHE, RS, and IterativeRS. We set 2 for RiC. Model training was performed on four V100 GPUs. For IterativeRS the number of selected objectives was 3.
B.3 Implementation Details for Text Summarization
To fine-tune the 1eae-3..-33--nsttrree model using MORLHF, RS, and IterativeRS, we employed PPO from the TRL. library. We first passed all prompt-response pairs from the Reddit dataset through three oracle reward models to construct a multi-labeled dataset. For each objective, proxy reward model was trained using the dataset and the corresponding objective-specifi labels. Each proxy model consists of a Llama-3.2-3B I-In-uuntt backbone with a two-layer MLP head, with only the MLP head being trained. We used the training set from the Reddit dataset for training and randomly split its validation set into two subsets serve validation and test sets. The validation set was used both for training the proxy reward models and for supervised fine-tuning in RiC. During inference, prompts from the test set were provided to the fine-tuned models to generate text summaries. The maximum summary length was set to 32.
Before applying PPO fine-tuning, we first trained an SFT model. The PPO fine-tuning was then performed using this SFT model. We observed that initializing PPO with an SFT model leads to improved ROUGE scores in the generated summaries. To construct the SFT training dataset, for each prompt in the training set, we selected the summary that outperformed the alternative in the majority of objectives based on reward scores. SFT was performed for two epochs with a learning rate of 1.41 x 10-6 using this constructed dataset. For fine-tuning with PPO, we selected a random rollout of 1.024 samples per epoch and used a batch size of 128. Each model was fine-tuned for 20 epochs using a learning rate of 1.41 x 10-6 and the Adam optimizer. The RiC baseline was fine-tuned on the entire training set for 2 epochs, using the same learning rate and batch size. We set p = 2 for RiC. All other hyperparameters were kept consistent across MORLHF, RS, and IterativeRS. four A A100 GPUs For IterativeRS, the number of selected objectives was set to 3. For both RS and IterativeRS, model merging was performed by selecting from seven candidate merged models obtained using seven different sets of merging weights. The model with the highest average reward, as evaluated by the reward models, was selected. The seven sets of merging weights consisted of [1/3, 1/3, 1/3], all permutations of [1/6, 1/6, 2/3). and all permutations of [1/6, 5/12, 5/12]. During training, at each merging step, we computed the reward of each merged model on 256 samples from the training data and selected the model with the highest average reward for the next iteration. After training. to obtain the final merged model, we evaluated all seven merged models on 1,024 validation samples and selected the one with the highest average reward. We then assessed its performance on the test set. Note that RS does not perform merging during training.
Training was conducted
B.4 Supplementary Results
We performed additional experiments on the DNA sequence generation task to evaluate the perfor- mance of RL-based fine-tuning methods MORLHF, RS, and IterativeRS, using RLOO, which does not rely on value model. The results are presented in Table S The results show that IterativeRS achieves a higher average reward than both MORLHF and RS when using RLOO.
To investigate the influence of merging on the performance of IterativeRS, we conducted supplemen- tary experiments. For DNA sequence generation task, we considered ten different sets of merging weights, including [1 11/ 1/3, 1/3), permutations of [1/6, 2/3), permutations of [1/6, 5/12, 5/12] and permutations of [1/2, 1/4, 1/4). To assess the performance of each merged model, we generated 8, 192 samples per model and identified the Pareto-optimal sequences based on scores from reward
20

---

Table 5: Average performance of Pareto-from DNA sequences generated by multi- multi-objective ap-
proaches using RLOO.
K562
HepG2
SKNSH
Avg Reward
ICV
MORLHF
0.3754
0.6747
0.6882
0.5794
3,5907
RS
0,4080
0.6571
0.6786
0,5812
5.7214
IterativeRS
0.3559
0.6860
0.7061
0.5826
3,9890
Table 6: Comparison of selective merging and fixed merging on the performance of IterativeRS on DNA sequence generation genuenatggneettion task.
K562
HepG2
SKNSH
Avg Reward
fixed merging
0.3559
0.6860
0.7061 0.6923
0.5826
selective merging
0.3678
0.6778
0.5793
models. The final selection was based on the model that achieved the highest average reward score on its Pareto optimal sequences. This method is referred to as selective merging, whereas merging with uniform weights is referred to as fixed merging. presentss the results obtained using RI.O.. The results suggest that selective merging offers no improvement over fixed merging. One possible reason for the limited effectiveness in the DNA sequence generation task is the evaluation method, which relies on Pareto-optimal samples generated by the model. This makes identifying the best- performing model more challenging. During training, merging allows experts to transfer cross-task knowledge, which can help the final merged model generate higher-quality sequences. However, selecting the best merged model among candidates is difficult because it depends on evaluating the Pareto-optimality of generated sequences using reward models. These reward models trained on limited data derived from an oracle model used during evaluation, leading to a performance gap between the reward models and the oracle. As result, assessing Pareto-optimality using these reward models may not yield reliable outcomes.
To examine the effect of the merging strategy both RS and IterativeRS in the molecule generation task, we applied MolMoE 7 to each method. MolMoE is an expert merging method designed for molecular applications, whereas IterativeRS focuses expert training. Therefore, these two methods can be used in conjunction. We applied MolMoE to both IterativeRS and RS, and the results are presented in Table As shown, IterativeRS with MolMoE outperforms RS with MolMoE across all objectives. Comparing Table Table we observe that incorporating MolMoE improves the performance of RS across all objectives. While IterativeRS with MolMoF achieves nearly the same average reward as IterativeRS without MolMoF, it yields an 11% improvement in ICV. It is worth noting that one of the main advantages of using MolMoE is its ability handle scenarios where preferences over objectives change dynamically over time, which is outside the scope of this paper's experimental study.
C Supplementary Related Works
Federated Learning. Federated learning involves a group of users, called clients, who collaborate with each other through communication with central server to train a global model [39 19 57 There is an analogy between federated learning and multi-objective reinforcement learning in context of foundation model fine-tuning. In federated learning, the clients and the server work together to train a model that performs optimally across all clients' data. However, this can be challenging since the data may be distributed non-i.i.d. among clients, which similar to multi- objective reinforcement learning can lead to conflicting objectives during model training. To address this issue, several personalized federated learning algorithms have been proposed in the literature
48
15
32
38
58 [6].
GFlowNet. GFlowNets. initially proposed by 4 were introduced as a generative reinforcement learning framework designed to effectively handle scenarios with multiple paths leading to a common state. They have been widely applied to biological sequence [23
28 17 and molecule design 61 22 27] tasks, where their effectiveness has been well documented. In this paper, we focus on policy gradient- gradient-based methods such PPO, due to their computational efficiency for foundation
21

---

Table Average performance of Pareto-optimal molecules generated by RS and IterativeRS employ- ing MolMoE for merging.
a energy
gap
Uo energy
Avg Reward
ICV
RS+MolMoE ItrraiveRS+MolMoE
1.4499
0.9715
1,5988
1.3400
4.1120
1.5651
0.9941
1.6420
1.4004
3.9938
model fine-tuning. It also worth noting that several methods have been proposed in the literature to improve the learning efficiency of GFlowNets [S] 37 36 47]. Furthermore, GFlowNets have recently been utilized enhance the reasoning capabilities of large language models and vision-language
models
25 55 54
D Societal Impact
In this paper, we addressed the problem of fine-tuning large language models (LLMs) on multiple objectives- challenge with significant implications in areas such small molecule design for drug discovery and biological sequence design. Methods that enable LLMs to generate molecules or biological sequences with desirable functionalities hold great promise for accelerating the discovery of new drugs and therapeutics, potentially benefiting society at large. However, we recognize the dual-use nature of this research. There is also the risk that such technologies could be misused or exacerbate existing health disparities, particularly among marginalized communities. As researchers, underscore the importance of carefully considering both the societal benefits and the potential unin- tended consequences of this work. We remain optimistic that the broader impact of our contributions will lean toward positive, equitable outcomes.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly explained the main contributions in both introduction and abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the experimental section, we reported that some baselines perform better than ours in certain aspects.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

Justification: In section 4.2 we provide the accurate definition of assumptions. We provide the complete proof for the Theorem in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provided the full implementation details in both the paper and Appendix
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification: The datasets are publicly available, and we provide the code used to generate the experimental results in the corresponding GitHub repository of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https /mips.ccc/pulic//uuui guwde//ddee/eee Cot//ooooeeesiseeeeeee hwtpibbmsssiooooooooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Appendix B we provided the implementation details including training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the inverse coefficient of variation (ICV) scores in experiments section to provide information about statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
25

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix B we explained our computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttp:://eurr//i httpa//rpp....ppp.aac/cccaaaaaaa
Answer: [Yes]
Justification: The research conducted in this paper conform, in every respect, with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed societal impact in Appendix D
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We believe the paper pose no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cited all models and datasets used by the paper in the experimental section.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
28

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
29


Reviews:
[{"review_id": "DHU66NyfyZ", "reviewer": "Reviewer_e3du", "summary": "The paper proposed a multi-objective reinforcement learning algorithm called IterativeRS to fine-tune foundation models on multiple reward signals. Specifically, the algorithm fine-tunes a separate \"expert\" policy for each objective to learn objective-specific skills , and then mitigates divergence between these experts by iteratively merging their parameters into a single shared policy, which is then used as the starting point for the next round of fine-tuning. The experiments on diverse tasks including small molecule generation, DNA sequence generation, and text summarization show that IterativeRS generally achieves higher average rewards.", "questions": "See weaknesses.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\nThe proposed method generalizes existing methods into a single flexible framework that offers greater potential for performance improvement by controlling the trade-off between specialization and knowledge sharing.\n\n**Weaknesses**\n\n1. Marginal performance gains in text summarization. IterativeRS barely outperforms MORLHF on average reward and is outperformed by RS on the \"faithful score\" metric. This suggests that the proposed method might have limited applications.\n2. The assumption A.3 is too strong and not commonly used in previous literature. The authors should provide a convincing reason on why this is realistic or necessary.\n3. Following 2, the theoritical analysis might not be helpful for explaining the practicality of the proposed method.", "was_revised": true, "final_justification": "The authors seem to have persistent confusion regarding bounded variance and  bounded random variables. But since theory is not the main contribution, I maintain my stance on weak acceptance.", "rebuttal": "Thank you for reviewing our paper and letting us know your valuable comments. Please find below our responses.\n\n## Experiments\nWe performed further experiments on the text summarization task. To improve the performance of the proposed algorithm, we considered seven different sets of merging weights, including [1/3, 1/3, 1/3], permutations of [1/6, 1/6, 2/3], and permutations of [1/6, 5/12, 5/12]. Previously, we only considered the uniform merging weights [1/3, 1/3, 1/3]. During training, at each merging step, we calculated the reward of each merged model on the training data and chose the one with the highest average reward for the next iteration. To obtain the final merged model after training, we measured the average reward of all seven merged models and chose the one with the highest average reward on the validation data. We then tested the performance on the test set. We followed the same procedure for RS as well. Note that RS does not perform merging during training. The table below shows the new results. These results demonstrate the effectiveness of the flexible merging strategy in IterativeRS, as it significantly outperforms other methods on all metrics.\n\n|  | faithful  | summary | deberta | Avg Reward | Sharp-like |\n| :--------- | --------: | :---------: | :--------: | :------------: | :----------: |\n| RS | 0.6549 | 0.4644 | 0.3178 | 0.4791 | 3.3719 |\n| IterativeRS | 0.7023 | 0.5160 | 0.5834 | 0.6006 | 6.2180 |\n\n## Theory\nPlease note that Assumption A.3, which is the boundedness of gradients, is a standard and common assumption in convex optimization (see for example [19]). The conclusions drawn from the theoretical analysis in the paper are not guaranteed to hold for non-convex deep neural networks, which are typically used in practice. However, the goal of the theoretical analysis is not to fully explain the behavior of the proposed algorithm in practical scenarios. Instead, it demonstrates that the algorithm’s design is both sensible and grounded in a rigorous theoretical foundation. Finally, the theoretical convergence analysis of deep neural networks used in practice remains an open research problem and is beyond the scope of this paper."}, {"review_id": "WhXf1cAu7T", "reviewer": "Reviewer_Bpig", "summary": "This paper proposes a new multi-objective reinforcement learning method (IterativeRS) that generalizes both multi-objective RLHF (MORLHF) and Rewarded Soups. Existing MORLHF approaches typically optimize a single expert using a scalar-weighted sum of multiple objectives. This approach lacks expert diversity (since there’s only one expert) and is prone to collapsing toward the objective with the dominant reward signal—a well-known challenge in balancing multiple rewards. In contrast, the rewarded soups method trains separate experts for each objective and mixes their parameters post-training. While this allows for expert diversity, it can result in high variance and conflict between experts due to a lack of synchronization. The proposed method introduces **periodic synchronization** using the hyperparameter *m*, effectively balancing objective-level and expert-level variance. By tuning *m*, the method achieves stable performance across three tasks: QM9 molecular generation, DNA sequence design, and Reddit summarization.", "questions": "- This method appears to be a general multi-objective RL approach, not limited to fine-tuning. Have you tried applying it to RL training from scratch?\n- In the context of LLM fine-tuning—currently the most common use case for RLHF—what are realistic scenarios where multiple objectives are necessary? Can this approach be applied effectively to tasks like text summarization?", "limitations": "there is no section for the limitations, please include these.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper proposes an elegant algorithm that unifies two existing approaches to multi-objective RL: MORLHF and rewarded soups.\n- It highlights the importance of balancing trade-offs between expert-level and objective-level variance.\n- The work provides both theoretical insight and empirical validation.\n\n**Weaknesses:**\n\n- The method is only evaluated using PPO. It is unclear how well it generalizes to other RL methods, such as value-based approaches.\n- The scale of the applications is relatively small. It is questionable whether this method is necessary for fine-tuning foundation models, as the tasks could potentially be handled by training from scratch.", "was_revised": true, "final_justification": "I have read the authors' rebuttal. They addressed my concerns related to generalizability, but still, my concerns about scalability are not fully resolved, since the experiments are conducted on short sequence generation tasks. I'll keep my score 4.", "rebuttal": "We would like to express our gratitude for reviewing our paper and letting us know your valuable comments. Please find below our responses to your comments.\n\n## Fine-Tuning Frameworks\nThe experiments are conducted using PPO, a commonly used reinforcement learning (RL) method for LLM fine-tuning. Based on your comment, we performed additional experiments on the DNA sequence generation task to evaluate the performance of RL-based fine-tuning methods MORLHF, RS, and IterativeRS, using RLOO, which does not rely on a value model. The experimental setup for RLOO is the same as that used with PPO. The results are presented in the table below. They indicate that using PPO leads to better performance for IterativeRS. It is possible that improved tuning of the hyperparameter $m$ could enhance the performance of IterativeRS when using RLOO. Furthermore, the results show that IterativeRS achieves a higher average reward than both MORLHF and RS when using RLOO.\n\n|  | K562 | HepG2 | SKNSH | Avg Reward | Sharp-like |\n| :----: | :----: | :----: | :----: | :----: | :----: |\n| MORLHF | 0.3754 | 0.6747 | 0.6882 | 0.5794 | 3.5907| \n| RS | 0.4080 | 0.6571 | 0.6786 | 0.5812 | 5.7214 |\n| IterativeRS | 0.3559 | 0.6860 | 0.7061 | 0.5826 | 3.9890|  \n\n## Reinforcement Learning from Scratch\nThe focus of this paper is on RL for LLM fine-tuning. However, the proposed method can also be applied to RL from scratch. It is worth noting that even in relatively small-scale applications where training from scratch is feasible, fine-tuning LLMs can offer significant advantages. First, LLMs can be pre-trained on both unsupervised and supervised data, while RL from scratch utilizes labeled data. This pre-training provides LLMs with prior knowledge that can be beneficial for downstream tasks. Moreover, fine-tuning an LLM might be more computationally efficient than training an RL model from scratch. For these reasons, we chose to focus on LLM fine-tuning in this work.\n\n## Applications\nThere are many practical applications for multi-objective RL in LLM fine-tuning. For example, LLMs are currently used for small molecule generation in drug discovery. These models learn to generate valid SMILES representations of molecules through pre-training on large molecular datasets. During generation, the goal is to produce molecules that satisfy multiple desirable properties. Each property typically has its own labeled dataset. An effective fine-tuning approach should enable the LLM to optimize all target properties simultaneously. Similarly, in text summarization, multiple criteria such as helpfulness, conciseness, and faithfulness are important for evaluating the quality of a summary. Relying on a single reward model to capture all these criteria may be suboptimal. Therefore, multi-objective RL offers a necessary solution for a range of important tasks in LLM fine-tuning."}, {"review_id": "GUwq5cmnAf", "reviewer": "Reviewer_ysgq", "summary": "In this paper, the authors introduce an algorithm for performing fine-tuning on foundation models with multi-objective rewards. In particular, they propose a generalization of current methods which either (1) train separate models on each objective and merge parameters at the end or (2) train a single model on all parameters with a weighted reward function. The generalization simply performs an iterative merging procedure where separate models are trained in parallel for $m$ steps and then merged before continuing another round of independent training. They derive a theoretical bound on the performance gap in the training loss between their proposed method and the optimal policy for the combined reward function and offer some insights on hyperparameter selection based on this bound. Finally, they provide experimental justification for their method on three practical and pertinent fine-tuning domains, demonstrating out-performance of their method in terms of average reward across all domains.", "questions": "Questions: \n\n(1 How do you compute the Pareto front? Are the results in Table 1-3 only for generated results on the Pareto front? Wouldn’t it be necessary to report how many results generated were on this front for each algorithm?\n\n(2) What exactly is the Sharp-like score? Is it better to be high or low? How should we balance the Sharp-like score versus the average reward?\n\n(3) How can we interpret Figs 1-3. e.g. in Line 265 “Notably, Figure 1 shows that the highest-scoring molecules are produced by IterativeRS” – From the plots, it is hard to varify this claim from the plots – the Iterative RS and MORLHF distributions seem to be mostly overlapping. This is true in most of the Fig 1-3 results.", "limitations": "The authors do not discuss any limitations of their work. They do note a limitation of the theoretical analysis, that it does not apply to generalization results, but this should probably be restated in the conclusion. Other limitations may include the reliance on knowing the $w$ weights to balance the value of various rewards, which is impractical in most scenarios.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n(1) The paper is very clearly written and easy to follow.\n(2) The experiments cover three diverse use cases and hence show a broad range of practical application domains for the proposed algorithm. The practical domains examined are relevant and important domains for the research community.\n(3) The authors provide both theoretical and experimental results for their proposed algorithm\n\nWeaknesses:\n(1) The results are not wholly convincing. From Figs 1-3, the only clear out-performance is versus RiC and it is somewhat unclear why this is a relevant baseline. It would be more interesting to see out-performance versus MORLHF and RS since, as the authors note, these are two extremes of the proposed IterativeRS algorithm. While the results in Tables 1-3 show that IterativeRS consistently outperforms in terms of average reward on the Pareto front, it is not exactly clear how to interpret this result (see questions below). Moreover, it is not clear if the out-performance is statistically significant since no confidence intervals are reported. I am also not clear on how to weigh the average reward results versus the Sharp-like score (and it is not clear what the Sharp-like score is measuring).I would be willing to increase my score if these issues are resolved.\n\n(2) The utility of the insights from the theoretical analysis seem limited. The theoretical results do not indicate any theoretical justification for IterativeRS over MORLHF or RS. The authors claim that the theoretical results provide insights into the behavior of these various algorithms, however, this claim is not clearly demonstrated. The authors themselves note that the results are limited to the training loss and do not apply to generalization capabilities which is of most interest. Moreover, the analysis of the hyperparameter choices, particularly the choice of $m$ do not seem to offer much practical use for selecting the hyperparameters. How was $m$ chosen in each of the three experimental cases? Was the theory provided leveraged to provide any insight into the choice of $m$ or was a hyperparameter sweep conducted?", "was_revised": true, "final_justification": "The original results in the paper do not provide sufficient performance improvement over prior methods to be convincing. While the authors provided some additional results during the rebuttal period on the text domain that show stronger outperformance, they did not show similar outperformance in the other domains.", "rebuttal": "Thank you so much for reviewing our paper and letting us know your valuable comments. Please find below our responses to your comments.\n\n## Theory\nThe theoretical analysis in this paper focuses on convex settings. The statement regarding generalization in practice refers to extending the results to non-convex cases, which is the relevant scenario for LLM fine-tuning. The theoretical convergence analysis of deep neural networks remains an open research problem and is beyond the scope of this paper. The purpose of our theoretical analysis is not to fully explain the algorithm’s behavior in practical settings, but rather to show that the algorithm’s design is well-motivated and grounded in a rigorous theoretical foundation. \n\nFurthermore, the goal of this paper is not to propose an algorithm whose superiority over RS and MORLHF is theoretically proven. Instead, the paper suggests that better empirical results than both RS and MORLHF can be achieved by appropriately tuning the hyperparameter $m$ of the proposed IterativeRS.\n\n## Metrics and Interpretation of Results\n**Sharp-like score:** For a given sample, the Sharpe-like score is defined as the average reward across all objectives divided by the standard deviation of those rewards. A higher Sharpe-like score indicates lower variability and more balanced performance across objectives. Multiplying the reward vector by a scalar $\\lambda>1$ increases the standard deviation, but the Sharpe-like score remains unchanged. In this case, the relative performance across objectives does not change, even though all objective values improve. A higher Sharpe-like score reflects more consistent performance across objectives, which is generally desirable. However, in practice, higher average performance (i.e., average reward) may be preferred over higher consistency. As an example, if two objectives remain unchanged while one objective improves, this leads to a reduction in the Sharpe-like score, while the overall performance has improved. The definition of the Sharpe-like score is provided in the first paragraph of Section 5. We can highlight the definition by providing math formulation.\n\n**Pareto-front:** Let us first explain the difference between the goals in text summarization and biological applications. In text summarization, each input consists of a prompt, and the goal is to generate a high-quality summary for that prompt. In contrast, for small molecule and DNA sequence generation, there is no prompt; the objective is to discover a small number of high-performing sequences using the LLM. In our experiments, we used the fine-tuned model to generate 10,000 molecules or DNA sequences, which were then evaluated using oracle models. Note that in our setup, oracles are distinct from reward models. Reward models are trained on labeled training data, where the labels are provided by oracle models. In biological applications, if one generated molecule outperforms another (both produced by the same model) across all objectives, the former dominates the latter and is included in the Pareto front, while the dominated sample is considered suboptimal. Therefore, for molecule and DNA sequence generation, we report the performance based on the Pareto front. In contrast, for text summarization, we generate a response for each prompt in the test set and report the scores assigned by the oracle reward models. As with the biological tasks, the LLMs did not interact with the oracle models during fine-tuning for text summarization. \n\n**Figures:** The comparison between RiC and IterativeRS is emphasized in the figures because RS and MORLHF can be viewed as special cases of IterativeRS. In the results shown in Figure 1, it is desirable to have samples in the top-right corner. Since the task is molecule generation, generating a few dominant samples is considered optimal. As shown in Figure 1, IterativeRS achieves the highest density of samples in the top-right corner.\n\n## Experiments\nWe performed further experiments on the text summarization task. To avoid relying on predefined merging weights, we considered seven different sets of merging weights, including [1/3, 1/3, 1/3], permutations of [1/6, 1/6, 2/3], and permutations of [1/6, 5/12, 5/12]. Previously, we only considered the uniform merging weights [1/3, 1/3, 1/3]. During training, at each merging step, we calculated the reward of each merged model on the training data and chose the one with the highest average reward for the next iteration. To obtain the final merged model after training, we measured the average reward of all seven merged models and chose the one with the highest average reward on the validation data. We then tested the performance on the test set. We followed the same procedure for RS as well. Note that RS does not perform merging during training. The table below shows the new results. These results demonstrate the effectiveness of the flexible merging strategy in IterativeRS, as it significantly outperforms other methods on all metrics.\n\n|  | faithful  | summary | deberta | Avg Reward | Sharp-like |\n| :--------- | --------: | :---------: | :--------: | :------------: | :----------: |\n| RS | 0.6549 | 0.4644 | 0.3178 | 0.4791 | 3.3719 |\n| IterativeRS | 0.7023 | 0.5160 | 0.5834 | 0.6006 | 6.2180 |\n\n**Hyperparameter Tuning:** To tune the hyperparameter $m$, we perform a few experiments with different values of $m$ and evaluate performance on the validation set. Based on these results, we select the value of $m$.\n\nPlease let us know if you have further comments and concerns."}, {"review_id": "gZ7gVTNcdn", "reviewer": "Reviewer_WYNS", "summary": "This paper addresses the fine-tuning problem of foundation models in multi-reward scenarios and proposes the IterativeRS algorithm. By iteratively fine-tuning expert policies for each objective and periodically merging parameters, it balances objective-specific skills and policy consistency. Theoretical analysis proves its convergence superiority over baselines such as MORLHF and Rewarded Soups under convex loss assumptions, providing a more flexible and effective solution for multi-objective reinforcement learning fine-tuning.", "questions": "Please refer to the above weakness.", "limitations": "1. Theoretical analysis relies on assumptions like convexity and smoothness of loss functions (e.g., A1–A3), but loss functions in practical multi-objective optimization are often non-convex, possibly causing deviations between theoretical performance bounds and actual performance when objectives conflict.  \n\n2. The mechanism of iteratively training expert policies and merging parameters leads to superlinear growth of computational cost with the number of objectives N and merging frequency m. For example, in DNA sequence generation experiments, the training time is about 3 times that of MORLHF, restricting large-scale applications.  \n\n3. Tasks such as text summarization do not analyze the impact of weight configurations on results by combining domain-standard metrics (e.g., ROUGE scores), and lack targeted discussions on practical scenarios (e.g., news summarization).  \n\n4. Experiments only compare methods like MORLHF and Rewarded Soups, without including emerging multi-objective optimization algorithms (e.g., Multi-Objective GFlowNets), and lack comparison with professional methods (e.g., Mol-MOE) in biological sequence generation, affecting the universality of conclusions.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The paper proposes a novel IterativeRS method. Through iterative training and merging of expert policies, it provides a unique alternative to existing methods such as MORLHF, and addresses multi - objective optimization problems in a more principled manner.\n2. It conducts a comprehensive theoretical analysis of the convergence behavior of IterativeRS, which not only explains the characteristics of this method but also provides a theoretical basis for understanding related baseline methods.\n3. Extensive experiments are carried out in multiple tasks such as small - molecule generation, DNA sequence generation, and text summarization. By comparing with a variety of baseline methods, the advantages of IterativeRS in terms of average reward and cross - objective consistency are verified.\nWeakness\n1. The theoretical analysis relies on assumptions such as the convexity of the loss function, which may not hold in real - world scenarios. This leads to limited universality of the theoretical results, and the performance of the method in complex practical applications may deviate from theoretical predictions. \n2. The method involves iterative merging and multi - objective optimization, resulting in a relatively high computational complexity. Although the trade-offs of hyper-parameter tuning are discussed, it may still face challenges in terms of computational resources under complex configurations, restricting its practical application and promotion.\n3. Although the method is applied to the text summarization task, there is a lack of in - depth interpretation of the actual application scenarios of the generated summaries. It fails to analyze the specific impact of different reward model weights on the quality of summaries in combination with domain - specific indicators such as faithfulness and readability. \n4. The number of baseline methods used in the experiments is relatively small. Incorporating more baseline methods widely adopted in relevant sub - fields could lead to a more rigorous and comprehensive evaluation of IterativeRS.", "was_revised": true, "final_justification": "1. The theory focuses on justifying the algorithm' s design is logically consistent. \n2. The subset strategy raises critical issues: vague selection rules, added hyperparameter complexity, and reduced generality for full-objective optimization. Not accepted. We recommend either clarifying subset selection rules or explicitly discussing their limitations.  \n3. Expanded experiments strengthen empirical support.\n4. The rationale for excluding GFlowNet (computational infeasibility for LLM fine-tuning) is valid. Adding MoMoE comparisons improves baseline coverage. \nIn view of the explanations from authors, I will increase my score accordingly.", "rebuttal": "We would like to express our gratitude for your valuable comments and suggestions. Please find our responses to your comments below.\n## Theoretical Analysis\nThe conclusions drawn from the theoretical analysis in the paper are not guaranteed to hold for non-convex deep neural networks, which are typically used in practice. However, the goal of the theoretical analysis is not to fully explain the behavior of the proposed algorithm in practical scenarios. Instead, it demonstrates that the algorithm’s design is both sensible and grounded in a rigorous theoretical foundation. Finally, the theoretical convergence analysis of deep neural networks used in practice remains an open research problem and is beyond the scope of this paper. It is also useful to mention that assumptions A1–A3 are common in convex optimization.\n## Complexity\nThe worst-case computational and memory complexities of IterativeRS are of the same order as those of RS. Both algorithms update the model over $T$ batches. IterativeRS performs merging every $m$ rounds; however, the cost of merging is negligible compared to expert training, as it typically does not involve additional training. Furthermore, as described in line 7 of Algorithm 1, between two merging rounds, a subset of objectives can be selected for fine-tuning. This can significantly reduce the complexity of IterativeRS, especially when compared to RS. Setting $M=1$ results in a complexity comparable to that of MORLHF.\n## Experiments\nWe performed further experiments on the text summarization task. To improve the performance of the proposed algorithm, we considered seven different sets of merging weights, including [1/3, 1/3, 1/3], permutations of [1/6, 1/6, 2/3], and permutations of [1/6, 5/12, 5/12]. Previously, we only considered the uniform merging weights [1/3, 1/3, 1/3]. During training, at each merging step, we calculated the reward of each merged model on the training data and chose the one with the highest average reward for the next iteration. To obtain the final merged model after training, we measured the average reward of all seven merged models and chose the one with the highest average reward on the validation data. We then tested the performance on the test set. We followed the same procedure for RS as well. Note that RS does not perform merging during training. The table below shows the new results. These results demonstrate the effectiveness of the flexible merging strategy in IterativeRS, as it significantly outperforms other methods on all metrics.\n\n|  | faithful  | summary | deberta | Avg Reward | Sharp-like |\n| :--------- | --------: | :---------: | :--------: | :------------: | :----------: |\n| RS | 0.6549 | 0.4644 | 0.3178 | 0.4791 | 3.3719 |\n| IterativeRS | 0.7023 | 0.5160 | 0.5834 | 0.6006 | 6.2180 |\n\nIn addition, to incorporate more standard metrics into our evaluation, we define a new metric as the average of all rewards and the ROUGE score. The results are presented in the table below. We can either replace the \"Avg Reward\" column in Table 3 with this new metric or update the definition of Avg Reward accordingly. As can be seen, IterativeRS achieves the highest average score.\n\n| methods | MORLHF  | RS | RiC | IterativeRS |\n| :--------- | --------: | :---------: | :--------: | :------------: |\n| Avg Score | 0.3943 | 0.3925 | 0.4518 | 0.4785 |\n\n### Baselines\nTo the best of our knowledge, we have compared the proposed algorithm with all relevant state-of-the-art multi-objective reinforcement learning methods for LLM fine-tuning. Sequence generation with GFlowNet can be viewed as a soft Q-learning problem [15]. However, applying Q-learning to LLM fine-tuning may not be computationally feasible. Therefore, we did not consider employing or comparing against existing GFlowNet methods. Note that in all experiments, we employ LLMs for sequence generation. \n\nIterativeRS and MolMoE have distinct and non-overlapping functionalities and applications. MolMoE is an expert merging method designed for molecular applications, whereas IterativeRS focuses on expert training. Therefore, these two methods can be used in conjunction. We applied MolMoE to both IterativeRS and RS, and the results are presented below. One of the main advantages of using MolMoE is its ability to handle scenarios where preferences over objectives change dynamically over time, which is outside the scope of this paper's experimental study.\n\n|  | $\\alpha$ energy | gap | $U_0$ energy | Avg Reward | Sharp-like |\n| :--------- | --------: | :---------: | :--------: | :------------: | :----------: |\n| RS+MolMoE | 1.4499 | 0.9715 | 1.5988 | 1.3400 | 4.1120 |\n| IterativeRS+MolMoE | 1.5651 | 0.9941 | 1.6420 | 1.4004 | 3.9938 |\n\nPlease let us know if you have further comments and concerns."}, {"review_id": "8Gm6aRw9IB", "reviewer": "Reviewer_2pTb", "summary": "This paper introduces a framework for fine-tuning foundation models by leveraging multiple given rewards functions. The proposed approach unifies two existing techniques, Reward Combining and Reward Soups. Furthermore, the authors provide theoretical support for their method, including an upper bound on the error between the optimal policy and the policy trained using their framework.", "questions": "Below are some questions and suggestions for the authors. Addressing these could help clarify certain aspects of the work and potentially strengthen the paper:\n\n1) The proof of Theorem 1 appears to utilize notation that differs from what is used in the main text. This inconsistency can create confusion for the reader. Would the authors consider revising the proof to maintain consistent notation with the rest of the paper? Also, clarity of Theorem 1 proof and Lemma 1: The proof of Theorem 1, especially the transition in notation around lines 432-435 and its connection to Lemma 1, could be made more accessible. The current presentation has some notational overlaps that harm comprehension. To improve clarity and consistency, please consider: 1.1) Adding a table of notations at the beginning of the appendix. 1.2) Providing a detailed proof of Lemma 1 using the same consistent notation as presented in the main text. 1.3) Breaking down the proof of Theorem 1 into more clearly delineated logical steps.\n\n2) The experiments and analysis seem to exclusively use weights $w_i = \\frac{1}{N}$. The paper mentions that other settings for these importance weights are possible. Have the authors investigated alternative parametrizations for $w_i$? Presenting results from such explorations could demonstrate the sensitivity of the method to this choice or some different behaviour.\n\n3) Have the authors considered comparing diversity-specific metrics (e.g., entropy, or other established measures) between their method and baselines? In addition, authors mention some GFlowNets baselines [1]. GFlowNet-based approaches are also good at generating more diverse candidates, have you considered a comparison with GFlowNets?\n\n4) In conclusion authors write: “We showed that both MORLHF and Rewarded Soups can be viewed as special cases of IterativeRS, highlighting its greater flexibility for performance enhancement.” Though, in my humble opinion, it is a little hidden, I would rather make a detailed section pinpointing this relation. Although, it is a minor issue of mine. \n\nReferences: \n\n[1] Jain et al. Multi-Objective GFlowNets. ICML 2023.", "limitations": "The paper does not include a dedicated section discussing the potential limitations of the proposed algorithm. Here are my thoughts:\n\n1) The current experimental results do not conclusively demonstrate that the proposed method offers a significant advantage over standard fine-tuning baselines, particularly for tasks outside the biological domain. Further evidence would be needed to establish its superiority in a broader range of applications.\n\n2) While the paper unifies existing techniques, the extent to which this unification yields fundamentally novel insights beyond the combination itself is not fully clarified.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Quality**:\nThe paper appears to be technically sound in its formulation. However, the claims made could be more robustly supported by the experimental results.\n\nStrengths: The theoretical derivation of an error bound is a positive aspect. The core idea of unifying Reward Combining and Reward Soups is presented.\n\nWeaknesses: The experimental validation seems limited. There is a need for more extensive experiments to demonstrate the stability and generalizability of the proposed method. For instance, the method shows promise primarily on biological tasks. The results on the text summarization task, however, indicate a marginal improvement only in average reward, while falling behind in other metrics, which is not really representative of the method’s utility. The work could benefit from a more thorough empirical evaluation.\n\n**Clarity**: The paper is generally well-organized, with a coherent structure and logical flow between sections.\n\nStrengths: The overall organization is good.\n\nWeaknesses: At times, the exposition could be clearer, making it somewhat challenging to fully grasp the rationale behind certain choices or implications of some arguments. (See Questions 1 and 4)\n\n**Significance**:\nThe potential impact of this work on the community is an area of concern.\n\nStrengths: The paper attempts to unify existing techniques, which can be valuable. The inclusion of theoretical analysis is commendable.\n\nWeaknesses: The proposed method appears to be a relatively straightforward combination of two prior techniques. For a method that builds upon existing ideas, especially if the combination seems intuitive, a comprehensive suite of experiments demonstrating clear advantages across diverse tasks would be expected. The current set of experiments (two biological tasks and one text-based task where performance is not compelling) does not fully meet this expectation. \n\n**Originality**:\nThe work does present a novel combination of existing methods.\n\nStrengths: To the best of my knowledge, the specific unified framework presented has not been proposed previously.\n\nWeaknesses: While the combination is new, the novelty would be more impactful if accompanied by surprising empirical results or deeper, unexpected theoretical insights.", "was_revised": true, "final_justification": "I believe that the main issues I pointed out were adequately addressed in the rebuttal: 1) poor performance on the text summarization task, 2) concern in the novelty because of the method being a combination of previous approaches. As they are resolved, I decided to raise my score to borderline accept. I also find the authors rationale about not comparing to GFlowNet-based methods to be reasonable.", "rebuttal": "Thank you for taking the time to review our paper and sharing your valuable thoughts and suggestions. Please find our responses to your comments below.\n\nWe would like to clarify that the proposed algorithm is **not** a combination of MORLHF and RS.  The paper provides a new perspective that MORLHF can be viewed as the case where merging is done after every batch. However, if merging happens after multiple batches, the algorithm after each merging is no longer equivalent to MORLHF.\n\n## Experiments\nWe performed further experiments on the text summarization task. To improve the performance of the proposed algorithm, we considered seven different sets of merging weights, including [1/3, 1/3, 1/3], permutations of [1/6, 1/6, 2/3], and permutations of [1/6, 5/12, 5/12]. During training, at each merging step, we calculated the reward of each merged model on the training data and chose the one with the highest average reward for the next iteration. To obtain the final merged model after training, we measured the average reward of all seven merged models and chose the one with the highest average reward on the validation data. We then tested the performance on the test set. We followed the same procedure for RS as well. Note that RS does not perform merging during training. The table below shows the new results. These results demonstrate the effectiveness of the flexible merging strategy in IterativeRS, as it significantly outperforms other methods on all metrics.\n|  | faithful  | summary | deberta | Avg Reward | Sharp-like |\n| :--------- | --------: | :---------: | :--------: | :------------: | :----------: |\n| RS | 0.6549 | 0.4644 | 0.3178 | 0.4791 | 3.3719 |\n| IterativeRS | 0.7023 | 0.5160 | 0.5834 | 0.6006 | 6.2180 |\n\n### Diversity and GFlowNet\nThe focus of this paper is on reinforcement learning for LLM fine-tuning. Due to pre-training on vast amounts of unlabeled data, LLMs are known to generate diverse outputs. During fine-tuning, we monitor the KL divergence between the policy and a reference policy to ensure that the fine-tuned model does not deviate too far from the initial pre-trained model. For example, we set kl_coef = 0.05 during PPO fine-tuning. \n\nSequence generation with GFlowNet can be viewed as a soft Q-learning problem [15]. However, applying Q-learning to LLM fine-tuning may not be computationally feasible. Therefore, we did not consider employing or comparing against existing GFlowNet methods. Note that in all experiments, we employ LLMs for sequence generation.\n\n## Notation and Presentation:\nSince Lemma 1 is taken from [19], we wanted to retain its original notation. We understand the confusion this may have caused. To address this, we will unify the notation of Lemma 1 with the rest of the paper and add its proof to the appendix. Furthermore, based on your suggestion, we will include a notation table at the beginning of the appendix.\n\nPlease let us know if you have further comments and concerns."}]

Author rebuttal:

Meta-review: This paper proposes a method for RL fine-tuning of foundation models using several rewards, which the authors term IterativeRS. This is achieved by carefully controlling the trade-off between specializing policies towards different rewards and fusing them into a single shared policy. 

 

On the positive side, all reviewers generally considered the paper technically sound and well-written. They noted that IterativeRS elegantly unifies two existing RL approaches (RS on one end, and MORLHF on the other). Most reviewers also praised the diversity of domains in which IterativeRS is evaluated. 

 

On the other hand, some reviewers were unconvinced by relatively modest performance improvements in some settings; although during rebuttal authors provided improved results by tuning the task weighing, this approach only yielded improvements in a subset of domains. Some reviewers also considered the novelty of the work to be somewhat limited. 

 

After rebuttal, four reviewers were leaning towards acceptance, while one (Reviewer ysgq) towards rejection. In subsequent Reviewer-AC discussion, the negative reviewer kept their score, but also was not strongly opposed to acceptance. 

 

Taking all of the above into account, I believe the paper is technically sound, has extensive evaluation, offers modest but consistent performance gains, and is somewhat novel although heavily building on prior work. I recommend acceptance, and encourage authors to take reviewer feedback into account when preparing a camera-ready version of the paper.
Decision: Accept (poster)