Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards
Authors: Alexandre Rame
Guillaume Couairon
Corentin Dancette
Jean-Baptiste Gaya
Mustafa Shukor
Laure Soulier
Matthieu Cord
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep learning
Foundation models
Fine-tuning
Reward optimization
Linear mode connectivity
Weight averaging
Model soups
Robustness
Generalization
Alignment
Multi objective learning.
TL;DR: We introduce rewarded soup, a strategy to trade-off between multiple rewards when RL fine-tuning foundation models; we first learn one network for each reward, and then linearly interpolate their weights despite the architecture's non-linearities.

Abstract: Foundation models are first pre-trained on vast unsupervised datasets and then fine-tuned on labeled data. Reinforcement learning, notably from human feedback (RLHF), can further align the network with the intended usage. Yet the imperfections in the proxy reward may hinder the training and lead to suboptimal results; the diversity of objectives in real-world tasks and human opinions exacerbate the issue. This paper proposes embracing the heterogeneity of diverse rewards by following a multi-policy strategy. Rather than focusing on a single a priori reward, we aim for Pareto-optimal generalization across the entire space of preferences. To this end, we propose rewarded soup, first specializing multiple networks independently (one for each proxy reward) and then interpolating their weights linearly. This succeeds empirically because we show that the weights remain linearly connected when fine-tuned on diverse rewards from a shared pre-trained initialization. We demonstrate the effectiveness of our approach for text-to-text (summarization, Q&A, helpful assistant, review), text-image (image captioning, text-to-image generation, visual grounding), and control (locomotion) tasks. We hope to enhance the alignment of deep models, and how they interact with the world in all its diversity.

Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards
Alexandre Ramel Guillaume Couairon 1.2t, Mustafa Shukor
Corentin Dancette', Jean-Baptiste Gayal Laure Soulier', Matthieu Cord 1.3
Meta AI VValco.ai 'Sorbonne Université, CNRS, ISIR, Paris, France
Abstract
Foundation models are first pre-trained on vast unsupervised datasets and then fine-tuned on labeled data. Reinforcement learning, notably from human feedback (RLHF), can further align the network with the intended usage. Yet the imperfee- tions in the proxy reward may hinder the training and lead to suboptimal results; the diversity of objectives in real-world tasks and human opinions exacerbate the issue. This paper proposes embracing the heterogeneity of diverse rewards by following mulli-policy strategy. Rather than focusing on single a priori reward, we aim for Pareto- Pareto-optimal generalization across the entire space of preferences. To this end, we propose rewarded soup, first specializing multiple networks independently (one for each proxy reward) and then interpolating their weights linearly. This succeeds empirically because we show that the weights remain linearly connected when fine- tuned diverse rewards from shared pre-trained initialization. We demonstrate the effectiveness of our approach for text-to-text (summarization, Q&A, helpful assistant, review), text-image (image captioning, text-to-image generation, visual grounding), and control (locomotion) tasks. We hope to enhance the alignment of deep models, and how they interact with the world in all its diversity.
Introduction
Foundation models [1] have emerged the standard paradigm learn neural networks' weights. They are typically first pre-trained through self-supervision [2, 3, 4, 5] and then fine-tuned [6, 7] via supervised learning [8]. Yet, collecting labels is expensive, and thus supervision may not cover all possibilities and fail to perfectly align [9, 10, 11] the trained network with the intended applications. Recent works [12, 13, 14] showed that deep reinforcement learning (DRL. helps by learning from various types of rewards.
prominent example is reinforcement learning from human feedback (RLHF) [12, 15, 16, 17], which appears as the current go-to strategy refine large language models (LLMs) into powerful conversational agents such as ChatGPT [13, 18]. After pre-training on next token prediction [19] using Web data, the LLMs are fine-tuned follow instructions [20, 21, 22] before reward maximization. This RL strategy enhances alignment by evaluating the entire generated sentence instead of each token independently, handling the diversity of correct answers and allowing for negative feedback [23]. Similar strategies have been useful in computer vision (CV) [14, 24], for instance to integrate human aesthetics into image generation [25, 26, 27].
Diversity of proxy rewards. RI. is usually seen as more challenging than supervised training [28], notably because the real reward-ideally reflecting the users' preferences-is often not specified at training time. Proxy rewards are therefore developed to guide the learning, either as hand-engineered metrics [29, 30, 31] more recently in RLHF models trained reflect human preferences
Project lead, main contributor, correspondence to alexandre.rom rame @ee@@@iiiii......... Equall experimental contribution, order determined at random. Further information and resources related to this project can be found on this website,
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

RL fine-tuning
MORI for 0.5
Rg rounded
Se - 2-
R$ for A 0.5
Unsupervised pre-training large datssed
Supervised fine-tuning on target bask
Rewarded soups by
weight interpo
fine-tuning pracy revonn Ray
-144
RS froat (C - 1) + 123
As
MORL foot: (9u slo to)
Selection
LZMMA
( 2 2m 11
(A)/i, solected by users docording to their preferences
ON
(a) Illustration of our proposed rewarded soup (RS).
(b) LLaMA RLHF for summarization.
Figure I: Figure I(a) details the different steps in rewarded soup. After unsupervised pre-training and supervised fine-tuning, we launch N independent RI. fine-tunings on the proxy rewards { RR ] Then we combine the trained networks by interpolation in the weight space. The final weights are adapted at test time by selecting the coefficient A. Figure ((b) shows our results (extended in Figure 2(a)) with L--b [44] instruct fine-tuned
on Alpaca [22]. when RL fine-tuning for news summarization [12] with N 2 reward models assessing diverse preferences of summaries. With only two trainings (Rl and R2 rewarded
Figure 1(b)), the A-interpolation (0 S X S 1) reveals the green front of Pareto-optimal solutions, i.e. that cannot be improved for one reward without sacrificing the other. RS matches the costly yellow front of objectvv (MORL) [45, 46] requiring multiple trainings on different linear weightings over the rewards (1 - 11) x R1 + x R2 with 0 S S 1.
32, 33]. Nonetheless, designing reliable proxy rewards for evaluation is difficult. This reward misspecificattion [9, 34] between the proxy reward and the users' actual rewards can lead to unforeseen consequences [35]. Moreover, the diversity of objectives in real-world applications complicates the challenge. In particular, human opinions can vary significantly [36, 37, 38] on subjects such as aesthetics [39], politics or faimess [40]. Humans have also different expectations from machines:
for example, while [41] stressed aligning LLMs towards harmless feedback, [42] requested helpful
non-evasive responses, and others' [43] interests are to make LLMs engaging and enjoyable. Even
hand-enginecred metrics can be in tension: generating shorter descriptions with higher precision can increase the BLEU [29] score but decrease the ROUGE [30] score due to reduced recall.
Towards multi-policy strategies. Considering these challenges. a single model cannot be aligned with everyone's preferences [13]. Existing works align towards consensus-bare based user [47, 48], relying on the "wisdom of the crowd" [49]. inherently prioritizing certain principles [42, 50]. resulting in unfair representations of marginalized groups [51, The trade-off's [53] are decided priori before training, shifting the responsibility to the engineers, reducing transparency and explainability [54], and actually aligning towards the "researchers designing the study" [13, 55]. These limitations, discussed in Appendix 1. highlight the inability of single-policy alignment strategies to handle human diversity. Yet, "human- "human-aligned artificial intelligence a multi-objective problem" [56]. Thus, we draw inspiration from the multi-objective reinforcement learning (MORL) literature [45, 46, 57, 58, 59, 60, 61, 62] and [54]; they argue that tackling diverse rewards requires shifting from single- policy to multi-policy approaches. As optimality depends on the relative preferences across those rewards, the goal is not to learn single network but rather set of Pareto-optimal networks [63].
In this paper, we propose rewarded soup (RS), an efficient and flexible multi-policy strategy to fine-tune any foundation model. As shown in Figure 1(a), first use RL to learn one network for each proxy reward; then, we combine these expert networks according to user preferences. This a posteriori selection allows for better-informed trade-offs, improved transparency and increased fairness [54, 64]. The method to combine those networks is our main contribution: we do this through linear interpolation in the weight space, despite the non-linearities in the network. This is in line with recent findings on linear mode connectivity (LMC) [65, 66]: weights fine-tuned from shared pre- pre-trained initialization remain linearly connected and thus can be interpolated. This LMC inspired a plethora of weight interpolation (WI) strategies [67, 68, 69, 70, 71, 72], discussed in Section 4. Actually, the name rewarded soups follows the terminology of model soups [67], as we combine various ingredients each rewarded differently. Unlike previous works, which focused on supervised learning, we explore LMC in RI., in a challenging setup where each training run uses different reward. Perhaps surprisingly, we show that we can trade off the capabilities of multiple weights in

---

single final model, thus without any computational overhead. This enables the creation of custom weights for any preference over the diverse rewards. We summarize our contributions as follows:
We advocate a mulli-policy paradigm to align deep generative models with human prefer- ences and reduce reward misspecification.
We then propose new multi- pulti-policy strategy, rewarded soup, possible when fine-tuning foundation models with diverse rewards. By weight interpolation. it defines a continuous set of (close to) Pareto-optimal solutions, approximating more costly multi-policy strategies.
In Section 3, we consistently validate the linear mode connectivity and thus the effectiveness of RS across a variety of tasks and rewards: RLHF fine-tuning of LLaMA, multimodal tasks such as image captioning or text-to-image generation with diffusion models, as well as locomotion tasks.
Rewarded soups
2.1 RL fine-tuning with diverse rewards
We consider deep neural network f of a fixed non-linear architecture (e.g.. with batch normalization [73], ReLU layers [74] or self-attention [75]). It defines policy by mapping inputs x to f(xx,
when parametrized by 0. For a reward R (evaluating the correctness of the prediction according to some preferences) and a test distribution T of deployment, our goal is to maximize Seer (z, 0))-
For example. with a LLM, I would be textual prompts. å would evaluate if the generated text is harmless [76]. and T would be the distribution of users' prompts. Learning the weights 0 is now commonly a three-step process: unsupervised pre-training, supervised fine-tuning, and reward optimization. Yet At is usually not specified before test time, meaning we can only optimize a proxy reward R during training. This reward misspecification between and Å may hinder the alignment of the network with R. Moreover, the diversity of human preferences complicates the design of R.
Rather than optimizing one single proxy reward, our paper's first key idea is to consider family of diverse proxy rewards { Each of these rewards evaluates the prediction according to different (potentially conflicting) criteria. The goal then becomes obtaining coverage set of policies that trade-off between these rewards. To this end, we first introduce the costly MORL baseline. Its inefficiency motivates our rewarded soups, which leverages our second key idea: weight interpolation.
MORL baseline. The standard MORL scalarization strategy [45, 46] (recently used [62] to align LLMs) linearizes the problem by interpolating the proxy rewards using M different weightings. Specifically, during the training phase, M trainings are launched, with the j-th optimizing the reward E/=1 (1 R,, where Vj E [1, {n4 E AN the N N-simplex s.t. N H = 1 and 0 S pd S 1. Then, during the selection phase, the user's reward R becomes known and the j-th policy that maximizes f on some validation dataset is selected. We typically expect to select j such that ΣN LRRR R₁ Rs R linearly approximates the user's reward. Finally, this j-th weight is used during the inference phase on test samples. Yet, a critical issue is that "minor [preference] variations may result significant changes in the solution" [77]. Thus, a high level of granularity in the mesh of A N is necessary. This requires explicitly maintaining large set of M >> N networks, practically one for each possible preference. Ultimately, this MORL strategy unscalable in deep learning due to computational, memory, and engineering costs involved (see further discussion in Appendix A.2).
Rewarded soup (RS). In this paper, we draw inspiration from the weight interpolation literature. The idea is to learn expert weights and interpolate them linearly to combine their abilities. Specifically, we propose RS, illustrated in Figure ((a) and whose recipe described below. RS alleviates MORL's scaling issue as it requires only M = N trainings while being flexible and transparent.
1. During the training phase, we optimize set of N expert weights (0.) each correspond- ing to one of the N proxy rewards [ and all from shared pre-trained initialization.
2. For the selection phase, we linearly interpolate those weights to define a continuous set of rewarded soups policies: (EN x - (A.. Practically, uniformly sample M
3

---

interpolating coefficients (1)) } from the N-simplex AN and select the j-th that maximizes the user's reward ft validation samples, i.e., argmax A - NO.))
3. For the inference phase, we predict using the network f parameterized by Σi= X,O.
While MORL interpolates the rewards, RS interpolates the weights. This is a considerable advantage as the appropriate weighting X, which depends on the desired trade-off, can be selected a posteriori; the selection is achieved without additional training, only via inference on some samples. In the next Section 2.2 we explicitly state the Hypotheses and 2 underlying in RS. These are considered Working Hypotheses as they enabled the development of our RS strategy. Their empirical verification will be the main motivation for our experiments on various tasks in Section 3.
2.2 Exploring the properties of the rewarded soups set of solutions
2.2.1 Linear mode connectivity of weights fine-tuned on diverse rewards
We consider (0: (0 (or { for brevity) fine-tuned on {Ra]: from shared pre-trained initialization. Previous works [65, 66, 67, 72] defined linear mode connectivity (LMC) w.r.t. a single performance measure (e.g., accuracy loss) in supervised learning. We extend this notion in RL with N rewards, and define that the LMC holds if all rewards for the interpolated weights exceed the interpolated rewards. It follows that the LMC condition which underpins RS's s viability is the Hypothesis 1 below.
Working Hypothesis 1 (LMC). v(A:): E A and & E (1...., N), Rs E, xs 02) 2 Σ, (O.).
2.2.2 Pareto optimality of rewarded soups
The Pareto front (PF) is the set of undominated weights. for which no other weights can improve a reward without sacrificing another, i.e., {0 30' E O s.t. {R,(0'); >N {R:(0)}}} where > n the dominance relation in Rr. In practice, we only need to retain one policy for each possible value vector, i.e., Pareto coverage set (PCS). We now introduce the key Hypothesis 2, that state the Pareto-optimality of the solutions uncovered by weight interpolation in RS.
Working Hypothesis 2 (Pareto optimality). The set (Σ,>, As 0142 E AND is PCS of (Ri)e-
Empirically, in Section 3, we consistently validate Hypotheses and 2. Theoretically, in Appendix C.2, we prove they approximately hold, in simplified setup (quadratic rewards with co-diagonallzzble Hessians) justifiable when weights remain close.
Remark 1. Hypotheses! I and 2 rely on good pre-trained initialization, making RS particularly well- suited to fine-tune foundation models. This because pre-training prevents the weights from diverging during training [66]. When the weights remain close, we can theoretically justify Hypotheses I and (see Appendix C.2) and, more broadly, demonstrate that WI approximates ensembling [78, 79] (see Lemma 4). In contrast, the LMC does not hold when training from scratch [66]. Neuron permutations strategies [80, 81] tried to enforce connectivity by aligning the weights, though (so far) with moderate empirical results: their complementarity with RS is a promising research avenue.
Remark 2. Pareto- Cyreto-optimality in Hypothesis 2 is defined w.r.t. a set of possible weights θ. Yet, in full generality, improvements in initialization, RL algorithms, data, or specific hyperparameters could enhance performances. In other words, for real-world applications, the true PF is unknown and needs to be defined w.r.t. a training procedure. In this case, θ represents the set of weights attainable by fine-tuning within a shared procedure. As such, in Section 3 we analyze Hypothesis by comparing the fronts obtained by RS and scalarized MORL while keeping everything else constant.
2.2.3 Consequences of Pareto optimality the user's reward is linear in the proxy rewards
Lemma (Reduced reward misspecification in the linear case). If Hypothesis 2 holds, and for linear reward R = Σ, R4 with (ūrte E AN. then 3(2hh E AN such that E, A 0, is optimal for R.
The proof outlined in Appendix C.I directly follows the definition of Pareto optimality. In simpler terms, Lemma implies that if Hypothesis holds, RS mitigates reward misspecification for linear rewards: for any preference fú, there exists a À such that the X-interpolation over weights maximizes the ji-interpolation over rewards. In practice, as we see in Figure 5(a), we can set À = i., or valddava \ on other samples.

---

3 Experiments
In this section we implement RS across a variety of standard learning tasks: text-to-text generation, image captioning, image generation, visual grounding. visual question answering, and locomotion. We use either model or statistical rewards. We follow a systematic procedure. First, we independently optimize diverse rewards on training samples. For all tasks, we employ the default architecture, hyperparameters and RL. algorithm; the only variation being the reward used across runs. Second, we evaluate the rewards on the test samples: the results are visually represented in series of plots. Third, we verify Hypothesis by examining whether RS's rewards exceed the interpolated rewards. Lastly, as the true Pareto front is unknown in real-world applications, we present empirical support for Hypothesis 2 by comparing the front defined by RS (sliding À between 0 and 1) to the MORL's solutions optimizing the y-weighted rewards (sometimes only μ = 0.5 for computational reasons). Implementations are released on github, and this website provides additional qualitative results.
3.1 Text-to-text: LLaMA with diverse RLHFs
Given the importance of RLHF to train LLMs, we begin our experiments with text-to-text generation. Our pre-trained network is LLaMA-7b [44], instruction fine-tuned [20, 83] on Alpaca [22]. For RL training with PPO [84], we employ the trl package [85] and the setup from [86] with low-rank adapters (LoRA) [87] for efficiency. We first consider summarization [12, 17] tasks on two datasets:
Reuter news [88] in Figures 1(b) and 2(a) and Reddit TL:DR [89] in Figure 2(b). We also consider
answering Stack Exchange questions [90] in Figure 2(c), movie review generation in Figure 2(d), and
helpfulness conversational assistant [49] in Figures 2( and 2(f). To evaluate the generation in
344
Ri gewanded
0.5
-194
5-000
Buth pewmodd
-15-
Mt, nosadded
-30
LLMMA
RL: A
14
RL: Pig
-00
MORL Sigh
RSS. (2 A] 2.4
(a) News summary.
(b) Reddit TL:DR summary.
(c) Stack Exchange Q&A.
Ro remanded
rovarded
Rg renuddo
Buck mesrivd
18
0.5
-130
6
R. rewae 551
-130
BL: Ri
Siglt
13.0044
-43
LLMA mn
MORL:
RL: Ra
X
Bli, Ri
33
-.30
T.L.AA
somidddd
BL. Ri
BI. R.
MIXE: gohe
nS 12/
(d) Useful movie review.
(e) Helpful assistant.
(f) Helpful assistant: spider map.
Figure 2: RLHI results in NLP with LLaMA-7b [44] and reward models R. from HuggingFace [82]. The blue line reports checkpoints results along the training trajectory of 0, rewarding Rt, the red line 02 rewarding Ra. and the purple line the MORL rewarding R2, Our rewarded soup (RS) linearly interpolates between the weights 0, and 0z: sliding the interpolation coefficient from 0 to 1 reveals the green solid front of rewarded soups solutions. In Figures ((a) and 2(b), we additionally show the multiple MORL runs rewarding
(1 -1-10 μ) x R1 + R₂ with preferences 0 < μ < 1. It reveals a similar yellow front, yet more costly. In Figure 2(f), we uniformly (A, =1 average the weights fine-tuned for the assistant task on N = 4 reward models.
5

---

the absence of supervision, we utilized N 2 different reward models (RMs) for each task, except in Figure 2(f) where N These RMs were trained on human preferences datasets [15] and all open-sourced on FuggingFace [82]. For example in summarization, R1 follows the "Summarize from Human Feedback" paper [12] and focuses on completeness, while R2 leverages "contrast candidate generation" [91] to evaluate factuality. For other tasks, we rely on diverse RMs from OpenAssistant [92]: though they all assess the answer is adequate, they differ by their architectures and procedures. Table I details the experiments.
The results are reported in Figure The green front, defined by RS between the two weights special- ized on R1 and R2, is above the straight line connecting those two points, validating Hypothesis 1. Second, the front passes through the point obtained by MORL fine-tuning on the average of the two rewards, supporting Hypothesis 2. Moreover, when comparing both full fronts, they have qualitatively the same shape; quantitatively in hypervolume [93] (lower better, the area over the curve w.r.t. an optimal point), RS's hypervolume is 0.367 vs. 0.340 for MORL in Figure 2(a), while it is 1.176 vs. 1.186 Figure 2(b). Finally, Figure 2(f), we use N RMs for the assistant task and uniformly average the N = 4 weights, confirming that RS can scale and trade-off between more rewards.
3.2 Image-to-text: captioning with diverse statistical rewards
RI is also effective for multimodal tasks [14] such in image captioning [24], generate textual descriptions of images. Precisely evaluating the quality of a prediction w.r.t. a set of human-written
2-85
0:21
BLAVI- powarded
ngg
0.20
Ry BLEE2
De tratzed tilt
DEEU ewwrreee
Partaked MTATAA
0.24
RL: DILEVI
RL: BLET'S
RL: BeUGz
Ri. METKOR
MORE:
B8:
6,55
Rg SLEUT
BLEU
(a) BLEUI and ROUGE.
(b) BLEUI and BLEU4.
(c) Captioning: spider map.
Figure 3: Results in image captioning on COCO [94]. As rewards R1 (blue stars every epoch) and Ra (red stars), we consider standard statistical metrics: BLEU1 (1-gram overlap), BLEU4 (4-grams overlap), ROUGE. METEOR and CIDEr. Figure 3(a) include the MORL training trajectories optimizing (1 μ) x BLEU1 + x ROUGE, uncovering a yellow front similar to RS's green front. In Figure 3(c), RS uniformly averages the 5 weights (one for each resulinng in the largest and the best trade-off between the rewards.
22h DL pol lala
J00M
(a) BLEUI first.
(b) BLEU4 first.
(c) ROUGE first.
Figure 4: Those spider maps uniformly average M 5 weights for captioning. where 0: fine-tuned on BLEUI (B1), 02 on BLEU4 (B4), 03 on ROUGE (R), 04 METEOR (M) and Os CIDEr (C). To show different combinations among the (3) possible, we iterate in clockwise direction starting in Figure 4(a) from i == 1 (always including On optimized on BLEUI), in Figure 4(b) from = (always including 02 optimized on BLEU4), and in Figure 4(c) from (always including 0g optimized ROUGE).

---

ROUGE ronomeed
MUIL
660
A- G.
BLEUY
R:: R2T
6000
BLBUT Seanded
605-
6051
5)
CIDE.
6400
(1- 3) Il (ausrn) Maponcon
03
BAR
(a) Analysis of Lemma 1.
(b) Performances w.r.t. A.
(c) RS vs. prediction ensembling.
Figure 5: Results in captioning for Rı = BLEU1 and Ra = ROUGE. When normalized, rewards are set to I for the init and O for the worst model. Figure 5(a) validates Lemma by reporting results of RS (for varying A) and of MORL (for varying for varying user's preference Figure 5(b) evaluates different rewards as function of the interpolating coefficient. Figure 5(c) reports ensembling scores when interpolating predictions.
captions challenging, thus the literature relies on various non differentiable metrics: e.g., the precision-focusee BLEU [29], the recall-focused ROUGE [30], METEOR [95] handling synonyms and CIDEr [31] using TF-IDF. As these metrics are proxies for human preferences, good trade-offs are desirable. We conduct our experiments on COCO [94], with ExpansionNetv2 [96] network and a Swin Transformer [97] visual encoder, initialized from the state-of-the-art weights of [96] optimized on CIDEr. We then utilize the code of [96] and their self-critical [24] procedure (a variant of REINFORCE [98] to reward the network on BLEUI, BLEU4, ROUGE or METEOR. More details and results can be found in Appendix E.
We observe in Figure 3 that tuning solely BLEUI sacrifices some points on ROUGE or BLEU4, Yet interpolating between 02 and fg uncovers a convex set of solutions approximating the ones obtained through scalarization of the rewards in MORL. When comparing both full fronts in Figure 3(a), they qualitatively have the same shape, and quantitatively the same hypervolume [93] of 0.0 One of the strengths of RS is its ability to scale to any number of rewards. Figure 3(c). we uniformly (A2 = ) average N = 5 weights fine-tuned independently. It improves upon the initialization [96] and current sfate-of-the-art on all metrics, except for CIDEr, on which [96] was explicitly optimized. We confirm in Figure 4 that RS can handle more than 2 rewards through additional spider maps. Specifically, we compare the performances across all N = 5 metrics when averaging 1 M < N networks (each fine-tuned on one of the N rewards, thus leaving out N M rewards at training) and sequentially adding more networks to the weight average. We consistently observe that adding one additional network specialized
one additional reward extends the scope of the possible rewards that RS can tackle Pareto-optimally. Figure 5(a) validates Lemma 1: for any linear preference fa over the proxy rewards, there exists an optimal solution in the set described by RS. Two empirical strategies to set the value of À are close to optimal: selecting X = ja if ja is known, or cross-validating (CV) A if a different data split [99] is available. Moreover, Figure 5(b) (and Appendix E) investigate all metrics as evaluation. Excluding results' variance, we observe monotonicity in both training rewards, linear in BLEUI and quadratic in ROUGE. For other evaluation rewards that cannot be linearly expressed over the training rewards, the curves' concavity shows that RS consistently improves the endpoints, thereby mitigating reward misspecification. The optimal À depends on the similarity between the evaluation and training rewards: e.g., best BLEU2 are with small A. Lastly, as per [100] and Lemma 4, Figure 5(c) suggests that RS succeeds because WI approximates prediction ensembling [78, 79] when weights remain close, interpolating the predictions rather than the weights. Actually, ensembling performs better, but cannot be fairly compared as its inference cost is doubled.
3.3 Text-to-image: diffusion models with diverse RLHFs
Beyond text generation, we now apply RS to align text-to-image generation with human feedbacks [25, 26, 33]. Our network is diffusion model [101] with 2.2B parameters, pre-trained on internal dataset of 300M images; reaches similar quality as Stable Diffusion [102], which was not used for copyright reasons. To represent the subjectivity of human aesthetics, we employ N = 2 open-source
7

---

reward models: ava, trained on the AVA dataset [103], and café, trained on a mix of real-life and manga images. We first generate 10000 images; then, for each reward, we remove half of the images with the lowest reward's s' score and fine-tune 10% of the parameters [104] on the reward-weighted negative log-likelihood [25]. Details and generations for visual inspection are in Appendix F. The results displayed in Figure 6(a) validate Hypothesis 1, as the front described by RS when sliding from 0 and 1 is convex. Moreover, RS gives a better front than MORL, validating Hypothesis 2. Interestingly, the reward model seems to be more
than cafe, as RL training on ava also enhances the scores of cafe. In contrast, the model fe performs poorly in terms ava in Figure 6(a). Nonetheless, RS with (1 A) Gava + A Ocafe outperforms aava alone, not only in terms of cafe, but also of ava when X E f0.1, 0.23. These findings confirm that RS can better align text-to-ine text-to-image models with a variety of aesthetic preferences. This ability to adapt at test time paves the way for a new form of user interaction with text-to-image models, beyond prompt engineering.
02
Large roveikl
cutikoas rewande!
pikpeceniin rovpffl
RS (1 Al Beoc + Storge
Rs velectt Ed
A 0
revided
arwanded
pre- cre-trccow 17
MURL foods (e
celecity
3S boot. (1) A/ Pas A Nasda
tred inil
oclecity Σ of
MORE sefocity
-I
4.00
poo-indend 182t
orvenial
B& [1 2) -arty
Sy. selosdy
(a) Image generation: ava and cafe.
(b) VG: Small and Large.
(c) Locomotion: risky and cautious.
Figure 6: Figure 6(a) reports RLHF experiments text-to-image generation with diffusion models. From the pre-trained initialization, we learn Baee and feafe by optimizing the two reward models ava and cafe. Interpolation between them reveals the green Pareto-optimal front, above the yellow MORL front. Figure 6(b) report our results in visual grounding (VG) on ReICOCO+ [105], where we optimize to predict boxes with IoU> 0.5 w.r.t the ground-truth, for objects of either small, medium or large size. Finally, Figure 6(c) report results from Section 3.5 for the locomotion lask with humanoids.
3.4 Text-to-box: visual grounding of objects with diverse sizes
We now consider visual grounding (VG) [105]: the task is to predict the bounding box of the region described by an input text. We use UnIVAL [106], a seq-to-seq model that predicts the box sequence of location tokens [107]. This model is pre-trained on large image-text dataset, then fine-tuned with cross-entropy for VG; finally, we use weighted loss between the cross-entropy and REINFORCE in the RL stage. As the main evaluation metric for VG is the accuracy (i.e., intersection over union (IoU)= > 0.5), we consider non-differentiable rewards: the accuracy on small, medium, and large objects. We design this experiment because improving results on all sizes simultancously is challenging, as shown in Figure 18(c), where MORL performs similarly to the initialization. The results in Figure 6(b) confirm that optimizing for small objects degrades performance on large ones; fortunately, interpolating can trade-off. In conclusion, we can adapt users' preferences test time by adjusting A. which in turn changes the object sizes that the model effectively handles. On the one hand, if focusing on distant and small objects, a large coefficient should be assigned to Smatl- On the other hand, to perform well across all sizes, we can recover initialization's 's performances by averaging uniformly (in Figure 18()).. More details are in Appendix G.
3.5 Locomotion with diverse engineered rewards
Teaching humanoids to walk in a human-like manner [108] serves as benchmark to evaluate RL strategies [109] for continuous control. One of the main challenges is to shape a suitable proxy
reward [110, 111], given the intricate coordination and balance involved in human locomotion. It
velocity - a x Σ. af, controlling the agent's 's velocity while regularizing the actions {ar}s taken over time. Yet, the penalty coefficient challenging to set. To address this, we devised two rewards the Brax physies engine [113]: a
standard [112] to consider dense rewards of the form
risky R with 0, and more cautious R2 with 1. Like in all previous tasks, RS's front in
8

---

Figure 6(c) exceeds the interpolated rewards, per Hypothesis 1. Moreover, the front defined by RS indicates an effective balance between risk-taking and cautiousness, providing empirical support for Hypothesis 2, although MORI, with 0.5 (i.e., 0.5) slightly surpasses RS's front. We provide animations of our RI. agent's locomotion on our website, and more details are in Appendix H.
3.6 Efficiency gain of RS over MORL
The efficiency gain of RS versus MORL is by design; when considering 2 rewards, RS only requires 2 fine-tunings, while MORI actually requires an infinite number of fine-tunings to reveal the entire front of preferences. To end this experimental section, quantify this efficiency gain by introducing R2 and the expectation is over all the possible user's preferences p. We then measure the difference between the expected rewards for RS (with 2 runs) and MORL (with M runs). Plotting this expected reward advantage for different values of M shows that MORL needs M > 2 to match RS.
in Figure the expected reward E (p-Uni/(0.1) Ro where RG (1 - ( Rí
(a) News summary.
(b) Reddit summary.
(c) Captioning.
Figure 7: Expected reward advantage of RS (always requiring only 2 trainings) over MORL (with M trainings). defined Ep-Unif((01) mazse RR (0
EAS maz R2. (O where
RG (1 A) R. x R2 is the reward for linear preference sampled uniformly between 0 and 1, A 0.. 1.0) is the of the possible values for X, and where the expectation for the MORL term is over the (1f) possible combinations Au of M elements from A (representing the M linear weightings a used for MORL training). We observe that MORL matches RS only for M sufficiently big.
Related work
Our RS approach leans on two key components from traditional DRL. The first is proxy rewards, whose design is challenging. Statistical metrics (the standard in captioning [24]) are not practical to measure human concepts [32] such as helpfulness [49, 76]. Thus recent RL.HF works [12, 13, 15] leverage human comparison of prediction learn reward model. Second, RS relies existing RL algorithms to maximize the given rewards. RS succeeds with variants
two the most common, REINFORCE [98] and PPO [84], suggesting it could be applied to others [114, 115]. When dealing with multiple objectives in deep learning, the common strategy is to combine them into single reward [59, 60]. For example, [116] sum the predictions of preference RM (as proxy for helpfulness) and rule RM (detecting rules breaking); [62] assign different weightings to the dllevancc/hetuualiyy/oopeeeeette
rewards, thereby customizing how detailed and lengthy the LLMs responses should be. Yet, those single-policy approaches (optimizing over a single set of linear preferences) force priori and uncertain decisions about the required trade-offs [52, 54], further detailed Appendix 1. The multi-policy alternatives [45, 46, 57, 58, 61] are not suitable because of the computational costs required to learn set of policies. To reduce the cost, [117. 118, 119, 120] build experts and then train new model to combine them; [121, 122, 123] share weights across experts; [124, 125, 126, 127] directly train a single model; the recent and more similar work [128] learns one linear embedding per (locomotion) task. Yet, all those works were developed for academic benchmarks [1 12, 129]; moreover, in terms of Pareto-optimality they perform equal or worse than the linearized MORL. As far as we know, the only approaches that might improve performances are those inspired from the multitask literature [130], tackling gradients conflicts [131, 132] or different variance scales [133, 134] across tasks. Though they succeed for games such as ATARI [135], our attempts apply [131] our setups failed. Overall, as previous MORI works modify the training procedure and usually introduce specific hyperparameters, adapting them to RLHF for foundation

---

models with PPO is complex; in contrast, RS can be used on top of any RLHF system. Finally, performance and simplicity are not the only advantages of RS over other MORL approaches; in brief, and as discussed in Appendix A.2, RS is compatible with the iterative alignment process.
Recent works extended the linear mode connectivity when fine-tuning on different tasks [70, 71, 72, 136], modalities [106] or losses [68, 137], while [138] highlighted some failures in text classification. In contrast, we investigate the LMC in RL. The most similar works are for control system tasks: [139] averaging decision transformers and [140] explicitly enforcing connectivity in subspaces of policies trained from scratch on single reward. When the LMC holds, combining networks in weights combines their abilities [141, 142]; e.g. averaging an English summarizer and an English-to- Banllshhto-French translator can summarize in French [143]. In domain generalization, [67, 68, 144] showed that WI reduces model misspecification [145]; by analogy, we show that RS reduces reward misspecification.
5 Discussion: limitations and societal impacts
The recent and rapid scaling of networks presents both opportunities and major concerns [9, 146, 147]. Our approach is a step towards better empirical alignment [10, 11]. Yet, many challenges remain untackled. First, proxy rewards may lack robustness [148] or be hacked [149] via adversarial exploita- tion, making them unreliable. Second, overfitting during training may lead to poor generalization, with a risk of goal misgeneralization [150, 151]. RS could alleviate the impact of some badly shaped proxy rewards and some failed optimizations, as well tackling Goodhart's law [152]. Yet, without constraint on the test distribution, complete alignment may be impossible [153], for example for LLMs with prompts of arbitrary (long) length.
Theoretical guarantees for alignment are also needed [154]. Yet, RS (as all weight interpolation strategies) relies on an empirical finding: the LMC [65], which currently lacks full theoretical guarantees, even in the simplest case of moving averages [100]. That's 's why we state explicitly our Working Hypotheses I and 2 in Section 2.2. Nonetheless, we want to point out that in Appendix C.2 we provide theoretical guarantees for the near-optimality of RS when considering quadratic rewards; specifically, in Lemma 3, we bound the reward difference between the optimal policy and our interpolated policy. A remaining limitation is that we theoretically fix issues only for R linear over the proxy rewards. Such lincarization follows the linear utility functions setup from the MORL literature [60], that cannot encapsulate all types of (human) preferences [56, 77]. Nonetheless, showed in Figures 5(b) and 12 that RS improves results even when R is not linear. We may further improve results by continually training on new and diverse proxy rewards, to capture the essential aspects of all possible rewards, such that their linear mixtures have increasingly good coverage.
Finally, our a posteriori alignment with users facilitates personalization [155] of models. As dis- cussed in Appendix 1. and in [52], this could increase usefulness by providing tailored generation, notably to under-represented groups. Moreover, the distributed nature of RS makes it paralleliz- able thus practical in a federated learning setup [1566 where data must remain private. Yet, this personalization comes with risks for individuals of "reinforcing their biases [...] and narrowing their information diet"[52]. This may worsen the polarization of the public sphere. Under these concerns, we concur with the notion of "personalization within bounds" [52], with these boundaries potentially set by weights fine-tuned on diverse and carefully inspected rewards.
Conclusion. As AI systems are increasingly applied to crucial real- real-world tasks, there is a pressing issue to align them to our specific and diverse needs, while making the process more transparent and limiting the cultural hegemony of a few individuals. In this paper, we proposed rewarded soup, a strategy that efficiently yields Pareto-optimal solutions through weight interpolation after training. Our experiments have consistently validated our working hypotheses for various significant large- scale learning tasks, demonstrating that rewarded soup can mitigate reward misspecification. We hope to inspire further research in exploring how the generalization literature in deep learning can help for alignment, to create AIs handling the diversity of opinions, and benefit society as whole.
Acknowledgments
This work was granted access to the HPC resources of IDRIS under the allocations ADO1 101 1953R1 and A010061 1204 made by GENCI Sorbonne Université acknowledges the financial support by the ANR agency in the chair VISA AANR-20-CHIA CHIA-0022-01),
10

---

References
[1] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx. Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks foundation models. arXiv preprint, 2021. (p.
[2] Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019. (p.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz I Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In NewrIPS, 2020. (p. 1)
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. (p.
[5] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. ICML, 2021. (p.
[6] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivie. Learning and transferring mid-level image representations using convolutional neural networks. In CVPR, 2014. (p. 1)
[7] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? NeuriPS, 2014. (p.
[8] Vladimir N Vapnik. An overview statistical learning theory. In TNN, 1999. (p. 1)
[9] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint, 2016. (pp. 2, and 10)
[10] Jessica Taylor, Eliezer Yudkowsky, Patrick La Victoire, and Andrew Critch. Alignment for advanced machine learning systems. Ethics of AI, 2016. (pp. and 10)
[11] Richard Ngo, Lawrence Chan, and Soren Mindermann. The alignment problem from a deep and
learning perspective. arXiv preprint, 2022. (pp.
10)
[12] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. NeurIPS, 2020. (pp. 2, 6, 9, 30, and 31)
[13] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NeurIPS, 2022. (pp. 1, 2, and 9)
[14] André Susano Pinto, Alexander Kolesnikov, Yuge Shi, Lucas Beyer, and Xiaohua Zhai. Tuning computer vision models with task rewards. arXiv preprint, 2023. (pp. and 6)
[15] Paul Christiano, Jan Leike, Tom Brown, Miljan Martie, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NeurIPS, 2017. (pp. 1, 6, and 9)
[16] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint, 2019. (p.
[17] Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint, 2021. (pp.
and 5)
[18] OpenAL Gpt-4 technical report. arXiv preprint, 2023, (p.

---

[19] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. (p.
[20] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In ICLR, 2022. (pp. and 5)
[21] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak. Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuzmia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Morad- shahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. Super-Naturaalnnstcciinn Generalization via declarative instructions on 1600+ NLP tasks. In ACL. 2022. (p. 1)
[22] Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An mnstuctiion-folloiigg LLaMA model. https://ggt /ggb conn/atsu-llaa stanf ord ataaanorddalpccaaaaaaaaaaaaaaaae alpaca, 2023. (pp. 2, 5, 30, and 31)
[23] Yoav Goldberg. Reinforcement learning for language models. https://gists http:///gggi...gt....... com/ htwggg/bbffffeedddoo999o9eeebbb2
2023. (p.
[24] Steven J Rennie, Etienne Marcheret, Youssef Mrouch, Jerret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. In CVPR, 2017. (pp. 1, 6, 7,9, and 33)
[25] Kimin Lee, Hao Liu, Moonkyung Ryu. Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprins, 2023. (pp. 1, 7, 8, and 35)
[26] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to- image models with human preference. arXiv preprint, 2023. (pp. 1, 7, and 35)
[27] Shu Zhang, Xinyi Yang. Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang. Silvio Savarese, Stefano Ermon, et al. HIVE: Harnessing human feedback for instructional visual editing. arXiv preprint, 2023. (p. 1)
[28] Gabriel
Nir Levine, Daniel Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, bench- marks and analysis. Machine Learning, 2021. (p. 1)
[29] Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002. (pp. 1, 2, 7, and 33)
[30] Chin-Yew Lin and Eduard Hovy. Automatic evaluation of summaries using n-gram co- occurrence statistics. In NAACL, 2003. (pp. 1, 7, and 33)
[31] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Consensus-based image description evaluation. In ICCV, 2015. (pp. and 33)
[32] Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In ICLR, 2023, (pp. 2 and
[33] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. ImageReward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint, 2023. (pp. 2, 7, and 35)
[34] Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In ICLR, 2022. (p. 2)
[35] Eric J Michaud, Adam Gleave, and Stuart Russell. Understanding learned reward functions. arXiv preprint, 2020. (p.
12

---

[36] Aaron Wildavsky. Choosing preferences by constructing institutions: A cultural theory of preference formation. American political science review, 1987. (p. 2)
[37] CA Coello. Handling preferences in evolutionary multiobjective optimization: A survey. In CEC, 2000. (p.2)
[38] Shalom Schwartz et al. An overview of the schwartz theory of basic values. Online readings in Psychology and Culture, 2012. (p.
[39] Marcos Nadal and Anjan Chatterjee. Neuroaesthetics and art's diversity and universality, Wiley Interdisciplinary Reviews: Cognitive Science, 2019. (p.
[40] David Lopez-Paz, Diane Bouchacourt, Levent Sagun, and Nicolas Usunier. Measuring and signing fairness as performance under multiple stakeholder distributions. arXiv preprint, 2022.
(p. 2)
[41] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint. 2022. (p.
[42] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint, 2022. (p. 2)
[43] Robert Irvine, Douglas Boubert, Vyas Raina, Adian Liusie, Vineet Mudupalli, Aliaksei Kor- shuk, Zongyi Liu, Fritz Cremer, Valentin Assassi, Christie-Carol Beauchamp. et al. Rewarding chatbots for real-world engagement with millions of users. arXiv preprint, 2023. (p.
[44] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient founda- tion language models. arXiv preprint arXiv:2302. 13971, 2023. (pp. 2, 5, 30, and 31)
[45] Leon Barrett and Srini Narayanan. Learning all optimal policies with multiple criteria. In ICML, 2008. (pp. 2, 3.3.am and 9)
[46] Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement lcarning for multiobjective optimization. IEEE-T T-CEBERNETICCCS 2020. (pp. 2, and 9)
[47] Michiel A. Bakker, Martin J Chadwick, Hannah Sheahan. Michael Henry Tessler, Lucy Campbel--Gillnngaan, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matthew Botvinick, and Christopher Summerfield. Fine-tuning language models find agreement among humans with diverse preferences. In NeurlPS, 2022. (p.
[48] Aviv Ovadya. Generative CI through collective response systems. arXiv preprint, 2023. (p.
[49] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint, 2022. (pp. 2,5,9
and 31)
[50] Grgur Kovač, Masataka Sawayama, Rémy Portelas, Cédric Colas, Peter Ford Dominey, and Pierre-Yvee Oudeyer. Large language models as superpositions of cultural perspectives. arXiv preprint, 2023. 2)
[51] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint, 2021. (p. 2)
[52] Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. Personalisation within bounds: risk taxonomy and policy framework for the alignment of large language models with personalised feedback. arXiv preprint, 2023. (pp. 2, 10, and 22)
13

---

[53] Alexander Pan, Chan Jun Shern, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Jonathan Ng. Hanlin Zhang, Scott Emmons. and Dan Hendrycks. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the MACHIAVELLI benchmark. ICMI,, 2023. (p. 2)
[54] Conor F Hayes, Roxana Rádulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfar- lane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. JAAMAS, 2022. (pp. 2, 22)
[55] Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? arXiv preprint, 2023. (p.
[56] Peter Vamplew, Richard Dazeley, Cameron Foale, Sally Firmin, and Jane Mummery. Human- aligned artificial intelligence is multiobjective problem. Ethics and Information Technology. 2018. (pp. 2, 10, and 22)
[57] Fumihide Tanaka and Masayuki Yamamura. Multitask reinforcement learning on the distribu- tion of mdps. CIRA, 2003. (pp. and 9)
[58] Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of pareto dominating policies. JMLR, 2014. (pp. 2 and 9)
[59] Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making. JAIR, 2013. (pp. 2 and
[60] Roxana Rädulescu, Patrick Mannion, Diederik M Roijers, and Ann Nowé. Multi-objective multi-agent decision making: utility-based analysis and survey. AAMAS, 2020. (pp. 9,
and 10)
[61] Daniel Marta, Simon Holk, Christian Pek, Jana Tumova, and Iolanda Leite. Aligning human preferences with baseline objectives in reinforcement learning. In ICRA, 2023. (pp. and 9)
[62] Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. In NewriPS, 2023. (pp. 2, 3, and 9)
[63] Vilfredo Pareto. Cours d' d'économie politique. Librairie Droz, 1964. (p.
[64] Patrick Mannion, Fredrik Heintz, Thommen George Karimpanal, and Peter Vamplew, Multi- objective decision making for trustworthy ai. In MODeM Workshop, 2021. (p. 2)
[65] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In ICML,, 2020. (pp. 2,4, and 10)
[66] Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer lcarning? In NewrlPS, 2020. (pp. 2, 4, 23, and 24)
[67] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari Morcos, Hongseok Namkoong. Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML., 2022. (pp. 2, 4, 10, 23, 25, 30, and 34)
[68] Alexandre Ramé, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for -uito--distribution generalization. In NeurIPS, 2022. (pp. 2, 10, 30, and 34)
[69] Michael Matena and Colin Raffel. Merging models with Fisher- averaging. In NeurIPS, 2022. (pp. 2 and 28)
[70] Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open- open-vocabulay, models by interpolating weights. In NewrIPS, 2022. (pp. 2 and 10)
14

---

[71] Shachar Don-Yehing Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. ColD fusion: Collaborative descent for distributed multitask finetuning. arXiv preprint, 2022. (pp. and 10)
[72] Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez- Paz. Model Ratatouille: Recycling diverse models for out-of-distribution generalization. In ICML, 2023. (pp. 2, 4, 10, and 23)
[73] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelcrating deep network training by redueing internal covariate shift. In ICML, 2015. (p. 3)
[74] Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint, 2018.
(p. 3)
[75] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrlPS, 2017. (pp. and 31)
[76] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as a laboratory for alignment. arXiv preprint, 2021. (pp. and 9)
[77] Peter Vamplew, John Yearwood, Richard Dazeley, and Adam Berry. On the limitations of scalarisation for multi-objective reinforcement learning of pareto fronts. In AJCAIA, 2008.
(pp. 3 and 10)
[78] Lars Kai Hansen and Peter Salamon. Neural network ensembles. TPAMI, 1990. (pp. 4 and 7)
[79] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. NeurIPS, 2017. (pp. 4 and 7)
[80] Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In ICLR, 2022. (p.4) 4)
[81] Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git Re-Basin: Merging models modulo permutation symmetries. In ICLR, 2023. (p. 4)
[82] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In EMNLP. 2020. (pp. 5, 6, and 30)
[83] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions. arXiv preprint, 2022. 5)
[84] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint, 2017. (pp. 5,9, 3,9,31, 31, and 38)
[85] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, and Nathan Lambert. TRL: Transformer reinforcement learning. httpp://giiiee //gtthub.. com/lvwer ra/trl, 2020. (pp. 5 and 31)
[86] Edward Beeching, Younes Belkada, Leandro von Werra, Sourab Mangrulkar, Lewis Tunstall, and Kashif Rasul. Fine-tuning 20B LLMs with RLHF on 24GB consumer GPU. https: / //huggingface. ace. http///aww.......////oee/e/////e ho/blog//rrlppet
2023. (pp. S, 30, and 31)
[87] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. (pp. 5, 30, and 31)
[88] Hadeer Ahmed, Detecting opinion spam and fake news using n- n-gram analysis and semantic similarity. PhD thesis, 2017, (pp. and 31)
15

---

[89] Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TI: dr: Mining reddit to learn automatic summarization. In ACL Workshop, 2017. (pp. 5, 30, and 31)
[90] Nathan Lambert, Lewis Tunstall, Nazneen Rajani, and Tristan Thrush. Huggingface h4 stack exchange preference dataset, 2023. (pp. 5 and 31)
[91] Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection. NAACL, 2021. (pp. 6, 30,
and 31)
[92] Andreas Köpf, Yannic Kilcher. Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Due, Oliver Stanley, Richárd Nagyfi, et Ope- nassistant converations-demmrrrtinng large language model alignment. arXiv preprint, 2023.
(pp. 6 and 30)
[93] Gary Yen and Zhenan He. Performance metric ensemble for multiobjective evolutionary algorithms. TEVC, 2013. (pp. 6 and 7)
[94] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. (pp. 6, 7, and 33)
[95] Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for mt evaluation with improved correlation with human judgments. ACL Workshop, 2005. (pp. 7 and 33)
[96] Jia Cheng Hu, Roberto Cavicchioli, and Alessandro Capotondi. ExpansionNet v2: Block static expansion in fast end to end training for image captioning. arXiv preprint, 2022. (pp. 7 and 33)
[97] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin Transformer V2: Scaling up capacity and resolution. In CVPR, 2022. (pp. and 33)
[98] Ronald J Williams. Simple statistical pradient-followigg algorithms for connectionist rein- forcement learning. Reinforcement learning, 1992. (pp. 7, 9, 33, 37, and 38)
[99] Andrej Karpathy and Li Fei-Fei. Deep sisual-semantic alignments for generating image descriptions. In CVPR, 2015. (pp. and 33)
[100] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In UAI, 2018. (pp. 7, 10, 25, and 30)
[101] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020. (p. 7)
[102] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. (pp. 7 and 35)
[103] Naila Murray, Luca Marchesotti, and Florent Perronnin. AVA: A large-scale database for aesthetic visual analysis. In CVPR, 2012. (pp. and 35)
[104] Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Unlocking transferability of large diffusion models via simple parameter- efficient fine-tuning. arXiv preprint arXiv:2304. 0664086, 2023. (pp. 8 and 35)
[105] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara Berg. Modeling context in referring expressions. In ECCV, 2016. (pp. 8, 37, and 38)
[106] Mustafa Shukor, Corentin Dancette, Alexandre Rame, and Matthieu Cord. Unified model for image, video, audio and language tasks. arXiv preprint arXiv:2307. /6184, 2023. (pp. 8, 10, 37, and 38)
[107] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: unifying architectures, tasks, and modalities through simple sequence-to-sequenee learning framework. CoRR, 2022. (p. 8)
16

---

[108] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and P. Abbeel. Benchmarking deep reinforcement learning for continuous control. In ICML, 2016. (p. 8)
[109] Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transforma-
tions: Theory and application to reward shaping. In ICML, 1999. (p. 8)
[110] Marco Dorigo and Marco Colombetti. Robot shaping: Developing autonomous agents through learning. Artificial intelligence, 1994. (p. 8)
[111] Dan Dewey. Reinforcement learning and the reward engineering principle. AAAI, 2014.
(p.
[112] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012. (pp. and 9)
[113] C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint, 2021. (pp. and 38)
[114] Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. Aligning language models with preferences through f-divergence minimization. arXiv preprint, 2023. (p. 9)
[115] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF: Rank responses to align language models with human feedback without tears. arXiv preprint, 2023. (p.
[116] Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint, 2022. (p. 9)
[117] Jungdam Won, Deepak Gopinath, and Jessica Hodgins. A scalable approach to control diverse behaviors for physically simulated characters. TOG, 2020. (p. 9)
[118] Chuanyu Yang, Kai Yuan, Qiuguo Zhu, Wanming Yu, and Zhibin Li. Multi-expert learning of adaptive legged locomotion. Science Robotics, 2020. (p. 9)
[119] Abbas Abdolmaleki, Sandy Huang, Leonard Hasenclever, Michael Neunert, Francis Song, Martina Zambelli, Murilo Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A distributional view on multi-objective policy optimization. In ICML, 2020. (p. 9)
[120] Xi Lin, Zhiyuan Yang, Xiaoyuan Zhang, and Qingfu Zhang. Pareto set learning for expensive multi-objective optimization. In NeuriPS, 2022. (p. 9)
[121] Hossam Mossalam, Yannis M Assael, Diederik M Roijers, and Shimon Whiteson. Multi- objective deep reinforcement learning. arXiv preprint, 2016. (p. 9)
[122] Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: a hierarchical bayesian approach. In ICML, 2007. 9p.
[123] Thanh Thi Nguyen, Ngoc Duy Nguyen, Peter Vamplew, Saeid Nahavandi, Richard Dazeley, and Chee Peng Lim. A multi-objective deep reinforcement learning framework. EAAI, 2020.
(p. 9)
[124] Andrea Castelletti, Francesca Pianosi, and Marcello Restelli. A multiobjective reinforcement learning approach to water resources systems operation: Pareto frontier approximation in a single run. Water Resources Research, 2013. (p. 9)
[125] Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi- objective reinforcement learning and policy adaptation. In NeurlPS, 2019. (p. 9)
[126] Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In ICML, 2019. (p.9)
17

---

[127] Markus Peschl, Arkady Zgonnikov, Frans A Oliehoek, and Luciano C Siebert. Moral: Aligning ai with human norms through multi-objective reinforced active learning. arXiv preprint, 2021.
(p. 9)
[128] Pu Hua, Yubei Chen, and Huazhe Xu. Simple emergent action representations from multi-task policy training. In ICLR, 2023. (p. 9)
[129] Peter Vamplew, Richard Dazeley, Adam Berry, Rustam Issabekov, and Evan Dekker. Empirical evaluation methods for multiobjective reinforcement learning algorithms. Deakin University,
2011. (p. 9)
[130] Rich Caruana. Multitask learning. Machine learning, 1997. (pp. 9 and 23)
[131] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In NeurIPS, 2020. (pp. 9 and 23)
[132] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. NeurlPS, 2021. (pp. 9 and 23)
[133] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL. with importance weighted actor-learner architectures. In ICML, 2018. (pp. 9 and 23)
[134] Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Had- sell, Nicolas Heess, and Razvan Pascanu, Distral: Robust multitask reinforcement learning. NeurlPS, 2017. (pp. 9 and 23)
[135] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The Arcade learning environment:
An evaluation platform for general agents. JAIR, 2013. (pp. 9 and 23)
[136] Nikolaos Dimitriadis, Pascal Frossard, and François Fleuret. Pareto manifold learning: Tack- ling multiple tasks via ensembles of single-task models. arXiv preprint, 2022. (p. 10)
[137] Francesco Croce, Sylvestre-Alvis Rebuffi, Evan Shelhamer, and Sven Gowal. Seasoning model soups for robustness to adversarial and natural distribution shifts. CVPR, 2023. (p. 10)
[138] Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Linear connectivity reveals generalization strategies. In ICLR, 2023. (p. 10)
[139] Daniel Lawson and Ahmed H Qureshi. Merging decision transformers: Weight averaging for forming multi-task policies. In ICLR RRL Workshop, 2023. (p. 10)
[140] Bean-Baptiste Gaya, Laure Soulier, and Ludovie Denoyer. Learning a subspace of policies for online adaptation in reinforcement learning. In ICLR, 2022. (p. 10)
[141] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023. (pp. 10, 23, and 34)
[142] Nico Daheim, Nouha Dziri, Mrinmaya Sachan, Iryna Gurevych, and Edoardo M Ponti. Elastic weight removal for faithful and abstractive dialogue generation. arXiv preprint, 2023. (p. 10)
[143] Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning. arXiv preprint, 2023. (p. 10)
[144] Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improving model selection and boosting performance in domain generalization. In NeurlPS, 2021. (pp. and 25)
[145] Alexander D' Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. Un- derspecification presents challenges for credibility modern machine learning. JMLR, 2020.
(p. 10)
18

---

[146] Dan Hendrycks and Mantas Mazeika. X-risk analysis for Al research. arXiv preprint, 2022.
(p.
10)
[147] Dan Hendrycks. Natural selection favors AIs over humans. arXiv preprint, 2023. (p. 10)
[148] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. arXiv preprint, 2022. (p. 10)
[149] Joar Max Viktor Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. In NeurIPS, 2022. (p. 10)
[150] Rohin Shah, Vikrant Varma, Ramana Kumar. Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralization: Why correct specifications aren't enough for correct goals. arXiv preprint. 2022. (p. 10)
[151] Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization in deep reinforcement learning. In ICML, 2022. (p. 10)
[152] Ben Smith. A brief review of the reasons multi-objective RL could be important in Al Safery
wwt.posts////poooooooooootoossst http//osss//ooee /iddff 6m6FCexReK9 a- brief Research. https: / /www ali / www.annnonnnn.mmmmmmmm.mmm org/
-review- wwwfffth-eeeeeee reasonr-eeens--eeee multi httpssammiiiooooiiteeeeeeeeeeeee oww.ooiveeeeeeeeeooo could- be, 2021. (p. 10)
[153] Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. Fundamental limitations of alignment in large language models. arXiv preprint, 2023. (p. 10)
[154] Manel Rodriguez-Soto, Maite Lopez-Sanchez, and Juan A Rodríguez-Aguilar Guaranteeing the learning of ethical behaviour through multi-objective reinforcement learning. In AAMAS,
2021. (p. 10)
[155] Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani, Lamp: When large language models meet personalization. arXiv preprint, 2023. (pp. 10 and 22)
[156] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communictiion--fficcinnt learning of deep networks from decentralized data. In AISTATS, 2017. (pp.
10
23)
[157] Philip Tetlock. A value pluralism model of idcological reasoning. JPSP, 1986. (p. 22)
[158] Umer Siddique, Paul Weng, and Matthieu Zimmer. Learning fair policies in multi-objective (deep) reinforcement learning with average and discounted rewards. In ICML, 2020. (p. (2.
[159] Iason Gabriel and Vafa Ghazavi. The challenge of value alignment: From fairer algorithms to Al safety. arXiv preprint, 2021. (p. 22)
[160] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In ACM SIGSAC, 2016. (p. 22)
[161] Kristof Van Moffaert, Tim Brys, Arjun Chandra, Lukas Esterle, Peter R Lewis, and Ann Nowé. A novel adaptive weight selection algorithm for multi-objective multi-agent reinforcement learning. In IJCNN, 2014. (p. 23)
[162] Zafir Stojanovski, Karsten Roth, and Zeynep Akata. Momentum- based weight interpolation of strong zero-shot models for continual learning. In NeurIPS Interpolate Workshop, 2022. (p. 23)
[163] Steven Vander Feckt et al. Weight averaging: A simple yet effective method to overcome catastrophic forgetting in automatic speech recognition. arXiv preprint, 2022. (p. 23)
[164] Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Tranch-Train-Merge. Embarrassingly parallel training of expert language models. arXiv preprint, 2022. (p. 23)
[165] Colin Raffel. A Call to Build Models Like We Build Open-Source Software. https: //colinraffel. cwwmbo/m//////. wtt.ccoooccc/eeeccecc/oooooooo/t wwwoo//-caalttoooo rce- sieesoffwaare html, 2021. (3)
www.ddudoddeeeeee like- we- bui ooold-o
19

---

[166] Yann LeCun. Léon Bottou, Genevieve B. Oir, and Klaus-Robert Müller. Efficient backprop. In Neural Networks. 2012. (p. 27)
[167] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. (pp. 27, 31, and 35)
[168] Yann LeCun, J. S. Denker, Sara A. Solla, R. E. Howard, and L.D. Jackel. Optimal brain damage. In NeurIPS, 1990. (p. 27)
[169] Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr; Invariant gradient variances for ditto---ittrbbutionn generalization. In ICML, 2022. (p. 27)
[170] Sue Becker and Yann Le Cun. Improving the convergence of prok-propggttion learning with second order methods. In Connectionist models summer school, 1988. (p. 27)
[171] Ronald A Fisher. On the mathematical foundations theoretical statistics. Philosophical
Transactions of the Royal Society of London., 1922. (p. 28)
[172] Nicol N Schraudolph. Fast curvature matrix-vecte products for second-order gradient descent. In Neural computation, 2002. (p. 28)
[173] Valentin Thomas, Fabian Pedregosa, Bart van Merriënboer, Pierre-Antoine Manzagol, Yoshua Bengio, and Nicolas Le Roux. On the interplay between noise and curvature and its effect on optimization and generalization. In A/STATS, 2020. (p. 28)
[174] Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approximation for natural gradient descent. In NeurIPS, 2019. (p. 28)
[175] Eric J. Wang. Alpaca-LoRA. /ttps://gitie wttc:/ooo///o// awpaca-paaaaooa -ora, 2023. (p. 31)
[176] Hadeer Ahmed, Issa Traore, and Sherif Saad. Detecting opinion spams and fake news using text classification. Security and Privacy, 2018. (p. 31)
[177] Edward Beeching, Younes Belkada, Kashif Rasul, Lewis Tunstall, Leandro Werra, Nazneen Rajani, and Nathan Lambert. StackLLaMA: An RL Fine-tuned LLaMA Model for Stack Exchange Question and Answering, 2023. (p.31)
[178] Andrew Maas, Raymond F. Daly, Peter Pham, Dan Huang, Andrew Y. Ng, and Christo- pher Potts. Learning word vectors for sentiment analysis. In ACL., 2011. (p. 31)
[179] Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. Masked language model scoring. arXiv preprint, 2019. (p. 32)
[180] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 2019. (p. 32)
[181] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L.. Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. (p.
[182] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In ICLR, 2020. (p. 33)
[183] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Hanna Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tun tne-tunnng of zero-shot models. In CVPR, 2022. (p.34) 34)
[184] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint, 2021. 35)
[185] Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. arXiv preprint, 2023. 35)
20

---

[186] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. NeurIPS, 2017. (pp. 36 and 37)
[187] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: reference- free evaluation metric for image captioning. In EMNLP, 2021. (pp. 36 and 37)
21

---

Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards
Supplementary material
This supplementary material is organized as follows:
Appendix A further discusses the practical benefits of rewarded soups.
Appendix B anticipates questions that might arise from readers.
Appendix details some theoretical guarantees.
Appendix details our text-to-text generation experiments.
Appendix enriches our image captioning experiments.
Appendix enriches our image generation experiments.
Appendix G enriches our visual grounding experiments.
Appendix H enriches our locomotion experiments.
The shareable code is released on github. Moreover, you can find additional qualitative results of our experiments this website.
A
Discussion
In this section we discuss the benefits of our rewarded soup (RS) approach with respect to the two families of strategies: the single-policy and the multi-policy approaches.
A.1 Compared to single-policy approaches
The main reason why single-policy approaches are not suitable is because they optimize over single set of preferences. In contrast, we build a coverage set Pareto-optimal policies. This is important for the following reasons, mostly first discussed in Kirk er al. [52] and in Hayes et al. [54].
Indeed, the user's true reward is highly uncertain before training. This "semi-blind" [54] manual process forces a priori and uncertain decisions about the required trade-offs. It shifts the respon- sibility from the problem stakeholders to the system engineers, who need to anticipate the impact of their choices on the final performance. Critically, the RLHF process may cause the "tyranny of the crowdworker" [52], models are "tailored to meet the expectations of [...] small number of crowdworkers primarily based in the US with little to no representation of broader human cultures, geographies or languages." [52]. Moreover, biased are caused by chaotic engineering choices, and "are exacerbated by a lack of [...] documentation" [52]. In contrast, our approach makes personal- ization explicit, as argued by [52]. Moreover, we could support decision-making find good balance between (potentially conflicting) parties' interests. This value pluralism [157] can lead to fairer and more equitable outcomes [56, 158]. Single-polic cannot adapt to test time requirements; in contrast, RS facilitates personalized assistances [155]. This is all the more important as human preferences change from time to time. In this dynamic utility function scenario, RS can quickly adapt with fewer data, by simply adjusting the À to match new preferences (rather than the full network). Finally, RS could also improve the interpretability and explainability of the decisions. Letting the users decide would make the process more transparent [159], which is essential to ensure that the development process fair, unbiased, and inclusive [160].
A.2 Compared to multi-policy approaches
The main reason why existing multi-policy approaches through multitasking are not suitable is because of their computational costs required to learn a dense set policies. In contrast, RS only trains the proxy rewards independently and enables the selection of the interpolating coefficient
22

---

posteriori. This is especially useful with large number of rewards and thus growing number of combinations. Second, multitask [130] is challenging: for example, even if the true reward is actually a linear weighted sum of some proxy rewards and those coefficients are known, using those preferences during training can lead to suboptimal results [161], because of conflicting gradients [131, 132] or different variance scales [133, 134]. This has been tackled in RI., but so far mostly for games such as ATARI [135]. Third, our strategy is compatible with the inherent iterative engineering process of alignment. Indeed, RS can continually include adjusted opinions while preventing forgetting of the old behaviours. This relates to the continual learning challenge, and the empirical observations that weight averaging can reduce catastrophic forgetting [162, 163]. Moreover, as shown in [141] and confirmed in Figure 13(c), negative editing by weight interpolation can fix and force the removal of some behaviours. Finally, RS is computationally effective, requiring communication across servers, thus enabling "embarrassingly simple parallelization" [164]. This facilitates its use in federated learning scenario [1566] where the data should remain private. Actually, RS follows the updatable machine learning paradigm [165], "allowing for the collaborative creation of increasingly sophisticated AI system" [72]. In the future, may develop open-source personalized models, rewarded on decentralized private datasets, and combine them continuously.
B FAQs
We addressed below questions that might arise from readers.
B.1 What is the difference between rewarded soups and model soups?
Rewarded soups (RS) and model soups (MS) [67] both average weights of models fine-tuned from a shared pre-trained initialization. That's why we chose the same terminology as "model soups" and named our method "rewarded soups". Yet, we want to clarify that RS and MS tackle different problems, have different goals, leading different methods and implementations.
RS challenges single-policy approaches improve alignment in reinforcement learning, and aims at reducing reward misspecification by revealing Pareto front of solutions across the entire space of preferences: thus RS considers different training objectives for fixed hyperparameters across runs, and non- -on-unnform interpolating coefficients À set posteriori.
MS challenges the standard model selection after a grid search to improve generalization in supervised learning, and aims at reducing model underspecification and reducing variance by combining all fine-tuned models: thus MS considers different hyperparameters for fixed training objective across runs, and (usually) uniform interpolating coefficients À
These differences mean that MS cannot be applied to reduce reward misspecification, as validated empirically in Figure 13(b) for the captioning task. This Figure 13(b) also shows that RS and MS are actually complementary and can combine their benefits; specifically, reward misspecification and variance reduction.
B.2
Limitations for the LMC?
B.2.1 Limitations for the design of networks for the LMC?
In our experiments, we consider different network architectures (transformers, CNNs, and MLPs). We also investigate different training procedures: with low-rank adapters, partial or end-to-end fine-tunings. We do so for many different tasks and modalities: text generation, image captioning, image-to-test generation, visual grounding, etc. Our empirical observation is that, across those setups, the LMC is architecture-agnostic, procedure-agnostic, task-agnostic and modality-agnostic.
The main condition we require is the shared pre-trained initialization [66], so that the weights remain close (as detailed in Remark 1). As a side note, there is another condition suggested by the literature
[164, 141]: the LMC would work better when the architecture has enough trainable parameters. For example, according to [141], larger networks may facilitate the orthogonality of the fine-tuned updates; then [141] "speculate that this [orthogonality) enables the combination of task vectors via addition with minimal interference".
23

---

B.2.2 Limitations for the number of training steps for the LMC?
As argued above, good performances are guaranteed when weights remain close; thus longer trainings may be worrisome, as the models may potentially diverge in the weight space. We investigate this question in Figure 8, for the news summarization and the captioning task; we double the number of training steps, and report multiple RS fronts over the course of fine-tuning. Fortunately, we consistently observe good performances for RS along fine-tuning. This confirms that the only condition for the LMC is the shared pre-trained initialization [66].
BLEVI
(a) News summary.
(b) Captioning.
Figure 8: Those figures show how RS's s fronts evolve over the course of fine-tuning, and confirms the LMC even when doubling the number of training epochs (previously 2 for news summarization and for image captioning).
B.2.3 How does the number of rewards (and networks) affects the LMC?
For visualization clarity, the fronts were mostly shown for N 2 rewards, one of the z-axis, the other on the y-axis. Yet, RS can scale and trade-off between more rewards. We validated this empirically in the spider maps from Figure 2(f) (for text generation), from Figures 3(c) and 4 (for image captioning). and from Figure 18(c) (for visual grounding), where we respectively consider up to N 4, N = 5 and N 3 networks fine-tuned on N different rewards, one reward each.
B.3
Comparison of MORL and RS
B.3.1
How to evaluate Pareto-optimality?
Given fixed preference ü between two rewards R1 and R2, we would like to compare our RS policy to an oracle policy maximizing (1 -1-))) (i) R1 + μ x R2 in test. Yet, this oracle policy (and the true Pareto front) is unknown in real-world applications.
That's why, in practice, and argued in Remark we presented empirical support for Hypothesis by considering the MORL's solutions fine-tuned to optimize (1 - ( x Rí + x R2 in train, for 0 S μ S 1. In other words, the linearized MORL our reference evaluate Pareto optimality. Overall, in Section 3, MORL and RS usually perform similarly (with small differences further discussed below in Appendices B.3.2 and B.3.3). Our conclusion is that rewarded soup is an empirical solution towards Pareto-optimaliy with indeed an experimental limitation highlighted in the paper's 's name.
B.3.2 How does reward diversity affect the effectiveness of RS?
Our experiments in captioning and image generation provide empirical evidence that the more similar the rewards, the higher the gains of RS versus MORL.
In the captioning experiment, by analyzing the transfer abilities across rewards in the spider maps from Figure 3(c), we can deduce that BLEU4 and ROUGE are more similar than BLEU1 and ROUGE, while METEOR is an outlier (fine-tuning on METEOR worsens the results for the other rewards). Then, we can observe that the gains of R$ versus MORL are consistent with these similarities across rewards. Specifically, when considering R₂ ROUGE, the RS green front is more convex and significantly above the MORL yellow front in Figure 11 (with R. BLFUU4 than Figure ((a) (with R₁ = BLEU1), In Figure 12(b), with R METEO R. MORL performs better than RS.
24

---

Similarly, in the image generation experiment, when we consider two (arguably similar) aesthetic rewards Figure 6(a) fine-tune a diffusion model, RS's front is to the right and above MORL's front. In contrast, performances get worse in Figure 14 where we also include nsfw reward inversely correlated with image quality.
In conclusion, despite using diverse and heterogeneous rewards that are in tension, we consistently obtain positive results. Yet, the case where rewards are fully antagonist, acknowledge that RS is likely to produce less favorable results. This empirical limitation of weight interpolation can be explained in two different ways. (i) Intuitively from a loss landscape perspective: weights fine-tuned on antagonist rewards will be more distant, thus potentially breaking the linear mode connectivity. (ii) Theoretically thanks to Lemma 3, where we bound the difference between the optimal reward and RS's s reward by RHS term growing the maximum of eigenvalues ratio for rewards' Hessians: if the rewards are more diverse, their Hessians would have more different eigenvalues, thus maximum of eigenvalues ratio would grow, the RHS term would grow in Lemma 3, and our guarantees for the optimality of RS would get loose.
As final note, to tackle this limitation under antagonist rewards, the complementarity of MORL and RS appears as a promising research direction; this is further discussed in the legend of Figure 13(a) for the captioning task and in Appendix F.2 for the image generation task.
B.3.3 Why RS is sometimes superior MORL?
We observe few times that the RS solutions are actually above the linearized MORL solutions. We speculate this is related to the multiple benefits of weight interpolation. The main benefit that we discuss in our paper is the ability to interpolate between different policies: from this benefit, we would expect RS to perform similarly to MORL. The second benefit from weight averaging is the implicit regularization, causing variance reduction and stabilizing performances [100, 144]. This is the main focus of the traditional weight averaging literature, for example in model soups [67]. In conclusion, we speculate that this second benefit (combined with the first) can explain why RS sometimes outperforms MORL.
25

---

C Theoretical insights
C.1 Proof of Lemma
Proof. Considering 0 maximizing R. we first show that 0 is on the PF of [Ra... Otherwise, consid- ering 0' >N 0 and as Vi, f4 2 O, we have 2; (a,)(00)) > E; it (0). This implies that 0' would produce a better policy than @ for R. E. R4 and thus the contradiction. Finally, as 0 is on the PF and by definition PCS, there exists À s.t. vk, (E, A, 02) Rg (0).
C.2 Theoretical guarantees with quadratic rewards
In this section, we provide theoretical guarantces for the near-optimality of RS when considering quadratic rewards. This simplification amounts to replacing the rewards by their second-order Taylor approximation, which is realistic assumption when the weights remain within a small neighborhood.
C.2.1 Simple case with Hessians proportional to the Identity matrix
For the first Lemma 2, we make the following simplifying Assumption 1.
Assumption (Hessians proportional to the Identity matrix.). Every reward RG is quadratic, with Hessians proportional to lá- Specifically, les θ C R.d be the set of possible weights, and let (Ra ) be the N rewards, we can write for i (1......0 N}:
40 O, R2( Ra((,) 1111 0,12
(1)
where Tis E R4 and 0g is the global maximum for reward Rú.
Lemma 2. Let ju = (ji....., (uN) AN. Then, under Assumption 1, the reward Rp - E, As x R. is maximized on the convex hull of (Oi..... Ox).
Proof. The function R2 is quadratic thus has an unique global maximum Ó, that we find analytically:
VoRa
Σ pmm (Ō 0.)
0 Σ Pim O₂ Ei= PimTi
Since all the A.m are positive or zero, and at least one is greater than zero, is indeed in the convex hull of [01....,NN}
Remark Under Assumption 1, the reward functions are concave; thus we can reasonably assume that each fine-tuning procedure for R₁ reaches its global optimum 0g for E {1...,N] N). Then, Lemma 2 tells us that the maximum value for linear user's reward Ra. is obtainable by weight interpolation between the {0:)N the interpolating coefficients in An such that X x Pan, make rewarded soups optimal.
C.2.2 Advanced case with diagonal Hessians
We now consider the more complex with the relaxed Assumption 2. For simplicity, only consider 2 rewards Rg and Rg.
Assumption (Diagonal Hessians). The rewards are quadratic, with Hessians diagonal negative definite. Specifically,
write for E {1,2}:
v0 = (0) 0d) E R₂(0) = R.(O.) - Σ 105 02)2
(2)
1=1
where ....)))))))) {RR7) and 0, (o!.... 0g)) is the global maximum for reward R.,
26

---

Remark 4. This diagonal Assumption 2 of the Hessian is common: for example in optimization
[166, 167], to prune networks [168] or in out-of-distribution generalization [169]. This strong assumption is supported by the empirical observation [170] that Hessians are diagonally dominant, particular at the end of training. Also, we note that our findings remain valid assuming only that the Hessians are co-diagonalizable.
Lemma 3. We consider the user's reward R2 = (1 - (1) x R1 + ja x Rs with ja E [0, 1}. and
Rp
max Rp(0) max XE
(21 X) 01
02).
AEE
R corresponds to the difference in terms of R& between the global maximum and the maximum reachable by weight interpolation through rewarded soups (with single interpolating coefficient for all dimensions). Then, under Assumption 2, we have:
1² (1 (MM,, A2) -1) P S (A(1 - (M 1)2 + M)((1 ()](1_))A1 + P
(4)
o z 0/2 गं
where M = maxje(L...... df max
is the maximum of eigenvalues ratio, A = R1((1)
R1 (02) and 22 R2(02) - (01)
When A A2, the bound simplifies into:
R S ( - 1)2 A
(5)
1)2 M
Furthermore, when the Hessians are equal, then M 1 and ARp = 0: RS is optimal.
Proof. This novel proof is three steps. First, we find i maximizing (0) for on the full set of weights @. Second, we find maximizing (2 - A) 01 + x x-02) for E [0, 1] and thus defining the best interpolation between the expert weights. Finally, we bound A Rg. the differences between their rewards, by applying the Bhatia-Davis inequality.
First step. Let's first find the maximum of R2 on O. Denoting S = (1 (1--A0 XxxxRR1 (01)+ (22))). we have for all 8 E O:
Ro(0)= = S Σ (1- W)n (00 ((
00)= + inn' ( (0) - 0z) 2)
02)²)
(6)
j=1
Since R2 is a sum of concave quadratic functions, it has a unique global maximum reached at point
we note 0 = (0'....,60) 0²).. The global maximum can be computed by differentiating R₁ with respect to each variable 03, which gives:
, = 11 0f - 2. .0
where the interpolating coefficients per dimension & are defined for j E {l, as:
M juná
E 1).
(1 pi)m pmmp
Second step. With E [0, 1] and (1 -))-1
1-01 + X 62. we can write (f) as a function of
(0) = S
Σ
d
(11 pimi + pmb) (^ (A-))) 2
+
(1 1-)) jung (0) 0z)
pu( - fi)m Tá.
1=1
=R)(0) Σ Py (x- -xx)
(8)
1=1
where pj is defined as pj = (11 (1-))) + (m13)) (0: 02)
27

---

From Equation (8), we can compute the maximum reward obtainable for weight averaging
maxxelo.1] R2 0) + X- 82). Since the function + (p - A) 0, + + x2) is con-
cave quadratic function, there a unique value À maximizing Rp equal to
Ā = 25-1 P2
(9)
2j
Since all py are positive and all X are between and 1, also between 0 and 1. Therefore, (11-1 - X) 01 + x- 02) is indeed the maximum reward for rewarded soups.
Third step.
Applying Equation (8) to Ā gives:
Rpp RQ(0)- - Ra - -X) -x). + x.x22) 02)
(10)
d Σ
A - (x-)))
Pj
(11)
j=1
d
Σ
Pj (1 20) 2
(x-)0)²
Σしj Pj
(12)
Σ 1i=1
j=1
The second term in Equation (12) can be simplified as:
Σ
Pj =1 و) AA2. j=1
(13)
The core component of this proof is the upper bounding of the first term in Equation (12). The key
idea to recognize the variance of a discrete random variable A with P(A = X,) = Σ pj then,
from Equation (9) is actually the expectation of A. Then, can apply the Bhatia-Davis inequality, as recalled in Equation (14), on the variance of a bounded random variable a S A < b:
Var(A) S (b (E(A))((A)) - a)
(14)
Therefore Equation (12) is bounded by:
max X x x) Isi
RR S
min
20 ) ((1 -)) + AA2).
(15)
15isd"
Now, we bound the variables X, since 1/M S n/ /1 S M. Then for all we have:
P
AM
(16)
(1 P)M + f
(1 (D) + RM
and thus:
1) (1+MM-TT) -x)
RM
P
AR I
M A(M - 1) ((1 (1-A)AI+ +AZ2).
(17)
Finally, noting that A, = Eÿ T/É -4) we deduce from Equation (9) that Ā (I-P)A +AA; Replacing this in the previous Equation (17) gives the final Equation (4), concluding the proof. [
AAg
Remark 5. As a final remark, please note that the suboptimality of RS comes from the need of having one single interpolating coefficient for all d parameters (6' .....d") of the network. Yet, the advanced merging operations in [69] remove this constraint, with interpolating coefficients propor- tional to the eigenvalues of the Fisher matrices [171], which actually approximate the eigenvalues of the Hessian [172, 173]. Combining [69] and our RS is a promising research direction, the key issue being the computation of the Fisher matrices [174] for networks with billions of parameters.
28

---

C.2.3 Bound visualization
We visualize in Figure 9 the bound given by Lemma 3. We show that for small values of M like M 2, the value of Ra for RS is quite close to the global optimum. Also, recall that RS theoretically matches this upper bound when M 1. For larger values like M = 10, the bound is less tight, and we note that the maximum value R2 approaches the constant function as + cc.
1.0
0.9
= (1- ( X R1 + Ra
0.8
0.6
Rp
0.5
Upper bound, M=10
Rewarded soups
Upper bound, M=2
LMC lower bound
0.4
0.0
0.2
0.4
0.6
0.8
1.0
i
Figure 9: Illustration of the bound given by Lemma 3 under Assumption 2. For simplicity, showcase the case where Ri (8:) R2(02)
thus A A2 = In green, we plot the rewards obtained with rewarded soups for the optimal 3, i.c., Ra ((1 X) 01 x -8). whose value independent of M in this case. In blues, we plot the maximum value of R2 given by Equation (5) in Lemma 3, for M = 2 and M 10. For reference, also plot the values for the lower bound in the LMC Hypothesis 1, i.c., equal to A) (0,) (02). As RS outperforms this lower bound, validates Hypothesis in this
H1 (02) = R2(0;)
29

---

C.3 Similarity between weight interpolation and functional ensembling
Lemma (A-interpolation of weights approximates the X-ensembling of predictions. Adapted from [67, 68, 100]]), Given 81 and 82 optimized for R1 and R2 s.t. they remain close, ie.. ||0] 02112 ~ O. Denoting 0x the interpolated weights 8 (1 X) 01 + 02 and fx the ensembling of predictinns
fx()) X) [1; f(;002)
f(,,8x) = f2(-)
for (1,
R&(J(,00)) & RE(AA))
Proof. This proof follows [68] and has two components.
Functional approximation. First, we perform Taylor expansion at the first order of the models' predictions w.r.t parameters for T:
f(x,01) f(x, Vef(2, - 8x) +0 -ox11)
= + Vof(=, Vol(,,))+ -. O 0(18, 0z
and similarly:
02) + +0(10, 02 12)
f(x, 0x) + Vos((,
Then by X-weighted sum over i, the term multiplying Vof(x,0x) cancels out and we obtain:
fa(2) -1-))(111)) X) f(z + f(x, f(x,02) f(z, 0x) + -6211)
(18)
Reward approximation. Second, we obtain the reward approximation with Taylor expansion at zeroth order of the reward Rs for & E {1, 2] and injecting Equation (18):
Rx(fx(=)) (x(f(z (z)) + (z) f(z 12)
(z. (z)) o (18) 821
We obtain the results when 0, and remain close, i.e., when we can ignore the O term.
D Text-to-text: LLaMA with diverse RLHFs
D.1 Experimental details
We summarize the key implementation details of our text-to-text generation experiments Table 1. The pre-trained network is LLaMA-7b [44]; then low-rank adapters [87] were fine-tuned on Alpaca [22] to follow instructions. We eventually fine-tune via PPO on the different considered tasks. Our code is adapted from [86]; we kept most of their hyperparameter values, only dividing by the batch size to in our GPU and extending the output length. For each task, we consider existing open-source datasets and available reward models, that we download from HuggingFace. Regarding the reward models, summarization tasks, R₁ was apen-sourced Summarize from Human Feedback paper [12], while R₂ [91] aimed at improved "faithfulness in abstractive summarization with contrast candidate generation". For other dialog tasks, we mostly rely different reward models from OpenAssistant [92]; though they all aim at evaluating whether an answer is adequate given a question, they differ their predictions due to differences in their architecture and training procedures. In practice, leverage these reward models as block-box classification pipelines, implemented in the transformers library [82].
an effort to reproduce the
Foor example, the TL:DR dataset is previously existing dataset extracted and obtained by [89] that contains preprocessed comments posted the social network Reddit and hosted on HuggingFace.
30

---

Table LLaMA with RLHF experiments: key implementation details.
Model
Architecture
Transformner [75]
Pre-traiming
LLaMA-7b [+4]
Instruction FT
Alpaca [22]
RL procedure
Fine-tuning strutegy
LoRA [87]
following Alpacu-LoRA [175]
LoRA alpha
16
LoRA dropout
0.05
following trl-peft f85, 86]
Optimizer
Adam [167]
Learning
1.4le-5
Batch size
128
Output length
Uniformly sampled between 16 and 32
RI. algorithm
PPO [84]
PPO
0.05 summary tasks
Epochs
2 for Reuter summary else
Hardware
NVIDIA RTX A6000 49 Go 4000 GPUh
Compute budget
Task name
Reuter summary
Description
Generate a concise and clear summary of newspaper articles from Reuters.
Prompt
"Generate onc-sentence summary of this post."
Dataset
Reuter news from [88, 176] from news-summary
Ri
gpt2-reward (e12--rwarl-summaare summaarzaziio trained here.
Ra
burt- faithfai faithful- -uls-smmmmmm-eeeeeee
[91]
Figure
Figures 1(b) and 2(a)
Task name Description Prompt Datasct
Reddit TL;DR summary
Generate a concise und clear summary of posts from Reddit across a variety of topics (subreddits).
"Generate one-sentence summary of this post."
Reddit crawl from the TL:DR datasct [89] from summarize from-foedbaack [12]
R1
gpt2-reward-e gnt-reward-summariiee -smmaaizzaaiiinn trained here.
R₂
hart- faitt-ffafful--see waw.faafullluummaarrra -ass-ssummmaeeeeeee [91]
Figure
Figure 2(b)
Task name Description
Stack Exchange
Answer accurately to technical questions from Stack Exchange.
Prompt Dataset
No prompt, only users' questions.
Q&A from Stack Exchange (90, 177] from stack- wwweeeecccceeeeeeeeeeeeeeeeeeeee
Ri
reward-modelldeberra. -3-baase
Rg
Figure
Figure 2(c)
Task name
Movie review
Description
Generate movic reviews that accurately describe a movic.
Prompt
"Generate movie
Dataset
IMDB reviews [178] from IMDB
Rí
reward model-deberra---
Ra
reward-model- electra-large-disseeeee
Figure
Figure 2(d)
Task name
Helpful assistant
Description
Provide helpful and harmless answers to potentially complex and sensitive questions.
Prompt
No prompt. only users' questions.
Dataset
Helpfulness and harmlessness datasets [49] from hh-rthf -3-rr-----lrg---
Rí
reward reward-model lat--laagg-diisimmnntttee
Ra
reward-moule coredd-udel-elle wwweel--eerr--laageeee
Ra
reward- model-oleeeeeo -eeeeeeteee---bbeeeeee
Ra
reward-modes www-mmoddl--eeeteee saeerta--3-bbee
Figure
Figures 2(c) and 2(f)
31

---

D.2 Inspection of the predictions by the interpolated models
This supplementary material includes multiple visualizations of predictions generated by models with interpolated weights. For example, Table 2 provides generated summaries, that remain grammatically coherent. We include more generated samples on this website.
To quantitatively validate that samples generated by interpolated models do not suffer from reduced quality, we provide evaluations with general-purpose quality metrics. Specifically, in Figure 10, we measure the perplexity (of MLMS [179] or of GPT2 [180)) for summaries generated when X-interpolating between two LLMs fine-tuned on the two summary rewards. We observe that, by interpolating the weights, we also interpolate the metrics. In conclusion, we confirm quantitatively that RS does not deteriorate quality.
Table Examples of generated summaries about vampire vigilantes Malawi news, for A) + O2 interpolating between 0, and 0₂ fine-tuned on R& (evaluating completeness) and Ra (evaluating faithfulness).
Generation
0.0
Malawi vigilante mobs have been killing people suspected of being vumpires,
the affected in the affected the country Caited Nations the US embassy declare widespread country go
the UN and US embussy declare
0.1
Malaui vigilante muhs have bren killing people suspected of being
promping the UN and US embassy declare
Malowi vigilante mobs have killed people mid September, UN and US embassy declure parts
0,3
Malass vigilante
killed people, promptnng the mobss killed people. promptnn
Malowi
country no-go President Peter
Malawi vigilante
Malowi arresed killed suepected sampires, promping killed snepectod vampines. prompting UN US ambassy duclare ne-go arressed suspected vampires, resuttng
ne-go
Malawi vigilante Malasi vigilastc
anestal suspected vampless, resuiting deaths embassy to declare to sones.
prompting
vigilame violence has caused widespecad paulic desth, prompting authortics anest suspected members investigute the belief vampirise. Malawi vigilante violenor has caused widespread punic death.
450
Perplexity (1)
Quality (t) News quality
MLMS
300
GPT2
250
Quality (1)
Perplexity
200
-21
150
-030
100
0.0
0.6
0.5
Figure 10: News summaries generated by X-intcrpolated LLMs evaluated in terms of perplexity (by MLMS [179] or by GPT2 [180]) or news quality by this classifier.
32

---

E Image-to-text: captioning with diverse statistical rewards
E.1 Experimental details
We summarize the key implementation details of our captioning experiments in Table 3. In short, we took the state-of-the-art network [96] for captioning on COCO, fine-tuned with their code and only changed the reward. In more details, since the self-critical paper [24] (a variant of REINFORCE
[98] with a specific estimation of the baseline score) is now common in captioning to optimize the CIDEr reward [31] after first step of supervised fine-training The recent ExpansionNetv2 [96] follows this strategy to reach state-of ofut-off-hhe-aee results, with a Swin Transformer [97] visual encoder and block static expansion for efficiency. We investigate whether additional RL trainings on the other traditional statistical metrics can help. We use the code from [96] and their hyperparameters, only reducing the batch size from 24 to 18 to fit in our GPUs and consequently adapting the learning rate.
Table 3: Captioning experiments: key implementation details.
Model
Architecture Visual encoder
ExpansionNetv2 [96]
Swin Transformer [97]
Visual encoder pre vreitrannnn Fine-tuning
ImageNet 22k [181]
Cross-entropy then CIDEr RL [24] on COCO [94]
RL procedure
Fine- tuning strategy
Usually frozen visual backbone, but end to-enn in Figure 13(d)
RI. algorithm
Self-critical [24]] variant of REINFORCE [98]
Optimizer
Radam [182]
Dataset
COCO [94] and Karpathy split [99]
Rewards
BLEU [29] (with I-gram or 4-grams), ROUGE [30], METEOR [95], CIDEr [31]
Learning rate
le:5
Batch size
18
Gradient accumulation
2
Warmup
Anneal 0.8 during epoch
Epochs
6
Hardware
GPU V100 32G 1500 GPUh
Compute budget
E.2 Additional results
042
trinn
SUNE Crnze mosandal
HOUGE rewanded
0:04
0.5
Ry BOT
M EGR renarded
CLDE:
remande
St. ewwnnded
BLEUA pownedy
pretooeed mit
HLL BEAACCA
BL: OLEVI
BL. BIAUU
RODGEE
METBOR
C1BS. ANAMINGTIA
28: (1 Sucase Paniou
B8: (1 dassos A Russson
RS: 11 A) Ronsor Anns
ein six
Ry. ALEUS
BLEU
(a) BLBU4 and ROUGE.
(b) BLEUI and METEOR.
(c) BLEUI and CIDEr.
Figure 11: Additional results in captioning with more rewards, complementing Figure 3. Specifically, Figure (1 uses Ri BLEU4 and Rs ROUGE; then, with Rı BLEU1, Figure (bb uses Rı M ET EOR and Figure (c))) uses R₂ Cr. In particular, the latter shows the failure when optimizing CIDEr; indeed, let's recall that the pre-trained initialization [96] has already been trained by optimizing CIDEr [24]. Thus optimizing CIDEr second time does not help, neither in CIDEr nor in other rewards. That's 's why in Figure 3(c) we consider the initialization as the network parametrization optimized for CIDEr.
33

---

B5, [1-AA
18 At Stuay Apoo OLTV)
KAASS
HOARS
411 111
SRAIE C- -0 OGAVE A ROUUS
BLSTT
DRAGE CLNN
0
(a) = BLEU4.
(b) R2 METEOR.
(c) R2 CIDEr.
(d) MORL.
Figure 12: Additional results in captioning when measuring performances on all rewards and varying the interpolating coefficients, complementing Figure 5(b). In Figures 12(a) to 12(c). we extend the results for Rs with Rı = BLEU1 and for varying R2; the optimal À depends on the similarity between the evaluation metric and R1 and R2. We also see in Figure 12(c) that all rewards are normalized to for the CIDEr-initie EDDEr-nitialliztion. In Figure 12(d), perform the same analysis for MORL while varying the weighting μ over the proxy rewards R1- BLEU1 and R2= ROUGE; we recover similar curves than in Figure 5(b) for RS.
BOUGE pouerded
DEPUT WODGER
0S
650-
mwandad
day BOAIG
13
ROUUEE
6:00
Rs ROUGE
mrezial
HLG BLEEE
BLEU
vourdel
R1. BEAEI Lf
WISE: [1- Al Bax 1 PALLAT
WISE: [1 2) dux RacINE
HIc NOU 126
RS. A0 degarr A BaoeGr
(186-
145 v3 4g
HJ. ROD DS v2
desann Buom
135 fruntsc (1 Al
TSS: (1- -gporss 3. -aron
155 of M8h and 09g
521
Rs: BLEUT
5:40
BLEUT
(a) Exploring new WI strategies.
(b) Results variances and model soups (MS).
A 0.5
3005
ROUGE
A-1.5
Ri: ROUGE
ROUGE rewarded
PLEt
rownoded
18.0
poetzuine iait
BLEU dew
A -03
19:04
0.5
0.00
RLI: ROLGGS
RL: DLEVI
MURL PARUE-ENESS
FOLETI
33:00 A04
Rg BLEV1
925
R BLEU
(c) Extrapolation with À outside of [0, 1].
(d) End-to-end training.
Figure 13: Additional results in captioning with R. BLEU1 and R₂ ROUGE. In Figure 13(a), investigate interpolating the fine-tuned networks with the pre-trained initialization as in WiSE [183]: this only reveals a small portion of the front. In contrast, the interpolation with O (μ= 0.5) solution improves RS's front: this highlights some limitations in Hypothesis and strict Pareto optimality of RS. Adding the MORL solutions as intermediate weights may help interpolate between two weights too distant. This suggests some practical complementarity between RS and MORL; given a training budget larger than the number of rewards, one may learn few MORL for varying 0 S μ < 1, and then interpolate the obtained solutions. Figure 13(b) shows results' variance with two RL trainings for BLEUI, and two for ROUGE, cach time with a different seed defining the data ordering and augmentations. Though we observe some randomness, the Hypothesis is consistently validated. Moreover, it presents the fronts described when interpolate weights fine-tuned on shared reward, as in model soups (MS) [67, 68]; mostly reduces variance and reveals only small portion of the spectrum of preferences. validating the need to fine-tune on different rewards (as proposed RS) to reveal the front across the entire space of preferences. Finally, the orange line shows that RS and MS can be complementary. by A-interpolating the MS for BLEUI and the MS for ROUGE with x = 0.5. Figure 13(c) presents the extrapolation results when À goes outside of [0, 1). This suggests that can artificially reduce reward with negative cocfficients, as studied in [141]. Finally, Figure 13(d) shows the results when the networks are trained end-to-end, rather than keeping the backbonddfrozen. This validates the efficiency of rewarded soups in a new more general setting where all layers are trainable.

---

F Text-to-image: diffusion models with diverse RLHFs
E1
Experimental details
Task description. Several works have studied the problem of aligning the output of diffusion models with human feedbacks [25, 26, 33]. Notably, diffusion models can be fine-tuned match human aesthetic perception. As for any subjective metric, there is a variety of reward models capturing different aesthetics. In our experiments, the two first reward models were trained in supervised setting to match human quality ratings collected on large image datasets, Specifically, the first R& is the ava aesthetic model, available here, trained on 250.000 images from the AVA dataset [103], based on CLIP features. The second R₂ is the cafe aesthetic model, available here, trained on 3500 real-life and anime/manga images. Moreover, Figure 14, we also consider a nsfw detector, estimating the probability of an image being safe by computing the cosine similarity with the CLIP embeddings of a set of unsafe words, as already done to filter the LAION dataset [1 [184].
Implementation details. We use a 2.2B parameters diffusion model trained on an internal dataset of 300M images, which reaches similar generation quality Stable Diffusion [102] in terms of CLIP alignment and FID scores on prompts from the 5000 images of the COCO test dataset (CLIPScore 30.0 30.2 for Stable Diffusion, FID 19.0 19.1 for Stable Diffusion). Given reward model Rt we first generate 10000 images with the pre-trained diffusion model on prompts from the COCO dataset, and compute the rewards for every generated image. For computational efficiency, we keep only dataset D' containing the 50% images with the best scores, and rescale rewards R linearly into r so that eD r r(zn)= = 0 and [DS ExSED r(xo) 1. Then, we fine-tune the diffusion model on the weighted negative log-likelihoon [25]:
L Fxx...0)ED..--V001114ccaaaa/o//
r(xo) x [xeo L, Q) =11²,
(19)
where 6p is the noise estimation network, T is the total number of training steps, r(xn) is the rescaled reward of image Xg and Q is the text associated to image Xn- As side note, on-policy RL. would require performing loops of image generations and model fine-tunings [185], but we only perform a single offline iteration for simplicity. Moreover, for efficiency, we only fine-tune 10% of the diffusion model's 's weights [104] corresponding to the cross-attention layers and the bias/scaling parameters. As further described in Table we apply the Adam [167] optimizer for 4000 steps with a batch size of 64 and a learning rate of 5e-6. To report results for each model (fine-tuned or interpolated via RS), we generate 1000 images from held-out set of COCO prompts and then we average the scores given by the reward models. To reduce the variance in image generation, each prompt has unique seed for all models, so that the input noise given the diffusion model only depends on the text prompt.
Table 4: Image generation experiments: key implementation details.
Model
Architecture Pre-training
GLIDE (2.2B parameters)
Internal dataset of 300M captioned images
RL Procedure
Fine-tuning objective
Reward-weighted diffusion loss
Fine-tuned parameters
Cross-attention layers and bias/scale
Optimizer
Adam [167]
Dataset
Generated with COCO prompts ava [103] and cafe and nsfw 5e-6
Rewards
Learning rate
Batch size
64
Epochs
25
Hardware
Single GPU V000 32G 500 GPUh
Compute budget
F22
Additional results
RS can trade-off between the two aesthetic rewards Figure 6(a), allowing adaptation to the user's preferences at test time. Yet, we show some limitations in the spider map of Figure 14, when
35

---

computing MORL and RS on all three rewards: ava, cafe and also the nsfw. In this case, MORL has higher scores than RS. We speculate this is because the nsfw very different from aesthetic preferences. Actually, the nsfw is inversely correlated with image quality: lower quality images result are less flagged as unsafe. This shows some limitations of weight interpolation when combining antagonist rewards. An improved strategy would first learn the MORL of the N = 3 rewards, and then optimize each reward independently from this improved initialization, before applying RS.
REE ons
RE cafe
MORL > R//
DS E 0,5
Figure 14: Image generation: spider map, with ava, cafe and nsfw reward models.
F.3 Visualization of generated images from interpolated models
We show in Appendix F.3 images generated by rewarded soups when varying the interpolation coefficient À between the two models fine-tuned for the ava and the cafe aesthetic rewards. You can find additional qualitative results for this experiment on this website.
Moreover, in Appendix F.3, we measure the FID [186] and the CLIPScore [187] of the images generated by the same interpolated models. This confirms quantitatively that images generated by interpolated models remain coherent.
Prompt
= 0.0
A= 0.2
== 0.4
A= 0.6
== 0.8
A= 1.0
tlg stands inside of it at camera
family with
wood floor and
beige walls
mattress leaning
against stone
wall.
man sitting on
top of chuir hold- ing up phone.
man sitting underneath umbrella and other structures.
Figure 15: Visualization images generated with rewarded soups for a varying interpolation coefficient between the two models fine-tuned for the ava (corresponding to X = 0) and cafe (corresponding to À = 1) reward models. We can see that all interpolated models produce images of similar quality compared to fine-tuned models, demonstrating linear mode connectivity between the two mnnettnnd models.
36

---

10.23
FID (4)
CLiPScare
11,00
31.15
22.
31:130
Pecere (17
32.7
33.005
32.1
31,050
323
31,025
04
0.2
0.1
0.5
0.8
1.0
Figure 16: Images generated by X-interpolated diffusion models evaluated terms realism by FID [186] or text alignment by CLIPScore [187].
G Text-to-box: visual grounding of objects with diverse sizes
G.1 Experimental details
We show the implementation details in Table 5. We use UnIVAL [106], a model pre-trained solely on public benchmarks, solve a variety of multimodal tasks such VQA. visual grounding and image captioning. It is then fine-tuned on RefCOCO+ dataset for visual grounding. During the last fine-tuning phase, we complement the cross-entropy loss with an additional REINFORCE [98] term rewarding accuracy when the object is of the considered size. This means that the loss for Smmall (log()) ()) + (area(()) small) LUCC(,.))000 x log(y)) for an object with ground -ruth box and prediction The image is discretized into 1000 x 1000 bins before calculating the box areas. The task is illustrated in Figure 17.
pu ple shi womd
Model
curv
air
Curly hair Purple shirt Green shirt man
Figure 17: Illustration of the visual grounding task. The RS model results from the average N 3 weights specialized to detect respectively small, medium and large objects. The model takes text (one description at a time) as input and outputs the bounding box in the corresponding region of the image. We show an example of small, medium and large predictions, and the associated ground truths in green. These texts and image are from the validation of RefCOCO+ [105].
37

---

Table 5: Visual grounding experiments: key implementation details.
Model
Architecture
UnIVAL [106]
Visual encoder
ResNet-101
Pre-training
Cross-Entropy on Public datasets (VQA, VG, Captioning)
Supervised fine-tuning
Cross- Eessopy on RefCOCO+ [105]
RL procedure
Fine-tuning strategy
end-to-end
Dataset
RefCOCO+ [105]
RL algorithm
Cross-entropy 5x REINFORCE
Reward Small
IoU>0.5 for object with area < 30000
Reward Medium
IoU>0.5 for object with 30000 < area < 100000
Reward Large
IoU>0.5 for object with 100000 S area
Optimizer
Adam
Learning rate
3e-5
Batch size
256
Epochs
10
Hardware
8 GPU 60GB 800 GPUh
Compute budget
G.2 Additional results
Catd neurl
Andiay avaslal
Gmais fodes
(1 3) Riiee 1.A. A.
(a) Small and Medium.
(b) Medium and Large.
(c) Spider map.
(d) CE RL.
Figure 18: Results in visual grounding RefCOCO+ [105]. We REINFORCE, [98] to improve directly the non-differentiable accuracy, i.c., predict boxes with ToU> 0.5 w.r.t the ground-truth. Fine-tunings are specialized on either small, medium, or large objects. These experiments complement Figures 6(b) and 18(c). Figure 18(c) shows that improving results on all sizes simultaneously is challenging, as MORL performs similarly to the initialization. Finally, Figure 18(d) motivates the of RL to fine-tune different sizes. Indeed, the results for (the proposed) RS of RL are significantly better than the results RS of CE, where we average weights specialized on different sizes by fine-tuning with cross-entropy (rather than with REINFORCE).
H Locomotion with diverse engineered rewards
Task description. This experiment takes on the intricate challenge of controlling a running humanoid in the Brax [113] physics engine. The complexities involved in achieving natural or fast movement in continuous control environments serve a testament to the robustness of our approach. The fine-tuning procedure is carried out on two distinct reward functions, with the aim of refining the running behavior of the humanoid, potentially resulting in smoother motion patterns. You can find qualitative results of this experiment on this website.
Pre-training, According to Remark 1, the LMC requires pre-training the base policy before fine- tuning. Thus, as the pre-training task, we use the default dense reward implemented in Brax: R = velocity 0.1 x Σ, oi. This pre-training phase also serves to collect statistics about observa- tions and normalize them before inputting to the model (as it facilitates training). We used the Brax implementation of PPO [84]. The pre-trained policy is saved while the value function is discarded.
38

---

Fine-tuning, We keep the same environment as in pre-training. We also use the normalization procedure inherited from pre-training but freeze the statistics. Two reward functions designed: risky one for R1 velocity and a cautious one where R2 velocily Σ at We tried a few hyperparameters (see the values in brackets Table 6) but results (see Figure 19) remain close and consistently validate our working hypotheses.
Table 6: Locomotion experiments: key implementation details.
PPO Pre-training
Interactions
5e8
Reward Scaling
1.0
Episode Length
1000
Unroll Length
Discounting
0.99
Learning Rate
5e-5
Entropy Cost
le-3
Number of environments in parallel
4096
Batch Size
1024
Hardware
IGPU Tesla V100-SXM2-16GB
Runtime per experiment
80min
PPO Fine-tuning
Interactions
le8
Reward Scaling
1.
Normalize observations
True
Unroll Length
10
Discounting
(0.97, 0.99, 0.999)
Learning Rate
(1e-5, 3e-5, le-4)
Entropy Cost
[1e-3, le-2)
Number of environments in parallel
4096
Batch Size
1024
Hardware
IGPU Tesla V100-5XM2-16GB
Runtime per experiment
20min
Model architecture
Policy
Architecture
MLP
Nb of Layers
6
Hidden Size
512
Value
Architecture
MLP
Nb of Layers
Hidden Size
256
39

---

Ry: contoaate
R2 constumus
- Prestranoe Inil
R8: 13 - Ai Graby A Bastion
6.10
4.40
200+
410
200
200
Figure 19: Analysis of results' variance for the locomotion task when varying the hyperparameters. Each column i corresponds to the i-th Orisky. interpolated in case (i, towards the j-th Occutious. The Figure 6(c) is actually the plot from case (1, 1).
40


Reviews:
[{"review_id": "4w9zqikdxE", "reviewer": "Reviewer_stHc", "summary": "This paper proposes a method of using linear interpolated weight finetuned on different rewards instead of using linear combination of rewards to finetune weight, which is a solution to applying model under different and multiple preference scenarios. The idea is intuitive but works well, it (the RS) can achieve similar performance compared with MORL, while RS reduces the computational cost significantly. The author makes some mathematical hypotheses which empirically hold when all the weight to be interpolated is finetuned from a pretrained model. Also, the author has done a lot of experiments to show the feasibility of the proposed method.", "questions": "Does the method harm the absolute quality of the produced samples? Evaluation other than reward functions should be provided.\n\nMay be you can show some extrapolated examples generated through this method.", "limitations": "All right in total.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written. The experimental work is sufficiently done.\n\nThe author did a lot of experiments on different task which shows that this interpolating strategy is universal under different application scenarios, while with good performance.\n\nThe proposed method reduces the heavy computational requirements for pretuning compared with previous work, which makes it much more applicable and flexible to complex application scenarios.\n", "weaknesses": "Novelty: The most heavy workload in this paper is applying the strategy to different tasks and testing their performance, while less novel concepts or theories is presented.\n \nCondition for Hypothesis: The hypothesis used in the method, such as the linear mode connectivity which states that the combined reward is concave to the weights requires research on the model structure and activation function. There should be some limitations for the design of networks which ensures that the hypothesis holds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank R.stHc for the positive feedback on the clarity of our idea and the experiments. We would like to respond to R.stHc's review as follows.\n\n---\n\n### Q1. Novelty\n\nOur approach is novel from two perspectives.\n\nThe first **conceptual** novelty is arguing for Pareto-optimality and a **multi-objective paradigm** to reduce **reward misspecification** when aligning deep generative models. This first novelty is critical to \"handle the diversity of human preferences\" (l.50), and as further detailed in Appendix A.1, to \"support decision-making\" (l.872), \"interpretability and explainability\" (l.878).\n\nThe second **empirical** novelty is proposing rewarded soups, based on new setups/conditions where the **linear mode connectivity** holds (in reinforcement learning, with diverse rewards, even in the multimodal case) and thus where weight interpolation can be used. This second novelty is critical to reduce \"the computational, memory, and engineering costs involved\" (l.106) in traditional MORL approaches, and as further detailed in Appendix A.2, to be \"compatible with the inherent iterative engineering process of alignment\" (l.890).\n\nMoreover, we want to point out that in Appendix B.2 \"we provide **theoretical** guarantees for the **near-optimality of RS** when considering quadratic rewards\" (l.908), as referenced l.141-143 and l.146 in Remark 1. Specifically, in Lemma 3, we bound the reward difference between the optimal policy and our interpolated policy. We give more theoretical details in our response to [R.bXWy.Q2](https://openreview.net/forum?id=lSbbC2VyCu&noteId=rSiwrlT8Be).\n\n---\n\n### Q2. Limitations for the design of networks for LMC?\n\nIn our experiments, we consider different network architectures (transformers, CNNs, and MLPs), with various activation functions.\nWe also investigate different training procedures: with low-rank adapters, partial or end-to-end fine-tunings.\nWe do so for many different tasks and modalities: text generation, image captioning, image-to-test generation, visual grounding, visual question answering, etc.\nOur empirical observation is that, across those setups, the **LMC is architecture-agnostic, procedure-agnostic, task-agnostic and modality-agnostic**.\n\nThe main condition we require is the shared pre-trained initialization [Neyshabur2020], as emphasized in Remark 1; this \"prevents the weights from diverging\" (l.145) and forces them \"to remain close\" (l.146).\nThe other condition, suggested by the literature [Li2022,Ilharco2023] and as also discussed in [R.TSwH.Q4](https://openreview.net/forum?id=lSbbC2VyCu&noteId=np0m5ACZtx), is that the architecture has enough trainable parameters. Indeed, larger networks may facilitate the orthogonality of the fine-tuned updates; then [Ilharco2023] \"speculate that this [orthogonality] enables the combination of task vectors via addition with minimal interference\". In conclusion, our experiments and the literature suggest **that the network design is not critical for the LMC, as long as the network is pre-trained and sufficiently parameterized**. Those constraints are arguably minimal given the predominance of the foundation model paradigm and the scaling trend in deep learning.\n\n[Neyshabur2020] What is being transferred in transfer learning? NeurIPS.\\\n[Li2022] Branch-Train-Merge: Embarrassingly parallel training of expert language models.\\\n[Ilharco2023] Editing models with task arithmetic. ICLR.\n\n---\n\n### Q3. Does the method harm the absolute quality of the produced samples? show the generated samples, and provide more evaluation\n\n**Qualitatively**, samples generated by weight interpolated models do not suffer from reduced quality. This was visible for text-to-image generation with diffusion models in Figure 12 from Appendix E.3, where we state: \"we can see that all interpolated models produce images of similar quality compared to fine-tuned models\". Moreover, **our anonymous website** (referenced l.856, l.1065, and l.1084), also includes generated samples for the locomotion task and for the text-to-text summarization task. For the sake of completeness, we now include **examples of generated summaries** in the Table 1 from the one-page rebuttal pdf; qualitatively, the summaries generated by interpolated models remain grammatically coherent.\n\nTo **quantitatively** validate this insight, the one-page rebuttal pdf includes new plots evaluating the samples generated by RS.\n\n- Figure 1.a evaluates the generated summaries when $\\lambda$-interpolating between two LLMs fine-tuned on two summary rewards. We leverage two text metrics; the first is (i) **perplexity** (exponentiated average NLL of the generated summaries) according to MLMS [Salazar2020] and GPT2 (following [Lee2021] and this [blog](https://huggingface.co/docs/transformers/perplexity)); the second is (ii) **quality**, as estimated by this [newspaper quality model](https://huggingface.co/valurank/distilbert-quality).\n- Figure 1.b evaluates the generated images when $\\lambda$-interpolating between two diffusion models fine-tuned on two aesthetic rewards. We leverage two standard image metrics; the first is (i) **FID** [Heusel2018] measuring image realism; the second is (ii) **CLIPScore** [Hessel2021] measuring image-text alignment.\n\nIn conclusion, we confirm quantitatively that **RS does not deteriorate quality**. More precisely, by interpolating the weights, we also interpolate the metrics; intermediate values of $\\lambda$ even sometimes increase quality. We will detail this analysis in the revised paper, and would be pleased to include any other suggested quality metrics.\n\n[Salazar2020] Masked Language Model Scoring. ACL.\\\n[Lee2021] Towards Few-Shot Fact-Checking via Perplexity. ACL.\\\n[Heusel2018] GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. NeurIPS\\\n[Hessel2021] CLIPScore: A Reference-free Evaluation Metric for Image Captioning. ACL."}, {"review_id": "C7qQ5cE61M", "reviewer": "Reviewer_DNE5", "summary": "This manuscript studies a way to interpolate trained networks' parameters for diverse rewards in a reinforcement learning manner. To be specific, the proposed method introduces a way to achieve Pareto-optimal solutions through linearly weighted parameters after training. Extensive experiments showed the effectiveness of the proposed method on various domains such as text2text, image2text, and text2image.\n", "questions": "I already mentioned above in the weakness section. I would like to ask the authors how the proposed method is significantly better than other weight interpolation methods like model soup, not only the reward interpolation method, MoRL.", "limitations": "I think there is no potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The manuscript is well-written and organized overall.\n- The proposed idea is effective yet efficient as it does not require additional training.\n- Extensive experimental results demonstrate the effectiveness of the proposed method on various domains; text generation, image captioning, and diffusion model.", "weaknesses": "Interpolating weights for better performance is not a new concept; model soups, which is mentioned in the manuscript. However, the authors did not provide any comparison with it. When we have N fine-tuned models, rewarded soup can perform better than model soup? \nFor example, in the case of an image captioning task, the experimental setup assumes only two rewarded models, differently fined-tuned models on AVA and cafe datasets, respectively. I think there is no reason to hesitate to apply a way of model soup.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank R.DNE5 for highlighting the organization of the paper and the diversity of our experiments.\n\n---\n\n### Q1. Similarity and differences with model soups\n\nR.DNE5's main concern relates to the similarity between rewarded soups (RS) and model soups (MS).\nFirst, we totally acknowledge similarity with MS; actually, as stated l.65, \"the name rewarded soups follows the terminology of model soups\".\nIndeed, RS and MS both average the weights of models fine-tuned from a shared pre-trained initialization.\n\nYet, we want to clarify that **RS and MS tackle different problems, have different goals, leading to different methods and implementations**.\n\n- MS challenges the standard model selection after a grid search to improve generalization in supervised learning, and aims at reducing the variance of the predictions by combining the fine-tuned models: thus MS considers different hyperparameters for a fixed training objective across runs, and (usually) uniform interpolating coefficients $\\lambda=\\frac{1}{M}$.\n- In contrast, RS challenges single-policy approaches to improve alignment in reinforcement learning, and aims at reducing reward misspecification by revealing a Pareto front of solutions across the entire space of preferences: thus RS considers different training objectives for fixed hyperparameters across runs, and non-uniform interpolating coefficients $\\lambda$ set a posteriori.\n\nOverall, these differences mean that **RS can but MS cannot be applied to reduce reward misspecification**.\nWe refer R.DNE5 to the Figure 10.b from Appendix D.2 (reproduced and enriched in the Figure 3.c from the one page rebuttal). The experiments are for the captioning task, when considering BLEU1 and ROUGE as rewards. The green lines only consider one fine-tuning per reward (standard RS), while  the light-blue (for BLEU1) and pink (for ROUGE) lines consider two fine-tunings on one single reward (standard MS).  As stated in the legend from Figure 10.b , \"it presents the fronts described when we interpolate weights fine-tuned on a shared reward, as in model soups. This also only reveals a small portion of the spectrum of preferences, validating the need of diverse rewards to satisfy all users’ preferences\". Specifically, MS mostly reduces variance; in contrast, considering **weights specialized on different rewards** (as proposed in this work) is key to reveal the front across the entire space of preferences.\n\nIn summary, regarding the exact statements from R.DNE5:\n\n- \"Interpolating weights for better performance is not a new concept\": indeed, but we are the first to use weight interpolation for alignment, for models RL fine-tuned with different rewards, in particular for generative and multimodal tasks.\n- \"the authors did not provide any comparison with model soups\": actually we already did, in Figure 10.b from Appendix D.2.\n- \"When we have $N$ fine-tuned models, rewarded soup can perform better than model soup?\": RS will be better in terms of Pareto optimality. Yet, if the true reward is available before training and thus there is no reward misspecification, fine-tuning $N$ models on this exact reward (as in MS) will certainly provide better results.\n- \"For example, in the case of an image captioning task, the experimental setup assumes only two rewarded models, differently fined-tuned models on AVA and cafe datasets\". In the image captioning task (Section 3.2), we consider multiple metrics such as BLEU1, BLEU4, ROUGE, and METEOR. In the image generation task (Section 3.3), we consider two models fine-tuned on reward models trained on AVA and cafe datasets. In the latter case, fine-tuning multiple times on the cafe reward would fail to improve the AVA reward, as \"the model $\\theta_{\\text{cafe}}$ performs poorly in terms of AVA\" (l.245).\n- \"how the proposed method is significantly better than other weight interpolation methods like model soup\": RS is the only weight-interpolation method seeking Pareto optimality across diverse rewards, thus the other methods will only optimize a metric given a priori, without tackling reward misspecification.\n\nAs a final note, in the Figure 3.c from the one page rebuttal, we also try to combine RS and MS, and thus plot:\n$\\lambda \\to \\frac{1-\\lambda}{2} \\cdot (\\theta_{BLEU1}^{v1} + \\theta_{BLEU1}^{v2}) + \\frac{\\lambda}{2} \\cdot (\\theta_{ROUGE}^{v1} + \\theta_{ROUGE}^{v2}),$ where $\\theta_{BLEU1}^{v1}$ and $\\theta_{BLEU1}^{v2}$ are from two independent RL fine-tunings on BLEU1 (and similarly for ROUGE). This orange line $\\lambda$-interpolates between two MS weights specialized on different rewards, which are themselves the averages of models fine-tuned on the same reward. The convexity of the interpolation and the slightly better performances at the endpoints  show that **we can combine the benefits from RS (reward misspecification) and MS (variance reduction)**.\n\n---\n\nWe hope this answer clarifies the difference between rewarded soups and model soups; we remain available for any further discussion."}, {"review_id": "IrQqHhrb48", "reviewer": "Reviewer_ntSF", "summary": "The authors present a new strategy to address the heterogeneity of diverse rewards in reinforcement learning. Specifically, they propose 'rewarded soup,' which involves individually training multiple networks, each assigned to a different proxy reward, and then linearly combining these networks. Compared to the multi-objective reinforcement learning baselines, the proposed rewarded soup demonstrates its superiority on several benchmarks, including text-to-text, text-to-image, and control benchmarks.", "questions": "- This is aligned with the weakness part. I am curious about the effectiveness of the rewarded soup under different scenarios:\n\n1. How does the difference/gap of rewards affect the effectiveness of the MORL baseline and the rewarded soup?\n2. How does the number of networks affect the results?\n\nThe reason why I am particularly interested in these two questions is that a prior work [1] indicates when the models are quite different (based on their objectives), the linear combination is likely to produce less favorable results. I am wondering about \"the limit\" of the rewarded soup and in what situations the proposed method might fail.\n\n\n[1] \"Robust fine-tuning of zero-shot models\" Wortsman et al., CVPR 2022", "limitations": "The authors have provided sufficient information of limitations and societal impact in the paper. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors presented a comprehensive study on the topic and the research field, as most arguments in introduction are supported by some references. The motivation is strongly supported and the path to the proposed method is reasonable.\n\n- Presentation is clear, concise, and easy-to-understand, and the idea is simple yet effective. \n\n- The authors conducted extensive experiments to verify the effectiveness of the proposed rewarded soup, and this includes multiple text-to-text tasks (shown in Section 3.1), image-to-text tasks (Section 3.2), text-to-image tasks (Section 3.3), and control tasks (Section 3.5). For most of the experiments, the improvement against MORL is obvious.", "weaknesses": "Although there are many strengths in the paper, there is a weakness that can be further enhance the overall quality. \n\n- Ablation studies could be added: While the authors presented many reinforcement learning applications and also showed the improvement, it would be better to include more fine-grained ablation studies, such as how the difference of rewards affects the effectiveness of MORL baseline and the rewarded soup or how the number of networks affects the results.  \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank R.ntSF for the deep understanding of the paper, for highlighting its strengths and for asking two intriguing questions - that we try to answer below.\n\n---\n\n### Q1. How does the difference/gap of rewards affect the effectiveness of the MORL baseline and the rewarded soup?\n\nOur experiments in captioning and image generation provide empirical evidence that **the more similar the rewards, the higher the gains of RS versus MORL**.\n\nFirst, in the **captioning** experiment from in Figure 3.c, by analyzing the transfer abilities across the 4 main metrics (BLEU1, BLEU4, ROUGE, and METEOR), we can deduce that:\n\n- BLEU4 and ROUGE are very similar.\n- BLEU1 and BLEU4 are more similar than BLEU1 and ROUGE.\n- METEOR is an outlier, quite different from other metrics, in particular from BLEU1.\n\nNow having set these similarities across rewards, we observe that the gains of RS versus MORL are consistent with these similarities across rewards.\nSpecifically,\n\n- with $R_1=\\text{BLEU4}$ and $R_2=\\text{ROUGE}$, we observe large performance gains for RS versus MORL (in Figure 8.a), where the green front is highly convex far above the solution provided by the MORL objective.\n- with $R_1=\\text{BLEU1}$, we observe larger gains (and cleaner convexity) for RS versus MORL with $R_2=\\text{BLEU4}$ (in Figure 3.b) than with $R_2=\\text{ROUGE}$ (in Figure 3.a).\n- with $R_1=\\text{BLEU1}$ and $R_2=\\text{METEOR}$, we observe better performances for MORL than for RS (in Figure 8.b).\n\nOverall all captioning rewards remain sufficiently similar to favor RS over MORL when combining all rewards in Figure 3.c.\n\nSimilarly, in the **image generation** experiment, when we consider two (arguably similar) aesthetic rewards in Figure 5.a to fine-tune a diffusion model, RS's front is to the right and above MORL's front.\nThen, when the rewards are very different or antagonist, we totally agree with your statement that \"the linear combination is likely to produce less favorable results\": in Appendix R.2 we include a *nsfw* reward \"inversely correlated with image quality\" (l.1058), and observe in this case that \"MORL has higher scores than RS\" (l.1056). This result \"shows some limitations of weight interpolation when combining antagonist rewards\" (l.1059). These insights were already briefly mentioned in the main paper (we state l.140 that \"we report a few limitations in Appendix and research directions to fix them\"), but will be clarified in the limitation section from the revised version. They can be explained in two different ways:\n\n- intuitively, from a **loss landscape perspective**, weights fine-tuned on diverse rewards will be more distant, thus potentially breaking the linear mode connectivity.\n- theoretically, thanks to **Lemma 3 in Appendix B.2.2**, where we bound the difference between the optimal reward and RS's reward by a RHS term growing with \"the maximum of eigenvalues ratio\" (l.942) across rewards' Hessians. This RHS term is illustrated in Figure 7. Then, if the rewards are more diverse, their Hessians would have more different eigenvalues, thus maximum of Hessian's eigenvalues ratio would grow, the RHS term would grow in Lemma 3, and our guarantees for the optimality of RS would get loose.\n\n---\n\n### Q2. How does the number of networks affect the results?\n\nThough most of our experiments are with $N=2$ rewards and networks for visualization clarity, \"RS can scale and trade-off between more rewards\" (l.201).\nWe validate this empirically in the spider maps from Figure 3.c (for image captioning), from Figure 2.f (for text generation),, and from Figure 5.c (for visual grounding), where we uniformly combine respectively $M=5$, $M=4$, and $M=3$ networks fine-tuned on $N=M$ rewards, one reward each.\n\nMost importantly, we **refine this analysis** for the captioning task through additional spider maps in Figure 1.c and 1.d (from the one-page rebuttal). Specifically, we compare the performances across all $N=5$ metrics when averaging $1\\leq M \\leq 5$ networks (each fine-tuned on one of the $N$ rewards, thus leaving out $N-M$ rewards at training) and sequentially adding more networks to the weight average. We consistently observe that **adding one additional network specialized on one additional reward extends the scope of the possible rewards that RS can tackle Pareto-optimally**.\n\nAs a short note, another possibility to scale the number of networks is at fixed number of rewards, by learning multiple networks on the same reward as in model soups (MS) [Wortsman2022].\nWe consider this in the Figure 3.c from the one page rebuttal for the captioning task and plot $\\lambda \\to \\frac{1-\\lambda}{2} \\cdot (\\theta_{BLEU1}^{v1} + \\theta_{BLEU1}^{v2}) + \\frac{\\lambda}{2} \\cdot (\\theta_{ROUGE}^{v1} + \\theta_{ROUGE}^{v2})$, where $\\theta_{BLEU1}^{v1}$ and $\\theta_{BLEU1}^{v2}$ are from two independent RL fine-tunings on BLEU1 (and similarly for ROUGE). This shows that we can combine RS and MS, i.e., $\\lambda$-interpolating between two MS weights specialized on different rewards, which are themselves the averages of models fine-tuned on the same reward.\n\nIn conclusion, in all our experiments, **performances consistently increase for more networks**; when they are trained on different rewards, this reduces reward misspecification; when they are fine-tuned on the same reward, this reduces variance. This is consistent with the findings from previous works, eg, the Figure B.1 from model soups [Wortsman2022] which showed than increasing the number of averaged models consistently helps.\n\n[Wortsman2022] Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICML.\n\n---\n\nWe hope these explanations and these additional ablations clarify your questions. Please let us know if there is anything else we can do to further strengthen our submission."}, {"review_id": "fcVHV6bido", "reviewer": "Reviewer_bJvT", "summary": "This paper explores a model-soup strategy to efficiently adapt to diverse reward functions from various real-world users. By fine-tuning a pre-trained LLM multiple times each with a specialized reward function and interpolating their weights linearly, the proposed method is able to adapt to various reward functions without having to train a new LLM per user.", "questions": "Noted above.", "limitations": "Noted above.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed method is much more efficient than the baselines, which have to train a separate model when a new reward function is given.\n- The evaluation is through and conducted on a variety of LLM tasks.\n- The performance of the proposed method does not fall behind compared to the more costly baselines.", "weaknesses": "- The novelty of the paper is weak. It seems the main contribution of the paper is applying the weight interpolation (model soup [1]) technique, which was well-explored in supervised learning, to RLHF. I suggest the authors clarify the paper's novelty (other than applying the model-soup technique to another domain) more clearly.\n- The authors point out that in RLHF, the reward function is different per each model, unlike supervised learning where the training objective is the same. I agree with the authors on this point, but the rewards used in the paper's experiments do not seem very heterogeneous to back up the authors' claim. A more realistic scenario would be to experiment with a set of rewards that contradict each other directly (e.g., different reward functions learned from users with conflicting interests).\n- The main strength of the proposed method is that it is more efficient in terms of training (fine-tuning) cost and inference cost. However, the paper does not provide a quantitative comparison of these costs between the proposed method and the baselines. Therefore, it is hard to assess how much efficiency gain the proposed method will provide.\n\n[1] Wortsman et al., Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, ICML 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank R.bJvT for reviewing our work, and try to address the expressed concerns below.\n\n---\n\n### Q1. Novelty and difference with model soups (extended in [R.DNE5.Q1](https://openreview.net/forum?id=lSbbC2VyCu&noteId=6LMJAJD6vx))\n\nThe first conceptual novelty is arguing for a **multi-objective paradigm** to align deep models with human preferences and reduce reward misspecification. The second empirical novelty is observing new setups/conditions where the **linear mode connectivity** holds, and thus where weight interpolation can be used; eg, in reinforcement learning with different rewards, for multimodal tasks. This weight interpolation strategy was indeed used in model soups (MS). Yet, we want to clarify that **RS and MS tackle different problems, have different goals, leading to different methods and implementations**.\n\n- MS challenges the standard model selection after a grid search to improve generalization in supervised learning, and aims at reducing the variance of the predictions by combining the fine-tuned models: thus MS considers different hyperparameters for a fixed training objective across runs, and (usually) uniform interpolating coefficients $\\lambda=\\frac{1}{M}$.\n- In contrast, RS challenges single-policy approaches to improve alignment in reinforcement learning, and aims at reducing reward misspecification by revealing a Pareto front of solutions across the entire space of preferences: thus RS considers different training objectives for fixed hyperparameters across runs, and non-uniform interpolating coefficients $\\lambda$ set a posteriori.\n\nOverall, these differences mean that **RS can but MS cannot reduce reward misspecification**. We will clarify this difference between RS and MS in the revised version.\n\n---\n\n### Q2. Rewards diversity\n\nWe respectfully disagree with R.bJvT, and argue that **we already use diverse and heterogeneous rewards that are in tension**. For example:\n\n- for the summarization tasks (in Figure 1.b, 2.a and 2.b): $R_1$ rewards completeness, while $R_2$ focuses on \"faithfulness\" (l.1005).\n- for the captioning experiments (in Figure 3.a and 8.b): BLEU1 measures accuracy while ROUGE captures recall (see l.42), and METEOR handles synonyms.\n- for the visual grounding experiments (in Figure 5.b and 14), the different rewards consider objects of different sizes.\n\nThe dissimilarities between these rewards are quantitatively validated by our experiments; when fine-tuning on one reward, the performances are usually worsened on the others.\nFor example, for captioning \"tuning solely BLEU1 sacrifices some points on ROUGE\" (l.213); for visual grounding \"optimizing for small objects degrades performance on large ones\" (l.259).\nThese examples are arguably representative of \"different reward functions learned from users with conflicting interests\" (R.bJvT).\n\nYet, we acknowledge (in Appendix E.2 and in our response to [R.ntSF.Q1](https://openreview.net/forum?id=lSbbC2VyCu&noteId=UEM7DRMGYu)) some \"limitations of weight interpolation when combining antagonist rewards\" (l.1059). This was suggested by the results for text-to-image generation in Figure 10, where RS underperforms MORL when considering a *nsfw* reward \"very different from aesthetic preferences\" (l.1057) and \"inversely correlated with image quality\" (l.1058). This limitation will be clarified in the limitation section from the revision. However, we want to emphasize that, in this kind of situation with fully antagonist rewards, the **complementarity of MORL and RS is a promising research direction**, as discussed l.1060: \"an improved strategy would first learn the MORL [...], and then optimize each reward independently from this improved [MORL] initialization, before applying RS\". As another research direction, we suggest (in the legend from Figure 10.a) that: \"adding the MORL solutions as intermediate weights may help interpolate between two weights too distant\".\n\n---\n\n### Q3. Quantify the efficiency gain (close duplicate of [R.QPfR.Q3](https://openreview.net/forum?id=lSbbC2VyCu&noteId=58jAIy2tXU))\n\nIndeed, \"the main strength of the proposed method is that it is more efficient in terms of training (fine-tuning) cost\" (R.bJvT) than the MORL baseline. For example, as stated in the legend from Figure 1.b, \"with only two trainings [RS] reveals the green front of Pareto-optimal solutions [...] and matches the costly yellow front of MORL requiring [11] trainings on different linear weightings\".\nAs a side note, truly revealing the full MORL front would actually require an infinite number of trainings.\n\nTherefore, we argue that this **efficiency gain is by design**; when considering $N$ rewards, RS only requires $M=N$ fine-tunings, while MORL \"requires explicitly maintaining a large set $M \\gg N$ networks, practically one for each possible preference\" (l.105).\nIndeed, as stated l.106, a critical issue in MORL is that “minor [preference] variations may result in significant changes in the solution. Thus, a high level of granularity in the mesh is necessary\".\n\nTo quantify the efficiency gain of RS, we now provide an analysis in Figure 2 from the one-page rebuttal pdf, where we define a new measure of success; the expected reward $E_{\\hat{\\mu}\\sim Unif\\left(0,1\\right)} \\hat{R_{\\hat{\\mu}}}$ where $\\hat{R_{\\hat{\\mu}}} = (1-\\hat{\\mu})\\times R_1 + \\hat{\\mu} \\times R_2$ and the expectation is over the possible user's preferences $\\hat{\\mu}$. Then we compute the difference between (i) the expected reward for RS (always with $2$ training runs), and (ii) the expected reward for MORL with $M$ training runs. **Plotting this expected reward advantage for different values of $M$ confirms that MORL needs $M \\gg 2$ to match RS**. Moreover, because of the dimensional curse, we expect the number of MORL trainings required to match RS to grow exponentially with the number of rewards $N$. In conclusion, these new experiments quantitatively validate that RS is more efficient than MORL, and will be included in the revised paper."}, {"review_id": "bGB3i3apKd", "reviewer": "Reviewer_QPfR", "summary": "In this paper, the authors propose a multi-policy strategy called \"rewarded soups\" to fine-tune any foundation model, embracing the heterogeneity of diverse rewards. The method combines multiple networks through linear interpolation in the weight space, despite the non-linearities in the network, which efficiently yields Pareto-optimal solutions after training. The authors demonstrate the effectiveness of the approach for text-to-text, text-image, and locomotion control tasks, showing that \"rewarded soups\" can mitigate reward misspecification. The proposed approach aims to enhance the alignment of deep models and how they interact with the world in all its diversity. The authors highlight the issue of aligning AI systems to specific and diverse needs while making the process more transparent and limiting the cultural hegemony of a few individuals.", "questions": "1. Why does the front passing through the point obtained by MORL fine-tuning on the average of the two rewards support Hypothesis 2 in Figure 2?\n2. Some formulas need clarification. For example, is  $\\lbrace\\lambda_i\\rbrace_{i}$ equals to $\\lbrace\\lambda_i\\rbrace_{i=1}^N$?\n3. $\\lbrace\\lambda_i\\rbrace_{i=1}^N$ is selected by users according to their preferences. How do users select these coefficients? For example, the user will give a preference label over pair-wise (or k-wise) instances in the standard RLHF paradigm.", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper addresses the reward misspecification problem caused by single priori rewards in current RLHF frameworks for foundation models. In order to solve the problem, this paper proposes a relatively complete framework called rewarded soup (RS). RS combines multiple networks (fine-tuning on different proxy rewards) through linear interpolation in the weight space and selects relative coefficients according to the user’s preferences, yielding Pareto-optimal solutions. The content of the whole paper is complete, and the experiments are sufficient.", "weaknesses": "1. The writing logic of the article could be more coherent in the reviewer's opinion. For example, in 3.3, \"Moreover, RS gives a better front than MORL, validating Hypothesis 2.\" and in 3.5, \"Moreover, the front defined by RS indicates an effective balance between risk-taking and cautiousness, providing empirical support for Hypothesis 2, although MORL with $\\mu$ = 0.5 (i.e., $\\alpha$ = 0.5) slightly surpasses RS's front.\" \n2. Validations on Hypothesis 1 and 2 are based on empirical results. The authors are encouraged to give some theoretical analysis. \n3. Lack of experiments on computational costs. According to the results, the proposed method has no strengths compared to MORL. The authors state that RS can reduce computational costs. However, no relative experiments showed in this paper.\n4. Most experiments are assigned an N=2 reward model, which is not aligned enough with the \"diverse\" in the title.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank R.QPfR for reviewing our work, and try to address the expressed concerns below.\n\n---\n\n### Q1. Empirical validation of Hypothesis 2\n\nOur introduction in Section 3 and the Remark 2 explain why \"the front passing through the point obtained by MORL fine-tuning on the average of the two rewards support Hyp. 2\" (R.QPfR). This is because we use **this MORL as the reference to evaluate the Pareto-optimality of RS**. Specifically, we state l.176 that: \"as the true Pareto front is unknown in real-world applications, we present empirical support for Hyp. 2 by comparing the front defined by RS (sliding $\\lambda$ between $0$ and $1$) to the MORL's solutions optimizing the $\\mu$-weighted rewards for $0\\leq\\mu\\leq 1$ (sometimes only $\\mu=0.5$ for computational reasons)\". We provide more details l.151: \"in full generality, improvements in initialization, RL algorithms, data, or specific hyperparameters could enhance performances. [Thus] the true Pareto front [...] needs to be defined with regard to a training procedure. [...] As such, in Section 3 [and Figure 2] we analyze Hyp. 2 by comparing the fronts obtained by RS and scalarized MORL\".\n\nThen our experiments consistently show that MORL and RS perform similarly (with minor differences in different setups). Overall, in absence of the true Pareto front, this process provides empirical support **towards** Pareto-optimality of RS, with indeed a limitation highlighted in the paper's name.\n\n---\n\n### Q2. Theoretical analysis (extended in [R.bXWy.Q2](https://openreview.net/forum?id=lSbbC2VyCu&noteId=rSiwrlT8Be))\n\nIndeed, the *full validation* of \"Hyp. 1 and 2 are based on empirical results\" (R.QPfR). That\"s why we state l.322: \"RS relies on an empirical finding: the LMC, which currently lacks full theoretical guarantees, even in the simplest case of moving averages\" in supervised learning.\n\nYet, we respectfully disagree with R.QPfR as **our work already gives theoretical analysis**, in particular in Appendix B.2 where \"we provide theoretical guarantees for the near-optimality of RS when considering quadratic rewards\" (l.908). Specifically, in Lemma 3, we bound the reward difference between the optimal policy and our interpolated policy. This is referenced in the main paper at two different places, where we state: (i) l.141-143 \"we theoretically prove in Appendix B.2 [that our Hypotheses 1 and 2] approximately hold when rewards are replaced by their second-order Taylor expansion with co-diagonalizable Hessians\"; and (ii) l.146 \"when the weights remain close, we can theoretically justify Hypotheses 1 and 2 (see Appendix B.2) and, more broadly, demonstrate that WI approximates ensembling (see Lemma 4)\".\n\n---\n\n### Q3. Computational costs (close duplicate of [R.bJvT.Q3](https://openreview.net/forum?id=lSbbC2VyCu&noteId=Hztsf2jtoo))\n\nRS reduces the computational costs of other MORL strategies; eg, as stated in Figure 1.b, \"with only two trainings [RS] reveals the green front of Pareto-optimal solutions [...] and matches the costly yellow front of MORL requiring [11] trainings on different linear weightings\". As a side note, truly revealing the full MORL front would actually require an infinite number of trainings. Thus, we argue that this **efficiency gain is by design**; when considering $N$ rewards, RS only requires $M=N$ fine-tunings, while MORL \"requires explicitly maintaining a large set $M\\gg N$ networks, practically one for each possible preference\" (l.105).\n\nTo quantify the efficiency gain of RS, in Figure 2 (from the one-page rebuttal pdf) we define a new measure of success; the expected reward $E_{\\hat{\\mu}\\sim Unif\\left(0,1\\right)} \\hat{R_{\\hat{\\mu}}}$ where $\\hat{R_{\\hat{\\mu}}} = (1-\\hat{\\mu})\\times R_1 + \\hat{\\mu} \\times R_2$ and the expectation is over all the possible user's linear preferences $\\hat{\\mu}$ over the $N=2$ rewards. Then we compute the difference between (i) the expected reward for RS (always with $2$ training runs), and (ii) the expected reward for MORL with $M$ training runs. **Plotting this expected reward advantage for different values of $M$ confirms that MORL needs $M \\gg 2$ to match RS**. Moreover, because of the dimensional curse, we expect the number of MORL trainings required to match RS to grow exponentially with the number of rewards $N$. In conclusion, these new experiments quantitatively validate that RS is more efficient than MORL, and will be included in the revised paper.\n\n---\n\n### Q4. Number of rewards (extended in [R.ntSF.Q2](https://openreview.net/forum?id=lSbbC2VyCu&noteId=UEM7DRMGYu))\n\nFor visualization clarity, the Pareto fronts were for $N=2$ rewards, one of the $x$-axis, the other on the $y$-axis. Yet, \"RS can scale and trade-off between more rewards\" (l.201). We validate this in the spider maps from Figure 2.f (for text generation), from Figure 3.c  (for image captioning, adapted in Figure 1.c and 1.d in the rebuttal), and from Figure 5.c (for visual grounding), where we respectively consider $4$, $5$ and $3$ different rewards.\n\n---\n\n### Q5. Formulas clarity\n\nYes, both formulas refer to $\\\\{ \\lambda_i \\\\}_{i=1}^N$. Bounds will be explicited in the revision.\n\n---\n\n### Q6. Selecting the $\\lambda$\n\nAs detailed l.163, and latter l. 223, we already **consider two practical strategies to select the values** of the interpolating coefficients $\\lambda$:\n\n1. if the user defines a linear preference $\\hat{\\mu}$, we can select $\\lambda=\\hat{\\mu}$ .\n2. if the user provides some labelled validation samples, we can cross-validate $\\lambda$.\n\nWe validate in Figure 4.a that both strategies perform well. If the user only provides preference comparisons (as suggested by R.QPfR), we could indeed select the $\\lambda$ similarly as in reward modeling in RLHF.\n\n---\n\nWe would greatly appreciate it if you took those clarifications below into account during discussions."}, {"review_id": "UpsGeAE6m8", "reviewer": "Reviewer_bXWy", "summary": "This paper presents reward soups (RS) which is the idea of starting with a pre-trained network, which is finetuned to multiple proxy rewards (say, multiple different criteria), and at test time, infers a reward as a linear combination of these proxy rewards and uses this to linearly combine the corresponding weights which is then used for prediction/generation. In contrast to naive variants of multi-objective RL which trains many different policies (far greater than the number of proxy rewards) to obtain a high fidelity policy for preferences encountered at test time, the proposed approach ends up working while training a number of policies that is equal to the number of proxy rewards while showing reasonable empirical performance.\n\n==> post rebuttal: updated score.", "questions": "Something that appears unclear is that linear mode connectivity etc. typically make sense in the context of supervised learning when training with the same set of labels. Here, the paper attempts to argue a similar perspective with different sets of labels, and in the RLHF context. Attempting to argue about both of these (particularly the latter) is non-trivial sense saying that a linear combination of parameters of a policy network optimizes the long term (i.e. generation level) linear combination of reward posits very strong assumptions on the structure of the optimal policy etc. for which I do not yet see a clear argument in this paper.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The problem formation and proposed approach are topics of increasing interest and relevance to the community. The paper presents interesting results for many practically relevant and useful benchmarks.", "weaknesses": "- There are not much comparisons to approaches in multi-objective RL, which makes it unclear as to how one can imagine this paper's result to be approaching notions of pareto optimal trade-offs. While the paper acknowledges this issue, it leaves open huge gaps as to what can be achieved using a suite of approaches that exist in multi-objective RL (for e.g. even starting from reference 130 cited in this paper).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank R.bXWy for reviewing our work. Yet, with all due respect, there is an inaccuracy in the summary by R.bXWy: \"at test time, [we do **not**] infer a reward as a linear combination of these proxy rewards\". More precisely, we show that interpolated weights can approximate the optimal policy for interpolated rewards, both empirically (in Section 3) and theoretically (in Appendix B.2), as detailed below.\n\n---\n\n### Q1. Towards Pareto-optimality and empirical comparisons with other MORL strategies\n\nAs stated l.292, \"when dealing with multiple objectives in deep learning, the common strategy is to combine them into a single reward\"; in particular, the linear MORL is now standard to train LLMs with RLHF [Glaese2022]. Thus, \"as the true Pareto front is unknown in real-world applications\" (l.177), we use this linear MORL as the reference to evaluate Pareto-optimality. As explained in [R.QPfR.Q1](https://openreview.net/forum?id=lSbbC2VyCu&noteId=58jAIy2tXU), the key reason is that: \"in full generality, improvements in initialization, RL algorithms, data, or specific hyperparameters could enhance performances. [Thus] the Pareto front [...] needs to be defined with regard to a training procedure\" (l.151). Our conclusion was that RS is an empirical solution **towards** Pareto-optimality, with a limitation highlighted in the paper's name.\n\nNow, regarding the other MORL strategies, please note that **they are not practical** for large-scale experiments, as acknowledged by R.stHc who stated: \"compared with previous work, [our approach is] much more applicable and flexible to complex application scenarios\". For example, (i) \"these works are mostly for **academic benchmarks**\" (l.299) or \"games such as ATARI\" (l.890), and (ii) none have been used for RLHF, for fine-tuning foundation models, or for deep networks with billions of parameters.\nCritically, their implementations are complex, as most introduce **specific hyperparameters** or even \"**modify the training procedure**\" (l.300); for example, the reference 130 [Yang2019] requires a change in Bellman equations. In contrast, RS requires zero modification to the optimization algorithm (such as PPO), and thus can be used on top of any RLHF system (such as [TRL](https://github.com/lvwerra/trl)). If R.bXWy is aware of any open-source implementation of any MORL algorithm for RLHF of LLMs, we would run the experiments, report the numbers, but also verify if the LMC holds and **apply RS on top of those refined solutions**.\n\nFinally, **performance and simplicity are not the only advantages of RS over other MORL strategies, as discussed at length in Appendix A.2**. In brief, RS \"is compatible with the inherent iterative engineering process of alignment\" (l.890): \"RS can continually include adjusted opinions while preventing forgetting of the old behaviours\" (l.891). For example, if a new reward is defined, RS requires one single additional training, whereas the other MORL would require starting again from scratch.\n\n[Glaese2022] Improving alignment of dialogue agents via targeted human judgements.\\\n[Yang2019] A generalized algorithm for multi objective reinforcement learning and policy adaptation. NeurIPS.\n\n---\n\n### Q2.  Linear mode connectivity and theoretical guarantees for the near-optimality of RS\n\nOur Hypothesis 1 tries to properly define the LMC when considering multiple metrics: R.TSwH \"found [it] to be very helpful in understanding how RS works\". Its empirical validation was arguably far from obvious; yet, we consistently obtain positive results in Section 3, for various setups and scenarios, even for generation task involving long term dependencies such as text-to-text with LLaMA, or image generation with diffusion models. Then, we state l.322: \"RS relies on an empirical finding: the LMC, which currently lacks full theoretical guarantees [in our complex RL setup with multiple rewards, but actually] even in the simplest case of moving averages [Izmailov2018]\" in supervised learning with one single set of labels.\n\nHowever, we'd like to respectfully emphasize that **we do provide a theoretical and novel \"argument in this paper\"**; in Appendix B.2 \"we provide theoretical guarantees for the near-optimality of RS when considering quadratic rewards\" (l.908). This is referenced in the main paper l.146 in Remark 1 and also l.141-143, where we state: \"we theoretically prove in Appendix B.2 [that our Hypotheses 1 and 2] approximately hold when rewards are replaced by their second-order Taylor expansion with co-diagonalizable Hessians, a simplified setup justifiable when weights remain close\", and a common assumption in deep learning (as argued in Remark 4). Specifically, considering a linear preference $\\hat{\\mu}$ over two rewards $R_1$ and $R_2$, Lemma 3 bounds the difference $\\Delta R_{\\hat{\\mu}}$ between the rewards obtained by (i) the optimal policy and (ii) our interpolated solution by:\n$$ \\Delta R_{\\hat{\\mu}} \\leq \\frac{\\hat{\\mu}^2(1-\\hat{\\mu})^2(M \\Delta_1 - \\Delta_2)(M \\Delta_2 - \\Delta_1)}{\\left(\\hat{\\mu}(1-\\hat{\\mu})(M-1)^2 + M\\right)\\left(\\left( 1 - \\hat{\\mu}\\right) \\Delta_1 + \\hat{\\mu} \\Delta_2\\right)},$$\nwhere $M$ is the maximum of eigenvalues ratio across the Hessians of the two rewards, $\\Delta_1 = R_1(\\theta_1) - R_1(\\theta_2)$ and $\\Delta_2 = R_2(\\theta_2) - R_2(\\theta_1)$. This bound is illustrated in Figure 7.\n\nIn conclusion, we **provide guarantees with assumptions on the rewards** being quadratic and co-diagonalizable, thus with indirect \"assumptions on the structure of the optimal policy\" (R.bXWy). This is acknowledged by R.TSwH who stated: \"the approach is well-motivated theoretically\" and that \"the theory part connects with experiments very well\". This theoretical analysis will be put forward in the revision.\n\n[Izmailov2018] Averaging weights leads to wider optima and better generalization. UAI.\n\n---\n\nIf this clarifies our empirical and theoretical analyses, we would be extremely grateful if R.bXWy could update their review accordingly."}, {"review_id": "2JGH9VsV26", "reviewer": "Reviewer_TSwH", "summary": "The paper proposes rewarded soup (RS), a simple technique for combining policies trained for different rewards into a single policy performing well on a particular convex combination of those rewards. The technique consists in linearly interpolating weights of individual policies, using the fact that they share the same initialisation and remain linearly connected during finetuning on different rewards. The approach is well-motivated theoretically and thoroughly evaluated on a diverse array of tasks ranging from language generation, image captioning and image generation to robot locomotion. ", "questions": "1. What’s the relation between scatter points and curves on the plots: are curves a smoothed based on scatter points? What smoothing technique did you use?\n2. How much does the performance of RS depend on policies being close to their shared base models? How does the RS front evolve over the course of finetuning; does it degenerate after some number of gradient updates?\n3. Relatedly, does using parameter-efficient finetuning (e.g. LoRA) play a role? In the paper, we claim you use LoRA only for computational efficiency reasons, but shouldn’t it also significantly help to maintain linear mode connectivity? Does RS work without LoRA?", "limitations": "Limitations and societal impacts are discussed thoroughly. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper addresses crucial problems of accounting for diverse preferences and adapting to changes in reward specification that frequently arise in the emerging and important field of aligning generative models with human preferences.\n2. Rewarded soup is well-motivated theoretically as a Pareto coverage set of policies for linear combinations of individual reward functions. I found working hypotheses 1 and 2 to be very helpful in understanding how RS works. I’m also convinced by empirical evidence for these hypotheses being true.\n3. The fact that linear mode connectivity also holds for RL policies trained for different rewards is an interesting finding about deep learning overall. I found it somewhat surprising that interpolated weights outperform the interpolated rewards so consistently.\n4. The paper is well-written and easy to follow despite being very dense. The theory part connects with experiments very well. I also appreciate the plots (e.g. Figure 2) being easy to navigate while conveying a lot of information.\n5. The experiments are very thorough and diverse and I find them compelling.", "weaknesses": "I think the discussion of reward misspecification could be more nuanced. I think the claims that RS “mitigates reward misspecification” (line 75) and “If Hypothesis 2 is true, then RS can mitigate reward misspecification” (line 161) should be framed a bit more cautiously, making it clear that it’s a very particular kind of reward misspecification: when the real reward is linear in a set of proxy rewards. I don’t think this is representative of most kinds of reward misspecification that we see and that we should me worried about such as context-dependence of human preferences or biases of data workers providing feedbacks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank R.TSwH for this positive review and the great understanding of the empirical and theoretical components of our work.\n\n---\n\n### Q1. Reward misspecification\n\nIn the revised version of the paper, we will clarify the discussion l.75 and l.161 on reward misspecification being mostly for linear rewards. Yet, please note that in Figure 4b (and in Figure 9 from Appendix D.2) we actually observe that \"despite the lack of theoretical guarantees, weight interpolation improves results **even for non-linear reward**\" (l.167). We speculate RS actually maximizes the projection of the user's reward on the linear subspaces defined by the different proxy rewards.\n\n---\n\n### Q2. Smoothing functions in plots\n\nThe curves fit the points with a **Savitzky-Golay smoothing filter** (inspired from this [blog](https://www.datatechnotes.com/2022/05/smoothing-example-with-savitzky-golay.html)) and a **quadratic interpolation** (inspired from this [stack overflow](https://stackoverflow.com/questions/52014197/how-to-interpolate-a-2d-curve-in-python)). The code is detailed below.\n\n```python\nimport numpy as np\nfrom scipy.interpolate import interp1d\nfrom scipy.signal import savgol_filter\n\ndef smoothing(x, y):\n    x_smooth = savgol_filter(x, 3, 1)\n    x_smooth[0], x_smooth[-1] = x[0], x[-1]\n    y_smooth = savgol_filter(y, 3, 1)\n    y_smooth[0], y_smooth[-1] = y[0], y[-1]\n\n    points = np.array([x_smooth, y_smooth]).T\n    distance = np.cumsum(np.sqrt(np.sum(np.diff(points, axis=0)**2, axis=1)))\n    distance = np.insert(distance, 0, 0) / distance[-1]\n    alpha = np.linspace(0, 1, 75)\n    interpolator = interp1d(distance, points, kind=\"quadratic\", axis=0)\n    curve = interpolator(alpha)\n    return curve.T\n```\n\n---\n\n### Q3. How much does the performance of RS depend on policies being close to their shared base models? How does the RS front evolve over the course of finetuning; does it degenerate after some number of gradient updates? (see [R.ntSF.Q1](https://openreview.net/forum?id=lSbbC2VyCu&noteId=UEM7DRMGYu))\n\nAs detailed in Remark 1, \"when the weights remain close, we can theoretically justify Hypotheses 1 and 2 (see Appendix B.2), and, more broadly, demonstrate that WI approximates ensembling (see Lemma 4 [in Appendix B.3])\" (l.146). In other words, good performances are guaranteed when weights are close; thus longer trainings may be worrisome, as the models may potentially diverge in the weight space.\nWe investigate this question in the one-page rebuttal pdf, for the news summarization task (in Figure 3.a) and for the captioning task (in Figure 3.b); we double the number of training steps, and report multiple RS fronts over the course of fine-tuning.\nFortunately, we **consistently observe good performances for RS along the full fine-tuning**, confirming that the pre-trained initialization is sufficient to enforce the LMC, and validating the insights from previous works [Neyshabur2020] in supervised learning.\n\n[Neyshabur2020] What is being transferred in transfer learning? NeurIPS.\n\n---\n\n### Q4. Does RS work without LoRA?\n\nActually, **most of our experiments are without LoRA**.\n\n- for the image generation task, we fine-tune 10% of the weights of the diffusion model, \"corresponding to the cross-attention layers and the bias/scaling parameters\" (l.1047).\n- for the visual grounding task, we fine-tune the transformer end-to-end.\n- for the locomotion task, we fine-tune the MLP end-to-end.\n- for the captioning task, we usually fine-tune the text decoder with the convolutional visual encoder frozen, but we show in Figure 10.d that RS convexity is actually even better when training end-to-end.\n\nTherefore, we argue that RS is agnostic to the \"parameterization\" strategy in training.\n\nAs a final note, [Li2022] have observed in NLP that weight interpolation works even better in larger architectures. Intuitively, a large number of parameters may facilitate the orthogonality of the fine-tuned updates observed in [Ilharco2023], which \"speculate that this [orthogonality] enables the combination of task vectors via addition with minimal interference\". This fact may explain why end-to-end fine-tuning in captioning provides better convexity in Figure 10.d than when keeping the visual encoder frozen in Figure 3.a. Moreover, this insight suggests that, as LoRA reduces the number of trainable weights, **performances might actually get better with full end-to-end fine-tuning than with LoRA** (as currently done in our text-to-text experiments). This is a promising research direction for future work.\n\n[Li2022] Branch-Train-Merge: Embarrassingly parallel training of expert language models.\\\n[Ilharco2023] Editing models with task arithmetic. ICLR.\n\n---\nThank you once more for your feedback. We remain open to further suggestions and discussions."}]

Author rebuttal:
We sincerely thank the reviewers for their time and their insightful feedbacks. We're encouraged by the positive comments, which highlight the main features of our submission.

- (*topic*) We "address the reward misspecification problem [...] in current RLHF frameworks" (R.QPfR), a problem "that frequently arise in the emerging and important field of aligning generative models with human preferences" (R.TSwH). This "topic [is] of increasing interest and relevance to the community" (R.bXWy).
- (*methodology*) We propose rewarded soup (RS) which "involves individually training multiple networks, each assigned to a different proxy reward, and then linearly combining these networks" (R.ntSF). "The proposed idea is effective yet efficient as it does not require additional training" (R.DNE5) contrary to "to the more costly baselines" (R.bJvT) such as MORL.
- (*experiments*) Empirically, we "did a lot of experiments on different task which shows that this interpolating strategy is universal under different application scenarios, while with good performance" (R.stHc). "The paper presents interesting results for many practically relevant and useful benchmarks" (R.bXWy).
- (*theory*) "The approach is well-motivated theoretically" (R.TSwH) and "the theory part connects with experiments very well" (R.TSwH).

We have taken note of the questions and suggested weaknesses, that we directly answer in response to each reviewer.
Most of our answers are based on quotes from the main paper or the Appendix, that the reviewers might have overlooked (in particular the theoretical Appendix B.2 referenced in Section 2.2.2). In contrast, a few questions required new plots to be answered, that we gather in the one-page rebuttal pdf. Specifically:

- Table 1 shows generated samples by our interpolated models for the summarization task. This qualitative inspection is enriched by quantitative evaluations in Figure 1.a and 1.b, with general-purpose quality metrics such as perplexity for summaries and FID for image generations. We validate that the generated samples from our method do not suffer from reduced quality ([R.stHc.Q3](https://openreview.net/forum?id=lSbbC2VyCu&noteId=GlQnTpp2gl)).
- Figure 1.c and 1.d show that performances get better (Pareto-optimally) when increasing the number of averaged weights ([R.ntSF.Q2](https://openreview.net/forum?id=lSbbC2VyCu&noteId=UEM7DRMGYu)) and thus the number of training rewards ([R.QPfR.Q4](https://openreview.net/forum?id=lSbbC2VyCu&noteId=58jAIy2tXU)).
- Figure 2 quantifies the average efficiency gain from RS with regard to the MORL baseline ([R.bJvT.Q3](https://openreview.net/forum?id=lSbbC2VyCu&noteId=Hztsf2jtoo) and [R.QPfR.Q3](https://openreview.net/forum?id=lSbbC2VyCu&noteId=58jAIy2tXU)).
- Figure 3.a and 3.b plot RS's fronts over the course of fine-tuning, and validates that the LMC holds even for longer trainings ([R.TSwH.Q3](https://openreview.net/forum?id=lSbbC2VyCu&noteId=np0m5ACZtx)).
- Figure 3.c illustrates the empirical difference and the complementarity of rewarded soups and model soups ([R.DNE5.Q1](https://openreview.net/forum?id=lSbbC2VyCu&noteId=6LMJAJD6vx)).

We hope our responses clarify the expressed concerns. If there is anything else we can do to further improve our work, please let us know.

Meta-review: This paper is about rewarded soup yielding Pareto optimal solutions through weight interpolation after training. As raised by many reviewers, I suggest the authors to clarify the novelty with respect to model soup and with respect to MORL in the final version.

I encourage the authors to incorporate the share concerns among the reviewers: (1) quantitative computational advantage over MORL - move figure 2 in the one page rebuttal to main paper. (2) clarify the RS targets a particular kind of reward misspecification (linear). (3) spider plots for the number of networks averaged (N>2) - move figure 1 (c,d) in the one page rebuttal to main paper.

Overall, this paper deserves acceptance at NeurIPS as a poster paper.
Decision: Accept (poster)