Rewarded soups: towards Pareto-optimal alignment
by interpolating weights fine-tuned on diverse rewards

Alexandre Ramé∗ 1 Guillaume Couairon† 1 2 Corentin Dancette† 1 Jean-Baptiste Gaya† 1 2 Mustafa Shukor† 1
Laure Soulier 1 Matthieu Cord 1 3

Abstract

Yosinski et al., 2014) via supervised learning (Vapnik, 1999).
Yet, collecting labels is expensive, and thus supervision may
not cover all possibilities and fail to perfectly align (Amodei
et al., 2016; Taylor et al., 2016; Ngo, 2022) the trained
network with the intended applications. Recent works (Stiennon et al., 2020; Ouyang et al., 2022; Pinto et al., 2023)
showed that deep reinforcement learning (DRL) helps bylearning from various types of rewards. A prominent example is reinforcement learning from human feedback (RLHF)(Stiennon et al., 2020; Christiano et al., 2017; Ziegler et al.,2019; Wu et al., 2021), which appears as the current go-to
strategy to refine large language models (LLMs) into conversational agents such as ChatGPT (Ouyang et al., 2022;
OpenAI, 2023). After pre-training on next token prediction(Radford et al., 2018), the LLMs are fine-tuned to follow
instructions (Wei et al., 2022; Wang et al., 2022c; Taoriet al., 2023) before reward maximization. This RL strategy enhances alignment by evaluating the entire generated
sentence instead of each token independently, handling thediversity of correct answers and allowing for negative feedback (Goldberg, 2023). Similar strategies have been usefulin computer vision (CV) (Pinto et al., 2023; Rennie et al.,2017), for example to integrate human aesthetics into image
generation (Lee et al., 2023; Wu et al., 2023b).

Foundation models are first pre-trained on vast
unsupervised datasets and then fine-tuned on labeled data. Reinforcement learning, notably fromhuman feedback (RLHF), can further align the
network with the intended usage. Yet the imperfections in the proxy reward may hinder the
training and lead to suboptimal results; the diversity of objectives in real-world tasks and humanopinions exacerbate the issue. This paper proposes embracing the heterogeneity of diverse rewards by following a multi-policy strategy. Rather
than focusing on a single a priori reward, we aim
for Pareto-optimal generalization across the entire space of preferences. To this end, we propose rewarded soup, first specializing multiple
networks independently (one for each proxy reward) and then interpolating their weights linearly. This succeeds empirically because we showthat the weights remain linearly connected whenfine-tuned on diverse rewards from a shared pretrained initialization. We demonstrate the effectiveness of our approach for text-to-text (summarization, helpful assistant), text-image (image captioning, text-to-image generation, visual grounding), and control (locomotion) tasks. We hope toenhance the alignment of deep models, and how
they interact with the world in all its diversity.

Diversity of proxy rewards. RL is usually seen as morechallenging than supervised training (Dulac-Arnold et al.,2021), notably because the real reward—ideally reflecting
the users’ preferences—is often not specified at training
time. Proxy rewards are therefore developed to guide thelearning, either as hand-engineered metrics (Papineni et al.,
2002; Lin & Hovy, 2003; Vedantam et al., 2015) or more
recently in RLHF as models trained to reflect human preferences (Christiano et al., 2017; Kwon et al., 2023; Xuet al., 2023). Nonetheless, designing reliable proxy rewards for evaluation is difficult. This reward misspecification (Amodei et al., 2016; Pan et al., 2022) between theproxy reward and the users’ actual rewards can lead to unforeseen consequences (Michaud et al., 2020). Moreover,the diversity of objectives in real-world applications complicates the challenge. In particular, human opinions canvary significantly (Wildavsky, 1987; Coello, 2000; Schwartz
et al., 2012) on subjects such as aesthetics (Nadal & Chat-

1. Introduction
Foundation models (Bommasani et al., 2021) have emerged
as the standard paradigm to learn neural networks’ weights.
They are typically first pre-trained through self-supervision(Devlin et al., 2019; Brown et al., 2020; Caron et al., 2021;
Radford et al., 2021) and then fine-tuned (Oquab et al., 2014;*
Project lead, main contributor,† Equal experimental contribution 1 Sorbonne Université, CNRS, ISIR, Paris, France 2 Meta
AI 3 Valeo.ai. Correspondence to: Alexandre Ramé <alexandre.rame@isir.upmc.fr>.

ILHF Workshop ICML 2023.

1

In this paper, we propose rewarded soup (RS), an efficientand flexible multi-policy strategy to fine-tune any foundation model. As shown in Section 1, we first use RL tolearn one network for each proxy reward; then, we combine these expert networks according to user preferences.
This a posteriori selection allows for better-informed tradeoffs, improved transparency and increased fairness (Hayes
et al., 2022; Mannion et al., 2021). The method to combine
those networks is our main contribution: we do this throughlinear interpolation in the weight space, despite the nonlinearities in the network. This is in line with recent findings
on linear mode connectivity (LMC) (Frankle et al., 2020;
Neyshabur et al., 2020): weights fine-tuned from a sharedpre-trained initialization remain linearly connected and thus
can be interpolated. This LMC inspired a plethora of weightinterpolation (WI) strategies (Wortsman et al., 2022a; Raméet al., 2022; Matena & Raffel, 2022; Ilharco et al., 2022;
Don-Yehiya et al., 2022; Ramé et al., 2023), discussed in
Section 4. Actually, the name rewarded soups follows theterminology of model soups (Wortsman et al., 2022a), as we
combine various ingredients each rewarded differently. Unlike previous works, which focused on supervised learning,
we explore LMC in RL, in a challenging setup where eachtraining run uses a different reward. Perhaps surprisingly,
we show that we can trade off the capabilities of multiple
weights in a single final model, thus without any computational overhead. This enables the creation of custom weights
for any preference over the diverse rewards.

RL fine-tuningof proxy reward

Unsupervised
pre-training
on large dataset

Supervised
fine-tuning on
target task

Rewarded soups
by weight interpolation

RL fine-tuningof proxy reward

Selection
selected by theusers according to theirpreferences

Figure 1: We detail the steps in rewarded soup (RS). After unsupervised pre-training and supervised fine-tuning, we launch N
independent RL fine-tunings on the proxy rewards {Ri }N
i=1 . Then
we combine the trained networks by interpolation in the weightspace. The final weights are adapted at test time by selecting λ.

terjee, 2019), politics or fairness (Lopez-Paz et al., 2022).
Humans have also different expectations from machines:
for example, while (Bai et al., 2022a) stressed aligning
LLMs towards helpful, honest, and harmless (Askell et al.,2021) feedback, others’ interests are to make LLMs mostly
engaging and enjoyable (Irvine et al., 2023). Even handengineered metrics can be in tension: generating shorter
descriptions with higher precision can increase the BLEU
(Papineni et al., 2002) score but decrease the ROUGE (Lin& Hovy, 2003) score due to reduced recall.

• We propose a new practical strategy named rewarded
soup for fine-tuning foundation models with diverse
rewards. It defines a continuous set of (close to) Paretooptimal solutions by weight interpolation, approximating more costly multi-policy strategies.

Towards multi-policy strategies. Considering these challenges, it may not be feasible to develop a single model simultaneously aligned with everyone’s preferences (Ouyang
et al., 2022). Current strategies tend to align towards a
consensus-based user (Bakker et al., 2022; Ovadya, 2023),
inherently prioritizing certain values over others, potentiallyresulting in unfair representations of marginalized groups(Kirk et al., 2023). Moreover, these trade-offs (Pan et al.,2023) are decided a priori before training, shifting the responsibility to the engineers and reducing transparency andexplainability (Hayes et al., 2022). These limitations, further
discussed in Appendix A.2.1, highlight a key limitation of
single-policy alignment strategies; their inability to handle
the diversity of human preferences. Yet, “human-aligned artificial intelligence is a multi-objective problem” (Vamplew
et al., 2018). Thus, we draw inspiration from the multiobjective reinforcement learning (MORL) literature (Barrett& Narayanan, 2008; Li et al., 2020; Tanaka & Yamamura,
2003; Van Moffaert & Nowé, 2014; Roijers et al., 2013;Rădulescu et al., 2020) and notably (Hayes et al., 2022)
arguing that tackling diverse rewards requires shifting fromsingle-policy to multi-policy approaches. As optimality
depends on the relative preferences across those rewards,
the goal is not to learn a single network but rather a set of
Pareto-optimal networks (Pareto, 1964).

• We analyze the linear mode connectivity between
weights fine-tuned on diverse rewards.
• We validate that RS mitigates reward misspecification.
In Section 3, we demonstrate the consistent effectiveness ofrewarded soup across a variety of tasks: RLHF fine-tuningof LLaMA, multimodal tasks such as image captioning,text-to-image generation with diffusion models or visualgrounding, as well as locomotion tasks. More results are on
our anonymized website.

2. Rewarded soups
2.1. RL fine-tuning with diverse rewards
We consider a deep neural network f of a fixed nonlinear architecture (e.g., with batch normalization (Ioffe& Szegedy, 2015), ReLU layers (Agarap, 2018) or selfattention (Vaswani et al., 2017)). It defines a policy by2mapping inputs x to f (x, θ) when parametrized by θ. Fora reward R̂ and a test
R distribution T of deployment, ourgoal is to maximize x∈T R̂(f (x, θ)). Learning θ is now
commonly a three-step process: unsupervised pre-training,supervised fine-tuning, and reward optimization. Yet R̂ is
usually not specified, meaning we can only optimize a proxyreward R during training. This reward misspecification
between R and R̂ may hinder the alignment of the networkwith R̂. Moreover, the diversity of human preferencescomplicates the design of R.

2. For the selection phase, we linearly interpolate thoseweights to definePa continuous set of rewardedNsoups policies: { i=1 λi · θi }{λi }N. Practii=1 ∈∆Ncally, we uniformly sample M interpolating coeffiM
cients {{λji }Ni=1 }j=1 from the N -simplex ∆N and select the j-th that maximizes the user’s
reward R̂ onPNjM
validation samples, i.e., argmaxj=1 R̂
i=1 λi θi .
3. For the inference phase,
PN we predict using the networkf parameterized by i=1 λji θi .

Rather than optimizing one single proxy reward, our paper’s first key idea is to consider a family of N diverse
proxy rewards {Ri }N
i=1 . Each of these rewards evaluates
the prediction according to different (potentially conflicting)criteria. The goal then becomes obtaining a coverage set ofpolicies that trade-off between these rewards. To this end,
we first introduce the costly MORL baseline. Its inefficiency
motivates our rewarded soups, which leverages our second
key idea: weight interpolation.

While MORL interpolates the rewards, RS interpolates
the weights. This is a considerable advantage as the appropriate weighting λ, which depends on the desired trade-off,can be selected a posteriori; the selection is achieved without additional training, only via inference on some samples.
In the next Section 2.2 we explicitly state the Hypotheses 1
and 2 underlying in RS. These are considered Working Hypotheses as they enabled the development of our RS strategy.
Their empirical verification will be the main motivation forour experiments on various tasks in Section 3.

MORL baseline. The standard MORL scalarization strategy (Barrett & Narayanan, 2008; Li et al., 2020) linearizes
the problem by interpolating the proxy rewards using M
different weightings. Specifically, during the training phase,
M trainings
PN are launched, with the j-th optimizing the reward i=1 µji Ri , where ∀j ∈ {1, ..., M }, {µji }N
i=1 ∈ ∆NPN jj
the N -simplex s.t. i=1 µi = 1 and 0 ≤ µi ≤ 1. Then,
during the selection phase, the user’s reward R̂ becomes
known and the j-th policy that maximizes R̂ on some validation dataset
PN is selected. We typically expect to select jsuch that i=1 µji Ri ≈ R̂ linearly approximates the user’s
reward. Finally, this j-th weight is used during the inference
phase on test samples. Yet, a critical issue is that “minor[preference] variations may result in significant changes inthe solution” (Vamplew et al., 2008). Thus, a high level of
granularity in the mesh of ∆N is necessary. This requires
explicitly maintaining a large set of M ≫ N networks, practically one for each possible preference. Ultimately, this
MORL strategy is unscalable in deep learning due to thecomputational, memory, and engineering costs involved(see further discussion in Appendix A.2.2).

2.2. Exploring the properties of the rewarded soups setof solutions
2.2.1. L INEAR MODE CONNECTIVITY OF WEIGHTSFINE - TUNED ON DIVERSE REWARDS
N
We consider {θi }N
i=1 fine-tuned on {Ri }i=1 from a sharedpre-trained initialization. Previous works (Frankle et al.,2020; Neyshabur et al., 2020; Wortsman et al., 2022a; Raméet al., 2023) defined linear mode connectivity (LMC) w.r.t.
a single performance measure (e.g., accuracy or loss) insupervised learning. We extend this notion in RL with N
rewards, and define that the LMC holds if all rewards forthe interpolated weights exceed the interpolated rewards.
It follows that the LMC condition which underpins RS’s
viability is the Hypothesis 1 below.

Working Hypothesis
P1 (LMC). For
P all {λi }i ∈ ∆N andk ∈ {1, ..., N }, Rk ( i λi · θi ) ≥ i λi Rk (θi ).
2.2.2. PARETO OPTIMALITY OF REWARDED SOUPS

Rewarded soup (RS). In this paper, we draw inspirationfrom the weight interpolation literature. The idea is to learn
expert weights and interpolate them linearly to combine
their abilities. Specifically, we propose RS, illustrated in
Section 1 and whose recipe is described below. RS alleviates
MORL’s scaling issue as it requires only M = N trainings
while being flexible and transparent.

The Pareto front (PF) is the set of undominated weights,
for which no other weights can improve a reward withoutNsacrificing another, i.e., {θ | ∄θ′ ∈ Θ s.t. {Ri (θ′ )}i=1 >NN{Ri (θ)}i=1 } where >N is the dominance relation in RN .
In practice, we only need to retain one policy for each possible value vector, i.e., a Pareto coverage set (PCS). We now
introduce the key Hypothesis 2.
Working
Hypothesis 2 (Pareto optimality). The setP{ i λi · θi |{λi }i ∈ ∆N } is a PCS of {Ri }i .

1. During the training phase, we optimize a set of N
expert weights {θi }Ni=1 , each corresponding to one of
the N proxy rewards {Ri }N
i=1 , and all from a sharedpre-trained initialization.

Hypothesis 2 holds if the rewarded soups solutions, uncovered by interpolation, are Pareto-optimal. Overall, we

33. Experiments

empirically validate Hypotheses 1 and 2 in Section 3, yetalso report a few limitations in Appendix and research directions to fix them. Moreover, we theoretically prove inAppendix B.2 they approximately hold when rewards are
replaced by their second-order Taylor expansion with codiagonalizable Hessians, a simplified setup justifiable whenweights remain close.

In this section we implement RS across a variety of standard
learning tasks: text-to-text generation, image captioning,
image generation, visual grounding, and locomotion. We
use either model or statistical rewards. We follow a systematic procedure. First, we independently optimize diverse
rewards on training samples. For all tasks, we employ thedefault architecture, hyperparameters and RL algorithm; theonly variation being the reward used across runs. Second,
we evaluate the rewards on the test samples: the results
are visually represented in series of plots. Third, we verify
Hypothesis 1 by examining whether RS’s rewards exceed
the interpolated rewards. Lastly, as the true Pareto front is
unknown in real-world applications, we present empiricalsupport for Hypothesis 2 by comparing the front defined by
RS (sliding λ between 0 and 1) to the MORL’s solutions optimizing the µ-weighted rewards for 0 ≤ µ ≤ 1 (sometimes
only µ = 0.5 for computational reasons).

Remark 1. Hypotheses 1 and 2 rely on a good pre-trained
initialization, making RS particularly well-suited to finetune foundation models. This is because pre-training prevents the weights from diverging during training (Neyshabur
et al., 2020). When the weights remain close, we can theoretically justify Hypotheses 1 and 2 (see Appendix B.2) and,more broadly, demonstrate that WI approximates ensembling (Hansen & Salamon, 1990; Lakshminarayanan et al.,2017) (see Lemma 4). In contrast, the LMC does not holdwhen training from scratch (Neyshabur et al., 2020).
Remark 2. Pareto-optimality in Hypothesis 2 is defined
w.r.t. a set of weights Θ. Yet, in full generality, improvements
in initialization, RL algorithms, data, or specific hyperparameters could enhance performances. In other words, forreal-world applications, the true PF is unknown and needs
to be defined w.r.t. a training procedure. In this case, Θ represents the set of weights attainable by fine-tuning within ashared procedure. As such, in Section 3 we analyze Hypothesis 2 by comparing the fronts obtained by RS and scalarized
MORL while keeping everything else constant.

3.1. Text-to-text: LLaMA with diverse RLHFs
Given the significance of RLHF to train LLMs, we begin
our experiments with text-to-text generation tasks. Our
pre-trained network is LLaMA-7b (Touvron et al., 2023),
instruction fine-tuned (Wei et al., 2022; Wang et al., 2022b)
on Alpaca (Taori et al., 2023). For RL training with PPO(Schulman et al., 2017), we employ the trl package (vonWerra et al., 2020) and the setup from (Beeching et al.,2023) with low-rank adapters (LoRA) (Hu et al., 2022a) for
efficiency. We consider the following tasks: summarization (Stiennon et al., 2020; Wu et al., 2021) on two datasets(Reuter news (Ahmed, 2017) in Figure 2(a) and Reddit posts(Völske et al., 2017) in Figure 2(b)), and helpfulness as aconversational assistant (Bai et al., 2022a) in Figures 2(c)and 2(d). To evaluate the generation in the absence of supervision, we utilized N = 2 different reward models (RMs)for each task, except in Figure 2(d) where N = 4. TheseRMs were trained on human preferences datasets (Christiano et al., 2017) and all open-sourced on HuggingFace(Wolf et al., 2020). For example in summarization, R1
follows the “Summarize from Human Feedback” paper (Stiennon et al., 2020), while R2 leverages “contrast candidate
generation” (Chen et al., 2021). For the assistant task, werely on diverse RMs from OpenAssistant (Köpf et al., 2023);
they differ by their training procedures.

2.2.3. C ONSEQUENCES OF PARETO OPTIMALITY IF THE
USER ’ S REWARD IS LINEAR IN THE PROXYREWARDS

Lemma 1 (Reduced reward misspecification in the linearcase). If Hypothesis 2 holds, and for linear reward R̂ =P
µ̂ R with {µ̂i }i ∈ ∆N , then ∃{λi }i ∈ ∆N such that
Pi i ii λi · θi is optimal for R̂.

The proof outlined in Appendix B.1 directly follows thedefinition of Pareto optimality. In simpler terms, Lemma 1
implies that if Hypothesis 2 is true, then RS can mitigate
reward misspecification. For any preference µ̂, there exists a
λ such that the λ-interpolation over weights maximizes theµ̂-interpolation over rewards. In practice, as we will see in
Figure 4(a), we can set λ = µ̂, or cross-validate λ on othersamples. Yet, this theoretically holds only for R̂ linear overthe proxy rewards. This follows the linear utility functions
setup from the MORL literature (Rădulescu et al., 2020),
whose limitations (Vamplew et al., 2008) are discussed inAppendix A.1. This motivates having sufficiently rich anddiverse proxy rewards to capture the essential aspects ofall possible users’ rewards. Despite the lack of theoreticalguarantees, we will show in Figures 4(b) and 9 that weightinterpolation improves results even for non-linear R̂.

The results are reported in Figure 2. The green front, defined by RS between the two weights specialized on R1 andR2 , is above the straight line connecting those two points,
validating Hypothesis 1. Second, the front passes throughthe point obtained by MORL fine-tuning on the averageof the two rewards, supporting Hypothesis 2. Moreover,when comparing both full fronts, they have qualitatively
the same shape; quantitatively in hypervolume (Yen & He,
4

λ = 0.7

R2 rewarded(µ = λ = 1.0)
λ = 0.9−1.0

0.4

λ = 0.6
λ = 0.5

λ = 0.8

0.2

λ = 0.4

R2
−1.4

−1.6

MORL front
RS front

−1.8
0.4

0.6

λ = 0.3−0.2

λ = 0.2
λ = 0.1−0.4

LLaMA init
0.8

0.0

R2

MORL: µ = 0.1
MORL: µ = 0.2
MORL: µ = 0.3
MORL: µ = 0.4
MORL: µ = 0.5
MORL: µ = 0.6
MORL: µ = 0.7
MORL: µ = 0.8
MORL: µ = 0.9
RL R2 : µ = 1.0

−1.2

1.0

R1

1.2

R1 rewarded(µ = λ = 0.0)
1.4(a) News summary.

1.6

1.8

−0.6
−0.2

−1.50

R2 rewarded(µ = λ = 1.0)

MORL: µ = 0.1
MORL: µ = 0.2
MORL: µ = 0.3
MORL: µ = 0.4
MORL: µ = 0.5
MORL: µ = 0.6
MORL: µ = 0.7
MORL: µ = 0.8
MORL: µ = 0.9
RL R2 : µ = 1.0

0.2-2.451

λ = 0.5−2.00

λ = 0.3
λ = 0.2
LLaMA init

-2.626−2.25

RL: R1
RL: R2−2.50

λ = 0.1
R1 rewarded(µ = λ = 0.0)

RS: (1 − λ) · θ1 + λ · θ2

−3.00

0.4

R1

0.6

0.8

1.0

−3.25

1.2

(b) Reddit summary.

-1.063

LLaMA init
−2.9

−2.8

−2.7

-1.736-2.409-2.604

R4

2
MORL: R1+R
2

−2.75

MORL front
RS front
0.0-2.276

R1 +R2
rewarded2

R2 rewarded−1.75

λ = 0.4

RL R1 : µ = 0.0

RL R1 : µ = 0.0

R1

λ = 0.9λ = 0.8λ = 0.7λ = 0.6
λ = 0.5

R2

−0.8

−2.6

R1

−2.5

R1 rewarded−2.4(c) Helpful assistant.

−2.3

−2.2

-2.079-1.555

R2
LLaMA init
RL: R1
RL: R2
RL: R3
RL: R4P =4
MORL: Ni=1 Ri /4
PN =4
RS: i=1
θi/4-0.937-0.712-0.487R3(d) Helpful assistant: spider map.

Figure 2: RLHF results in NLP with LLaMA-7b (Touvron et al., 2023) and reward models Ri from HuggingFace (Wolf et al., 2020). The
blue line reports checkpoints’ results along the training trajectory of θ1 rewarding R1 , the red line θ2 rewarding R2 , and the purple line2
the MORL rewarding R1 +R. Our rewarded soup (RS) linearly interpolates between the weights θ1 and θ2 ; sliding the interpolation2
coefficient λ from 0 to 1 reveals the green solid front of rewarded soups solutions. In Figures 2(a) and 2(b), we additionally show themultiple MORL runs rewarding (1 − µ) × R1 + µ × R2 with preferences 0 ≤ µ ≤ 1. It reveals a similar yellow front, yet more costly.
In Figure 2(d), we uniformly (λi = 14 ) average the weights fine-tuned for the assistant task on N = 4 reward models.

2013) (lower is better, the area over the curve w.r.t. an optimal point), RS’s hypervolume is 0.367 vs. 0.340 for MORLin Figure 2(a), while it is 1.176 vs. 1.186 in Figure 2(b).
Finally, in Figure 2(d), we use N = 4 RMs for the assistanttask and uniformly average the N = 4 weights, confirmingthat RS can scale and trade-off between more rewards.

quantitatively the same hypervolume (Yen & He, 2013) of
0.140. One of the strengths of RS is its ability to scale
to any number of rewards. In Figure 3(c), we uniformly(λi = 15 ) average N = 5 weights fine-tuned independently.
It improves upon the initialization (Hu et al., 2022b) andcurrent state-of-the-art on all metrics, except for CIDEr, on
which (Hu et al., 2022b) was explicitly optimized.

3.2. Image-to-text: captioning with statistical rewards

Figure 4 refines our analysis of RS. In Figures 4(a) and 4(b),rewards are normalized to 1 for the initialization and 0 for
the worst model. Figure 4(a) validates Lemma 1: for any
linear preference µ̂ over the proxy rewards, there exists an
optimal solution in the set described by RS. Two empiricalstrategies to set the value of λ are close to optimal: selecting λ = µ̂ if µ̂ is known, or cross-validating (CV) λ if
a different data split (Karpathy & Fei-Fei, 2015) is available. Moreover, Figure 4(b) (and Figure 9 in Appendix D)
investigate all metrics as evaluation. Excluding results’ variance, we observe monotonicity in both training rewards,
linear in BLEU1 and quadratic in ROUGE. For other evaluation rewards that cannot be linearly expressed over the
training rewards, the curves’ concavity shows that RS consistently improves the endpoints, thereby mitigating reward
misspecification. The optimal λ depends on the similarity between the evaluation and training rewards: e.g., best
BLEU2 are with small λ. Lastly, as per (Izmailov et al.,2018) and Lemma 4, Figure 4(c) suggests that RS succeeds
because WI approximates deep ensembling (Hansen & Salamon, 1990; Lakshminarayanan et al., 2017), interpolating
the predictions rather than the weights. Actually, ensembling performs better, but it cannot be fairly compared as its
inference cost is doubled.

RL training is also effective for multimodal tasks (Pinto
et al., 2023), for example in image captioning (Rennie et al.,2017) where the task is to generate textual descriptions ofimages. Precisely evaluating the quality of a prediction
w.r.t. a set of human-written captions is a challenging task,
thus the literature relies on various hand-engineered, nondifferentiable metrics: e.g., the precision-focused BLEU
(Papineni et al., 2002), the recall-focused ROUGE (Lin &Hovy, 2003), METEOR (Banerjee & Lavie, 2005) handlingsynonyms and CIDEr (Vedantam et al., 2015) using TF-IDF.
As these metrics are proxies for human preferences, goodtrade-offs are desirable. We conduct our experiments on
COCO (Lin et al., 2014), with an ExpansionNetv2 (Hu et al.,2022b) network and a Swin Transformer (Liu et al., 2022)
visual encoder, initialized from the state-of-the-art weightsof (Hu et al., 2022b) optimized on CIDEr. We then utilize
the code of (Hu et al., 2022b) and their self-critical (Rennie
et al., 2017) procedure (a variant of REINFORCE (Williams,
1992)) to reward the network on BLEU1, BLEU4, ROUGE
or METEOR. More details are in Appendix D.
We observe in Figure 3 that tuning solely BLEU1 sacrifices some points on ROUGE or BLEU4. Yet interpolating
between θ1 and θ2 uncovers a convex set of solutions approximating the ones obtained through scalarization of the
rewards in MORL. When comparing both full fronts in
Figure 3(a), they qualitatively have the same shape, and5

0.611µ=λ=1λ = 0.9

λ = 0.8

λ = 0.7 λ = 0.6

BLEU 1BLEU 1+BLEU 4
rewarded2

0.424

λ = 0.5

0.85

0.610
0.422
0.828

λ = 0.4
RL BLEU 1: µ = 0
MORL: µ = 0.1
MORL: µ = 0.2
MORL: µ = 0.3
MORL: µ = 0.4
MORL: µ = 0.5
MORL: µ = 0.6
MORL: µ = 0.7
MORL: µ = 0.8
MORL: µ = 0.9
RL ROU GE: µ = 1
MORL front
RS front

0.608

0.607

0.606

0.605

0.604

0.603

0.825

0.830

0.420

λ = 0.3

λ = 0.2
λ = 0.1

R2: BLEU 4

R2: ROU GE

0.609

BLEU 4 rewarded

0.418

0.416

0.410

RL: BLEU 1
RL: BLEU 44
MORL: BLEU 1+BLEU
2

0.408

RS: (1 − λ) · θBLEU 1 + λ · θBLEU 4

0.412

0.840

0.845

0.850

0.806

0.825

(a) BLEU1 and ROUGE.

0.830

0.835

0.840

R1: BLEU 1

0.845

0.424

1.37

0.405
1.343

Pre-trained init
RL: BLEU 1
RL: BLEU 4
RL: ROU GE
RL: M ET EOR
P =5/5
MORL: Ni=1 Ri0.307
PN =5
RS: i=1 θi /5

0.414

µ=λ=0

0.835

BLEU 4

1.396

pre-trained init
BLEU 1 rewarded

pre-trained init

R1: BLEU 1

CIDEr

λ = 0.5

0.386

0.303

0.603

0.607

0.312

0.61

M ET EOR

0.850(b) BLEU1 and BLEU4.

ROU GE(c) Captioning: spider map.

Figure 3: Results in image captioning on COCO (Lin et al., 2014). As rewards R1 (blue stars every epoch) and R2 (red stars), we consider
standard statistical metrics: BLEU1 (1-gram overlap), BLEU4 (4-grams overlap), ROUGE, METEOR and CIDEr. Figure 3(a) include the
MORL training trajectories optimizing (1 − µ) × BLEU 1 + µ × ROU GE, uncovering a yellow front similar to RS’s green front. In
Figure 3(c), RS uniformly averages the 5 weights (one for each reward), resulting in the largest area and the best trade-off.
(1 − µ̂) × BLEU 1N orm + µ̂ × ROU GEN orm

0.611

1.6

1.0

0.8

MORL µ:
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
µ̂
CV µ

0.610

1.25

0.609

R2: ROU GE

RS λ:
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
µ̂
CV λ

1.2

Normalized rewards

1.4

1.50

1.00

0.75

R1 : BLEU 1

0.25

BLEU 2BLEU 3BLEU 4
R2 : ROU GE

0.00

M ET EOR
CIDEr

0.50

ROU GE rewarded

λ = 0.5

0.608

0.607

0.606

BLEU 1 rewarded

0.605

0.604

RS: (1 − λ) · θBLEU 1 + λ · θROU GE

ENS: (1 − λ) · f (·, θBLEU 1 ) + λ · f (·, θROU GE )

0.603

0.6
0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

µ̂

(a) Analysis of Lemma 1.

0.6

0.8

λ

(b) Performances w.r.t. λ.

1.0

0.825

0.830

0.835

0.840

R1: BLEU 1

0.845

0.850

(c) RS vs. prediction ensembling.

Figure 4: Refined results in captioning with R1 = BLEU 1 and R2 = ROU GE. Figure 4(a) empirically validates Lemma 1 by reporting
results of RS (for varying λ) and of MORL (for varying µ) for varying user’s preference µ̂. In Figure 4(b), all rewards are used forevaluation as a function of the interpolating coefficient. In Figure 4(c), we report the front of the costly ensembling (Hansen & Salamon,
1990; Lakshminarayanan et al., 2017) of predictions (rather than of weights).

3.3. Text-to-image: diffusion models with RLHFs

of aesthetic preference. In our experiments, we employ
N = 2 open-source reward models: ava, trained on the
AVA dataset (Murray et al., 2012), and cafe, trained ona mix of real-life and manga images. These two models
are trained in a supervised setting to match human quality
ratings collected. We first generate 10000 images; then, foreach reward, we remove half of the images with the lowest
reward’s score, and fine-tune 10% of the parameters (Xieet al., 2023) on the reward-weighted negative log-likelihood(Lee et al., 2023). More details are in Appendix E.

Beyond text generation, we now apply RS to align textto-image generation with human feedbacks, as previously
done in recent papers(Lee et al., 2023; Wu et al., 2023b; Xu
et al., 2023). This alignment is expected to improve specificvisual control signals like colors, counts, and backgrounds.
Notably, diffusion models can be fine-tuned to match humanaesthetic preferences. Our network is a diffusion model (Hoet al., 2020) with 2.2B parameters, pre-trained on an internal
dataset of 300M images; it reaches similar quality as Stable
Diffusion (Rombach et al., 2022), which was not used for
copyright reasons. As for any subjective metric, there is avariety of reward models that can capture different aspects

60.820cafe rewarded

12.7

0.805

ava rewarded
0.800

0.795

Small+Large
rewarded
2

12.6

22.14

21.57

11.95

RS: (1 − λ) · θava + λ · θcaf e43.917

12.5

MORL: (1 − µ) × Rava + µ × Rcaf e

0.790

Small
22.71

Pre-trained init
RL: Small
RL: M edium
RL: Large
P =3
MORL: Ni=1 Ri /3
P =3
RS: N
i=1 θi /3

RS: (1 − λ) · θSmall + λ · θLargeµ = 0.5

R3: Large

R2: caf e

Large rewarded
12.8

λ = 0.1

0.815

0.810

λ = 0.2

λ = 0.5

0.785

12.4

pre-trained init
Large

Small rewarded

12.4

44.833

12.85

45.75

Mediumpre-trained init
0.780
5.22

5.24

5.26

R1: ava

5.28

5.30

(a) Image generation.

21.6

21.8

22.0

22.2

R1: Small

22.4

22.6

(b) VG: Small and Large.

(c) VG: spider map.

Figure 5: Figure 5(a) reports our RLHF experiments on text-to-image generation with diffusion models. From the pre-trained initialization,
we learn θava and θcaf e by optimizing the two reward models ava and cafe. Interpolation between them reveals the green Pareto-optimal
front, above the yellow MORL front. Figures 5(b) and 5(c) report our results in visual grounding (VG) on RefCOCO+ (Yu et al., 2016),
where we optimize to predict boxes with IoU> 0.5 w.r.t. the ground-truth, for objects of either small, medium or large size.

The results displayed in Figure 5(a) validate Hypothesis 1,
as the front described by RS when sliding λ from 0 and 1
is convex. Moreover, RS gives a better front than MORL,
validating Hypothesis 2. Interestingly, the ava reward modelseems to be more general-purpose than cafe, as RL training
on ava also enhances the scores of cafe. In contrast, themodel θcaf e performs poorly in terms of ava in Figure 5(a).
Nonetheless, RS with (1 − λ) · θava + λ · θcaf e outperformsθava alone, not only in terms of cafe, but also of ava whenλ ∈ {0.1, 0.2}. These findings confirm that RS can betteralign text-to-image models with a variety of aesthetic preferences. This ability to adapt at test time paves the way fora new form of user interaction with text-to-image models,
beyond prompt engineering.

cautious rewarded
risky+cautious
rewarded
2

3000

R2: velocity −

P

2
t at

4000

λ = 0.5
risky rewarded

2000

pre-trained init

1000

RL: velocity
P
RL: velocity − t a2tP
MORL: velocity − 12 t a2t0

RS: (1 − λ) · θrisky + λ · θcautious
3500

4000

4500

5000

5500

R1: velocity

6000

6500

Figure 6: Locomotion results.

3.4. Text-to-box: visual grounding
We now consider visual grounding (VG) (Yu et al., 2016):
the task is to predict the bounding box of the region described by an input text. We use a seq-to-seq unified model
predicting the box auto-regressively as a sequence of location tokens (Wang et al., 2022a). This model is pre-trainedon a large image-text dataset, then fine-tuned with crossentropy for VG; finally, we use a weighted loss between
the cross-entropy and REINFORCE in the RL stage. As
the main evaluation metric for VG is the accuracy (i.e.,intersection over union (IoU) > 0.5), we consider 3 nondifferentiable rewards: the accuracy on small, medium, andlarge objects. We design this experimental setup because
improving results on all sizes simultaneously is challenging,as shown in Figure 5(c), where MORL performs similarly
to the initialization. The results in Figure 5(b) confirm that
optimizing for small objects degrades performance on large
ones; fortunately, interpolating can trade-off. In conclusion,

we can adapt to users’ preferences at test time by adjusting
λ, which in turn changes the object sizes that the model
effectively handles. On the one hand, if focusing on distantand small objects, a large coefficient should be assigned toθSmall . On the other hand, to perform well across all sizes,
we can recover initialization’s performances by averaging
uniformly (in Figure 5(c)). More details are in Appendix F.
3.5. Locomotion with diverse engineered rewards
Teaching humanoids to walk in a human-like manner (Duanet al., 2016) serves as a benchmark to evaluate RL strategies(Ng et al., 1999) for continuous control. One of the main
challenges is to shape a suitable proxy reward (Dorigo &Colombetti, 1994; Dewey, 2014), given the intricate coordination and balance involved in human locomotion. It is standard (Todorov et al., 2012) to consider dense rewards of the7

P
form R = velocity − α × t a2t , controlling the agent’s
velocity while regularizing the actions {at }t taken overtime. Yet, the penalty coefficient α is challenging to set. To
address this, we devised two rewards in the Brax physics
engine (Freeman et al., 2021): a risky R1 with α = 0, and amore cautious R2 with α = 1.

cost, (Won et al., 2020; Yang et al., 2020) build experts
and then train a new network to combine them; (Mossalam
et al., 2016; Wilson et al., 2007; Nguyen et al., 2020) shareweights across experts; (Castelletti et al., 2013; Yang et al.,2019; Abels et al., 2019; Peschl et al., 2021) directly train asingle model; the recent and more similar (Hua et al., 2023)
learns one linear embedding per (locomotion) task that canbe interpolated. Yet, these works are mostly for academic
benchmarks (Todorov et al., 2012; Vamplew et al., 2011);
adapting them to larger tasks (e.g., RLHF for foundation
models with PPO) is challenging as they modify the training
procedure. Finally, we relate to multitask learning (Caruana, 1997), where predictions are evaluated for multiple
tasks; in contrast, we have a single prediction evaluated bymultiple rewards.

Like in all previous tasks, RS’s front in Figure 6 exceeds
the interpolated rewards, as per Hypothesis 1. Moreover,
RS’s front indicates an effective balance between risk-taking
and cautiousness, providing empirical support for Hypothesis 2, although MORL with µ = 0.5 (i.e., α = 0.5) slightly
surpasses RS’s front. For a more qualitative and intuitive
assessment, we provide animations of our RL agent’s locomotion on our anonymized website. More details areavailable in Appendix G.

Recent works extended the linear mode connectivity whenfine-tuning on different tasks (Ilharco et al., 2022; DonYehiya et al., 2022; Ramé et al., 2023; Wu et al., 2023a)
or with different losses (Ramé et al., 2022; Croce et al.,2023), while (Juneja et al., 2023) highlighted some failures in NLP for classification. In contrast, we investigate
the LMC in RL. The most similar works are for control
system tasks: (Lawson & Qureshi, 2023) averaging decision transformers and (Gaya et al., 2022) explicitly enforcing connectivity in subspaces of policies trained fromscratch on a single reward. When the LMC holds, combining networks in weights combines their abilities (Ilharco
et al., 2023; Daheim et al., 2023); e.g., averaging an English summarizer and an English-to-French translator cansummarize in French (Jang et al., 2023). In domain generalization, (Wortsman et al., 2022a; Ramé et al., 2022; Arpit
et al., 2021) showed that WI reduces model misspecification (D’Amour et al., 2020); by analogy, we show that RS
reduces reward misspecification.

4. Related work
Our RS approach leans on two key components from traditional DRL. The first is proxy rewards, whose design ischallenging. Statistical metrics, the standard in captioning(Rennie et al., 2017) or language translation (Ranzato et al.,2016), are not practical to measure human concepts (Kwonet al., 2023) such as helpfulness (Bai et al., 2022a; Askell
et al., 2021). Reward models can be trained via inverse DRL(Ng et al., 2000; Abbeel & Ng, 2004) when supervisionfrom experts is available, otherwise from prediction comparison in recent RLHF works(Stiennon et al., 2020; Ouyang
et al., 2022; Christiano et al., 2017). The latest (Kwon et al.,2023; Bai et al., 2022b; Madaan et al., 2023; Scheurer et al.,2023; Sun et al., 2023) further reduce the labeling costs byusing the in-context abilities of LLMs.
Second, RS relies on existing RL algorithms to maximize
the given rewards. RS succeeds with variants of two of themost common, REINFORCE (Williams, 1992) and PPO(Schulman et al., 2017), suggesting it could be applied toothers (Go et al., 2023; Yuan et al., 2023). Among theensembling-like RL strategies (Wang et al., 2010; Mordatch
et al., 2015; Rajeswaran et al., 2017) handling multiple policies, some (Parker-Holder et al., 2020; Osa et al., 2022) aim
to explicitly increase the diversity, yet never with foundation
models nor weight interpolation. Moreover, pre-training
could address stability and exploration issues (Xie et al.,2022; Yang et al., 2023; Sekar et al., 2020).

5. Conclusion
As AI systems are increasingly applied to crucial real-world
tasks, there is a pressing issue to align them to our specificand diverse needs, while making the process more transparent and limiting the cultural hegemony of a few individuals.
In this paper, we proposed rewarded soup, a strategy that
efficiently yields Pareto-optimal solutions through weightinterpolation after training. Our experiments have consistently validated our working hypotheses for various significant large-scale learning tasks, demonstrating that rewarded
soup can mitigate reward misspecification. We hope to inspire further research in exploring how the generalization
literature in deep learning can help for alignment, to create
AIs that benefit society as a whole.

When dealing with multiple objectives in deep learning, thecommon approach is to combine them into a single reward(Roijers et al., 2013; Rădulescu et al., 2020): (Glaese et al.,2022) multiply the predictions of a preference RM (evaluating factfullness) and a rule RM (detecting rules breaking).
The multi-policy alternatives (Barrett & Narayanan, 2008;
Li et al., 2020; Tanaka & Yamamura, 2003; Van Moffaert& Nowé, 2014) are usually more costly. To reduce the8References

Bakker, M. A., Chadwick, M. J., Sheahan, H., Tessler, M. H.,
Campbell-Gillingham, L., Balaguer, J., McAleese, N.,
Glaese, A., Aslanides, J., Botvinick, M., and Summerfield, C. Fine-tuning language models to find agreement
among humans with diverse preferences. In NeurIPS,
2022. (p. 2)

Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B.,
Mironov, I., Talwar, K., and Zhang, L. Deep learningwith differential privacy. In ACM SIGSAC, 2016. (p. 18)
Abbeel, P. and Ng, A. Y. Apprenticeship learning via inverse
reinforcement learning. In ICML, 2004. (p. 8)

Banerjee, S. and Lavie, A. METEOR: An automatic metricfor mt evaluation with improved correlation with humanjudgments. In ACL Workshop, 2005. (p. 5)

Abels, A., Roijers, D., Lenaerts, T., Nowé, A., and Steckelmacher, D. Dynamic weights in multi-objective deep
reinforcement learning. In ICML, 2019. (p. 8)

Barrett, L. and Narayanan, S. Learning all optimal policies
with multiple criteria. In ICML, 2008. (pp. 2, 3, and 8)

Agarap, A. F. Deep learning using rectified linear units(relu). arXiv preprint, 2018. (p. 2)

Becker, S. and Le Cun, Y. Improving the convergence ofback-propagation learning with second order methods. In
Connectionist models summer school, 1988. (p. 20)

Ahmed, H. Detecting opinion spam and fake news using ngram analysis and semantic similarity. PhD thesis, 2017.

Beeching, E., Belkada, Y., von Werra, L., Mangrulkar, S.,
Tunstall, L., and Rasul, K. Fine-tuning 20B LLMs with
RLHF on a 24GB consumer GPU. https://hugg
ingface.co/blog/trl-peft, 2023. (pp. 4, 23,(pp. 4 and 24)

Ahmed, H., Traore, I., and Saad, S. Detecting opinion
spams and fake news using text classification. Security
and Privacy, 2018. (p. 24)

and 24)

Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. Concrete problems in AI safety.
arXiv preprint, 2016. (pp. 1 and 17)

Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling,
M. The Arcade learning environment: An evaluation
platform for general agents. JAIR, 2013. (p. 18)

An, G., Moon, S., Kim, J.-H., and Song, H. O. Uncertaintybased offline reinforcement learning with diversified qensemble. In Ranzato, M., Beygelzimer, A., Dauphin,Y., Liang, P., and Vaughan, J. W. (eds.), NeurIPS, 2021.

Bommasani, R., Hudson, D. A., Adeli, E., Altman, R.,
Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. On the opportunities and risks
of foundation models. arXiv preprint, 2021. (p. 1)(p. 17)

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G.,
Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J.,
Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M.,
Gray, S., Chess, B., Clark, J., Berner, C., McCandlish,
S., Radford, A., Sutskever, I., and Amodei, D. Language
models are few-shot learners. In NeurIPS, 2020. (p. 1)

Arpit, D., Wang, H., Zhou, Y., and Xiong, C. Ensembleof averages: Improving model selection and boosting
performance in domain generalization. In NeurIPS, 2021.
(p. 8)

Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D.,
Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez,D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D.,
Brown, T., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. A general language assistant as a laboratory foralignment. arXiv preprint, 2021. (pp. 2 and 8)

Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J.,
Bojanowski, P., and Joulin, A. Emerging properties inself-supervised vision transformers. In ICCV, 2021. (p. 1)
Caruana, R. Multitask learning. Machine learning, 1997.
(pp. 8 and 18)

Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T.,et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint,
2022a. (pp. 2, 4, 8, and 24)

Castelletti, A., Pianosi, F., and Restelli, M. A multiobjective
reinforcement learning approach to water resources systems operation: Pareto frontier approximation in a single
run. Water Resources Research, 2013. (p. 8)

Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J.,
Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional AI: Harmlessness from AI
feedback. arXiv preprint, 2022b. (p. 8)

Chen, S., Zhang, F., Sone, K., and Roth, D. Improving
Faithfulness in Abstractive Summarization with Contrast
Candidate Generation and Selection. In NAACL, 2021.
(pp. 4, 23, and 24)

9Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg,
S., and Amodei, D. Deep reinforcement learning fromhuman preferences. In NeurIPS, 2017. (pp. 1, 4, and 8)

Eeckt, S. V. et al. Weight averaging: A simple yet effective
method to overcome catastrophic forgetting in automatic
speech recognition. arXiv preprint, 2022. (p. 18)

Coello, C. Handling preferences in evolutionary multiobjective optimization: A survey. In CEC, 2000. (p. 1)

Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V.,
Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I.,
Legg, S., and Kavukcuoglu, K. IMPALA: Scalable distributed deep-RL with importance weighted actor-learnerarchitectures. In ICML, 2018. (p. 18)

Croce, F., Rebuffi, S.-A., Shelhamer, E., and Gowal, S.
Seasoning model soups for robustness to adversarial andnatural distribution shifts. arXiv preprint, 2023. (p. 8)

Fisher, R. A. On the mathematical foundations of theoreticalstatistics. Philosophical Transactions of the Royal Societyof London., 1922. (p. 22)

Daheim, N., Dziri, N., Sachan, M., Gurevych, I., and Ponti,
E. M. Elastic weight removal for faithful and abstractive
dialogue generation. arXiv preprint, 2023. (p. 8)

Frankle, J., Dziugaite, G. K., Roy, D. M., and Carbin, M.
Linear mode connectivity and the lottery ticket hypothesis.
In ICML, 2020. (pp. 2, 3, and 17)

D’Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein,J., Hoffman, M. D., et al. Underspecification presents
challenges for credibility in modern machine learning.
JMLR, 2020. (p. 8)

Freeman, C. D., Frey, E., Raichuk, A., Girgin, S., Mordatch,I., and Bachem, O. Brax–a differentiable physics engine
for large scale rigid body simulation. arXiv preprint,
2021. (pp. 8 and 30)

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei,
L. ImageNet: A large-scale hierarchical image database.
In CVPR, 2009. (p. 25)

Gábor, Z., Kalmár, Z., and Szepesvári, C. Multi-criteria
reinforcement learning. In ICML, 1998. (p. 17)

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert:
Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019. (p. 1)

Gabriel, I. and Ghazavi, V. The challenge of value alignment: From fairer algorithms to AI safety. arXiv preprint,
2021. (p. 18)

Dewey, D. Reinforcement learning and the reward engineering principle. In AAAI Spring Symposia, 2014. (p. 7)

Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward
model overoptimization. arXiv preprint, 2022. (p. 17)

Di Langosco, L. L., Koch, J., Sharkey, L. D., Pfau, J., andKrueger, D. Goal misgeneralization in deep reinforcement learning. In ICML, 2022. (p. 17)

Gaya, J.-B., Soulier, L., and Denoyer, L. Learning a subspace of policies for online adaptation in reinforcement
learning. In ICLR, 2022. (p. 8)

Don-Yehiya, S., Venezian, E., Raffel, C., Slonim, N., Katz,
Y., and Choshen, L. ColD fusion: Collaborative descentfor distributed multitask finetuning. arXiv preprint, 2022.

Glaese, A., McAleese, N., Trebacz, M., Aslanides, J., Firoiu,
V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M.,
Thacker, P., et al. Improving alignment of dialogue agents
via targeted human judgements. arXiv preprint, 2022.

(pp. 2 and 8)(p. 8)

Dong, H., Xiong, W., Goyal, D., Pan, R., Diao, S., Zhang,J., Shum, K., and Zhang, T. RAFT: Reward ranked
finetuning for generative foundation model alignment.
arXiv preprint, 2023. (p. 27)

Go, D., Korbak, T., Kruszewski, G., Rozen, J., Ryu, N., andDymetman, M. Aligning language models with preferences through f-divergence minimization. arXiv preprint,
2023. (p. 8)

Dorigo, M. and Colombetti, M. Robot shaping: Developing
autonomous agents through learning. Artificial intelligence, 1994. (p. 7)

Goldberg, Y. Reinforcement learning for language models.
https://gist.github.com/yoavg/6bff0f
ecd65950898eba1bb321cfbd81, 2023. (p. 1)

Duan, Y., Chen, X., Houthooft, R., Schulman, J., andAbbeel, P. Benchmarking deep reinforcement learningfor continuous control. In ICML, 2016. (p. 7)

Hansen, L. K. and Salamon, P. Neural network ensembles.
TPAMI, 1990. (pp. 4, 5, 6, 17, and 26)
Hayes, C. F., Rădulescu, R., Bargiacchi, E., Källström, J.,
Macfarlane, M., Reymond, M., Verstraeten, T., Zintgraf,
L. M., Dazeley, R., Heintz, F., et al. A practical guideto multi-objective reinforcement learning and planning.
JAAMAS, 2022. (pp. 2 and 18)

Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J.,
Paduraru, C., Gowal, S., and Hester, T. Challenges of realworld reinforcement learning: definitions, benchmarks
and analysis. Machine Learning, 2021. (p. 1)
10Hendrycks, D. Natural selection favors AIs over humans.
arXiv preprint, 2023. (p. 17)

Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C.,
and Socher, R. Ctrl: A conditional transformer language
model for controllable generation. arXiv preprint, 2019.

Hendrycks, D. and Mazeika, M. X-risk analysis for AI
research. arXiv preprint, 2022. (p. 17)(p. 17)

Kingma, D. P. and Ba, J. Adam: A method for stochastic
optimization. In ICLR, 2015. (pp. 20, 24, and 27)

Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. NeurIPS, 2020. (p. 6)

Kirk, H. R., Vidgen, B., Röttger, P., and Hale, S. A. Personalisation within bounds: A risk taxonomy and policy
framework for the alignment of large language models
with personalised feedback. arXiv preprint, 2023. (pp. 2,

Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y.,
Wang, S., Wang, L., and Chen, W. LoRA: Low-rank
adaptation of large language models. In ICLR, 2022a.
(pp. 4, 23, and 24)

17, and 18)

Hu, J. C., Cavicchioli, R., and Capotondi, A. ExpansionNet
v2: Block static expansion in fast end to end training forimage captioning. arXiv preprint, 2022b. (pp. 5 and 25)

Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S.,
Tam, Z.-R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., et al. Openassistant conversations–democratizing large language model alignment. arXiv
preprint, 2023. (pp. 4 and 23)

Hua, P., Chen, Y., and Xu, H. Simple emergent action
representations from multi-task policy training. In ICLR,
2023. (p. 8)

Korbak, T., Shi, K., Chen, A., Bhalerao, R., Buckley, C. L.,
Phang, J., Bowman, S. R., and Perez, E. Pretraining
language models with human preferences. arXiv preprint,
2023. (p. 17)

Ilharco, G., Wortsman, M., Gadre, S. Y., Song, S., Hajishirzi,
H., Kornblith, S., Farhadi, A., and Schmidt, L. Patchingopen-vocabulary models by interpolating weights. In
NeurIPS, 2022. (pp. 2 and 8)

Kunstner, F., Hennig, P., and Balles, L. Limitations of
the empirical fisher approximation for natural gradientdescent. In NeurIPS, 2019. (p. 22)

Ilharco, G., Tulio Ribeiro, M., Wortsman, M., Gururangan,
S., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing
models with task arithmetic. In ICLR, 2023. (pp. 8 and 18)

Kwon, M., Xie, S. M., Bullard, K., and Sadigh, D. Reward
design with language models. In ICLR, 2023. (pp. 1 and 8)

Ioffe, S. and Szegedy, C. Batch normalization: Accelerating
deep network training by reducing internal covariate shift.
In ICML, 2015. (p. 2)

Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple
and scalable predictive uncertainty estimation using deep
ensembles. In NeurIPS, 2017. (pp. 4, 5, 6, and 17)

Irvine, R., Boubert, D., Raina, V., Liusie, A., Mudupalli, V.,
Korshuk, A., Liu, Z., Cremer, F., Assassi, V., Beauchamp,
C.-C., et al. Rewarding chatbots for real-world engagement with millions of users. arXiv preprint, 2023. (p. 2)

Lawson, D. and Qureshi, A. H. Merging decision transformers: Weight averaging for forming multi-task policies. In
ICLR RRL Workshop, 2023. (p. 8)

Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., andWilson, A. Averaging weights leads to wider optima andbetter generalization. In UAI, 2018. (pp. 5, 17, and 23)

LeCun, Y., Denker, J., Solla, S., Howard, R., and Jackel, L.
Optimal brain damage. In NeurIPS, 1990. (p. 20)
LeCun, Y., Bottou, L., Orr, G. B., and Müller, K.-R. Efficient backprop. In Neural Networks. 2012. (p. 20)

Jang, J., Kim, S., Ye, S., Kim, D., Logeswaran, L., Lee, M.,
Lee, K., and Seo, M. Exploring the benefits of training
expert language models over instruction tuning. arXiv
preprint, 2023. (p. 8)

Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y., Boutilier, C.,
Abbeel, P., Ghavamzadeh, M., and Gu, S. S. Aligning textto-image models using human feedback. arXiv preprint,
2023. (pp. 1, 6, and 27)

Juneja, J., Bansal, R., Cho, K., Sedoc, J., and Saphra, N.
Linear connectivity reveals generalization strategies. In
ICLR, 2023. (p. 8)

Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L.
Bart: Denoising sequence-to-sequence pre-training fornatural language generation, translation, and comprehension. In ACL, 2020. (p. 29)

Karpathy, A. and Fei-Fei, L. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015.
(pp. 5 and 25)

11

Li, K., Zhang, T., and Wang, R. Deep reinforcement learning
for multiobjective optimization. IEEE-T-CYBERNETICS,
2020. (pp. 2, 3, and 8)

Michaud, E. J., Gleave, A., and Russell, S. Understanding
learned reward functions. arXiv preprint, 2020. (p. 1)
Mordatch, I., Lowrey, K., and Todorov, E. Ensemble-cio:Full-body dynamic motion planning that transfers to physical humanoids. In IROS, 2015. (p. 8)

Li, M., Gururangan, S., Dettmers, T., Lewis, M., Althoff, T.,
Smith, N. A., and Zettlemoyer, L. Branch-Train-Merge:
Embarrassingly parallel training of expert language models. arXiv preprint, 2022. (p. 18)

Mossalam, H., Assael, Y. M., Roijers, D. M., and Whiteson,
S. Multi-objective deep reinforcement learning. arXiv
preprint, 2016. (p. 8)

Lin, C.-Y. and Hovy, E. Automatic evaluation of summaries
using n-gram co-occurrence statistics. In NAACL, 2003.

Murray, N., Marchesotti, L., and Perronnin, F. AVA: A largescale database for aesthetic visual analysis. In CVPR,
2012. (pp. 6 and 27)(pp. 1, 2, and 5)

Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P.,
Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft
coco: Common objects in context. In ECCV, 2014. (pp. 5,

Nadal, M. and Chatterjee, A. Neuroaesthetics and art’s diversity and universality. Wiley Interdisciplinary Reviews:
Cognitive Science, 2019. (p. 1)

6, and 25)

Liu, B., Liu, X., Jin, X., Stone, P., and Liu, Q. Conflictaverse gradient descent for multi-task learning. NeurIPS,
2021. (p. 18)

Neyshabur, B., Sedghi, H., and Zhang, C. What is beingtransferred in transfer learning? In NeurIPS, 2020. (pp. 2,

Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., andHan, J. On the variance of the adaptive learning rate andbeyond. In ICLR, 2020. URL https://openrevi
ew.net/forum?id=rkgz2aEKDr. (p. 25)

Ng, A. Y., Harada, D., and Russell, S. Policy invarianceunder reward transformations: Theory and application to
reward shaping. In ICML, 1999. (p. 7)

3, and 4)

Ng, A. Y., Russell, S., et al. Algorithms for inverse reinforcement learning. In ICML, 2000. (p. 8)

Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J.,
Cao, Y., Zhang, Z., Dong, L., Wei, F., and Guo, B. Swin
Transformer V2: Scaling up capacity and resolution. In
CVPR, 2022. (pp. 5 and 25)

Ngo, R. The alignment problem from a deep learning perspective. arXiv preprint, 2022. (pp. 1 and 17)
Nguyen, T. T., Nguyen, N. D., Vamplew, P., Nahavandi,
S., Dazeley, R., and Lim, C. P. A multi-objective deep
reinforcement learning framework. EAAI, 2020. (p. 8)

Lopez-Paz, D., Bouchacourt, D., Sagun, L., and Usunier,N. Measuring and signing fairness as performance undermultiple stakeholder distributions. arXiv preprint, 2022.
(p. 2)

OpenAI. Gpt-4 technical report. arXiv preprint, 2023. (pp. 1
and 17)

Lu, J., Clark, C., Zellers, R., Mottaghi, R., and Kembhavi,
A. Unified-io: A unified model for vision, language, andmulti-modal tasks. arXiv preprint, 2022. (p. 28)

Oquab, M., Bottou, L., Laptev, I., and Sivic, J. Learningand transferring mid-level image representations using
convolutional neural networks. In CVPR, 2014. (p. 1)

Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L.,
Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang,
Y., Welleck, S., Majumder, B. P., Gupta, S., Yazdanbakhsh, A., and Clark, P. Self-Refine: Iterative refinementwith self-feedback. arXiv preprint, 2023. (p. 8)

Osa, T., Tangkaratt, V., and Sugiyama, M. Discovering
diverse solutions in deep reinforcement learning by maximizing state–action-based mutual information. Neural
Networks, 2022. (p. 8)

Mannion, P., Heintz, F., Karimpanal, T. G., and Vamplew,
P. Multi-objective decision making for trustworthy ai. In
MODeM Workshop, 2021. (p. 2)

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.,
Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A.,et al. Training language models to follow instructions
with human feedback. NeurIPS, 2022. (pp. 1, 2, and 8)

Matena, M. and Raffel, C. Merging models with Fisherweighted averaging. In NeurIPS, 2022. (pp. 2 and 22)

Ovadya, A. Generative CI through collective response systems. arXiv preprint, 2023. (p. 2)

McMahan, B., Moore, E., Ramage, D., Hampson, S., andy Arcas, B. A. Communication-efficient learning of deep
networks from decentralized data. In AISTATS, 2017.

Pan, A., Bhatia, K., and Steinhardt, J. The effects of reward
misspecification: Mapping and mitigating misaligned
models. In ICLR, 2022. (p. 1)(p. 18)12

Pan, A., Shern, C. J., Zou, A., Li, N., Basart, S., Woodside,T., Ng, J., Zhang, H., Emmons, S., and Hendrycks, D.
Do the rewards justify the means? measuring trade-offs
between rewards and ethical behavior in the MACHIAVELLI benchmark. arXiv preprint, 2023. (p. 2)

Ramé, A., Kirchmeyer, M., Rahier, T., Rakotomamonjy, A.,
Gallinari, P., and Cord, M. Diverse weight averaging
for out-of-distribution generalization. In NeurIPS, 2022.
(pp. 2, 8, 17, 23, and 26)

Ranzato, M., Chopra, S., Auli, M., and Zaremba, W. Sequence level training with recurrent neural networks. In
ICLR, 2016. (p. 8)

Papineni, Kishore, Roukos, S., Ward, T., and Zhu, W.-J.
Bleu: a method for automatic evaluation of machine translation. In ACL, 2002. (pp. 1, 2, and 5)

Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., and Goel,
V. Self-critical sequence training for image captioning.
In CVPR, 2017. (pp. 1, 5, 8, and 25)

Pareto, V. Cours d’économie politique. Librairie Droz, 1964.
(p. 2)

Rodriguez-Soto, M., Lopez-Sanchez, M., and RodríguezAguilar, J. A. Guaranteeing the learning of ethical behaviour through multi-objective reinforcement learning.
In AAMAS, 2021. (p. 17)

Parker-Holder, J., Pacchiano, A., Choromanski, K. M., andRoberts, S. J. Effective diversity in population based
reinforcement learning. In Larochelle, H., Ranzato, M.,
Hadsell, R., Balcan, M., and Lin, H. (eds.), NeurIPS,
2020. (p. 8)

Roijers, D. M., Vamplew, P., Whiteson, S., and Dazeley, R.
A survey of multi-objective sequential decision-making.
JAIR, 2013. (pp. 2 and 8)

Peschl, M., Zgonnikov, A., Oliehoek, F. A., and Siebert,
L. C. Moral: Aligning ai with human norms throughmulti-objective reinforced active learning. arXiv preprint,
2021. (p. 8)

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., andOmmer, B. High-resolution image synthesis with latentdiffusion models. In CVPR, 2022. (pp. 6 and 27)

Pinto, A. S., Kolesnikov, A., Shi, Y., Beyer, L., and Zhai, X.
Tuning computer vision models with task rewards. arXiv
preprint, 2023. (pp. 1 and 5)

Salemi, A., Mysore, S., Bendersky, M., and Zamani, H.
Lamp: When large language models meet personalization.
arXiv preprint, 2023. (pp. 17 and 18)

Radford, A., Narasimhan, K., Salimans, T., and Sutskever,I. Improving language understanding by generative pretraining. 2018. (p. 1)

Scheurer, J., Campos, J. A., Korbak, T., Chan, J. S., Chen,
A., Cho, K., and Perez, E. Training language models withlanguage feedback at scale. arXiv preprint, 2023. (p. 8)

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G.,
Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J.,et al. Learning transferable visual models from natural
language supervision. In ICML, 2021. (p. 1)

Schraudolph, N. N. Fast curvature matrix-vector products
for second-order gradient descent. In Neural computation,
2002. (p. 22)

Rădulescu, R., Mannion, P., Roijers, D. M., and Nowé, A.
Multi-objective multi-agent decision making: a utilitybased analysis and survey. AAMAS, 2020. (pp. 2, 4, and 8)

Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk,
R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., andKomatsuzaki, A. Laion-400m: Open dataset of clipfiltered 400 million image-text pairs. arXiv preprint,
2021. (p. 27)

Raffel, C. A Call to Build Models Like We Build OpenSource Software. https://colinraffel.com/blog/a-call-to-build-models-like-w
e-build-open-source-software.html, 2021.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., andKlimov, O. Proximal policy optimization algorithms.
arXiv preprint, 2017. (pp. 4, 8, 24, and 30)(p. 18)

Rajeswaran, A., Ghotra, S., Ravindran, B., and Levine, S.
EPOpt: Learning robust neural network policies using
model ensembles. In ICLR, 2017. (p. 8)

Schwartz, S. H. et al. An overview of the schwartz theory ofbasic values. Online readings in Psychology and Culture,
2012. (p. 1)

Rame, A., Dancette, C., and Cord, M. Fishr: Invariant
gradient variances for out-of-distribution generalization.
In ICML, 2022. (p. 20)

Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D.,
and Pathak, D. Planning to explore via self-supervised
world models. In ICML, 2020. (p. 8)

Ramé, A., Ahuja, K., Zhang, J., Cord, M., Bottou, L., andLopez-Paz, D. Model Ratatouille: Recycling diverse
models for out-of-distribution generalization. In ICML,
2023. (pp. 2, 3, 8, and 18)

Shah, R., Varma, V., Kumar, R., Phuong, M., Krakovna,
V., Uesato, J., and Kenton, Z. Goal misgeneralization:
Why correct specifications aren’t enough for correct goals.
arXiv preprint, 2022. (p. 17)
13

Siddique, U., Weng, P., and Zimmer, M. Learning fairpolicies in multi-objective (deep) reinforcement learningwith average and discounted rewards. In ICML, 2020.

Thomas, V., Pedregosa, F., van Merriënboer, B., Manzagol,
P.-A., Bengio, Y., and Roux, N. L. On the interplay
between noise and curvature and its effect on optimizationand generalization. In AISTATS, 2020. (p. 22)(p. 18)Skalse, J. M. V., Howe, N. H. R., Krasheninnikov, D., andKrueger, D. Defining and characterizing reward gaming.
In NeurIPS, 2022. (p. 17)

Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics
engine for model-based control. In IROS, 2012. (pp. 7

Smith, B. A brief review of the reasons multi-objective
RL could be important in AI Safety Research. https://www.alignmentforum.org/posts/i5dLf
i6m6FCexReK9/a-brief-review-of-the-reasons- multi- objective- rl- could- be,2021. (p. 17)

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux,
M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E.,
Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. LLaMA: Open and efficient foundation language
models. arXiv preprint arXiv:2302.13971, 2023. (pp. 4, 5,

and 8)

23, and 24)

Vamplew, P., Yearwood, J., Dazeley, R., and Berry, A. On
the limitations of scalarisation for multi-objective reinforcement learning of pareto fronts. In AJCAIA, 2008.

Smith, B. J., Klassert, R., and Pihlakas, R. Soft maximin
approaches to multi-objective decision-making for encoding human intuitive values. In Multi-Objective Decision
Making Workshop, 2021. (p. 17)(pp. 3, 4, and 17)

Vamplew, P., Dazeley, R., Berry, A., Issabekov, R., andDekker, E. Empirical evaluation methods for multiobjective reinforcement learning algorithms. Deakin University, 2011. (p. 8)

Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R.,
Voss, C., Radford, A., Amodei, D., and Christiano, P. F.
Learning to summarize with human feedback. NeurIPS,
2020. (pp. 1, 4, 8, 23, and 24)

Vamplew, P., Dazeley, R., Foale, C., Firmin, S., and Mummery, J. Human-aligned artificial intelligence is a multiobjective problem. Ethics and Information Technology,
2018. (pp. 2, 17, and 18)

Stojanovski, Z., Roth, K., and Akata, Z. Momentum-based
weight interpolation of strong zero-shot models for continual learning. In NeurIPS Interpolate Workshop, 2022.
(p. 18)

Van Moffaert, K. and Nowé, A. Multi-objective reinforcement learning using sets of pareto dominating policies.
JMLR, 2014. (pp. 2 and 8)

Sun, Z., Shen, Y., Zhou, Q., Zhang, H., Chen, Z., Cox, D.,
Yang, Y., and Gan, C. Principle-driven self-alignmentof language models from scratch with minimal humansupervision. arXiv preprint, 2023. (p. 8)

Van Moffaert, K., Drugan, M. M., and Nowé, A. Scalarized multi-objective reinforcement learning: Novel design techniques. In ADPRL, 2013. (p. 17)

Tanaka, F. and Yamamura, M. Multitask reinforcement
learning on the distribution of mdps. In CIRA, 2003.

Van Moffaert, K., Brys, T., Chandra, A., Esterle, L., Lewis,
P. R., and Nowé, A. A novel adaptive weight selection
algorithm for multi-objective multi-agent reinforcement
learning. In IJCNN, 2014. (p. 18)(pp. 2 and 8)

Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X.,
Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford Alpaca: An instruction-following LLaMA model.
https://github.com/tatsu-lab/stanford_alpaca, 2023. (pp. 1, 4, 23, and 24)

Vapnik, V. N. An overview of statistical learning theory. In
TNN, 1999. (p. 1)
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,
L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In NeurIPS, 2017. (pp. 2 and 24)

Taylor, J., Yudkowsky, E., LaVictoire, P., and Critch, A.
Alignment for advanced machine learning systems. Ethics
of AI, 2016. (pp. 1 and 17)

Vedantam, R., Lawrence Zitnick, C., and Parikh, D.
Consensus-based image description evaluation. In ICCV,2015. (pp. 1, 5, and 25)

Teh, Y., Bapst, V., Czarnecki, W. M., Quan, J., Kirkpatrick,J., Hadsell, R., Heess, N., and Pascanu, R. Distral: Robust
multitask reinforcement learning. NeurIPS, 2017. (p. 18)

Völske, M., Potthast, M., Syed, S., and Stein, B. Tl; dr:Mining reddit to learn automatic summarization. In ACL
Workshop, 2017. (pp. 4 and 24)

Tetlock, P. E. A value pluralism model of ideological reasoning. JPSP, 1986. (p. 18)
14

von Werra, L., Belkada, Y., Tunstall, L., Beeching, E.,
Thrush, T., and Lambert, N. TRL: Transformer reinforcement learning. https://github.com/lvwer
ra/trl, 2020. (pp. 4 and 24)

Wolf, Y., Wies, N., Levine, Y., and Shashua, A. Fundamental limitations of alignment in large language models.
arXiv preprint, 2023. (p. 17)
Won, J., Gopinath, D., and Hodgins, J. A scalable approachto control diverse behaviors for physically simulated characters. TOG, 2020. (p. 8)

Wang, E. J. Alpaca-LoRA. https://github.com/t
loen/alpaca-lora, 2023. (p. 24)

Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R.,
Gontijo-Lopes, R., Morcos, A. S., Namkoong, H.,
Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L.
Model soups: averaging weights of multiple fine-tuned
models improves accuracy without increasing inference
time. In ICML, 2022a. (pp. 2, 3, 8, 23, and 26)

Wang, J. M., Fleet, D. J., and Hertzmann, A. Optimizing
walking controllers for uncertain inputs and environments.
ACM, 2010. (p. 8)
Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J.,
Zhou, C., Zhou, J., and Yang, H. OFA: unifying architectures, tasks, and modalities through a simple sequenceto-sequence learning framework. CoRR, 2022a. (pp. 7

Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Hajishirzi,
H., Farhadi, A., Namkoong, H., and Schmidt, L. Robust
fine-tuning of zero-shot models. In CVPR, 2022b. (p. 26)

and 28)Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A.,
Khashabi, D., and Hajishirzi, H. Self-instruct: Aligninglanguage model with self generated instructions. arXiv
preprint, 2022b. (p. 4)

Wu, C., Wang, T., Ge, Y., Lu, Z., Zhou, R., Shan, Y., andLuo, P. π-tuning: Transferring multimodal foundation
models with optimal multi-task interpolation. In ICML,
2023a. (p. 8)

Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y.,
Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S.,
Arunkumar, A., Stap, D., Pathak, E., Karamanolakis,
G., Lai, H., Purohit, I., Mondal, I., Anderson, J., Kuznia, K., Doshi, K., Pal, K. K., Patel, M., Moradshahi,
M., Parmar, M., Purohit, M., Varshney, N., Kaza, P. R.,
Verma, P., Puri, R. S., Karia, R., Doshi, S., Sampat, S. K.,
Mishra, S., Reddy A, S., Patro, S., Dixit, T., and Shen, X.
Super-NaturalInstructions: Generalization via declarative
instructions on 1600+ NLP tasks. In ACL, 2022c. (p. 1)

Wu, J., Ouyang, L., Ziegler, D. M., Stiennon, N., Lowe, R.,
Leike, J., and Christiano, P. Recursively summarizing
books with human feedback. arXiv preprint, 2021. (pp. 1
and 4)Wu, X., Sun, K., Zhu, F., Zhao, R., and Li, H. Betteraligning text-to-image models with human preference.
arXiv preprint, 2023b. (pp. 1, 6, and 27)
Xie, E., Yao, L., Shi, H., Liu, Z., Zhou, D., Liu, Z., Li, J.,
and Li, Z. Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning.
arXiv preprint arXiv:2304.06648, 2023. (pp. 6 and 27)

Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester,B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language
models are zero-shot learners. In ICLR, 2022. (pp. 1 and 4)

Xie, Z., Lin, Z., Li, J., Li, S., and Ye, D. Pretraining in deep
reinforcement learning: A survey. arXiv preprint, 2022.

Wildavsky, A. Choosing preferences by constructing institutions: A cultural theory of preference formation. American political science review, 1987. (p. 1)(p. 8)Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang,J., and Dong, Y. ImageReward: Learning and evaluating
human preferences for text-to-image generation. arXiv
preprint, 2023. (pp. 1, 6, and 27)

Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement learning, 1992. (pp. 5, 8, 25, 28, and 29)

Yang, C., Yuan, K., Zhu, Q., Yu, W., and Li, Z. Multi-expert
learning of adaptive legged locomotion. Science Robotics,
2020. (p. 8)

Wilson, A., Fern, A., Ray, S., and Tadepalli, P. Multi-task
reinforcement learning: a hierarchical bayesian approach.
In ICML, 2007. (p. 8)

Yang, R., Sun, X., and Narasimhan, K. A generalizedalgorithm for multi-objective reinforcement learning andpolicy adaptation. In NeurIPS, 2019. (p. 8)

Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C.,
Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M.,
Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite,
Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M.,
Lhoest, Q., and Rush, A. Transformers: State-of-the-art
natural language processing. In EMNLP, 2020. (pp. 4, 5,

Yang, S., Nachum, O., Du, Y., Wei, J., Abbeel, P., andSchuurmans, D. Foundation models for decision making:Problems, methods, and opportunities. arXiv preprint,
2023. (p. 8)

and 23)15

Yen, G. G. and He, Z. Performance metric ensemble formultiobjective evolutionary algorithms. TEVC, 2013.
(pp. 4 and 5)

Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. How transferable are features in deep neural networks? NeuriPS,
2014. (p. 1)
Yu, L., Poirson, P., Yang, S., Berg, A. C., and Berg, T. L.
Modeling context in referring expressions. In ECCV,2016. (pp. 7, 28, and 29)
Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., andFinn, C. Gradient surgery for multi-task learning. In
NeurIPS, 2020. (p. 18)
Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., and
Huang, F. RRHF: Rank responses to align language models with human feedback without tears. arXiv preprint,
2023. (p. 8)
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford,
A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning
language models from human preferences. arXiv preprint,
2019. (p. 1)

16

Rewarded soups: towards Pareto-optimality by interpolating weights fine-tuned ondiverse rewards
Supplementary material

This supplementary material is organized as follows:• Appendix A further discusses the societal impacts of our RS strategy.
• Appendix B details some theoretical aspects of our RS strategy.
• Appendix C details our experiments in RLHF with LLaMA for text-to-text generation.
• Appendix D details and enriches our experiments in image captioning.
• Appendix E details and enriches our experiments in image generation.
• Appendix F details and enriches our experiments in visual grounding.
• Appendix G details and enriches our locomotion experiments.
The shareable code will be released on this anonymized url page. Moreover, you can find additional qualitative results of ourexperiments on our anonymized website.

A. Discussion
A.1. Limitations and societal impacts
The recent and rapid scaling of networks presents both opportunities and major concerns (Amodei et al., 2016; Hendrycks& Mazeika, 2022; Hendrycks, 2023). Our approach is a step towards better empirical alignment (Taylor et al., 2016;
Ngo, 2022). Yet, reward misspecification is only one of the many challenges inherited from the RL paradigm. First, proxyrewards may lack robustness (Gao et al., 2022) or be hacked (Skalse et al., 2022) via adversarial exploitation, making
them unreliable. Second, RL algorithms may cause overfitting, leading to poor generalization in test, with a risk of goal
misgeneralization (Shah et al., 2022; Di Langosco et al., 2022). Third, RLHF has drawbacks, such as harming calibration(OpenAI, 2023). Our a posteriori multi-policy strategy could alleviate the impact of some badly shaped proxy rewards andsome failed optimizations, as well as tackling Goodhart’s law (Smith, 2021). Yet, without constraint on the test distribution,complete alignment may be impossible (Wolf et al., 2023), for example for LLMs with prompts of arbitrary (long) length.
Therefore, new training paradigms (Keskar et al., 2019; Korbak et al., 2023) beyond RL may be required.
Theoretical guarantees for alignment are also needed (Rodriguez-Soto et al., 2021). Yet, RS relies on an empiricalfinding: the LMC (Frankle et al., 2020), which currently lacks full theoretical guarantees, even in the simplest case ofmoving averages (Izmailov et al., 2018). The best existing explanation (Ramé et al., 2022; Izmailov et al., 2018) relies on
the similarities between weight interpolation and functional ensembling (Hansen & Salamon, 1990; Lakshminarayanan
et al., 2017) when weights remain close, as recalled in Lemma 4. Moreover, assuming the LMC, Lemma 1 theoretically
fixes issues only for R̂ linear over the proxy rewards. Yet, such linearization cannot encapsulate all types of (human)
preferences (Vamplew et al., 2018; 2008). Thus, considering more complex combinations (Gábor et al., 1998; An et al.,2021; Van Moffaert et al., 2013; Smith et al., 2021) is a promising direction. We may empirically overcome this limitation
within RS by continually adjusting and adding new proxy rewards, such that their linear mixtures have increasingly goodcoverage. Indeed, RS is flexible and was shown to handle variable numbers of rewards.
Finally, our a posteriori alignment with users facilitates personalization (Salemi et al., 2023) of models. As discussed
in Appendix A.2.1 and in (Kirk et al., 2023), this could increase usefulness by providing tailored generation, notably tounder-represented groups. Moreover, the distributed nature of RS makes it parallelizable thus practical in a federated
17

learning setup (McMahan et al., 2017) where data must remain private. Yet, this personalization comes with risks forindividuals of “reinforcing their biases [...] and narrowing their information diet”(Kirk et al., 2023). This may worsen thepolarization of the public sphere. Under these concerns, we concur with the notion of “personalization within bounds” (Kirket al., 2023), with these boundaries potentially set by weights fine-tuned on diverse and carefully inspected rewards.
A.2. Benefits of our approach
In this section we discuss the benefits of our rewarded soup (RS) approach with respect to the two families of strategies: thesingle-policy and the multi-policy approaches.
A.2.1. C OMPARED TO SINGLE - POLICY APPROACHES
The main reason why single-policy approaches are not suitable is because they optimize over a single set of preferences.
In contrast, we build a coverage set of Pareto-optimal policies. This is important for the following reasons, mostly first
discussed in Hayes et al. (Hayes et al., 2022) and in Kirk et al. (Kirk et al., 2023).
Indeed, the user’s true reward is highly uncertain before training. This “semi-blind” (Hayes et al., 2022) manual process
forces a priori and uncertain decisions about the required trade-offs. It shifts the responsibility from the problem
stakeholders to the system engineers, who need to anticipate the impact of their choices on the final performance. Critically,
the RLHF process may cause the “tyranny of the crowdworker” (Kirk et al., 2023), as models are “tailored to meet the
expectations of [...] a small number of crowdworkers primarily based in the US, with little to no representation of broader
human cultures, geographies or languages.” (Kirk et al., 2023). Moreover, these “biases are exacerbated by a lack of [...]
documentation” (Kirk et al., 2023). Thus (Kirk et al., 2023) argue that the personalization should be explicit—rather than
implicitly caused by hidden and chaotic engineering choices. In contrast, our strategy could support decision-making tofind a good balance between (potentially conflicting) parties’ interests. This value pluralism (Tetlock, 1986) can lead tofairer and more equitable outcomes (Vamplew et al., 2018; Siddique et al., 2020). Single-policy cannot adapt to test timerequirements; in contrast, RS facilitates personalized assistances (Salemi et al., 2023), with fewer prompts/inputs to themodel, as we only need to adapt interpolating coefficients and not the full network. This is all the more important as humanpreferences change from time to time: in this dynamic utility function scenario, RS can quickly adapt by adjusting the λ tomatch new preferences. Finally, RS could also improve the interpretability and explainability of the decisions. Letting theusers decide could make the process more transparent (Gabriel & Ghazavi, 2021), which is essential to ensure that thedevelopment process is fair, unbiased, and inclusive (Abadi et al., 2016).
A.2.2. C OMPARED TO MULTI - POLICY APPROACHES
The main reason why other multi-policy approaches through multitasking are not suitable is because of their computationalcosts required to learn a dense set of policies. In contrast, RS only trains the proxy rewards independently, and enables
the selection of the interpolating coefficient a posteriori. This is especially useful with large number of rewards and thus
growing number of combinations. Second, multitask (Caruana, 1997) is challenging; for example, even if the true reward isactually a linear weighted sum of some proxy rewards and those coefficients are known, using those preferences during
training can lead to suboptimal results (Van Moffaert et al., 2014), because of conflicting gradients (Yu et al., 2020; Liuet al., 2021) or different variance scales (Espeholt et al., 2018; Teh et al., 2017). This has been tackled in RL, but so far
mostly for games such as ATARI (Bellemare et al., 2013). Third, our strategy is compatible with the inherent iterativeengineering process of alignment. Indeed, RS can continually include adjusted opinions while preventing forgetting of theold behaviours. This relates to the continual learning challenge, and the empirical observations that weight averaging canreduce catastrophic forgetting (Stojanovski et al., 2022; Eeckt et al., 2022). Moreover, as shown in (Ilharco et al., 2023)
and confirmed in Figure 10(c), negative editing by weight interpolation can fix and force the removal of some behaviours.
Finally, RS is computationally effective, requiring no communication across servers, thus enabling “embarrassingly simple
parallelization” (Li et al., 2022). This facilitates its use in federated learning scenario (McMahan et al., 2017) where thedata should remain private. Actually, RS follows the updatable machine learning paradigm (Raffel, 2021), “allowing
for the collaborative creation of increasingly sophisticated AI system” (Ramé et al., 2023). In the future, we may develop
open-source personalized models, rewarded on decentralized private datasets, and combine them continuously.

18

B. Theoretical insights
B.1. Proof of Lemma 1
Proof. Considering θ maximizing
firstPshow that θ is on the PF of {Ri }i . Otherwise, considering θ′ >N θ
P R̂, we′′and as ∀i, µ̂i ≥ 0, we have i µ̂i Ri (θ ) >i µ̂i Ri (θ). This implies that θ would produce a better policy than θP
for R̂ =P i µ̂i Ri and thus the contradiction. Finally, as θ is on the PF and by definition of a PCS, there exists λ s.t.
∀k, Rk ( i λi · θi ) = Rk (θ).
B.2. Theoretical guarantees with quadratic rewards

In this section, we provide theoretical guarantees for the near-optimality of RS when considering quadratic rewards. This
simplification amounts to replacing the rewards by their second-order Taylor approximation, which is a realistic assumption
when the weights remain within a small neighborhood.
B.2.1. S IMPLE CASE WITH H ESSIANS PROPORTIONAL TO THE I DENTITY MATRIXFor the first Lemma 2, we make the following simplifying Assumption 1.
Assumption 1 (Hessians proportional to the Identity matrix.). Every reward Ri is quadratic, with Hessians proportional
to Id . Specifically, let Θ ⊂ Rd be the set of possible weights, and let {Ri }N
i=1 be the N rewards, we can write fori ∈ {1, ..., N }:∀θ ∈ Θ,Ri (θ) = Ri (θi ) − ηi ∥θ − θi ∥2

where ηi ∈ R∗+ and θi is the global maximum for reward Ri .

Lemma 2. Let µ̂ = (µ̂1 , ..., µ̂N ) ∈ ∆N . Then, under Assumption 1, the reward Rµ̂ =convex hull of {θ1 , ..., θN }.

(1)Pi µ̂i × Ri is maximized on the

Proof. The function Rµ̂ is quadratic thus has an unique global maximum θ̂, that we find analytically:∇θ Rµ̂ (θ̂) = 0 =⇒

N
X
i=1

=⇒ θ̂ =

µi ηi · (θ̂ − θi ) = 0
PN

i=1 µ̂i ηi · θiPN

i=1 µ̂i ηi

Since all the µ̂i ηi are positive or zero, and at least one is greater than zero, θ̂ is indeed in the convex hull of {θ1 , ..., θN }.
Remark 3. Under Assumption 1, the reward functions are concave; thus we can reasonably assume that each fine-tuning
procedure for Ri reaches its global optimum θi for i ∈ {1, ..., N }. Then, Lemma 2 tells us that the maximum value forlinear user’s reward Rµ̂ is obtainable by weight interpolation between the {θi }Ni=1 : the interpolating coefficients in ∆Nsuch that λi ∝ µ̂i ηi make rewarded soups optimal.
B.2.2. A DVANCED CASE WITH DIAGONAL H ESSIANS
We now consider the more complex case with the relaxed Assumption 2. For simplicity, we only consider N = 2 rewards
R1 and R2 .
Assumption 2 (Diagonal Hessians). The rewards are quadratic, with Hessians diagonal negative definite. Specifically, we
can write for i ∈ {1, 2}:1d∀θ = (θ , ..., θ ) ∈ Θ,dX
Ri (θ) = Ri (θi ) −ηij (θj − θij )2 ,j=1

where (ηi1 , ...ηid ) ∈ {R∗+ }d and θi = (θi1 , ..., θid ) is the global maximum for reward Ri .
19

(2)

Remark 4. This diagonal Assumption 2 of the Hessian is common: for example in optimization (LeCun et al., 2012; Kingma& Ba, 2015), to prune networks (LeCun et al., 1990) or in out-of-distribution generalization (Rame et al., 2022). This strong
assumption is supported by the empirical observation (Becker & Le Cun, 1988) that Hessians are diagonally dominant,in particular at the end of training. Also, we note that our findings remain valid assuming only that the Hessians are
co-diagonalizable.
Lemma 3. We consider the user’s reward Rµ̂ = (1 − µ̂) × R1 + µ̂ × R2 with µ̂ ∈ [0, 1], and∆Rµ̂ = max Rµ̂ (θ) − max Rµ̂ ((1 − λ) · θ1 + λ · θ2 ).
θ∈Θ

λ∈[0,1](3)∆Rµ̂ corresponds to the difference in terms of Rµ̂ between the global maximum and the maximum reachable by weightinterpolation through rewarded soups (with a single interpolating coefficient for all dimensions). Then, under Assumption 2,
we have:µ̂2 (1 − µ̂)2 (M ∆1 − ∆2 )(M ∆2 − ∆1 )∆Rµ̂ ≤,(4)(µ̂(1 − µ̂)(M − 1)2 + M )((1 − µ̂)∆1 + µ̂∆2 ) j j
ηηwhere M = maxj∈{1, ... ,d} max η1j , η2j is the maximum of eigenvalues ratio, ∆1 = R1 (θ1 ) − R1 (θ2 ) and ∆2 =
2

R2 (θ2 ) − R2 (θ1 ).

1When ∆1 = ∆2 , the bound simplifies into:∆Rµ̂ ≤µ̂2 (1 − µ̂)2 (M − 1)2∆1µ̂(1 − µ̂)(M − 1)2 + M(5)Furthermore, when the Hessians are equal, then M = 1 and ∆Rµ̂ = 0: RS is optimal .
Proof. This novel proof is in three steps. First, we find θ̂ maximizing Rµ̂ (θ) for θ on the full set of weights Θ. Second,
we find λ̄ maximizing Rµ̂ ((1 − λ) · θ1 + λ · θ2 ) for λ ∈ [0, 1] and thus defining the best interpolation between the expert
weights. Finally, we bound ∆Rµ̂ , the differences between their rewards, by applying the Bhatia-Davis inequality.
First step. Let’s first find the maximum of Rµ̂ on Θ. Denoting S = (1 − µ̂) × R1 (θ1 ) + µ̂ × R2 (θ2 ), we have for allθ ∈ Θ:Rµ̂ (θ) = S −

d 
X
j=1

(1 − µ̂)η1j



θ

j

− θ1j

2

+ µ̂η2j



θ

j

− θ2j

2 

(6)

Since
Rµ̂ is a sum of concave quadratic functions, it has a unique global maximum reached at a point we note θ̂ =1d
θ̂ , ..., θ̂ . The global maximum can be computed by differentiating Rµ̂ with respect to each variable θj , which gives:θ̂j = 1 − λ̂j · θ1j + λ̂j · θ2jwhere the interpolating coefficients per dimension λ̂j are defined for j ∈ {1, ..., d} as:
λ̂j =

µ̂η2j
(1 − µ̂)η1j + µ̂η2j∈ [0, 1].

(7)

Second step. With λ ∈ [0, 1] and θ = (1 − λ) · θ1 + λ · θ2 , we can write Rµ̂ (θ) as a function of λ:
!
d
j j


2

2
X
η
µ̂(1
−
µ̂)η
j
j
j
j
1 2
Rµ̂ (θ) = S −(1 − µ̂)η1 + µ̂η2 λ − λ̂j +
θ
−
θ
1
2
(1 − µ̂)η1j + µ̂η2j
j=1= Rµ̂ (θ̂) −

d
X
j=1


2
pj λ − λ̂j

(8)

20



2
where pj is defined as pj = (1 − µ̂)η1j + µ̂η2j θ1j − θ2j .

From Equation (8), we can compute the maximum reward obtainable for weight averaging maxλ∈[0,1] Rµ̂ ((1 −λ)·θ1 + λ · θ2 ). Since the function λ 7→ Rµ̂ ((1 − λ) · θ1 + λ · θ2 ) is a concave quadratic function, there is a uniquevalue λ̄ maximizing Rµ̂ equal toPd

j=1 pj λ̂λ̄ = Pd

j=1 pj

j

.

(9)


Since all pj are positive and all λ̂j are between 0 and 1, λ̄ is also between 0 and 1. Therefore, Rµ̂ (1 − λ̄) · θ1 + λ̄ · θ2 isindeed the maximum reward for rewarded soups.
Third step.

Applying Equation (8) to λ̄ gives:∆Rµ̂ = Rµ̂ (θ̂) − Rµ̂ (1 − λ̄) · θ1 + λ̄ · θ2
d

2
X
=
pj λ̄ − λ̂j
j=1


d
X
p

Pn j
=

i=1 pij=1λ̄ − λ̂j

2






n
X
j=1

(10)
(11)


pj 

(12)

The second term in Equation (12) can be simplified as:
dX
j=1

pj = (1 − µ̂)∆1 + µ̂∆2 .

(13)

The core component of this proof is the upper bounding of the first term in Equation (12). The key idea is to recognize thevariance of a discrete random variable Λ with P(Λ = λ̂i ) = Pnpi pj ; then, λ̄ from Equation (9) is actually the expectation
j=1
of Λ. Then, we can apply the Bhatia-Davis inequality, as recalled in Equation (14), on the variance of a bounded randomvariable a ≤ Λ ≤ b:
V ar(Λ) ≤ (b − E(Λ))(E(Λ) − a)Therefore Equation (12) is bounded by:



j
j
∆Rµ̂ ≤ max λ̂ − λ̄ λ̄ − min λ̂ ((1 − µ̂)∆1 + µ̂∆2 ).
1≤j≤d

1≤j≤d

(14)

(15)

Now, we bound the variables λ̂j , since 1/M ≤ η1j /η2j ≤ M . Then for all j we have:µ̂µ̂M≤ λ̂j ≤,
(1 − µ̂)M + µ̂
(1 − µ̂) + µ̂M(16)

and thus:∆Rµ̂ ≤

µ̂M− λ̄
1 + µ̂(M − 1)




µ̂
λ̄ −
((1 − µ̂)∆1 + µ̂∆2 ).
M − µ̂(M − 1)

(17)


2
Pd
µ̂∆2
Finally, noting that ∆i = j=1 ηij θ2j − θ1j , we deduce from Equation (9) that λ̄ = (1−µ̂)∆. Replacing this in the
1 +µ̂∆2previous Equation (17) gives the final Equation (4), concluding the proof.
21

Remark 5. As a final remark, please note that the suboptimality of RS comes from the need of having one single interpolating
coefficient λ̄ for all d parameters (θ1 , ..., θd ) of the network. Yet, the advanced merging operations in (Matena & Raffel,2022) remove this constraint, with interpolating coefficients proportional to the eigenvalues of the Fisher matrices (Fisher,1922), which actually approximate the eigenvalues of the Hessian (Schraudolph, 2002; Thomas et al., 2020). Combining(Matena & Raffel, 2022) and our RS is a promising research direction, the key issue being the computation of the Fisher
matrices (Kunstner et al., 2019) for networks with billions of parameters.
B.2.3. B OUND VISUALIZATION
We visualize in Figure 7 the bound given by Lemma 3. We show that for small values of M like M = 2, the value of Rµ̂ for
RS is quite close to the global optimum. Also, recall that RS theoretically matches this upper bound when M = 1. For
larger values like M = 10, the bound is less tight, and we note that the maximum value of Rµ̂ approaches the constantfunction 1 as M → ∞.

1.0

Rµ̂ = (1 − µ̂) × R1 + µ̂ × R2

0.9

0.8

0.7

0.6

0.5
Upper bound, M=10
Upper bound, M=2

Rewarded soups
LMC lower bound

0.4
0.0

0.2

0.4

0.6

0.8

1.0

µ̂
Figure 7: Illustration of the bound given by Lemma 3 under Assumption 2. For simplicity, we showcase the case where R1 (θ1 ) =R2 (θ2 ) = 1, R1 (θ2 ) = R2 (θ1 ) = 0, thus ∆1 = ∆2 = 1. In green, we plot the rewards obtained with rewarded soups for the optimal
λ̄, i.e., Rµ̂ (1 − λ̄) · θ1 + λ̄ · θ2 , whose value is independent of M in this case. In blues, we plot the maximum value of Rµ̂ given by
Equation (5) in Lemma 3, for M = 2 and M = 10. For reference, we also plot the values for the lower bound in the LMC Hypothesis 1,
i.e., equal to (1 − µ̂)(1 − λ̄)R1 (θ1 ) + µ̂λ̄R2 (θ2 ). As RS outperforms this lower bound, it validates Hypothesis 1 in this case.

22B.3. Similarity between weight interpolation and functional ensembling
Lemma 4 (λ-interpolation of weights approximates the λ-ensembling of predictions. Adapted from (Wortsman et al.,2022a; Ramé et al., 2022; Izmailov et al., 2018).). Given θ1 and θ2 optimized for R1 and R2 s.t. they remain close, i.e.,∥θ1 − θ2 ∥2 ≈ 0. Denoting θλ the interpolated weights θλ = (1 − λ) · θ1 + λ · θ2 and fλ the ensembling of predictionsfλ (·) = (1 − λ) · f (·, θ1 ) + λ · f (·, θ2 ):f (·, θλ ) ≈ fλ (·)
and for k ∈ {1, 2}:

Rk (f (·, θλ )) ≈ Rk (fλ (·))

Proof. This proof follows (Ramé et al., 2022) and has two components.
Functional approximation.
parameters θ for x ∈ T :and similarly:

First, we perform a Taylor expansion at the first order of the models’ predictions w.r.t.

2f (x, θ1 ) = f (x, θλ ) + ∇θ f (x, θλ )⊺ (θ1 − θλ ) + O ∥θ1 − θλ ∥22= f (x, θλ ) + ∇θ f (x, θλ )⊺ (λ · θ1 − λ · θ2 ) + O ∥θ1 − θ2 ∥22
f (x, θ2 ) = f (x, θλ ) + ∇θ f (x, θλ )⊺ ((λ − 1) · θ1 + (1 − λ) · θ2 ) + O ∥θ1 − θ2 ∥2Then by λ-weighted sum over i, the term multiplying ∇θ f (x, θλ )⊺ cancels out and we obtain:2
fλ (x) = (1 − λ) · f (x, θ1 ) + λ · f (x, θ2 ) = f (x, θλ ) + O ∥θ1 − θ2 ∥2 .

(18)

Reward approximation. Second, we obtain the reward approximation with a Taylor expansion at the zeroth order of the
reward Rk for k ∈ {1, 2} and injecting Equation (18):
Rk (fλ (x)) = Rk (f (x, θλ )(x)) + O(∥fλ (x) − f (x, θλ )∥2 )
2= Rk (f (x, θλ )(x)) + O ∥θ1 − θ2 ∥2 .

We obtain the results when θ1 and θ2 remain close, i.e., when we can ignore the O term.

C. Text-to-text: LLaMA with diverse RLHFs
We summarize the key implementation details of our text-to-text generation experiments in Table 1. The pre-trained network
is LLaMA-7b (Touvron et al., 2023); then low-rank adapters (Hu et al., 2022a) were fine-tuned on Alpaca (Taori et al.,2023) to follow instructions. We eventually fine-tune via PPO on the different considered tasks. Our code is adaptedfrom (Beeching et al., 2023); we kept most of their hyperparameter values, only dividing by 2 the batch size to fit in our
GPU and extending the output length. For each considered task, we downloaded the reward models from HuggingFace(Wolf et al., 2020). For example in summarization tasks, R1 was open-sourced in an effort to reproduce the Summarize
from Human Feedback paper (Stiennon et al., 2020), while R2 (Chen et al., 2021) aimed at improved “faithfulness inabstractive summarization with contrast candidate generation”. For the assistant task, we rely on different reward models
from OpenAssistant (Köpf et al., 2023). Though they all aim at evaluating whether an answer is adequate given a question,
they differ in their predictions due to differences in their architecture and training procedures. In practice, we simply leveragethem as block-box classification pipelines, implemented in the transformers library (Wolf et al., 2020).

23

Table 1: LLaMA with RLHF experiments: key implementation details.

Architecture
Pre-training
Instruction FT

ModelTransformer (Vaswani et al., 2017)
LLaMA-7b (Touvron et al., 2023)
Alpaca (Taori et al., 2023)

Optimizer
Learning rate
Batch size
Output length
RL algorithm
KL PPO
EpochsHardware
Compute budget

RL procedure
LoRA (Hu et al., 2022a)
following Alpaca-LoRA (Wang, 2023)
16
0.05
following trl-peft (von Werra et al., 2020; Beeching et al., 2023)
Adam (Kingma & Ba, 2015)
1.41e-5128
Between 16 and 32
PPO (Schulman et al., 2017)0.05 for summary tasks else 0.2
2 for Reuter summary else 1
NVIDIA RTX A6000 49 Go
4000 GPUh

Task name
Description
PromptDataset
R1
R2
FigureReuter summary
Generate a concise and clear summary of newspaper articles from Reuters.
“Generate a one-sentence summary of this post.”
Reuter news from (Ahmed, 2017; Ahmed et al., 2018) from news-summary
gpt2-reward-summarization trained here.
bart-faithful-summary-detector (Chen et al., 2021)
Figure 2(a)Task name
Description
PromptDataset
R1
R2
FigureReddit summary
Generate a concise and clear summary of posts from Reddit across a variety of topics (subreddits).
“Generate a one-sentence summary of this post.”
Reddit crawl from the TL;DR dataset (Völske et al., 2017) from summarize-from-feedback (Stiennon et al., 2020)
gpt2-reward-summarization trained here.
bart-faithful-summary-detector (Chen et al., 2021)
Figure 2(b)Task name
Description
PromptDataset
R1
R2
R3
R4
FigureHelpful assistant
Provide helpful and harmless answers to potentially complex and sensitive questions.
No prompt, only users’ questions.
Helpfulness and Harmlessness datasets (Bai et al., 2022a) from hh-rlhf
reward-model-deberta-v3-large-v2
reward-model-electra-large-discriminator
reward-model-deberta-v3-base-v2
reward-model-deberta-v3-base
Figures 2(c) and 2(d)Fine-tuning strategy
LoRA alpha
LoRA dropout

24

D. Image-to-text: captioning with diverse rewards
D.1. Experimental details
We summarize the key implementation details of our captioning experiments in Table 2. In short, we took the state-of-the-art
network (Hu et al., 2022b) for captioning on COCO, fine-tune with their code and only changing the reward. In more details,
since the self-critical paper (Rennie et al., 2017) (a variant of REINFORCE (Williams, 1992) with a specific estimation of
the baseline score) it is now common in captioning to optimize the CIDEr reward (Vedantam et al., 2015) after a first step ofsupervised fine-training. The recent ExpansionNetv2 (Hu et al., 2022b) follows this strategy to reach state-of-the-art results,
with a Swin Transformer (Liu et al., 2022) visual encoder and a block static expansion for efficiency. We investigate whether
additional RL trainings can help. We use their code and most of their hyperparameters, only reducing the batch size from 24
to 18 to fit in our GPUs and consequently adapt the learning rate.
Table 2: Captioning experiments: key implementation details.

Architecture
Visual encoder
Visual encoder pre-training
Fine-tuning

Model
ExpansionNetv2 (Hu et al., 2022b)
Swin Transformer (Liu et al., 2022)
ImageNet 22k (Deng et al., 2009)
Cross-entropy then CIDEr RL (Rennie et al., 2017) on COCO (Lin et al., 2014)

Fine-tuning strategy
RL algorithm
Optimizer
DatasetRewards
Learning rate
Batch size
Gradient accumulation
WarmupEpochsHardware
Compute budget

RL procedure
Usually frozen visual backbone, but end-to-end in Figure 10(d)self-critical (Rennie et al., 2017), a variant of REINFORCE (Williams, 1992)
Radam (Liu et al., 2020)
COCO (Lin et al., 2014) and Karpathy split (Karpathy & Fei-Fei, 2015)
BLEU (with 1-gram or 4-grams), ROUGE, METEOR, CIDEr
1e-5
18
2
Anneal 0.8 during 1 epoch
6
GPU V100 32G
1500 GPUh

D.2. Additional results
0.312

pre-trained init

1.395
0.310

ROU GE rewarded
R2: M ET EOR

R2: ROU GEBLEU 4+ROU GE
rewarded2
0.608

pre-trained init

BLEU 4 rewarded

RL: BLEU 4
RL: ROU GEGEMORL: BLEU 4+ROU
2

0.607

0.308

0.417

0.418

0.419

0.420

0.421

M ET EOR rewarded

R1: BLEU 4

BLEU 1 rewarded

0.304

pre-trained init
0.302

0.300

0.422

λ = 0.5

0.306

RS: (1 − λ) · θBLEU 4 + λ · θROU GE

0.606

RL: BLEU 1
RL: M ET EORET EOR
MORL: BLEU 1+M2(a) BLEU4 and ROUGE.

0.424

0.78

0.79

0.80

0.81

0.82

R1: BLEU 1

CIDEr rewarded
1.385

BLEU 1 rewarded

1.380

RL: BLEU 1
RL: CIDEr
MORL: BLEU 1+CIDEr
2

1.375

RS: (1 − λ) · θBLEU 1 + λ · θM ET EOR
0.423

λ = 0.5

1.390

λ = 0.5
0.609

BLEU 1+CIDEr
rewarded
2

BLEU 1+M ET EOR
rewarded2

R2: CIDEr

0.610

0.83

0.84

RS: (1 − λ) · θBLEU 1 + λ · θCIDEr

1.370
0.85

(b) BLEU1 and METEOR.

0.83000.83250.83500.83750.84000.8425

R1: BLEU 1

0.84500.84750.8500(c) BLEU1 and CIDEr.

Figure 8: Additional results in captioning with more rewards, complementing Figure 3. Specifically, Figure 8(a) uses R1 = BLEU 4 andR2 = ROU GE; then, with R1 = BLEU 1, Figure 8(b) uses R2 = M ET EOR and Figure 8(c) uses R2 = CIDEr. In particular, thelatter shows the failure when optimizing CIDEr; indeed, let’s recall that the pre-trained initialization (Hu et al., 2022b) has actually alreadybeen trained by optimizing CIDEr (Rennie et al., 2017). Thus optimizing CIDEr a second time does not help, nor in CIDEr neither inother rewards. That’s why in Figure 3(c) we consider the initialization as the network parametrization optimized for CIDEr.

25

RS: (1 − λ) · θBLEU 1+λ · θM ET EOR

RS: (1 − λ) · θBLEU 1+λ · θBLEU 4

0.8

R1 : BLEU 1
BLEU 2BLEU 3R2 : BLEU 4

0.6

0.4

ROU GEM ET EOR
CIDEr

0.2

CIDEr

1.5

1.0

0.0
0.2

0.4

1.2

1.0

1.4

0.8

0.6

1.2

1.0

0.8

MORL: (1 − µ) × BLEU 1 + µ × ROU GE
R1 : BLEU 1

BLEU 2BLEU 3BLEU 4
R2 : ROU GE

0.6

0.4

0.4

0.2

0.2

0.0

0.0

0.5

0.0
0.0

BLEU 2BLEU 3BLEU 4
ROU GEM ET EOR
R2 : CIDEr

1.4

Normalized rewards

1.0

1.6

R1 : BLEU 1

BLEU 2BLEU 3BLEU 4
ROU GER2 : M ET EOR

2.0

Normalized rewards

Normalized rewards

1.2

RS: (1 − λ) · θBLEU 1+λ · θCIDEr

1.6

R1 : BLEU 1

2.5
1.4

Normalized rewards

1.6

0.6

0.8

1.0

0.0

λ

0.2

0.4

0.6

0.8

λ

(a) R2 = BLEU 4.

(b) R2 = M ET EOR.

1.0

0.0

0.2

0.4

0.6

0.8

1.0

M ET EOR
CIDEr
0.0

0.2

0.4

0.6

λ

0.8

1.0

µ

(c) R2 = CIDEr.

(d) MORL.

Figure 9: Additional results in captioning when measuring performances on all rewards and varying the interpolating coefficients,
complementing Figure 4(b). In Figures 9(a) to 9(c), we extend the results for RS with R1 = BLEU 1 and for varying R2 ; the optimal λdepends on the similarity between the evaluation metric and R1 and R2 . We also see in Figure 9(c) that all rewards are normalized to
1 for the CIDEr-initialization. In Figure 9(d), we perform the same analysis for MORL while varying the weighting µ over the proxyrewards R1 = BLEU 1 and R2 = ROU GE; we recover similar curves than in Figure 4(b) for RS.
0.611

ROU GE rewarded

0.611

BLEU 1+ROU GE
2

rewarded

0.610

0.609

λ = 0.5

0.608

0.607

0.606

BLEU 1 rewarded
WiSE: (1 − λ) · θinit + λ · θBLEU 1

0.605

RS: (1 − λ) · θBLEU 1 + λ · θROU GE

0.825

0.830

0.835

0.840

R1: BLEU 1

pre-trained init

λ = 0.5
betweentwo BLEU 1

0.606

0.603

RS: (1 − λ) · θROU GE + λ · θM ORL

0.603

0.607

0.604

RS: (1 − λ) · θBLEU 1 + λ · θM ORL

ROU GE
rewarded

0.608

0.605

WiSE: (1 − λ) · θinit + λ · θROU GE

0.604

RL: BLEU 1 v1
RL: BLEU 1 v2
MS: (1 − λ) · θBLEU 1v1 + λ · θBLEU 1v2

RL: ROU GE v1
RL: ROU GE v2
MS: (1 − λ) · θROU GEv1 + λ · θROU GEv2

BLEU 1
rewarded

RS fronts: (v1 or v2) → (v1 or v2)

0.845

0.850

0.825

(a) Exploring new strategies.

0.830

0.835

0.840

R1: BLEU 1

0.845

0.850

(b) Results variances.
0.610

0.6100λ = 0.5 between
BLEU 1 and ROU GE

0.609

R2: ROU GE

R2: ROU GE

λ = 0.5
betweentwo ROU GE

0.610

λ = 0.5

λ = 1.3
λ = 0.50.608

0.6075

λ = 1.1 λ = 1.0
λ = 1.5

0.606

pre-trained init

0.6025λ = 0.00.6000λ = −0.1

0.5975λ = −0.3

0.5950λ = −0.5

R2: ROU GE

R2: ROU GE

0.60500.604

BLEU 1+ROU GE
rewarded
2

ROU GE rewarded
pre-trained init

BLEU 1 rewarded

0.602

RL: BLEU 1
RL: ROU GEGEMORL: BLEU 1+ROU
2

0.600

0.59250.598

RS: (1 − λ) · θBLEU 1 + λ · θROU GE

RS: (1 − λ) · θBLEU 1 + λ · θROU GE
0.5900
0.81

0.82

0.83

0.84

R1: BLEU 1(c) λ outside of [0, 1].

0.85

0.86

0.820

0.825

0.830

0.835

0.840

R1: BLEU 1

0.845

0.850

0.855

(d) End-to-end training.

Figure 10: Additional results in captioning with R1 = BLEU 1 and R2 = ROU GE. In Figure 10(a), we investigate interpolating thefine-tuned networks with the pre-trained initialization as in WiSE (Wortsman et al., 2022b); this only reveals a small portion of the front.
In contrast, the interpolation with θM ORL (µ = 0.5) solution improves RS’s front: this highlights some limitations in Hypothesis 2 andstrict Pareto optimality of RS. Adding the MORL solutions as intermediate weights may help interpolate between two weights too distant.
This suggests some practical complementarity between RS and MORL; given a training budget larger than the number of rewards, one
may learn a few MORL for varying 0 ≤ µ ≤ 1, and then interpolate the obtained solutions. Figure 10(b) shows results’ variance with two
RL trainings for BLEU, and two for ROUGE, each time with a different seed defining the data ordering and augmentations. Though weobserve some randomness, the Hypothesis 1 is consistently validated. Moreover, it presents the fronts described when we interpolate
weights fine-tuned on a shared reward, as in model soups (MS) (Wortsman et al., 2022a; Ramé et al., 2022). This also only reveals a small
portion of the spectrum of preferences, validating the need for diverse rewards to reveal the full Pareto front. In Figure 4(c), we report thefront of the costly ensembling (Hansen & Salamon, 1990) of predictions (rather than of weights). Actually, ensembling performs better,but it cannot be fairly compared as its inference cost is doubled. Finally, Figure 10(d) shows the results when the networks are trained
end-to-end, rather than keeping the backbone frozen. This validates the efficiency of rewarded soups in a new more general setting where
all layers are trainable.
26

E. Text-to-image: diffusion models with RLHFs
E.1. Experimental details
Several works have studied the problem of aligning the output of diffusion models with human feedbacks (Lee et al., 2023;
Wu et al., 2023b; Xu et al., 2023). Models are expected to understand specific visual control signals like colors, counts,
and backgrounds more accurately after alignment. Notably, diffusion models can be fine-tuned to match human aesthetics
preferences. As for any subjective metric, there is a variety of reward models that capture different aspects of aesthetic
preference. These models are trained in a supervised setting to match human quality ratings collected on large image datasets(Xu et al., 2023), like the AVA dataset (Murray et al., 2012). As in the previous sections, RS allows to efficiently optimize
multiple aesthetics reward models at test time, which allows adapting to the preferences of a single user.
We consider three metrics as rewards models: The cafe aesthetics model 1 , trained on 3500 real-life and anime/manga
images; An aesthetic score predictor based on CLIP features2 , trained on 250 000 images from the AVA dataset (Murray
et al., 2012); we also experiment with a CLIP-based NSFW detector that estimates the probability of an image being "safe"by computing the cosine similarity with the embeddings of a set of "unsafe" words. The last two reward models are used tofilter the LAION dataset (Schuhmann et al., 2021).
To fine-tune a diffusion model on a reward model R, we first generate 10000 images with the pre-trained diffusion model andcompute the rewards for every generated image. Then, we fine-tune the diffusion model on the reward-weighted negative
log-likelihood (Lee et al., 2023):
L = E(x0 ,Q)∈D,ϵ∼N (0,1),t∼U nif orm(0,T )r(x0 )∥ϵθ (xt , t, Q) − ϵ∥2(19)where ϵθ is the noise estimation network, T is the total number of training steps, r(x0 ) is the reward of image x0 and Q is
the text associated to image x0 .
On-policy Reinforcement Learning would normally require to perform loops of image generation and model fine-tuning(Dong et al., 2023), but we only perform a single optimization loop for simplicity.
Implementation details. We use a 2.2B parameters diffusion model trained on an internal dataset of 300M images, whichreaches similar generation quality as Stable Diffusion (Rombach et al., 2022) in terms of CLIP alignment and FID scores.
For efficient finetuning, we only fine-tune 10% of the diffusion model’s weights (Xie et al., 2023) corresponding to thecross-attention layers and the bias/scaling parameters. For computational efficiency,
Pwe remove the 50% images withthe worse scores, and rescale rewards linearly so that minx0 ∈D′ r(x0 ) = 0 and |D1′ | x0 ∈D′ r(x0 ) = 1. All models are
fine-tuned with Adam (Kingma & Ba, 2015) for 4000 steps with a batch size of 32 and learning rate 5e-6. Fine-tuned
checkpoints and checkpoints interpolated with RS are evaluated on 1000 images.
Table 3: Image generation experiments: key implementation details.

Model
Architecture
Fine-tuning objective
Fine-tuning strategy
Optimizer
DatasetRewards
Learning rate
Batch size
EpochsHardware
Compute budget

1
2

GLIDE
Reward-weighted Diffusion Loss
Fine-tuning cross-attention layers and bias/scale parameters
Adam (Kingma & Ba, 2015)
Generated with COCO prompts
ava (Murray et al., 2012) and cafe
5e-6
64
25
Single GPU V100 32G
500 GPUhavailable at https://huggingface.co/cafeai/cafe_aesthetic
available at https://github.com/christophschuhmann/improved-aesthetic-predictor/

27

E.2. Additional results
We show in Figure 11 the spider map when computing MORL and RS on all three metrics: ava, cafe and the nsfw detector.
In this case, MORL has higher scores than RS on the ava and cafe scores. We speculate that this is because the nsfw is very
different from aesthetics preferences and that it can be inversely correlated with image quality: we have indeed noticed thatlower quality images result in higher scores for the nsfw metric, being less often flagged as unsafe.
ava
5.296

Init
RL: ava
RL: caf e
RL: nsf w
P =3
MORL: Ni=1 Ri /3
P =3
RS: N
i=1 θi /3

5.271

5.245

0.934

0.795

0.942

nsfw

0.808

0.951

0.82

cafe

Figure 11: Image generation: spider map.

F. Text-to-box: visual grounding
F.1. Experimental details
We show the implementation details in Table 4. We use an internal unified model (Wang et al., 2022a; Lu et al., 2022)
which will be released soon. The model is pre-trained solely on public benchmarks, to solve a variety of multimodal
tasks such as VQA, visual grounding and image captioning. It is then fine-tuned on RefCOCO+ dataset for visualgrounding. During the last fine-tuning phase, we complement the cross-entropy loss with an additional REINFORCE(Williams, 1992) term rewarding accuracy when the object is of the considered size. This means that the loss for θSmall is− log(ŷ) + 5 × 1{area(ŷ) is small} × 1AU C(y,ŷ)>0.5 × log(y) for an object with ground-truth box ŷ and prediction y. The
image is discretized into 1000 × 1000 bins before calculating the box areas. The task is illustrated in Figure 12.

Model

Curly hair
Purple shirt woman
Green shirt man

Figure 12: Illustration of the Visual Grounding task. The RS model results from the average of N = 3 weights specialized to detect
respectively small, medium and large objects. The model takes the text (one at a time) as input and outputs the bounding box (i.e., colored
predictions) in the image region described by the text (the ground truths are shown in green). We show an example of small, medium andlarge predictions. The texts and image input are from the validation set of RefCOCO+ (Yu et al., 2016).

28

Table 4: Visual grounding experiments: key implementation details.

Model
Architecture
Visual encoder
Pretraining
Finetuning

Unified Model (ResNet-101+BART (Lewis et al., 2020))
ResNet-101
Cross-Entropy on Public datasets (VQA, VG, Captioning)
Cross-Entropy on RefCOCO+ (Yu et al., 2016)
RL procedureFine-tuning strategy
DatasetRL algorithm
Reward Small
Reward Medium
Reward Large
Optimizer
Learning rate
Batch size
EpochsHardware
Compute budget

end-to-end
RefCOCO+ (Yu et al., 2016)
Cross-entropy + 5× REINFORCE
IoU>0.5 for object with area < 30000
IoU>0.5 for object with 30000 ≤ area < 100000
IoU>0.5 for object with 100000 ≤ area
Adam
3e-5
256
10
8 GPU 60GB
800 GPUhF.2. Additional results
Large rewarded

45.75

Medium rewarded

Small
22.23

12.8

Init

45.50
21.85312.6

R3: Large

R2: M edium

45.25

45.00

Small+M edium
rewarded2
44.75

pre-trained init

44.50

M edium+Large
rewarded2

21.477

12.4

pre-trained init
12.343

12.2

RS: (1 − λ) · θSmall + λ · θM edium

44.25

44.00

P =3
Supervised: N
i=1 θi /3
PN =3
MORL: i=1 Ri/3
P =3
RS: N
i=1 θi /3

RS: (1 − λ) · θM edium + λ · θLarge
Medium rewarded

Small rewarded

44.27712.487Large

44.553

12.63

44.83

Medium

12.0
21.2

21.4

21.6

21.8

22.0

22.2

R1: Small

22.4(a) Small and Medium.

22.6

22.8

43.5

44.0

44.5

R2: M edium

45.0(b) Medium and Large.

45.5(c) CE vs. RL.

Figure 13: Results in visual grounding on RefCOCO+ (Yu et al., 2016). We use REINFORCE (Williams, 1992) to improve directly thenon-differentiable accuracy, i.e., predict boxes with IoU> 0.5 w.r.t. the ground-truth. Trainings are specialized on either small, medium
or large objects. These experiments complement Figures 5(b) and 5(c). Finally, Figure 13(c) compares between cross-entropy (CE)supervised fine-tuning (with Cross-entropy CE) and REINFORCE RL fine-tuning, using RS and MORL.

29

G. Locomotion with diverse engineered rewards
G.1. Experimental details
Setup and task. This experiment consists in fine-tuning a policy that has already learned how to make an humanoid run on
Brax physics engine (Freeman et al., 2021).
Pre-training. We used the Brax implementation of PPO (Schulman et al., 2017) algorithm to pre-train the base policy used
fine-tuning (see Table 5). The goal task used for pre-training is to make a Humanoid run with the default dense reward
implemented in Brax: R = velocity − 0.5 · aTt at . This phase is also used to collected statistics about observations andnormalize them before inputting to the model, which helps training a lot.
Fine-tuning. The pre-trained policy is saved while the value function is discarded. We use the normalization procedure
inherited from the pre-training but freeze it. We keep the same environment. Two reward functions are designed: a risky one
for R1 (t) = velocity and a cautious one where R2 (t) = velocity −·aTt at . We make a grid-search on a few hyperparameters
over 3 seeds (see the values between brackets in Table 5).
Table 5: Locomotion experiments: key implementation details.

PPO Pre-training
Interactions
Reward Scaling
Episode Length
Normalize observations
Unroll Length
Discounting
Learning Rate
Entropy Cost
Number of environments in parallel
Batch Size
Hardware
Runtime per experiment

5e8
1.0
1000
True
10
0.99
5e-5
1e-3
4096
1024
1GPU Tesla V100-SXM2-16GB
80min

PPO Fine-tuning
Interactions
Reward Scaling
Normalize observations
Unroll Length
Discounting
Learning Rate
Entropy Cost
Number of environments in parallel
Batch Size
Hardware
Runtime per experiment

1e8
1.
True
10
{0.97, 0.99, 0.999}(1e-5, 3e-5, 1e-4)
1e-3, 3e-3, 1e-2
4096
1024
1GPU Tesla V100-SXM2-16GB
20min

Model architecture
PolicyArchitecture
Nb of Layers
Hidden Size
Value
Architecture
Nb of Layers
Hidden Size

MLP
6
512
MLP
5
256

30

R2: cautious

G.2. Additional results
4000

4000

4000

3000

3000

3000

2000

2000

2000

1000

1000

1000

0

0

R2: cautious

4000

5000

6000

5000

6000

7000

4000

4000

4000

4000

3000

3000

3000

2000

2000

2000

1000

1000

1000

0

0
4000

R2: cautious

0
4000

5000

6000

5000

6000

7000
4000

3000

3000

3000

2000

2000

2000

1000

1000

1000

0

0
6000

R1: risky

Pre-trained init
RS: (1 − λ) · θrisky + λ · θcautious
4000

4000

5000

6000

0
4000

4000

4000

5000

5000

6000

5000

6000

0
4000

5000

6000

R1: risky4000

R1: risky

Figure 14: Some other runs for the locomotion task when varying the seed / hyperparameters.

31