SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For example, the TL;DR dataset is a previously existing dataset extracted and obtained by [89] that contains preprocessed comments posted on the social network Reddit and hosted on HuggingFace.
Table 1: LLaMA with RLHF experiments: key implementation details.
Model
Architecture	Transformer [75]
Pre-training	LLaMA-7b [44]
Instruction FT	Alpaca [22]
RL procedure
Fine-tuning strategy	LoRA [87]
following Alpaca-LoRA [175]
LoRA alpha	16
LoRA dropout	0.05
following trl-peft [85, 86]
Optimizer	Adam [167]
Learning rate	1.41e-5
Batch size	128
Output length	Uniformly sampled between 16 and 32
RL algorithm	PPO [84]
KL PPO	0.05 for summary tasks else 0.2
Epochs	2 for Reuter summary else 1
Hardware	NVIDIA RTX A6000 49 Go
Compute budget	4000 GPUh
Task name	Reuter summary
Description	Generate a concise and clear summary of newspaper articles from Reuters.
Prompt	“Generate a one-sentence summary of this post.”
Dataset	Reuter news from [88, 176] from news-summary
R1	gpt2-reward-summarization trained here.
R2	bart-faithful-summary-detector [91]
Figure	Figures 1(b) and 2(a)
Task name	Reddit TL;DR summary
Description	Generate a concise and clear summary of posts from Reddit across a variety of topics (subreddits).
Prompt	“Generate a one-sentence summary of this post.”
Dataset	Reddit crawl from the TL;DR dataset [89] from summarize-from-feedback [12]
R1	gpt2-reward-summarization trained here.
R2	bart-faithful-summary-detector [91]
Figure	Figure 2(b)
Task name	Stack Exchange
Description	Answer accurately to technical questions from Stack Exchange.
Prompt	No prompt, only users' questions.
Dataset	Q&A from Stack Exchange [90, 177] from stack-exchange/preferences
R1	reward-model-deberta-v3-base
R2	reward-model-electra-large-discriminator
Figure	Figure 2(c)
Task name	Movie review
Description	Generate movie reviews that accurately describe a movie.
Prompt	“Generate a movie review.”
Dataset	IMDB reviews [178] from IMDB
R1	reward-model-deberta-v3-base
R2	reward-model-electra-large-discriminator
Figure	Figure 2(d)
Task name	Helpful assistant
Description	Provide helpful and harmless answers to potentially complex and sensitive questions.
Prompt	No prompt, only users' questions.
Dataset	Helpfulness and harmlessness datasets [49] from hh-rlhf
R1	reward-model-deberta-v3-large-v2
R2	reward-model-electra-large-discriminator
R3	reward-model-deberta-v3-base-v2
R4	reward-model-deberta-v3-base
Figure	Figures 2(e) and 2(f)
D.2 Inspection of the predictions by the interpolated models
This supplementary material includes multiple visualizations of predictions generated by models with interpolated weights. For example, Table 2 provides generated summaries, that remain grammatically coherent. We include more generated samples on this website.
To quantitatively validate that samples generated by interpolated models do not suffer from reduced quality, we provide evaluations with general-purpose quality metrics. Specifically, in Figure 10, we measure the perplexity (of MLMS [179] or of GPT2 [180]) for summaries generated when  $\lambda$ -interpolating between two LLMs fine-tuned on the two summary rewards. We observe that, by interpolating the weights, we also interpolate the metrics. In conclusion, we confirm quantitatively that RS does not deteriorate quality.
Table 2: Examples of generated summaries about vampire vigilantes in Malawi news, for  $(1 - \lambda)\cdot \theta_{1} + \lambda \cdot \theta_{2}$  interpolating between  $\theta_{1}$  and  $\theta_{2}$  fine-tuned on  $R_{1}$  (evaluating completeness) and  $R_{2}$  (evaluating faithfulness).
λ	Generation
0.0	Malawi vigilante mobs have been killing people suspected of being vampires, prompting the UN and US embassy to declare no-go zones in the affected'
0.1	Malawi vigilante mobs have been killing people suspected of being vampires, prompting the UN and US embassy to declare no-go zones in the affected
0.2	Malawi vigilante mobs have killed nine people since mid-September, prompting the United Nations and the US embassy to declare some parts of the country no-
0.3	Malawi vigilante mobs have killed nine people, prompting the UN and US embassy to declare parts of the country no-go zones due to widespread
0.4	Malawi vigilante mobs have killed nine people, prompting the UN and US embassy to declare some parts of the country no-go zones.
0.5	Malawi vigilante mobs have arrested and killed suspected vampires, prompting the UN and US embassy to declare no-go zones and President Peter Muth
0.6	Malawi vigilante mobs have arrested and killed suspected vampires, prompting the UN and US embassy to declare no-go zones.
0.7	Malawi vigilante mobs have arrested suspected vampires, resulting in deaths and prompting the UN and US embassy to declare no-go zones.
0.8	Malawi vigilante mobs have arrested suspected vampires, resulting in deaths and violence.
0.9	Malawi vigilante violence has caused widespread panic and death, prompting authorities to arrest suspected members and investigate the belief in vampirism.
1.0	Malawi vigilante violence has caused widespread panic and death.
Figure 10: News summaries generated by  $\lambda$ -interpolated LLMs evaluated in terms of perplexity (by MLMS [179] or by GPT2 [180]) or news quality by this classifier.
E Image-to-text: captioning with diverse statistical rewards
E.1 Experimental details
We summarize the key implementation details of our captioning experiments in Table 3. In short, we took the state-of-the-art network [96] for captioning on COCO, fine-tuned with their code and only changed the reward. In more details, since the self-critical paper [24] (a variant of REINFORCE [98] with a specific estimation of the baseline score) it is now common in captioning to optimize the CIDEr reward [31] after a first step of supervised fine-training. The recent ExpansionNetv2 [96] follows this strategy to reach state-of-the-art results, with a Swin Transformer [97] visual encoder and a block static expansion for efficiency. We investigate whether additional RL trainings on the other traditional statistical metrics can help. We use the code from [96] and their hyperparameters, only reducing the batch size from 24 to 18 to fit in our GPUs and consequently adapting the learning rate.
Table 3: Captioning experiments: key implementation details.
Model
Architecture	ExpansionNetv2 [96]
Visual encoder	Swin Transformer [97]
Visual encoder pre-training	ImageNet 22k [181]
Fine-tuning	Cross-entropy then CIDEr RL [24] on COCO [94]
RL procedure
Fine-tuning strategy	Usually frozen visual backbone, but end-to-end in Figure 13(d)
RL algorithm	Self-critical [24], a variant of REINFORCE [98]
Optimizer	Radam [182]
Dataset	COCO [94] and Karpathy split [99]
Rewards	BLEU [29] (with 1-gram or 4-grams), ROUGE [30], METEOR [95], CIDEr [31]
Learning rate	1e-5
Batch size	18
Gradient accumulation	2
Warmup	Anneal 0.8 during 1 epoch
Epochs	6
Hardware	GPU V100 32G
Compute budget	1500 GPUh
(a) BLEU4 and ROUGE.
(b) BLEU1 and METEOR.
(c) BLEU1 and CIDEr.
Figure 11: Additional results in captioning with more rewards, complementing Figure 3. Specifically, Figure 11(a) uses  $R_{1} = \text{BLEU4}$  and  $R_{2} = \text{ROUGE}$ ; then, with  $R_{1} = \text{BLEU1}$ , Figure 11(b) uses  $R_{2} = \text{METEOR}$  and Figure 11(c) uses  $R_{2} = \text{CIDEr}$ . In particular, the latter shows the failure when optimizing CIDEr; indeed, let's recall that the pre-trained initialization [96] has already been trained by optimizing CIDEr [24]. Thus optimizing CIDEr a second time does not help, neither in CIDEr nor in other rewards. That's why in Figure 3(c) we consider the initialization as the network parametrization optimized for CIDEr.
(a)  $R_{2} = BLEU4$
(b)  $R_{2} = M E T E O R.$
(c)  $R_{2} = CIDEr$
(d) MORL.
Figure 12: Additional results in captioning when measuring performances on all rewards and varying the interpolating coefficients, complementing Figure 5(b). In Figures 12(a) to 12(c), we extend the results for RS with  $R_{1} = BLEU1$  and for varying  $R_{2}$ ; the optimal  $\lambda$  depends on the similarity between the evaluation metric and  $R_{1}$  and  $R_{2}$ . We also see in Figure 12(c) that all rewards are normalized to 1 for the CIDEr-initialization. In Figure 12(d), we perform the same analysis for MORL while varying the weighting  $\mu$  over the proxy rewards  $R_{1} = BLEU1$  and  $R_{2} = ROUGE$ ; we recover similar curves than in Figure 5(b) for RS.
(a) Exploring new WI strategies.
(b) Results variances and model soups (MS).
(c) Extrapolation with  $\lambda$  outside of [0, 1].
Figure 13: Additional results in captioning with  $R_{1} = BLEU_{1}$  and  $R_{2} =ROUGE$ . In Figure 13(a), we investigate interpolating the fine-tuned networks with the pre-trained initialization as in WiSE [183]; this only reveals a small portion of the front. In contrast, the interpolation with  $\theta_{MORL}(\mu = 0.5)$  solution improves RS's front: this highlights some limitations in Hypothesis 2 and strict Pareto optimality of RS. Adding the MORL solutions as intermediate weights may help interpolate between two weights too distant. This suggests some practical complementarity between RS and MORL; given a training budget larger than the number of rewards, one may learn a few MORL for varying  $0 \leq \mu \leq 1$ , and then interpolate the obtained solutions. Figure 13(b) shows results' variance with two RL trainings for BLEU1, and two for ROUGE, each time with a different seed defining the data ordering and augmentations. Though we observe some randomness, the Hypothesis 1 is consistently validated. Moreover, it presents the fronts described when we interpolate weights fine-tuned on a shared reward, as in model soups (MS) [67, 68]; it mostly reduces variance and reveals only a small portion of the spectrum of preferences, validating the need to fine-tune on different rewards (as proposed in RS) to reveal the front across the entire space of preferences. Finally, the orange line shows that RS and MS can be complementary, by  $\lambda$ -interpolating the MS for BLEU1 and the MS for ROUGE with  $\kappa = 0.5$ . Figure 13(c) presents the extrapolation results when  $\lambda$  goes outside of [0, 1]. This suggests that we can artificially reduce a reward with negative coefficients, as studied in [141]. Finally, Figure 13(d) shows the results when the networks are trained end-to-end, rather than keeping the backbone frozen. This validates the efficiency of rewarded soups in a new more general setting where all layers are trainable.
(d) End-to-end training.
F Text-to-image: diffusion models with diverse RLHFs
F.1 Experimental details
Task description. Several works have studied the problem of aligning the output of diffusion models with human feedbacks [25, 26, 33]. Notably, diffusion models can be fine-tuned to match human aesthetic perception. As for any subjective metric, there is a variety of reward models capturing different aesthetics. In our experiments, the two first reward models were trained in a supervised setting to match human quality ratings collected on large image datasets. Specifically, the first  $R_{1}$  is the ave aesthetic model, available here, trained on 250.000 images from the AVA dataset [103], based on CLIP features. The second  $R_{2}$  is the cafe aesthetic model, available here, trained on 3500 real-life and anime/manga images. Moreover, in Figure 14, we also consider a nsfw detector, estimating the probability of an image being safe by computing the cosine similarity with the CLIP embeddings of a set of unsafe words, as already done to filter the LAION dataset [184].
Implementation details. We use a 2.2B parameters diffusion model trained on an internal dataset of 300M images, which reaches similar generation quality as Stable Diffusion [102] in terms of CLIP alignment and FID scores on prompts from the 5000 images of the COCO test dataset (CLIPScore 30.0 vs 30.2 for Stable Diffusion, FID 19.0 vs 19.1 for Stable Diffusion). Given a reward model  $R$ , we first generate 10000 images with the pre-trained diffusion model on prompts from the COCO dataset, and compute the rewards for every generated image. For computational efficiency, we keep only a dataset  $\mathcal{D}'$  containing the  $50\%$  images with the best scores, and rescale rewards  $R$  linearly into  $r$  so that  $\min_{\mathbf{x}_0 \in \mathcal{D}'} r(x_0) = 0$  and  $\frac{1}{|\mathcal{D}'|} \sum_{\mathbf{x}_0 \in \mathcal{D}'} r(x_0) = 1$ . Then, we fine-tune the diffusion model on the reward-weighted negative log-likelihood [25]:
$$
\mathcal {L} = \mathbb {E} _ {\left(\mathbf {x} _ {0}, Q\right) \in \mathcal {D}, \epsilon \sim \mathcal {N} (0, 1), t \sim \text {U n i f o r m} (0, T)} \quad r \left(\mathbf {x} _ {0}\right) \times \| \epsilon_ {\theta} \left(\mathbf {x} _ {t}, t, Q\right) - \epsilon \| ^ {2}, \tag {19}
$$
where  $\epsilon_{\theta}$  is the noise estimation network,  $T$  is the total number of training steps,  $r(\mathbf{x}_0)$  is the rescaled reward of image  $\mathbf{x}_0$  and  $Q$  is the text associated to image  $\mathbf{x}_0$ . As a side note, on-policy RL would require performing loops of image generations and model fine-tunings [185], but we only perform a single offline iteration for simplicity. Moreover, for efficiency, we only fine-tune  $10\%$  of the diffusion model's weights [104] corresponding to the cross-attention layers and the bias/scaling parameters. As further described in Table 4, we apply the Adam [167] optimizer for 4000 steps with a batch size of 64 and a learning rate of 5e-6. To report results for each model (fine-tuned or interpolated via RS), we generate 1000 images from a held-out set of COCO prompts and then we average the scores given by the reward models. To reduce the variance in image generation, each prompt has a unique seed for all models, so that the input noise given to the diffusion model only depends on the text prompt.
Table 4: Image generation experiments: key implementation details.
Model
Architecture	GLIDE (2.2B parameters)
Pre-training	Internal dataset of 300M captioned images
RL Procedure
Fine-tuning objective	Reward-weighted diffusion loss
Fine-tuned parameters	Cross-attention layers and bias/scale
Optimizer	Adam [167]
Dataset	Generated with COCO prompts
Rewards	ava [103] and cafe and nsfw
Learning rate	5e-6
Batch size	64
Epochs	25
Hardware	Single GPU V100 32G
Compute budget	500 GPUh
RS can trade-off between the two aesthetic rewards in Figure 6(a), allowing adaptation to the user's preferences at test time. Yet, we show some limitations in the spider map of Figure 14, when
computing MORL and RS on all three rewards: ava, cafe and also the nsfw. In this case, MORL has higher scores than RS. We speculate this is because the nsfw is very different from aesthetic preferences. Actually, the nsfw is inversely correlated with image quality: lower quality images result are less flagged as unsafe. This shows some limitations of weight interpolation when combining antagonist rewards. An improved strategy would first learn the MORL of the  $N = 3$  rewards, and then optimize each reward independently from this improved initialization, before applying RS.
Figure 14: Image generation: spider map, with ava, cafe and nsfw reward models.
F.3 Visualization of generated images from interpolated models
We show in Appendix F.3 images generated by rewarded soups when varying the interpolation coefficient  $\lambda$  between the two models fine-tuned for the ave and the cafe aesthetic rewards. You can find additional qualitative results for this experiment on this website.
Moreover, in Appendix F.3, we measure the FID [186] and the CLIPScore [187] of the images generated by the same interpolated models. This confirms quantitatively that images generated by interpolated models remain coherent.
A family room with wood floor and beige walls and a mattress leaning against a stone wall.
A man sitting on top of a chair holding up a cell phone.
A man sitting underneath an umbrella and other structures.
Figure 15: Visualization of images generated with rewarded soups for a varying interpolation coefficient  $\lambda$  between the two models fine-tuned for the ave (corresponding to  $\lambda = 0$ ) and cafe (corresponding to  $\lambda = 1$ ) reward models. We can see that all interpolated models produce images of similar quality compared to fine-tuned models, demonstrating linear mode connectivity between the two fine-tuned models.
Figure 16: Images generated by  $\lambda$ -interpolated diffusion models evaluated in terms of realism by FID [186] or text alignment by CLIPScore [187].
G Text-to-box: visual grounding of objects with diverse sizes
G.1 Experimental details
We show the implementation details in Table 5. We use UnIVAL [106], a model pre-trained solely on public benchmarks, to solve a variety of multimodal tasks such as VQA, visual grounding and image captioning. It is then fine-tuned on RefCOCO+ dataset for visual grounding. During the last fine-tuning phase, we complement the cross-entropy loss with an additional REINFORCE [98] term rewarding accuracy when the object is of the considered size. This means that the loss for  $\theta_{Small}$  is  $-\left(\log (\hat{y}) + 5\times 1_{\{\text{area} (\hat{y})\text{is small}\}}\times 1_{AUC(y,\hat{y}) > 0.5}\times \log (y)\right)$  for an object with ground-truth box  $\hat{y}$  and prediction  $y$ . The image is discretized into  $1000\times 1000$  bins before calculating the box areas. The task is illustrated in Figure 17.
Figure 17: Illustration of the visual grounding task. The RS model results from the average of  $N = 3$  weights specialized to detect respectively small, medium and large objects. The model takes a text (one description at a time) as input and outputs the bounding box in the corresponding region of the image. We show an example of small, medium and large predictions, and the associated ground truths in green. These texts and images are from the validation set of RefCOCO+ [105].
Table 5: Visual grounding experiments: key implementation details.
Model
Architecture	UnIVAL [106]
Visual encoder	ResNet-101
Pre-training	Cross-Entropy on Public datasets (VQA, VG, Captioning)
Supervised fine-tuning	Cross-Entropy on RefCOCO+ [105]
RL procedure
Fine-tuning strategy	end-to-end
Dataset	RefCOCO+ [105]
RL algorithm	Cross-entropy + 5× REINFORCE
Reward Small	IoU>0.5 for object with area < 30000
Reward Medium	IoU>0.5 for object with 30000 ≤ area < 100000
Reward Large	IoU>0.5 for object with 100000 ≤ area
Optimizer	Adam
Learning rate	3e-5
Batch size	256
Epochs	10
Hardware	8 GPU 60GB
Compute budget	800 GPUh
G.2 Additional results
(a) Small and Medium.
(b) Medium and Large.
(c) Spider map.
Figure 18: Results in visual grounding on RefCOCO+ [105]. We use REINFORCE [98] to improve directly the non-differentiable accuracy, i.e., predict boxes with IoU  $>0.5$  w.r.t. the ground-truth. Fine-tunings are specialized on either small, medium, or large objects. These experiments complement Figures 6(b) and 18(c). Figure 18(c) shows that improving results on all sizes simultaneously is challenging, as MORL performs similarly to the initialization. Finally, Figure 18(d) motivates the use of RL to fine-tune on different sizes. Indeed, the results for (the proposed) RS of RL are significantly better than the results for RS of CE, where we average weights specialized on different sizes by fine-tuning with cross-entropy (rather than with REINFORCE).
(d) CE vs. RL.
H Locomotion with diverse engineered rewards
Task description. This experiment takes on the intricate challenge of controlling a running humanoid in the Brax [113] physics engine. The complexities involved in achieving natural or fast movement in continuous control environments serve as a testament to the robustness of our approach. The fine-tuning procedure is carried out on two distinct reward functions, with the aim of refining the running behavior of the humanoid, potentially resulting in smoother motion patterns. You can find qualitative results of this experiment on this website.
Pre-training. According to Remark 1, the LMC requires pre-training the base policy before finetuning. Thus, as the pre-training task, we use the default dense reward implemented in Brax:  $R = \text{velocity} - 0.1 \times \sum_{t} a_{t}^{2}$ . This pre-training phase also serves to collect statistics about observations and normalize them before inputting to the model (as it facilitates training). We used the Brax implementation of PPO [84]. The pre-trained policy is saved while the value function is discarded.
Fine-tuning. We keep the same environment as in pre-training. We also use the normalization procedure inherited from pre-training but freeze the statistics. Two reward functions are designed: a risky one for  $R_{1} = \text{velocity}$  and a cautious one where  $R_{2} = \text{velocity} - \sum_{t} a_{t}^{2}$ . We tried a few hyperparameters (see the values in brackets in Table 6) but results (see Figure 19) remain close and consistently validate our working hypotheses.
Table 6: Locomotion experiments: key implementation details.
PPO Pre-training
Interactions	5e8
Reward Scaling	1.0
Episode Length	1000
Unroll Length	10
Discounting	0.99
Learning Rate	5e-5
Entropy Cost	1e-3
Number of environments in parallel	4096
Batch Size	1024
Hardware	1GPU Tesla V100-SXM2-16GB
Runtime per experiment	80min
PPO Fine-tuning
Interactions	1e8
Reward Scaling	1.
Normalize observations	True
Unroll Length	10
Discounting	{0.97, 0.99, 0.999}
Learning Rate	{1e-5, 3e-5, 1e-4}
Entropy Cost	{1e-3, 3e-3, 1e-2}
Number of environments in parallel	4096
Batch Size	1024
Hardware	1GPU Tesla V100-SXM2-16GB
Runtime per experiment	20min
Model architecture
Policy
Architecture	MLP
Nb of Layers	6
Hidden Size	512
Value
Architecture	MLP
Nb of Layers	5
Hidden Size	256
Figure 19: Analysis of results' variance for the locomotion task when varying the hyperparameters. Each column  $i$  corresponds to the  $i$ -th  $\theta_{\text{risky}}$ , interpolated in case  $(i, j)$  towards the  $j$ -th  $\theta_{\text{cautious}}$ . The Figure 6(c) is actually the plot from case  $(1, 1)$ .