SlowGuess/NeurIPS_2023_OCR huggingface dataset:
captions is challenging, thus the literature relies on various non-differentiable metrics: e.g., the precision-focused BLEU [29], the recall-focused ROUGE [30], METEOR [95] handling synonyms and CIDEr [31] using TF-IDF. As these metrics are proxies for human preferences, good trade-offs are desirable. We conduct our experiments on COCO [94], with an ExpansionNetv2 [96] network and a Swin Transformer [97] visual encoder, initialized from the state-of-the-art weights of [96] optimized on CIDEr. We then utilize the code of [96] and their self-critical [24] procedure (a variant of REINFORCE [98]) to reward the network on BLEU1, BLEU4, ROUGE or METEOR. More details and results can be found in Appendix E.
We observe in Figure 3 that tuning solely BLEU1 sacrifices some points on ROUGE or BLEU4. Yet interpolating between  $\theta_{1}$  and  $\theta_{2}$  uncovers a convex set of solutions approximating the ones obtained through scalarization of the rewards in MORL. When comparing both full fronts in Figure 3(a), they qualitatively have the same shape, and quantitatively the same hypervolume [93] of 0.140. One of the strengths of RS is its ability to scale to any number of rewards. In Figure 3(c), we uniformly  $(\lambda_{i} = \frac{1}{5})$  average  $N = 5$  weights fine-tuned independently. It improves upon the initialization [96] and current state-of-the-art on all metrics, except for CIDEr, on which [96] was explicitly optimized. We confirm in Figure 4 that RS can handle more than 2 rewards through additional spider maps. Specifically, we compare the performances across all  $N = 5$  metrics when averaging  $1 \leq M \leq N$  networks (each fine-tuned on one of the  $N$  rewards, thus leaving out  $N - M$  rewards at training) and sequentially adding more networks to the weight average. We consistently observe that adding one additional network specialized on one additional reward extends the scope of the possible rewards that RS can tackle Pareto-optimally. Figure 5(a) validates Lemma 1: for any linear preference  $\hat{\mu}$  over the proxy rewards, there exists an optimal solution in the set described by RS. Two empirical strategies to set the value of  $\lambda$  are close to optimal: selecting  $\lambda = \hat{\mu}$  if  $\hat{\mu}$  is known, or cross-validating (CV)  $\lambda$  if a different data split [99] is available. Moreover, Figure 5(b) (and Appendix E) investigate all metrics as evaluation. Excluding results' variance, we observe monotonicity in both training rewards, linear in BLEU1 and quadratic in ROUGE. For other evaluation rewards that cannot be linearly expressed over the training rewards, the curves' concavity shows that RS consistently improves the endpoints, thereby mitigating reward misspecification. The optimal  $\lambda$  depends on the similarity between the evaluation and training rewards: e.g., best BLEU2 are with small  $\lambda$ . Lastly, as per [100] and Lemma 4, Figure 5(c) suggests that RS succeeds because WI approximates prediction ensembling [78, 79] when weights remain close, interpolating the predictions rather than the weights. Actually, ensembling performs better, but it cannot be fairly compared as its inference cost is doubled.
3.3 Text-to-image: diffusion models with diverse RLHFs
Beyond text generation, we now apply RS to align text-to-image generation with human feedbacks [25, 26, 33]. Our network is a diffusion model [101] with 2.2B parameters, pre-trained on an internal dataset of 300M images; it reaches similar quality as Stable Diffusion [102], which was not used for copyright reasons. To represent the subjectivity of human aesthetics, we employ  $N = 2$  open-source
reward models: ava, trained on the AVA dataset [103], and cafe, trained on a mix of real-life and manga images. We first generate 10000 images; then, for each reward, we remove half of the images with the lowest reward's score and fine-tune  $10\%$  of the parameters [104] on the reward-weighted negative log-likelihood [25]. Details and generations for visual inspection are in Appendix F. The results displayed in Figure 6(a) validate Hypothesis 1, as the front described by RS when sliding  $\lambda$  from 0 and 1 is convex. Moreover, RS gives a better front than MORL, validating Hypothesis 2. Interestingly, the ava reward model seems to be more general-purpose than cafe, as RL training on ava also enhances the scores of cafe. In contrast, the model  $\theta_{cafe}$  performs poorly in terms of ava in Figure 6(a). Nonetheless, RS with  $(1 - \lambda)\cdot \theta_{ava} + \lambda \cdot \theta_{cafe}$  outperforms  $\theta_{ava}$  alone, not only in terms of cafe, but also of ava when  $\lambda \in \{0.1,0.2\}$ . These findings confirm that RS can better align text-to-image models with a variety of aesthetic preferences. This ability to adapt at test time paves the way for a new form of user interaction with text-to-image models, beyond prompt engineering.
(a) Image generation: ava and cafe.
(b) VG: Small and Large.
(c) Locomotion: risky and cautious.
Figure 6: Figure 6(a) reports our RLHF experiments on text-to-image generation with diffusion models. From the pre-trained initialization, we learn  $\theta_{ava}$  and  $\theta_{cafe}$  by optimizing the two reward models  $ava$  and  $cafe$ . Interpolation between them reveals the green Pareto-optimal front, above the yellow MORL front. Figure 6(b) report our results in visual grounding (VG) on RefCOCO+ [105], where we optimize to predict boxes with IoU>0.5 w.r.t. the ground-truth, for objects of either small, medium or large size. Finally, Figure 6(c) report our results from Section 3.5 for the locomotion task with humanoids.
3.4 Text-to-box: visual grounding of objects with diverse sizes
We now consider visual grounding (VG) [105]: the task is to predict the bounding box of the region described by an input text. We use UnIVAL [106], a seq-to-seq model that predicts the box as a sequence of location tokens [107]. This model is pre-trained on a large image-text dataset, then fine-tuned with cross-entropy for VG; finally, we use a weighted loss between the cross-entropy and REINFORCE in the RL stage. As the main evaluation metric for VG is the accuracy (i.e., intersection over union  $(\mathrm{IoU}) > 0.5$ ), we consider 3 non-differentiable rewards: the accuracy on small, medium, and large objects. We design this experiment because improving results on all sizes simultaneously is challenging, as shown in Figure 18(c), where MORL performs similarly to the initialization. The results in Figure 6(b) confirm that optimizing for small objects degrades performance on large ones; fortunately, interpolating can trade-off. In conclusion, we can adapt to users' preferences at test time by adjusting  $\lambda$ , which in turn changes the object sizes that the model effectively handles. On the one hand, if focusing on distant and small objects, a large coefficient should be assigned to  $\theta_{Small}$ . On the other hand, to perform well across all sizes, we can recover initialization's performances by averaging uniformly (in Figure 18(c)). More details are in Appendix G.
3.5 Locomotion with diverse engineered rewards
Teaching humanoids to walk in a human-like manner [108] serves as a benchmark to evaluate RL strategies [109] for continuous control. One of the main challenges is to shape a suitable proxy reward [110, 111], given the intricate coordination and balance involved in human locomotion. It is standard [112] to consider dense rewards of the form  $R = \text{velocity} - \alpha \times \sum_{t} a_{t}^{2}$ , controlling the agent's velocity while regularizing the actions  $\{a_{t}\}_{t}$  taken over time. Yet, the penalty coefficient  $\alpha$  is challenging to set. To address this, we devised two rewards in the Brax physics engine [113]: a risky  $R_{1}$  with  $\alpha = 0$ , and a more cautious  $R_{2}$  with  $\alpha = 1$ . Like in all previous tasks, RS's front in
Figure 6(c) exceeds the interpolated rewards, as per Hypothesis 1. Moreover, the front defined by RS indicates an effective balance between risk-taking and cautiousness, providing empirical support for Hypothesis 2, although MORL with  $\mu = 0.5$  (i.e.,  $\alpha = 0.5$ ) slightly surpasses RS's front. We provide animations of our RL agent's locomotion on our website, and more details are in Appendix H.
3.6 Efficiency gain of RS over MORL
The efficiency gain of RS versus MORL is by design; when considering 2 rewards, RS only requires 2 fine-tunings, while MORL actually requires an infinite number of fine-tunings to reveal the entire front of preferences. To end this experimental section, we quantify this efficiency gain by introducing in Figure 7 the expected reward  $\mathbb{E}_{\hat{\mu}\sim Unif(0,1)}\hat{R}_{\hat{\mu}}$  where  $\hat{R}_{\hat{\mu}} = (1 - \hat{\mu})\times R_1 + \hat{\mu}\times R_2$  and the expectation is over all the possible user's preferences  $\hat{\mu}$ . We then measure the difference between the expected rewards for RS (with 2 runs) and MORL (with  $M$  runs). Plotting this expected reward advantage for different values of  $M$  shows that MORL needs  $M\gg 2$  to match RS.
(b) Reddit summary.
Figure 7: Expected reward advantage of RS (always requiring only 2 trainings) over MORL (with  $M$  trainings), defined as  $\mathbb{E}_{\hat{\mu} \sim Unif(0,1)}\left[max_{\lambda \in \Lambda}\hat{R}_{\hat{\mu}}(\theta_{\lambda}^{RS}) - \mathbb{E}_{\Lambda_M}\left[max_{\mu \in \Lambda_M}\hat{R}_{\hat{\mu}}(\theta_{\mu}^{MORL})\right]\right]$ , where  $\hat{R}_{\hat{\mu}} = (1 - \hat{\mu}) \times R_1 + \hat{\mu} \times R_2$  is the user reward for user linear preference  $\hat{\mu}$  sampled uniformly between 0 and 1,  $\Lambda = \{0,0.1,\dots,1.0\}$  is the set of the 11 possible values for  $\lambda$ , and where the expectation for the MORL term is over the  $\binom{11}{M}$  possible combinations  $\Lambda_M$  of  $M$  elements from  $\Lambda$  (representing the  $M$  linear weightings  $\mu$  used for MORL training). We observe that MORL matches RS only for  $M$  sufficiently big.
(c) Captioning.
Our RS approach leans on two key components from traditional DRL. The first is proxy rewards, whose design is challenging. Statistical metrics (the standard in captioning [24]) are not practical to measure human concepts [32] such as helpfulness [49, 76]. Thus recent RLHF works [12, 13, 15] leverage human comparison of prediction to learn a reward model. Second, RS relies on existing RL algorithms to maximize the given rewards. RS succeeds with variants of two of the most common, REINFORCE [98] and PPO [84], suggesting it could be applied to others [114, 115]. When dealing with multiple objectives in deep learning, the common strategy is to combine them into a single reward [59, 60]. For example, [116] sum the predictions of a preference RM (as a proxy for helpfulness) and a rule RM (detecting rules breaking); [62] assign different weightings to the relevance/factuality/completeness rewards, thereby customizing how detailed and lengthy the LLMs responses should be. Yet, those single-policy approaches (optimizing over a single set of linear preferences) force a priori and uncertain decisions about the required trade-offs [52, 54], as further detailed in Appendix A.1. The multi-policy alternatives [45, 46, 57, 58, 61] are not suitable because of the computational costs required to learn set of policies. To reduce the cost, [117, 118, 119, 120] build experts and then train a new model to combine them; [121, 122, 123] share weights across experts; [124, 125, 126, 127] directly train a single model; the recent and more similar work [128] learns one linear embedding per (locomotion) task. Yet, all those works were developed for academic benchmarks [112, 129]; moreover, in terms of Pareto-optimality, they perform equal or worse than the linearized MORL. As far as we know, the only approaches that might improve performances are those inspired from the multitask literature [130], tackling gradients conflicts [131, 132] or different variance scales [133, 134] across tasks. Though they succeed for games such as ATARI [135], our attempts to apply [131] in our setups failed. Overall, as previous MORL works modify the training procedure and usually introduce specific hyperparameters, adapting them to RLHF for foundation
models with PPO is complex; in contrast, RS can be used on top of any RLHF system. Finally, performance and simplicity are not the only advantages of RS over other MORL approaches; in brief, and as discussed in Appendix A.2, RS is compatible with the iterative alignment process.
Recent works extended the linear mode connectivity when fine-tuning on different tasks [70, 71, 72, 136], modalities [106] or losses [68, 137], while [138] highlighted some failures in text classification. In contrast, we investigate the LMC in RL. The most similar works are for control system tasks: [139] averaging decision transformers and [140] explicitly enforcing connectivity in subspaces of policies trained from scratch on a single reward. When the LMC holds, combining networks in weights combines their abilities [141, 142]; e.g., averaging an English summarizer and an English-to-French translator can summarize in French [143]. In domain generalization, [67, 68, 144] showed that WI reduces model misspecification [145]; by analogy, we show that RS reduces reward misspecification.
5 Discussion: limitations and societal impacts
The recent and rapid scaling of networks presents both opportunities and major concerns [9, 146, 147]. Our approach is a step towards better empirical alignment [10, 11]. Yet, many challenges remain untackled. First, proxy rewards may lack robustness [148] or be hacked [149] via adversarial exploitation, making them unreliable. Second, overfitting during training may lead to poor generalization, with a risk of goal misgeneralization [150, 151]. RS could alleviate the impact of some badly shaped proxy rewards and some failed optimizations, as well as tackling Goodhart's law [152]. Yet, without constraint on the test distribution, complete alignment may be impossible [153], for example for LLMs with prompts of arbitrary (long) length.
Theoretical guarantees for alignment are also needed [154]. Yet, RS (as all weight interpolation strategies) relies on an empirical finding: the LMC [65], which currently lacks full theoretical guarantees, even in the simplest case of moving averages [100]. That's why we state explicitly our Working Hypotheses 1 and 2 in Section 2.2. Nonetheless, we want to point out that in Appendix C.2 we provide theoretical guarantees for the near-optimality of RS when considering quadratic rewards; specifically, in Lemma 3, we bound the reward difference between the optimal policy and our interpolated policy. A remaining limitation is that we theoretically fix issues only for  $\hat{R}$  linear over the proxy rewards. Such linearization follows the linear utility functions setup from the MORL literature [60], that cannot encapsulate all types of (human) preferences [56, 77]. Nonetheless, we showed in Figures 5(b) and 12 that RS improves results even when  $\hat{R}$  is not linear. We may further improve results by continually training on new and diverse proxy rewards, to capture the essential aspects of all possible rewards, such that their linear mixtures have increasingly good coverage.
Finally, our a posteriori alignment with users facilitates personalization [155] of models. As discussed in Appendix A.1 and in [52], this could increase usefulness by providing tailored generation, notably to under-represented groups. Moreover, the distributed nature of RS makes it parallelizable thus practical in a federated learning setup [156] where data must remain private. Yet, this personalization comes with risks for individuals of "reinforcing their biases [...] and narrowing their information diet"[52]. This may worsen the polarization of the public sphere. Under these concerns, we concur with the notion of "personalization within bounds" [52], with these boundaries potentially set by weights fine-tuned on diverse and carefully inspected rewards.
Conclusion. As AI systems are increasingly applied to crucial real-world tasks, there is a pressing issue to align them to our specific and diverse needs, while making the process more transparent and limiting the cultural hegemony of a few individuals. In this paper, we proposed rewarded soup, a strategy that efficiently yields Pareto-optimal solutions through weight interpolation after training. Our experiments have consistently validated our working hypotheses for various significant large-scale learning tasks, demonstrating that rewarded soup can mitigate reward misspecification. We hope to inspire further research in exploring how the generalization literature in deep learning can help for alignment, to create AIs handling the diversity of opinions, and benefit society as a whole.
This work was granted access to the HPC resources of IDRIS under the allocations AD011011953R1 and A0100612449 made by GENCI. Sorbonne Université acknowledges the financial support by the ANR agency in the chair VISA-DEEP (ANR-20-CHIA-0022-01).
[186] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS, 2017. (pp. 36 and 37)
[187] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: a reference-free evaluation metric for image captioning. In EMNLP, 2021. (pp. 36 and 37)
This supplementary material is organized as follows:
- Appendix A further discusses the practical benefits of rewarded soups.
- Appendix B anticipates questions that might arise from readers.
- Appendix C details some theoretical guarantees.
- Appendix D details our text-to-text generation experiments.
- Appendix E enriches our image captioning experiments.
- Appendix F enriches our image generation experiments.
- Appendix G enriches our visual grounding experiments.
- Appendix H enriches our locomotion experiments.
The shareable code is released on github. Moreover, you can find additional qualitative results of our experiments on this website.
In this section we discuss the benefits of our rewarded soup (RS) approach with respect to the two families of strategies: the single-policy and the multi-policy approaches.
A.1 Compared to single-policy approaches
The main reason why single-policy approaches are not suitable is because they optimize over a single set of preferences. In contrast, we build a coverage set of Pareto-optimal policies. This is important for the following reasons, mostly first discussed in Kirk et al. [52] and in Hayes et al. [54].
Indeed, the user's true reward is highly uncertain before training. This "semi-blind" [54] manual process forces a priori and uncertain decisions about the required trade-offs. It shifts the responsibility from the problem stakeholders to the system engineers, who need to anticipate the impact of their choices on the final performance. Critically, the RLHF process may cause the "tyranny of the crowdworker" [52], as models are "tailored to meet the expectations of [...] a small number of crowdworkers primarily based in the US, with little to no representation of broader human cultures, geographies or languages." [52]. Moreover, biased are caused by chaotic engineering choices, and "are exacerbated by a lack of [...] documentation" [52]. In contrast, our approach makes personalization explicit, as argued by [52]. Moreover, we could support decision-making to find a good balance between (potentially conflicting) parties' interests. This value pluralism [157] can lead to fairer and more equitable outcomes [56, 158]. Single-policy cannot adapt to test time requirements; in contrast, RS facilitates personalized assistance [155]. This is all the more important as human preferences change from time to time. In this dynamic utility function scenario, RS can quickly adapt with fewer data, by simply adjusting the  $\lambda$  to match new preferences (rather than the full network). Finally, RS could also improve the interpretability and explainability of the decisions. Letting the users decide would make the process more transparent [159], which is essential to ensure that the development process is fair, unbiased, and inclusive [160].
A.2 Compared to multi-policy approaches
The main reason why existing multi-policy approaches through multitasking are not suitable is because of their computational costs required to learn a dense set of policies. In contrast, RS only trains the proxy rewards independently and enables the selection of the interpolating coefficient
a posteriori. This is especially useful with large number of rewards and thus growing number of combinations. Second, multitask [130] is challenging; for example, even if the true reward is actually a linear weighted sum of some proxy rewards and those coefficients are known, using those preferences during training can lead to suboptimal results [161], because of conflicting gradients [131, 132] or different variance scales [133, 134]. This has been tackled in RL, but so far mostly for games such as ATARI [135]. Third, our strategy is compatible with the inherent iterative engineering process of alignment. Indeed, RS can continually include adjusted opinions while preventing forgetting of the old behaviours. This relates to the continual learning challenge, and the empirical observations that weight averaging can reduce catastrophic forgetting [162, 163]. Moreover, as shown in [141] and confirmed in Figure 13(c), negative editing by weight interpolation can fix and force the removal of some behaviours. Finally, RS is computationally effective, requiring no communication across servers, thus enabling "embarrassingly simple parallelization" [164]. This facilitates its use in federated learning scenario [156] where the data should remain private. Actually, RS follows the updatable machine learning paradigm [165], "allowing for the collaborative creation of increasingly sophisticated AI system" [72]. In the future, we may develop open-source personalized models, rewarded on decentralized private datasets, and combine them continuously.
B FAQs
We addressed below questions that might arise from readers.
B.1 What is the difference between rewarded soups and model soups?
Rewarded soups (RS) and model soups (MS) [67] both average weights of models fine-tuned from a shared pre-trained initialization. That's why we chose the same terminology as "model soups" and named our method "rewarded soups". Yet, we want to clarify that RS and MS tackle different problems, have different goals, leading to different methods and implementations.
- RS challenges single-policy approaches to improve alignment in reinforcement learning, and aims at reducing reward misspecification by revealing a Pareto front of solutions across the entire space of preferences: thus RS considers different training objectives for fixed hyperparameters across runs, and non-uniform interpolating coefficients  $\lambda$  set a posteriori.
- MS challenges the standard model selection after a grid search to improve generalization in supervised learning, and aims at reducing model underspecification and reducing variance by combining all fine-tuned models: thus MS considers different hyperparameters for a fixed training objective across runs, and (usually) uniform interpolating coefficients  $\lambda = \frac{1}{M}$ .
These differences mean that MS cannot be applied to reduce reward misspecification, as validated empirically in Figure 13(b) for the captioning task. This Figure 13(b) also shows that RS and MS are actually complementary and can combine their benefits; specifically, reward misspecification and variance reduction.
B.2 Limitations for the LMC?
B.2.1 Limitations for the design of networks for the LMC?