SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In our experiments, we consider different network architectures (transformers, CNNs, and MLPs). We also investigate different training procedures: with low-rank adapters, partial or end-to-end fine-tunings. We do so for many different tasks and modalities: text generation, image captioning, image-to-test generation, visual grounding, etc. Our empirical observation is that, across those setups, the LMC is architecture-agnostic, procedure-agnostic, task-agnostic and modality-agnostic.
The main condition we require is the shared pre-trained initialization [66], so that the weights remain close (as detailed in Remark 1). As a side note, there is another condition suggested by the literature [164, 141]: the LMC would work better when the architecture has enough trainable parameters. For example, according to [141], larger networks may facilitate the orthogonality of the fine-tuned updates; then [141] "speculate that this [orthogonality] enables the combination of task vectors via addition with minimal interference".
B.2.2 Limitations for the number of training steps for the LMC?
As argued above, good performances are guaranteed when weights remain close; thus longer trainings may be worrisome, as the models may potentially diverge in the weight space. We investigate this question in Figure 8, for the news summarization and the captioning task; we double the number of training steps, and report multiple RS fronts over the course of fine-tuning. Fortunately, we consistently observe good performances for RS along fine-tuning. This confirms that the only condition for the LMC is the shared pre-trained initialization [66].
(b) Captioning.
Figure 8: Those figures show how RS's fronts evolve over the course of fine-tuning, and confirms the LMC even when doubling the number of training epochs (previously 2 for news summarization and 6 for image captioning).
B.2.3 How does the number of rewards (and networks) affects the LMC?
For visualization clarity, the fronts were mostly shown for  $N = 2$  rewards, one of the  $x$ -axis, the other on the  $y$ -axis. Yet, RS can scale and trade-off between more rewards. We validated this empirically in the spider maps from Figure 2(f) (for text generation), from Figures 3(c) and 4 (for image captioning), and from Figure 18(c) (for visual grounding), where we respectively consider up to  $N = 4$ ,  $N = 5$  and  $N = 3$  networks fine-tuned on  $N$  different rewards, one reward each.
B.3 Comparison of MORL and RS
B.3.1 How to evaluate Pareto-optimality?
Given a fixed preference  $\hat{\mu}$  between two rewards  $R_{1}$  and  $R_{2}$ , we would like to compare our RS policy to an oracle policy maximizing  $(1 - \hat{\mu})\times R_1 + \mu \times R_2$  in test. Yet, this oracle policy (and the true Pareto front) is unknown in real-world applications.
That's why, in practice, and as argued in Remark 2, we presented empirical support for Hypothesis 2 by considering the MORL's solutions fine-tuned to optimize  $(1 - \hat{\mu})\times R_1 + \mu \times R_2$  in train, for  $0\leq \mu \leq 1$ . In other words, the linearized MORL is our reference to evaluate Pareto optimality. Overall, in Section 3, MORL and RS usually perform similarly (with small differences further discussed below in Appendices B.3.2 and B.3.3). Our conclusion is that rewarded soup is an empirical solution towards Pareto-optimality, with indeed an experimental limitation highlighted in the paper's name.
B.3.2 How does reward diversity affect the effectiveness of RS?
Our experiments in captioning and image generation provide empirical evidence that the more similar the rewards, the higher the gains of RS versus MORL.
In the captioning experiment, by analyzing the transfer abilities across rewards in the spider maps from Figure 3(c), we can deduce that BLEU4 and ROUGE are more similar than BLEU1 and ROUGE, while METEOR is an outlier (fine-tuning on METEOR worsens the results for the other rewards). Then, we can observe that the gains of RS versus MORL are consistent with these similarities across rewards. Specifically, when considering  $R_{2} = ROUGE$ , the RS green front is more convex and significantly above the MORL yellow front in Figure 11(a) (with  $R_{1} = BLEU4$ ) than in Figure 3(a) (with  $R_{1} = BLEU1$ ). In Figure 12(b), with  $R_{2} = METEOR$ , MORL performs better than RS.
Similarly, in the image generation experiment, when we consider two (arguably similar) aesthetic rewards in Figure 6(a) to fine-tune a diffusion model, RS's front is to the right and above MORL's front. In contrast, performances get worse in Figure 14 where we also include an  $nsfw$  reward inversely correlated with image quality.
In conclusion, despite using diverse and heterogeneous rewards that are in tension, we consistently obtain positive results. Yet, in the case where rewards are fully antagonist, we acknowledge that RS is likely to produce less favorable results. This empirical limitation of weight interpolation can be explained in two different ways. (i) Intuitively from a loss landscape perspective: weights fine-tuned on antagonist rewards will be more distant, thus potentially breaking the linear mode connectivity. (ii) Theoretically thanks to Lemma 3, where we bound the difference between the optimal reward and RS's reward by a RHS term growing the maximum of eigenvalues ratio for rewards' Hessians: if the rewards are more diverse, their Hessians would have more different eigenvalues, thus maximum of eigenvalues ratio would grow, the RHS term would grow in Lemma 3, and our guarantees for the optimality of RS would get loose.
As a final note, to tackle this limitation under antagonist rewards, the complementarity of MORL and RS appears as a promising research direction; this is further discussed in the legend of Figure 13(a) for the captioning task and in Appendix F.2 for the image generation task.
B.3.3 Why RS is sometimes superior to MORL?
We observe a few times that the RS solutions are actually above the linearized MORL solutions. We speculate this is related to the multiple benefits of weight interpolation. The main benefit that we discuss in our paper is the ability to interpolate between different policies: from this benefit, we would expect RS to perform similarly to MORL. The second benefit from weight averaging is the implicit regularization, causing variance reduction and stabilizing performances [100, 144]. This is the main focus of the traditional weight averaging literature, for example in model soups [67]. In conclusion, we speculate that this second benefit (combined with the first) can explain why RS sometimes outperforms MORL.
C Theoretical insights
Proof. Considering  $\theta$  maximizing  $\hat{R}$ , we first show that  $\theta$  is on the PF of  $\{R_i\}_i$ . Otherwise, considering  $\theta' > _N\theta$  and as  $\forall i, \hat{\mu}_i \geq 0$ , we have  $\sum_{i} \hat{\mu}_i R_i(\theta') > \sum_{i} \hat{\mu}_i R_i(\theta)$ . This implies that  $\theta'$  would produce a better policy than  $\theta$  for  $\hat{R} = \sum_{i} \hat{\mu}_i R_i$  and thus the contradiction. Finally, as  $\theta$  is on the PF and by definition of a PCS, there exists  $\lambda$  s.t.  $\forall k, R_k(\sum_{i} \lambda_i \cdot \theta_i) = R_k(\theta)$ .
C.2 Theoretical guarantees with quadratic rewards
In this section, we provide theoretical guarantees for the near-optimality of RS when considering quadratic rewards. This simplification amounts to replacing the rewards by their second-order Taylor approximation, which is a realistic assumption when the weights remain within a small neighborhood.
C.2.1 Simple case with Hessians proportional to the Identity matrix
For the first Lemma 2, we make the following simplifying Assumption 1.
Assumption 1 (Hessians proportional to the Identity matrix.). Every reward  $R_{i}$  is quadratic, with Hessians proportional to  $\mathbb{I}_d$ . Specifically, let  $\Theta \subset \mathbb{R}^d$  be the set of possible weights, and let  $\{R_i\}_{i = 1}^N$  be the  $N$  rewards, we can write for  $i \in \{1,\dots ,N\}$ :
$$
\forall \theta \in \Theta , \quad R _ {i} (\theta) = R _ {i} \left(\theta_ {i}\right) - \eta_ {i} \| \theta - \theta_ {i} \| ^ {2} \tag {1}
$$
where  $\eta_{i}\in \mathbb{R}_{+}^{*}$  and  $\theta_{i}$  is the global maximum for reward  $R_{i}$ .
Lemma 2. Let  $\hat{\mu} = (\hat{\mu}_1,\dots ,\hat{\mu}_N)\in \Delta_N$ . Then, under Assumption 1, the reward  $R_{\hat{\mu}} = \sum_{i}\hat{\mu}_{i}\times R_{i}$  is maximized on the convex hull of  $\{\theta_1,\ldots ,\theta_N\}$ .
Proof. The function  $R_{\hat{\mu}}$  is quadratic thus has an unique global maximum  $\hat{\theta}$ , that we find analytically:
$$
\begin{array}{l} \nabla_ {\theta} R _ {\hat {\mu}} (\hat {\theta}) = 0 \Longrightarrow \sum_ {i = 1} ^ {N} \mu_ {i} \eta_ {i} \cdot (\hat {\theta} - \theta_ {i}) = 0 \\ \Rightarrow \hat {\theta} = \frac {\sum_ {i = 1} ^ {N} \hat {\mu} _ {i} \eta_ {i} \cdot \theta_ {i}}{\sum_ {i = 1} ^ {N} \hat {\mu} _ {i} \eta_ {i}} \\ \end{array}
$$
Since all the  $\hat{\mu}_i\eta_i$  are positive or zero, and at least one is greater than zero,  $\hat{\theta}$  is indeed in the convex hull of  $\{\theta_1,\dots ,\theta_N\}$ .
Remark 3. Under Assumption 1, the reward functions are concave; thus we can reasonably assume that each fine-tuning procedure for  $R_{i}$  reaches its global optimum  $\theta_{i}$  for  $i \in \{1, \dots, N\}$ . Then, Lemma 2 tells us that the maximum value for linear user's reward  $R_{\hat{\mu}}$  is obtainable by weight interpolation between the  $\{\theta_{i}\}_{i=1}^{N}$ : the interpolating coefficients in  $\Delta_N$  such that  $\lambda_{i} \propto \hat{\mu}_{i} \eta_{i}$  make rewarded soups optimal.
C.2.2 Advanced case with diagonal Hessians
We now consider the more complex case with the relaxed Assumption 2. For simplicity, we only consider  $N = 2$  rewards  $R_{1}$  and  $R_{2}$ .
Assumption 2 (Diagonal Hessians). The rewards are quadratic, with Hessians diagonal negative definite. Specifically, we can write for  $i \in \{1,2\}$ :
$$
\forall \theta = \left(\theta^ {1}, \dots , \theta^ {d}\right) \in \Theta , \quad R _ {i} (\theta) = R _ {i} \left(\theta_ {i}\right) - \sum_ {j = 1} ^ {d} \eta_ {i} ^ {j} \left(\theta^ {j} - \theta_ {i} ^ {j}\right) ^ {2}, \tag {2}
$$
where  $(\eta_i^1,\dots \eta_i^d)\in \{\mathbb{R}_+^*\} ^d$  and  $\theta_{i} = (\theta_{i}^{1},\dots,\theta_{i}^{d})$  is the global maximum for reward  $R_{i}$
Remark 4. This diagonal Assumption 2 of the Hessian is common: for example in optimization [166, 167], to prune networks [168] or in out-of-distribution generalization [169]. This strong assumption is supported by the empirical observation [170] that Hessians are diagonally dominant, in particular at the end of training. Also, we note that our findings remain valid assuming only that the Hessians are co-diagonalizable.
Lemma 3. We consider the user's reward  $R_{\hat{\mu}} = (1 - \hat{\mu}) \times R_1 + \hat{\mu} \times R_2$  with  $\hat{\mu} \in [0,1]$ , and
$$
\Delta R _ {\hat {\mu}} = \max  _ {\theta \in \Theta} R _ {\hat {\mu}} (\theta) - \max  _ {\lambda \in [ 0, 1 ]} R _ {\hat {\mu}} ((1 - \lambda) \cdot \theta_ {1} + \lambda \cdot \theta_ {2}). \tag {3}
$$
$\Delta R_{\hat{\mu}}$  corresponds to the difference in terms of  $R_{\hat{\mu}}$  between the global maximum and the maximum reachable by weight interpolation through rewarded soups (with a single interpolating coefficient for all dimensions). Then, under Assumption 2, we have:
$$
\Delta R _ {\hat {\mu}} \leq \frac {\hat {\mu} ^ {2} (1 - \hat {\mu}) ^ {2} \left(M \Delta_ {1} - \Delta_ {2}\right) \left(M \Delta_ {2} - \Delta_ {1}\right)}{(\hat {\mu} (1 - \hat {\mu}) (M - 1) ^ {2} + M) ((1 - \hat {\mu}) \Delta_ {1} + \hat {\mu} \Delta_ {2})}, \tag {4}
$$
where  $M = \max_{j\in \{1,\ldots ,d\}}\max \Bigl {(}\frac{\eta_1^j}{\eta_2^j},\frac{\eta_2^j}{\eta_1^j}\Bigr)$  is the maximum of eigenvalues ratio,  $\Delta_{1} = R_{1}(\theta_{1}) - R_{1}(\theta_{2})$  and  $\Delta_{2} = R_{2}(\theta_{2}) - R_{2}(\theta_{1})$ .
When  $\Delta_1 = \Delta_2$ , the bound simplifies into:
$$
\Delta R _ {\hat {\mu}} \leq \frac {\hat {\mu} ^ {2} (1 - \hat {\mu}) ^ {2} (M - 1) ^ {2}}{\hat {\mu} (1 - \hat {\mu}) (M - 1) ^ {2} + M} \Delta_ {1} \tag {5}
$$
Furthermore, when the Hessians are equal, then  $M = 1$  and  $\Delta R_{\hat{\mu}} = 0$ : RS is optimal.
Proof. This novel proof is in three steps. First, we find  $\hat{\theta}$  maximizing  $R_{\hat{\mu}}(\theta)$  for  $\theta$  on the full set of weights  $\Theta$ . Second, we find  $\bar{\lambda}$  maximizing  $R_{\hat{\mu}}((1 - \lambda) \cdot \theta_1 + \lambda \cdot \theta_2)$  for  $\lambda \in [0,1]$  and thus defining the best interpolation between the expert weights. Finally, we bound  $\Delta R_{\hat{\mu}}$ , the differences between their rewards, by applying the Bhatia-Davis inequality.
First step. Let's first find the maximum of  $R_{\hat{\mu}}$  on  $\Theta$ . Denoting  $S = (1 - \hat{\mu}) \times R_1(\theta_1) + \hat{\mu} \times R_2(\theta_2)$ , we have for all  $\theta \in \Theta$ :
$$
R _ {\hat {\mu}} (\theta) = S - \sum_ {j = 1} ^ {d} \left((1 - \hat {\mu}) \eta_ {1} ^ {j} \left(\theta^ {j} - \theta_ {1} ^ {j}\right) ^ {2} + \hat {\mu} \eta_ {2} ^ {j} \left(\theta^ {j} - \theta_ {2} ^ {j}\right) ^ {2}\right) \tag {6}
$$
Since  $R_{\hat{\mu}}$  is a sum of concave quadratic functions, it has a unique global maximum reached at a point we note  $\hat{\theta} = \left(\hat{\theta}^{1}, \dots, \hat{\theta}^{d}\right)$ . The global maximum can be computed by differentiating  $R_{\hat{\mu}}$  with respect to each variable  $\theta^{j}$ , which gives:
$$
\hat {\theta} ^ {j} = \left(1 - \hat {\lambda} ^ {j}\right) \cdot \theta_ {1} ^ {j} + \hat {\lambda} ^ {j} \cdot \theta_ {2} ^ {j}
$$
where the interpolating coefficients per dimension  $\hat{\lambda}^j$  are defined for  $j\in \{1,\dots ,d\}$  as:
$$
\hat {\lambda} ^ {j} = \frac {\hat {\mu} \eta_ {2} ^ {j}}{(1 - \hat {\mu}) \eta_ {1} ^ {j} + \hat {\mu} \eta_ {2} ^ {j}} \in [ 0, 1 ]. \tag {7}
$$
Second step. With  $\lambda \in [0,1]$  and  $\theta = (1 - \lambda)\cdot \theta_{1} + \lambda \cdot \theta_{2}$ , we can write  $R_{\hat{\mu}}(\theta)$  as a function of  $\lambda$ :
$$
\begin{array}{l} R _ {\hat {\mu}} (\theta) = S - \sum_ {j = 1} ^ {d} \left(\left((1 - \hat {\mu}) \eta_ {1} ^ {j} + \hat {\mu} \eta_ {2} ^ {j}\right) \left(\lambda - \hat {\lambda} ^ {j}\right) ^ {2} + \frac {\hat {\mu} (1 - \hat {\mu}) \eta_ {1} ^ {j} \eta_ {2} ^ {j}}{(1 - \hat {\mu}) \eta_ {1} ^ {j} + \hat {\mu} \eta_ {2} ^ {j}}\right) \left(\theta_ {1} ^ {j} - \theta_ {2} ^ {j}\right) ^ {2} \\ = R _ {\hat {\mu}} (\hat {\theta}) - \sum_ {j = 1} ^ {d} p _ {j} \left(\lambda - \hat {\lambda} ^ {j}\right) ^ {2} \tag {8} \\ \end{array}
$$
where  $p_j$  is defined as  $p_j = \left((1 - \hat{\mu})\eta_1^j + \hat{\mu}\eta_2^j\right)\left(\theta_1^j - \theta_2^j\right)^2$ .
From Equation (8), we can compute the maximum reward obtainable for weight averaging  $\max_{\lambda \in [0,1]} R_{\hat{\mu}}((1 - \lambda) \cdot \theta_1 + \lambda \cdot \theta_2)$ . Since the function  $\lambda \mapsto R_{\hat{\mu}}((1 - \lambda) \cdot \theta_1 + \lambda \cdot \theta_2)$  is a concave quadratic function, there is a unique value  $\bar{\lambda}$  maximizing  $R_{\hat{\mu}}$  equal to
$$
\bar {\lambda} = \frac {\sum_ {j = 1} ^ {d} p _ {j} \hat {\lambda} ^ {j}}{\sum_ {j = 1} ^ {d} p _ {j}}. \tag {9}
$$
Since all  $p_j$  are positive and all  $\hat{\lambda}^j$  are between 0 and 1,  $\bar{\lambda}$  is also between 0 and 1. Therefore,  $R_{\hat{\mu}}\big((1 - \bar{\lambda})\cdot \theta_1 + \bar{\lambda}\cdot \theta_2\big)$  is indeed the maximum reward for rewarded soups.
Third step. Applying Equation (8) to  $\bar{\lambda}$  gives:
$$
\begin{array}{l} \Delta R _ {\hat {\mu}} = R _ {\hat {\mu}} (\hat {\theta}) - R _ {\hat {\mu}} \left((1 - \bar {\lambda}) \cdot \theta_ {1} + \bar {\lambda} \cdot \theta_ {2}\right) (10) \\ = \sum_ {j = 1} ^ {d} p _ {j} \left(\bar {\lambda} - \hat {\lambda} ^ {j}\right) ^ {2} (11) \\ = \left(\sum_ {j = 1} ^ {d} \frac {p _ {j}}{\sum_ {i = 1} ^ {n} p _ {i}} \left(\bar {\lambda} - \hat {\lambda} ^ {j}\right) ^ {2}\right) \left(\sum_ {j = 1} ^ {n} p _ {j}\right) (12) \\ \end{array}
$$
The second term in Equation (12) can be simplified as:
$$
\sum_ {j = 1} ^ {d} p _ {j} = (1 - \hat {\mu}) \Delta_ {1} + \hat {\mu} \Delta_ {2}. \tag {13}
$$
The core component of this proof is the upper bounding of the first term in Equation (12). The key idea is to recognize the variance of a discrete random variable  $\Lambda$  with  $\mathbb{P}(\Lambda = \hat{\lambda}_i) = \frac{p_i}{\sum_{j=1}^n p_j}$ ; then,  $\bar{\lambda}$  from Equation (9) is actually the expectation of  $\Lambda$ . Then, we can apply the Bhatia-Davis inequality, as recalled in Equation (14), on the variance of a bounded random variable  $a \leq \Lambda \leq b$ :
$$
V a r (\Lambda) \leq (b - \mathbb {E} (\Lambda)) (\mathbb {E} (\Lambda) - a) \tag {14}
$$
Therefore Equation (12) is bounded by:
$$
\Delta R _ {\hat {\mu}} \leq \left(\max  _ {1 \leq j \leq d} \hat {\lambda} ^ {j} - \bar {\lambda}\right) \left(\bar {\lambda} - \min  _ {1 \leq j \leq d} \hat {\lambda} ^ {j}\right) \left((1 - \hat {\mu}) \Delta_ {1} + \hat {\mu} \Delta_ {2}\right). \tag {15}
$$
Now, we bound the variables  $\hat{\lambda}^j$ , since  $1 / M \leq \eta_1^j / \eta_2^j \leq M$ . Then for all  $j$  we have:
$$
\frac {\hat {\mu}}{(1 - \hat {\mu}) M + \hat {\mu}} \leq \hat {\lambda} ^ {j} \leq \frac {\hat {\mu} M}{(1 - \hat {\mu}) + \hat {\mu} M}, \tag {16}
$$
$$
\Delta R _ {\hat {\mu}} \leq \left(\frac {\hat {\mu} M}{1 + \hat {\mu} (M - 1)} - \bar {\lambda}\right) \left(\bar {\lambda} - \frac {\hat {\mu}}{M - \hat {\mu} (M - 1)}\right) ((1 - \hat {\mu}) \Delta_ {1} + \hat {\mu} \Delta_ {2}). \tag {17}
$$
Finally, noting that  $\Delta_{i} = \sum_{j=1}^{d} \eta_{i}^{j} \left( \theta_{2}^{j} - \theta_{1}^{j} \right)^{2}$ , we deduce from Equation (9) that  $\bar{\lambda} = \frac{\hat{\mu} \Delta_{2}}{(1 - \hat{\mu}) \Delta_{1} + \hat{\mu} \Delta_{2}}$ . Replacing this in the previous Equation (17) gives the final Equation (4), concluding the proof.
Remark 5. As a final remark, please note that the suboptimality of  $RS$  comes from the need of having one single interpolating coefficient  $\bar{\lambda}$  for all  $d$  parameters  $(\theta^1, \dots, \theta^d)$  of the network. Yet, the advanced merging operations in [69] remove this constraint, with interpolating coefficients proportional to the eigenvalues of the Fisher matrices [171], which actually approximate the eigenvalues of the Hessian [172, 173]. Combining [69] and our  $RS$  is a promising research direction, the key issue being the computation of the Fisher matrices [174] for networks with billions of parameters.
C.2.3 Bound visualization
We visualize in Figure 9 the bound given by Lemma 3. We show that for small values of  $M$  like  $M = 2$ , the value of  $R_{\hat{\mu}}$  for RS is quite close to the global optimum. Also, recall that RS theoretically matches this upper bound when  $M = 1$ . For larger values like  $M = 10$ , the bound is less tight, and we note that the maximum value of  $R_{\hat{\mu}}$  approaches the constant function 1 as  $M \to \infty$ .
Figure 9: Illustration of the bound given by Lemma 3 under Assumption 2. For simplicity, we showcase the case where  $R_{1}(\theta_{1}) = R_{2}(\theta_{2}) = 1$ ,  $R_{1}(\theta_{2}) = R_{2}(\theta_{1}) = 0$ , thus  $\Delta_{1} = \Delta_{2} = 1$ . In green, we plot the rewards obtained with rewarded soups for the optimal  $\bar{\lambda}$ , i.e.,  $R_{\hat{\mu}}\left((1 - \bar{\lambda})\cdot \theta_{1} + \bar{\lambda}\cdot \theta_{2}\right)$ , whose value is independent of  $M$  in this case. In blues, we plot the maximum value of  $\mathcal{R}_{\hat{\mu}}$  given by Equation (5) in Lemma 3, for  $M = 2$  and  $M = 10$ . For reference, we also plot the values for the lower bound in the LMC Hypothesis 1, i.e., equal to  $(1 - \hat{\mu})(1 - \bar{\lambda})R_{1}(\theta_{1}) + \hat{\mu}\bar{\lambda} R_{2}(\theta_{2})$ . As RS outperforms this lower bound, it validates Hypothesis 1 in this case.
C.3 Similarity between weight interpolation and functional ensembling
Lemma 4 ( $\lambda$ -interpolation of weights approximates the  $\lambda$ -ensembling of predictions. Adapted from [67, 68, 100].). Given  $\theta_{1}$  and  $\theta_{2}$  optimized for  $R_{1}$  and  $R_{2}$  s.t. they remain close, i.e.,  $\| \theta_{1} - \theta_{2}\|_{2} \approx 0$ . Denoting  $\theta_{\lambda}$  the interpolated weights  $\theta_{\lambda} = (1 - \lambda) \cdot \theta_{1} + \lambda \cdot \theta_{2}$  and  $f_{\lambda}$  the ensembling of predictions  $f_{\lambda}(\cdot) = (1 - \lambda) \cdot f(\cdot, \theta_{1}) + \lambda \cdot f(\cdot, \theta_{2})$ :
$$
f (\cdot , \theta_ {\lambda}) \approx f _ {\lambda} (\cdot)
$$
and for  $k \in \{1, 2\}$ :
$$
R _ {k} (f (\cdot , \theta_ {\lambda})) \approx R _ {k} (f _ {\lambda} (\cdot))
$$
Proof. This proof follows [68] and has two components.
Functional approximation. First, we perform a Taylor expansion at the first order of the models' predictions w.r.t. parameters  $\theta$  for  $x\in T$ :
$$
\begin{array}{l} f (x, \theta_ {1}) = f (x, \theta_ {\lambda}) + \nabla_ {\theta} f (x, \theta_ {\lambda}) ^ {\intercal} (\theta_ {1} - \theta_ {\lambda}) + \mathcal {O} \left(\left\| \theta_ {1} - \theta_ {\lambda} \right\| _ {2} ^ {2}\right) \\ = f (x, \theta_ {\lambda}) + \nabla_ {\theta} f (x, \theta_ {\lambda}) ^ {\intercal} (\lambda \cdot \theta_ {1} - \lambda \cdot \theta_ {2}) + \mathcal {O} \Big (\| \theta_ {1} - \theta_ {2} \| _ {2} ^ {2} \Big) \\ \end{array}
$$
and similarly:
$$
f (x, \theta_ {2}) = f (x, \theta_ {\lambda}) + \nabla_ {\theta} f (x, \theta_ {\lambda}) ^ {\intercal} ((\lambda - 1) \cdot \theta_ {1} + (1 - \lambda) \cdot \theta_ {2}) + \mathcal {O} \Big (\| \theta_ {1} - \theta_ {2} \| _ {2} ^ {2} \Big)
$$
Then by  $\lambda$ -weighted sum over  $i$ , the term multiplying  $\nabla_{\theta}f(x,\theta_{\lambda})^{\intercal}$  cancels out and we obtain:
$$
f _ {\lambda} (x) = (1 - \lambda) \cdot f (x, \theta_ {1}) + \lambda \cdot f (x, \theta_ {2}) = f (x, \theta_ {\lambda}) + \mathcal {O} \left(\left\| \theta_ {1} - \theta_ {2} \right\| _ {2} ^ {2}\right). \tag {18}
$$
Reward approximation. Second, we obtain the reward approximation with a Taylor expansion at the zeroth order of the reward  $R_{k}$  for  $k\in \{1,2\}$  and injecting Equation (18):
$$
\begin{array}{l} R _ {k} \left(f _ {\lambda} (x)\right) = R _ {k} \left(f \left(x, \theta_ {\lambda}\right) (x)\right) + \mathcal {O} \left(\left\| f _ {\lambda} (x) - f (x, \theta_ {\lambda}) \right\| _ {2}\right) \\ = R _ {k} \left(f (x, \theta_ {\lambda}) (x)\right) + \mathcal {O} \left(\left\| \theta_ {1} - \theta_ {2} \right\| _ {2} ^ {2}\right). \\ \end{array}
$$
We obtain the results when  $\theta_{1}$  and  $\theta_{2}$  remain close, i.e., when we can ignore the  $\mathcal{O}$  term.
D Text-to-text: LLaMA with diverse RLHFs
D.1 Experimental details
We summarize the key implementation details of our text-to-text generation experiments in Table 1. The pre-trained network is LLaMA-7b [44]; then low-rank adapters [87] were fine-tuned on Alpaca [22] to follow instructions. We eventually fine-tune via PPO on the different considered tasks. Our code is adapted from [86]; we kept most of their hyperparameter values, only dividing by 2 the batch size to fit in our GPU and extending the output length. For each task, we consider existing open-source datasets<sup>3</sup> and available reward models, that we download from HuggingFace. Regarding the reward models, in summarization tasks,  $R_{1}$  was open-sourced in an effort to reproduce the Summarize from Human Feedback paper [12], while  $R_{2}$  [91] aimed at improved "faithfulness in abstractive summarization with contrast candidate generation". For other dialog tasks, we mostly rely on different reward models from OpenAssistant [92]; though they all aim at evaluating whether an answer is adequate given a question, they differ in their predictions due to differences in their architecture and training procedures. In practice, we leverage these reward models as block-box classification pipelines, implemented in the transformers library [82].