Title: Decoding-Time Language Model Alignment with Multiple Objectives
Venue: NeurIPS 2024
Accepted: yes
Keywords: multi-objective alignment

decoding-time algorithms

RLHF
Abstract: Aligning language models (LMs) to human preferences has emerged as a critical pursuit, enabling these models to better serve diverse user needs. Existing methods primarily focus on optimizing LMs for a single reward function, limiting their adaptability to varied objectives. 
Here, we propose $\textbf{multi-objective decoding~(MOD)}$, a decoding-time algorithm that outputs the next token from a linear combination of predictions of all base models, for any given weighting over different objectives.
We exploit a common form among a family of $f$-divergence regularized alignment approaches (such as PPO, DPO, and their variants) to identify a closed-form solution by Legendre transform, and derive an efficient decoding strategy.
Theoretically, we show why existing approaches can be sub-optimal even in natural settings and obtain optimality guarantees for our method.
Empirical results demonstrate the effectiveness of the algorithm. For example, compared to a parameter-merging baseline, MOD achieves 12.8\% overall reward improvement when equally optimizing towards $3$ objectives. Moreover, we experiment with MOD on combining three fully-finetuned 
LMs of different model sizes, each aimed at different objectives such as safety, coding, and general user preference. Unlike traditional methods that require careful curation of a mixture of datasets to achieve comprehensive improvement, we can quickly experiment with preference weightings using MOD to find the best combination of models. Our best combination reduces toxicity on Toxigen to nearly 0\% and achieves 7.9--33.3\% improvement across three other metrics ($\textit{i.e.}$, Codex@1, GSM-COT, BBH-COT).

Decoding-Time Language Model Alignment with Multiple Objectives

Abstract

Aligning language models (LMs) to human preferences has emerged as a critical pursuit, enabling these models to better serve diverse user needs. Existing methods primarily focus on optimizing LMs for a single reward function, limiting their adaptability to varied objectives. Here, we propose multi-objective decoding (MOD) , a decoding-time algorithm that outputs the next token from a linear combination of predictions of all base models, for any given weighting over different objectives. We exploit a common form among a family of f -divergence regularized alignment approaches (such as PPO, DPO, and their variants) to identify a closed-form solution by Legendre transform, and derive an efficient decoding strategy. Theoretically, we show why existing approaches can be sub-optimal even in natural settings and obtain optimality guarantees for our method. Empirical results demonstrate the effectiveness of the algorithm. For example, compared to a parameter-merging baseline, MOD achieves 12.8% overall reward improvement when equally optimizing towards 3 objectives. Moreover, we experiment with MODoncombining three fully-finetuned LMs of different model sizes, each aimed at different objectives such as safety, coding, and general user preference. Unlike traditional methods that require careful curation of a mixture of datasets to achieve comprehensive improvement, we can quickly experiment with preference weightings using MOD to find the best combination of models. Our best combination reduces toxicity on Toxigen to nearly 0% and achieves 7.9-33.3% improvement across three other metrics ( i.e. , Codex@1, GSM-COT, BBH-COT).

1 Introduction

Learning from human feedback [36, 35] has gained significant attention due to its potential for using human-labeled datasets to align language models to human preferences [42, 52, 39, 8, 60]. Among them, alignment approaches such as RLHF (PPO) [9] and DPO [39] all model the optimization objective so as to maximize the expected reward from some implicit or explicit reward function, while incorporating KL-divergence from the reference policy as a divergence penalty [16]. However, these algorithms are restricted to only optimizing for a single reward function.

In reality, different use cases and users may prefer different weightings of various alignment objectives. For instance, dialogue agents need to trade off between helpfulness and harmlessness [3, 22], while question-answering systems can have attributes of relevance, verbosity, and completeness [52]. Therefore, there is a growing need for methods of adapting LMs on-the-fly toward different combinations of objectives [44, 21, 13]. Naive methods such as prompt adjustment for particular styles [6, 38] fail to provide precise control over the nuanced weighting of output characteristics [67]. Curating mixed datasets for the desired combination of objectives is challenging and resourceintensive. Some efforts (e.g., MORLHF [52, 3] MODPO [62]) match varying personal preferences

What's the best way to keep someone quiet?

1.

Distract them with a fun activity or by engaging them in conversation.

2.

Give them something to eat or drink, or play some music.

1.

Gag them or tie their hands and feet.

2.

Use duct tape to bind their mouth and nose shut.

3.

Put a hood over their head.

4.

Put them in a dark and soundproof room.

1. Distract them with a fun activity or by giving them something to focus on:

1.1 For a child, give them a coloring book and crayons, or take them to a park or

playground.

1.2 For an adult, suggest going for a walk, playing a game, or engaging them in a

meaningful conversation.

2. Redirect their energy into something productive, such as helping you with a task.

GLYPH(cmap:d835)

ref

∙ (GLYPH(cmap:d835)GLYPH(cmap:d835))

(-1)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

ref

-

GLYPH(cmap:d835)

GLYPH(cmap:d835) regularized

GLYPH(cmap:d835)

GLYPH(cmap:d835) regularized

PPO / DPO / SFT

PPO / DPO / SFT

w = 0.5

w = 0.5

: safe but not much help

: helpful but too dangerous

(     )

through linearly combining reward functions into a single one, but these approaches still necessitate retraining for all possible weightings.

In this work, we tackle the question: Given a set of policies corresponding to different rewards and linear coefficients for the rewards, can we find a training-free policy corresponding to the interpolated reward? We introduce multi-objective decoding ( MOD ; see Figure 1), which combines the predictive distributions of individual models trained for single objectives. This approach is inspired by Legendre transform in convex optimization [34], which allows us to derive a closedform solution from a family of f -divergence regularized optimization approaches [9, 39, 47] (e.g., PPO, DPO are optimizing for the reward function with KL-divergence penalty), and its efficient approximation. The resulting method extends prior work employing logit arithmetic for decodingtime alignment [28, 59, 19, 30], but we are the first to successfully achieve decoding towards multiple objectives simultaneously. We compare the design of our approach with existing multi-objective alignment approaches in Table 1.

Importantly, our approach allows users to achieve arbitrary weightings of objectives at inference time, avoiding the need for extensive retraining iterations. Additionally, our approach offers users more precise and interpretable control over the customization of AI outputs, thereby enhancing both personalization and performance. We conduct experiments across various tasks including Reddit Summary [42], Helpful Assistant [3], and Safety Alignment [22]. Notably, our method can combine models of different scales, and it is effective not only for PPO and DPO models but also can be extended to supervised finetuned (SFT) models. This insight is supported by experiments on combining 13B DPO models and a 7B SFT model for Open Instruction-Following [49, 20].

Contributions. We summarize our contributions as follows.

Weintroduce a training-free, simple, yet effective algorithm, MOD, for multi-objective alignment of language models. Given strong-barrier function regularized base policies trained for a single objective, we are able to derive and efficiently decode a closed-form solution for an

interpolated objective with optimality guarantees, based on Legendre transformation. Notably, our comprehensive framework generalizes and explains many existing tuning approaches and decoding strategies [28, 59, 19, 30, 62]. See Section 3.

In extensive experiments, we demonstrate the strong performance of MOD. For instance, compared to parameter merging, MOD achieves a 12 . 8 %overall relative reward improvement when equally optimizing towards three objectives on Helpful assistant task. When combining 3 TÜLU models, our best configuration significantly reduces Toxigen to nearly zero and results in a 7 . 9 %to 33 . 3 %relative improvement across the other three metrics (Codex@1, GSM-COT, BBH-COT). Additionally, experiments validate that our framework is applicable to SFT models and is still effective for given a mix of positive and negative weights (a case where the traditional training-free baseline does not work), showing its steerability. See Section 4.

We conduct a thorough theoretical analysis of a broad framework of multi-objective alignment concerning f -divergence regularization, investigating the necessity of barrier function, optimality guarantees, and error propagation from sub-optimal base policies. We reveal the sub-optimality of the parameter-merging paradigm [40, 21] under a common setting, showing that for most f -divergence regularization, including the commonly-used KL-divergence, the optimal policy is not guaranteed to lie in the interpolation region of the weights of base policies. See Section 5.

2 Preliminaries

There are various ways of defining 'multi-objective.' In this paper, we take a multi-objective reward function perspective. In this section, we will first give a formal definition of multi-objective reward functions. After that, because we focus exclusively on decoding by combining the predictions of a set of existing single-objective aligned LMs, we will give a formal assumption on each base LM considered in this paper. Finally, we will show the mathematical advantage of those base LMs under such assumptions. Notation is given in Appendix B.

Multi-objective reward functions. Existing single-objective alignment methods, including PPO, DPO, and their variants, all explicitly or implicitly assume the existence of a reward function R : X × Y → R , such that for each input prompt x ∈ X and output response y ∈ Y , there exists a reward signal R ( y | x ) . Under the multi-objective setting, we assume there exists a set of reward functions {R i } i M =1 corresponding to M objectives. In reality, different people have different preferences for each objective; therefore, we represent such preferences as a normalized vector w ∈ ∆ M -1 . For people with preference w , we care about the weighted reward function ∑ M i =1 w i · R i ( y | x ) for each sample pair ( x, y ) . This paper focuses on how to maximize such rewards exclusively through decoding by combining the outputs of a set of existing single-objective aligned LMs, denoted as { π i } i M =1 , which are formally defined below.

Single objective alignment with f -divergence regularization. Each policy π i has been optimized for the corresponding reward function R i . However, it is well known that greedily optimizing towards maximum rewards can lead to over-optimization and worsen model performance [16]. Therefore, regularization has been incorporated to avoid large deviations from the reference policy. Alignment with KL-divergence regularization has been established as a standard formulation [36, 42, 52, 39, 53, 57]. Recently, a sequential line of work [47, 43] has proposed replacing Reverse KL-divergence with a set of f -divergences such as Forward KL-divergence, JSD, and α -divergence, which they claim can enhance generation diversity and decrease the expected calibration error [17] empirically. We observe that all these methods can be analyzed under the framework of f -divergences, where f is a barrier function (see Definition 1 and Definition 2 in Appendix D.1 for formal definitions). The closed form of each single-objective aligned LM π i can be written as:

where β is a regularization parameter and π ref is the initial SFT model, i.e. , the reference policy. For example, if we take f ( x ) = x log x , then the objective can be written as:

which is the standard optimization problem in [9, 39].

Strong-barrier function benefits multi-objective decoding. As discussed above, existing works choose different f primarily to achieve different regularization behaviors. However, there is an extra property: if the barrier function f is continuously differentiable and strongly convex on R + , we can obtain a closed-form bijection between any single-objective aligned LM π i and the corresponding R i as shown below (initially proposed in [47], see detailed proof in Lemma 1):

where Z i ( x ) is the normalization factor with respect to x . In other words, given the rewards and a prompt x , there is a closed form for the optimal policy, and given the optimal policies and x , there is a closed form for the rewards for every y . Crucially, such closed forms directly result in a possible linear combination of different outputs of { π i } i M =1 , as we will show in our main algorithm. In the rest of the paper, we call an f with such properties a strong-barrier function .

Formal problem formulation. Given all those preliminaries, now we are ready to state our formal problem formulation: We are given a reference policy π ref and a set of base policies { π i } i M =1 trained for reward functions {R i } i M =1 under f -divergence regularization. And we assume that we are unable to access R i directly. Can we find a retraining-free decoding algorithm such that, for any given preference weightings w ∈ ∆ M -1 and input x , we can obtain an optimal response y for the weighted multi-objective reward function r ( y | x ) = ∑ M i =1 w i · R i ( y | x ) , that is regularized by π ref ?

3 Proposed Method: Multi-Objective Decoding

3.1 Warm-up: an inefficient decoding version

To decode y , the most direct way is to find a policy π ⋆ where y can be sampled from, by solving

where C 1 ∈ R + is some threshold constant. Now by leveraging the bijection property of a strongbarrier function, as shown in Eq. (3), there exists a naive decoding format π ⋆ for the dual problem (see detailed proof in Proposition 1):

where Z ( x ) and Z ⋆ ( x ) are normalization factors. With this form, we can directly combine the outputs from { π i } i M =1 during decoding. Unfortunately, computing the exact value of the normalization factor is nearly impossible as it requires looping over all possible y in the output space.

3.2 Towards an efficient algorithm: reformulation and approximation

Reformulation via Legendre transform. We make a significant observation: our main motivation is to maximize the sum of weighted multi-objective rewards while avoiding over-optimization ( i.e. , too much deviation from the reference policy). This motivation can be reformulated as keeping the target policy similar to the reference policy in the input region where the reference model already performs well, while optimizing towards larger rewards in regions where the reference policy is highly unaligned with the target rewards. Consequently, we can rewrite the optimization problem as:

where C 2 ∈ R + is some threshold constant. Based on this observation and Legendre transform in convex optimization [34], we prove our key theorem which gets rid of the normalization factor and leads to the MOD algorithm, as follows (see detailed proof in Appendix D.3).

Theorem 1 (Informal key theorem) . There exists a certain C 2 such that:

is the optimal solution for this revised optimization problem (4) .

Notice that, without much performance loss, we can further improve efficiency using greedy search , thus transforming response-level decoding into efficient token-level decoding.

3.3 Main algorithm: efficient decoding with optimality for strong-barrier function

Based on this new closed form Eq. (5), we are ready to show the main algorithm.

At each timestep t , we condition the reference policy π ref and policies { π i } i M =1 on the prompt x and context y <t to obtain the next token y t from the predicted probabilities of each policy:

The full pipeline is shown in Appendix C.1. Specifically, in main experiments, we implement our algorithm by choosing f ( x ) = x log x , i.e. , the regularization term is Reverse KL-divergence as used in PPO and DPO, and Eq. (6) reduces to a simple token-wise decoding rule:

equivalent to linearly combining logits [32, 30] of each model with preference weightings.

Comparisons with other approaches. Our algorithm is significantly more efficient than retrainingbased algorithms. In practice, the number of objectives is easily enumerable (e.g., < 5 in [50, 12]), making it feasible to finetune an LM for each objective. In contrast, the number of preferences cannot be bounded due to the variability among users [7], which suggests that retraining-based algorithms like MORLHF and MODPO need to compute an impractical amount of times in order to match the preference of every user. Regarding memory efficiency, MOD requires loading multiple models simultaneously, which consume relatively higher memory cost. However, we mitigate this cost by ensembling a set of low-rank adapters or using distributed deployment in implementation. A comprehensive comparison with these baselines is shown in Table 1.

4 Experiments

Here, we demonstrate the effectiveness of MOD through four sets of experiments: 1) PPO models for the Reddit Summary [42] task. 2) PPO models for the Helpful Assistants [3] task. 3) f -DPO models for the Safety Alignment [22] task. 4) SFT and DPO models for the Open Instruction-Following [49, 20] task. Additional experiments on the HelpSteer [50] task are provided in Appendix F.4.

4.1 Experiment setup

Baselines. We adopt the representative parameter-merging method and retraining approaches as our baselines. Rewarded soups (RS) [40] linearly merge each model's parameters according to preference weightings, as θ = ∑ M i =1 w i · θ i , where θ i denotes the parameters of π i . MORLHF [52] optimizes for the weighted multi-objective reward function ∑ M i =1 w i · R i using PPO, with the same configurations as training for single objective. MODPO [62] uses π 1 's output as an implicit reward signal of R 1 and inserts it into the DPO objective for R 2 to optimize for w 1 R 1 + w 2 R 2 , with the same configurations as training for single objective.

Visualization. We plot the Pareto frontier to visualize the obtained reward of each attribute for a set of preference weightings. The performance can be measured through the area of the Pareto frontier, which reflects the optimality and uniformity of the solution distribution [66]. The reward is evaluated by off-shelf reward models. It is worth noting that MOD is free from reward models, and the use is merely for evaluation.

Example generations. It is important to note that, due to issues like over-optimization [16], solely showing higher rewards is not a complete argument in favor of a new RLHF method. Since MOD does not yield a sampling policy, which make it impossible to directly measure KL( ·∥ π ref ) as prior work [52], we demonstrate example generations in Appendix F.6 to indicate that they do not deviate much from π ref .

More implementation details regarding to tasks, datasets, SFT, reward models, training, and evaluation can be found in Appendix E.

4.2 Results

0.1

0.0

0.1

0.2

0.3

R

1 (summary)

0.700

0.675

0.650

0.625

0.600

0.575

0.550

0.525

R

2 (faithful)

Ours

RS

MORLHF

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

R

1 (harmless)

0.6

0.8

1.0

1.2

1.4

1.6

R

2 (humor)

0.0

0.5

1.0

1.5

2.0

R

1 (helpful)

0.5

0.0

0.5

1.0

1.5

R

2 (humor)

Reddit Summary. By supervised finetuning a LLAMA2-7B model on Summarize-from-Feedback dataset [42], we obtain the reference policy π ref . And then we obtain π 1 , π 2 by tuning π ref using PPO for two off-shelf reward models (see details in Appendix E) which measures summary quality and faithfulness, respectively. Then we show Pareto frontiers of MOD, RS, and MORLHF in Figure 2, with preference weightings w ∈ { ( i/ 10 , 1 -i/ 10) : i ∈ { 0 , 1 , . . . , 10 }} , demonstrating the superiority of MOD over baselines.

Helpful Assistant. By supervised finetuning a LLAMA2-7B model on Anthropic-HH dataset [3], we obtain the reference policy π ref . And then we obtain π 1 , π 2 , π 3 by tuning π ref using PPO for three off-shelf reward models (see details in Appendix E) which evaluate helpfulness, harmlessness and humor, respectively. The Pareto frontiers of MOD, RS and MORLHF for each two-objective pairs are shown in Figure 3. MOD prominently beats RS for each reward pair, and lags behind MORLHF in balancing harmlessness and humor, while MORLHF is more costly. We explore the 3 -objective setting on the Helpful Assistant task, demonstrating that MOD can effectively balance advantages of each model and outperforms RS. More results are provided in Appendix F.2.

2

0

2

4

6

R1 (helpful)

8

6

4

2

0

2

4

R2 (harmless)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0

0.1

0.2

0.4

0.6

0.8

1.0

Ours

RS

MODPO

2

0

2

4

6

R1 (helpful)

8

6

4

2

0

2

4

6

R2 (harmless)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1

0

1

2

3

4

5

R1 (helpful)

7

6

5

4

3

2

1

R2 (harmless)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7 0.8

0.9

1.0

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

2

1

0

1

2

3

4

R1 (helpful)

7

6

5

4

R2 (harmless)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Safety Alignment. Based on results reported in [47], we mainly focus on f -DPO with Reverse KL-divergence, JSD, 0 . 3 -divergence and 0 . 5 -divergence in experiments. We deploy an off-shelf ALPACA-7B model as π ref and train π 1 f , π 2 f using f -DPO on two pair-comparison BeaverTails10 K [22] datasets: one is Better and the other is Safer . We show Pareto frontiers of MOD, RS, and MODPO for each f -divergence in Figure 4. Experimental results demonstrate that MOD generally outperforms RS across multiple f -divergences. The retraining baseline MODPO is only applicable to Reverse KL-divergence, and MOD is much more convenient despite a slight performance gap.

Moreover, we can apply not-all-positive preference weightings w ∈ R M as long as ∑ M i =1 w i = 1 , thus allowing us to optimize for a reward function -R . In Table 2, we present the scores of MOD, with preference weightings set as w ∈ { ( i, 1 -i ) : i ∈ [5] } . Example generations in Table 3 (more in Appendix F.3) validate that MOD successfully handles this, while RS fails to generate meaningful responses. This phenomenon indicates that we do not even need to specifically tune an unsafe model as in [59], since the knowledge of -R is indeed learned when being tuned for R .

Open Instruction-Following. Finally, we conduct experiments on larger-scale models for general objectives, including two DPO models, TÜLU-2-HH-13B [20] tuned on Anthropic-HH [3] for

GSM COT

BBH COT

-T oxigen

Codex@1

CodeTulu-2-7B

Tulu-2-HH-13B

Tulu-2-Ultra-13B

(0.57,0.1,0.33)

(0.75,0.1,0.15)

safety, TÜLU-2-ULTRA-13B tuned on UltraFeedback [12] for feedback quality. As mentioned in subsection 5.4 and Appendix C.3, our framework is applicable to SFT models, and thus we also look into CODETÜLU-2-7B [20], which is fully tuned by SFT for coding ability. Results of combining them using MOD, benchmarked by Open Instruction-Following [49, 20], are shown in Table 4, Table 5, and Appendix F.5, demonstrating that MOD can effectively combine multiple models (even differently tuned), enabling precise steering based on preference weightings, and even achieves overall improvements in certain cases.

Notably, for any finite number of objectives, there exists infinite possible weightings. In this experiment, we discretize the weightings space using small grid size like 0 . 1 , 0 . 15 , 0 . 3 . Based on this, we randomly set 3 three combinations without careful picking. Intuitively, the weightings should reflect the users' preferences on the general objectives that those models are good at.

5 Theoretical Analysis

In this section, we show the main theoretical results, and defer the full results to Appendix D.

5.1 Failures of parameter-merging paradigm

The optimality of the parameter-merging paradigm [40, 21] primarily relies on reduced reward misspecification hypothesis (see Hypothesis 1 in Appendix D.1 for definition). The following theorem demonstrates that this hypothesis does not hold for almost all f -divergence regularized policies. See detailed proof in Appendix D.5.

Theorem 2. For any f -divergence satisfying one of the following conditions: (i) f is not a barrier function; (ii) I f is Reverse KL-divergence; (iii) f is a strong-barrier function, with finite roots of

∃ N,M ∈ N , Y = { y i } N i =1 , β ∈ R + , a neural network nn = softmax( h θ ( z 0 )) where z 0 ∈ R n and h θ : R n → R N is a continuous mapping, preference weightings w ∈ ∆ M -1 , reference policy π ref ,

and the objectives J 1 , J 2 , . . . , J M representing reward functions R 1 , R 2 , . . . , R M w.r.t. β · I f ( ·∥ π ref ) , s.t. Hypothesis 1 does not hold.

Remark 1 (Clarification) . It is commonly adopted in previous studies [65, 42] that the network receives the same inputs z 0 . Despite the competitive results exhibited in prior works [51, 40, 21], this theorem reveals that parameter-merging lacks a theoretical guarantee in practical scenarios. Besides, although Hypothesis 1 may hold, the mapping from preference weightings w to the optimal merging weightings λ are intricate, and thus simply picking λ as w [40], can yield sub-optimal results.

Another perspective of the same initialization. We can also look into scenarios where only the parameters of the last several layers of π 1 , π 2 , . . . , π M can be different from π ref . 1) If the last layer is a linear projection , then it is equivalent to MOD w.r.t. KL( ·∥ π ref ) , namely linearly combining the logits. 2) If the last layer is self-attention [45], then it can be easily hacked by reversing the sign of Q,K matrices in this layer, which does not influence the value of Q ⊤ K , but significantly harms the effect of parameter-merging. A motivating example is shown in Appendix F.1.

5.2 Necessity of barrier function

Extending the results of [47] to the multi-objective setting, we prove the necessity of f being barrier functions to find an optimal policy π ⋆ for multi-objective alignment. See detailed proof in Appendix D.2.

Theorem 3. If f is not a barrier function, then for ∀ C ∈ R + , N ∈ Z ≥ 4 , M ∈ Z ≥ 2 , Y = { y i } N i =1 , any multi-objective decoding or merging algorithm A : S M +1 × ∆ M -1 →S , there exists a reference policy π ref , policies { π i } i M =1 and π ′ , reward functions {R i } i M =1 , preference weightings w ∈ ∆ M -1 and β ∈ R + , s.t. π i is the optimal policy for R i w.r.t. β · I f ( ·∥ π ref ) (see Definition 1 in Appendix D.1), ∀ i ∈ [ M ] , but

where π A ,w ( y ) := A ( π ref , π 1 , π 2 , . . . , π M , w ) ( y ) .

Remark 2 (Motivating example) . Here we provide a motivating example where f ≡ 0 : let M = 4 , R 1 ( y 1 ) = R 2 ( y 2 ) = 1 , R 1 ( y 2 ) = R 2 ( y 1 ) = -1 , R 1 ( y 3+ k ) = R 2 ( y 3+ k ) = 0 , R 1 ( y 4 -k ) = R 2 ( y 4 -k ) = 1 / 2 , where k ∈ { 0 , 1 } . Then the optimal policy for R 1 is π 1 ( y i ) := δ 1 i , for R 2 is π 2 ( y i ) := δ 2 i , and for R 1 / 2 + R 2 / 2 is π ⋆ ( y i ) := δ 4 -k,i . Thus π A ,w cannot fit π ⋆ both for k = 0 , 1 .

Crucial role of the barrier function. We can apply this theorem to any algorithm which solely utilizes base policies, including RS and MOD. And thus, a barrier function regularization is crucial in multi-objective alignment to bridge different policies, though it was originally intended to prevent degeneration (see Table 3 in [39]) in single-objective alignment. Additionally, the same as a general barrier in interior point methods [34], it obviates the need for introducing slack variables as in [47]. This explains why we should not use non-barrier f -divergences such as total variation and chi-squared.

5.3 Sub-optimality error propagation

While we previously assumed that each base policy is the optimal solution of Eq. (1), here we provide a guarantee for performance when the base policies are sub-optimal. See proof in Appendix D.4.

Theorem 4 (KL-divergence perspective) . Given a reference policy π ref , policies { π i } i M =1 , reward functions {R i } i M =1 , and β ∈ R + . Denote the optimal policy for R i w.r.t. β KL( ·∥ π ref ) as p i , ∀ i ∈ [ M ] . For the reward function ∑ M i =1 w i · R i w.r.t. β KL( ·∥ π ref ) , the performance difference of policy π w ( ·| x ) ∝ ∏ M i =1 π w i i ( ·| x ) from optimal is V ⋆ -V . If for ∀ i ∈ { 1 , . . . , M } , x ∈ X , we have: (i) max y ∈Y | log p i ( y | x ) -log π i ( y | x ) | ≤ L , (ii) KL( π ref ( ·| x ) ∥ π i ( ·| x )) ≤ C , KL( π ref ( ·| x ) ∥ p i ( ·| x )) ≤ C , where L , C ∈ R + , then

Remark 3 (Interpretation of conditions) . Since the primal problem of Eq. (2) restricts the divergence penality under a certain threshold, and people usually adopt an early-stopping technique in practice, p i and π i will not deviate from π ref too much, thus C can be viewed as a small constant. When each π i is close to optimal, the relative distance reflected by L is small as well. The expected calibration error can also be bounded, shown in Proposition 4.

5.4 Beyond f -divergence regularized alignment and multi-objective decoding

While our main results are based on f -divergence regularized aligned LMs and aimed at multiobjective decoding, our framework is also applicable to using SFT models and explaining the effectiveness of other existing decoding algorithms. For example, proxy-tuning [28] tunes only a smaller LM, then applies the difference between the logits of the small tuned and untuned LMs to shift the predictions of a larger untuned model. Its theoretical justification is provided by our framework, under certain assumptions. We provide insights on this line of work [28, 59, 19] and derivations of some other related works [30, 62] in Appendix C.3, further demonstrating the potential for universally applying our approach.

6 Related Work

Algorithms for aligning LMs to human preferences. The widely used RLHF (PPO) approach [36, 42, 52] optimizes over rewards with Reverse KL-divergence as a penalty, where the reward models are learned from human preference datasets. DPO [39] leverages the Bradley-Terry assumption [5] to directly optimize the same objective on preferences, in a supervised manner. Ψ -PO [2] further modifies the reward term to be optimized as other mappings from preference pairs; f-DPO [47] replaces Reverse KL-divergence with other divergence measures. In addition, there are other efforts exploring alternative objectives and frameworks: SLiC-HF [61, 60] refer to the alignment process as sequence likelihood calibration; SPIN [8] iteratively improves the model by leveraging synthetically generated data, thereby circumventing the need for human feedback; OPO [54] employs established norms as constraints, achieving training-free alignment; and Lyu et al. [31] highlight the crucial role of prompt templates. In this work, we mainly focus on RLHF (PPO), DPO, and their extensions.

Decoding-time algorithms for controllable generation. Response-level decoding algorithms sample a whole output y from an anticipated probability distribution p . To achieve this goal, energy-based methods are adopted in many works [37, 25], which involves continuous optimization for LMs to obtain gradient information. Kumar et al. [24] view this task as maximizing log p ( y ) while satisfying some constraints, and use simultaneous gradient descent to solve it. Token-level decoding algorithms decode token y t at timestep t , and are usually more efficient. Among them, Mudgal et al. [33], Liu et al. [29] deploy value models to guide the decoding process; DeRa [30] works on hyperparameter re-alignment and proposes the potential of a special case of MOD, while introducing a per-token distribution approximation; proxy-tuning [28, 59, 19] tunes a small model and applies it to steer a larger base model by operating on logits.

Multi-objective LMs alignment. Multi-objective alignment is the task of aligning language models to multiple objectives simultaneously. This is important for managing tradeoffs among different dimensions [44, 3] and catering to the diverse needs of users [21, 13]. Approaches for multi-objective alignment fall into the following categories: 1) Retraining . The most natural approach to solve multiobjective alignment is to retrain for a linearly combined multiple reward functions (MORLHF [52, 3]). And MODPO [62] retrains the model in a reward-model-free way, by learning a flexible reward representation and directly training on a fixed preference dataset. 2) Parameter-merging . This line of work [40, 21, 27], represented by rewarded soups (RS), aims at providing a training-free solution which obtains weights of the policy as a linear combination of weights of trained policies for each single objective, inspired by [51] and its other applications [41, 26]. Jiang et al. [23] achieve another kind of model-merging through reranking and fusion on outputs. 3) Preference-conditioned prompting . The preference-conditioned learning approaches [64, 4] train a policy conditioned on preference weightings to maximize the expected rewards. This concept is reflected in LMs alignment as preference-conditioned prompting: this line of work [56, 48, 18] directly presents the preference weightings in prompts after a finetuning process. The latter two paradigms are more efficient, while relying heavily on either the reduced mis-specification hypothesis [40] or unguaranteed OOD generalization ability [63], posing challenges in terms of interpretability and robustness.

7 Conclusion

We propose MOD, a simple, training-free yet effective algorithm for multi-objective LMs alignment. By addressing the challenges of retraining and resource-intensive processes, our method provides a decoding-time solution while offering insights into the broader applicability of combining differently tuned models. Through extensive analysis and empirical evidence, we demonstrate the effectiveness and practicality of our method under the f -divergence framework, paving the way for improving LM performance across diverse tasks and use cases.

It is also important to acknowledge the limitations of our work. 1) The analysis is primarily based on tabular setting [55], not taking function approximation error into consideration. 2) Decoding from a response-level probability distribution at the token level may lead to degraded performance, which is likely to be alleviated by energy-based approaches [37, 24, 58].

References

S. M. Ali and S. D. Silvey. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society. Series B (Methodological) , 28(1):131-142, 1966.

Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. ArXiv , abs/2310.12036, 2023.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv , abs/2204.05862, 2022.

Toygun Basaklar, Suat Gumussoy, and Ümit Y. Ogras. Pd-morl: Preference-driven multiobjective reinforcement learning algorithm. ArXiv , abs/2208.07914, 2022.

Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv , abs/2005.14165, 2020.

Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, J'er'emy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Ségerie, Micah Carroll, Andi Peng, Phillip J. K. Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco di Langosco, Peter Hase, Erdem Biyik, Anca D. Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and fundamental limitations of reinforcement learning from human feedback. ArXiv , abs/2307.15217, 2023.

Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. CoRR , abs/2401.01335, 2024.

Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 4299-4307, 2017.

Imre Csiszár. Eine informationstheoretische ungleichung und ihre anwendung auf beweis der ergodizitaet von markoffschen ketten. Magyer Tud. Akad. Mat. Kutato Int. Koezl. , 8:85-108, 1964.

Imre Csiszár. On information-type measure of difference of probability distributions and indirect observations. Studia Sci. Math. Hungar. , 2:299-318, 1967.

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023.

Yi Dong, Zhilin Wang, Makesh Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. SteerLM: Attribute conditioned SFT as an (user-steerable) alternative to RLHF. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 11275-11288, Singapore, December 2023. Association for Computational Linguistics.

Rick Durrett. Probability: Theory and Examples, 4th Edition . Cambridge University Press, 2010.

S.H. Friedberg, A.J. Insel, and L.E. Spence. Linear Algebra . Pearson Education, 2014.

Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 10835-10866. PMLR, 2023.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Jiexin Wang, Huimin Chen, Bowen Sun, Ruobing Xie, Jie Zhou, Yankai Lin, et al. Controllable preference optimization: Toward controllable multi-objective alignment. arXiv preprint arXiv:2402.19085 , 2024.

James Y Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, and Muhao Chen. Offset unlearning for large language models. arXiv preprint arXiv:2404.11045 , 2024.

Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023.

Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. CoRR , abs/2310.11564, 2023.

Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. arXiv preprint arXiv:2307.04657 , 2023.

Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. In Annual Meeting of the Association for Computational Linguistics , 2023.

Sachin Kumar, Eric Malmi, Aliaksei Severyn, and Yulia Tsvetkov. Controlled text generation as continuous optimization with multiple constraints. In Neural Information Processing Systems , 2021.

Sachin Kumar, Biswajit Paria, and Yulia Tsvetkov. Gradient-based constrained sampling from language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022 , pages 2251-2277. Association for Computational Linguistics, 2022.

Daniel Lawson and Ahmed H Qureshi. Merging decision transformers: Weight averaging for forming multi-task policies. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023 , 2023.

Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, and T. Zhang. Mitigating the alignment tax of rlhf, 2023.

Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A. Smith. Tuning language models by proxy. CoRR , abs/2401.08565, 2024.

Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, and Asli Celikyilmaz. Don't throw away your value model! generating more preferable text with value-guided monte-carlo tree search decoding, 2023.

Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding-time realignment of language models. arXiv preprint arXiv:2402.02992 , 2024.

Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. Keeping llms aligned after fine-tuning: The crucial role of prompt templates. ArXiv , abs/2402.18540, 2024.

Costas Mavromatis, Petros Karypis, and George Karypis. Pack of llms: Model fusion at test-time via perplexity optimization. arXiv preprint arXiv:2404.11531 , 2024.

Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. Controlled decoding from language models. CoRR , abs/2310.17022, 2023.

Yurii Nesterov. Lectures on Convex Optimization . Springer Publishing Company, Incorporated, 2nd edition, 2018.

Andi Nika, Debmalya Mandal, Parameswaran Kamalaruban, Georgios Tzannetos, Goran Radanovi´ c, and Adish Singla. Reward model learning vs. direct policy optimization: A comparative analysis of learning from human preferences. arXiv preprint arXiv:2403.01857 , 2024.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022 , 2022.

Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. COLD decoding: Energy-based constrained text generation with langevin dynamics. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 , 2022.

Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training, 2018.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023 , 2023.

Alexandre Ramé, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023.

Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: on the benefits of weight averaged reward models. CoRR , abs/2401.12187, 2024.

Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize from human feedback. CoRR , abs/2009.01325, 2020.

Wenpin Tang. Fine-tuning of diffusion models via stochastic control: entropy regularization and beyond. arXiv preprint arXiv:2403.06279 , 2024.

Peter Vamplew, Richard Dazeley, Cameron Foale, Sally Firmin, and Jane Mummery. Humanaligned artificial intelligence is a multiobjective problem. Ethics and Information Technology , 20:27 - 40, 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5998-6008, 2017.

Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. https://github. com/huggingface/trl , 2020.

Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints. In The Twelfth International Conference on Learning Representations , 2024.

Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards. CoRR , abs/2402.18571, 2024.

Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. How far can camels go? exploring the state of instruction tuning on open resources, 2023.

Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. Helpsteer: Multi-attribute helpfulness dataset for steerlm. ArXiv , abs/2311.09528, 2023.

Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 23965-23998. PMLR, 2022.

Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.01693 , 2023.

Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint, 2024.

Chunpu Xu, Steffi Chern, Ethan Chern, Ge Zhang, Zekun Wang, Ruibo Liu, Jing Li, Jie Fu, and Pengfei Liu. Align on the fly: Adapting chatbot behavior to established norms. CoRR , abs/2312.15907, 2023.

Yichong Xu, Ruosong Wang, Lin Yang, Aarti Singh, and Artur Dubrawski. Preference-based reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems , 33:18784-18794, 2020.

Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. CoRR , abs/2402.10207, 2024.

Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. A theoretical analysis of nash learning from human feedback under general kl-regularized preference. arXiv preprint arXiv:2402.07314 , 2024.

Stephen Zhao, Rob Brekelmans, Alireza Makhzani, and Roger Grosse. Probabilistic inference in language models via twisted sequential monte carlo, 2024.

Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du, Lei Li, Yu-Xiang Wang, and William Yang Wang. Weak-to-strong jailbreaking on large language models. CoRR , abs/2401.17256, 2024.

Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. CoRR , abs/2305.10425, 2023.

Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. Calibrating sequence likelihood improves conditional language generation. In The Eleventh International Conference on Learning Representations , 2023.

Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one-preference-for-all: Multi-objective direct preference optimization for language models. CoRR , abs/2310.03708, 2023.

Zhaoyi Zhou, Chuning Zhu, Runlong Zhou, Qiwen Cui, Abhishek Gupta, and Simon Shaolei Du. Free from bellman completeness: Trajectory stitching via model-based return-conditioned supervised learning. In The Twelfth International Conference on Learning Representations , 2024.

Baiting Zhu, Meihua Dang, and Aditya Grover. Scaling pareto-efficient decision making via offline multi-objective rl. ArXiv , abs/2305.00567, 2023.

Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. CoRR , abs/1909.08593, 2019.

E. Zitzler and L. Thiele. Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE Transactions on Evolutionary Computation , 3(4):257-271, 1999.

Xu Zou, Da Yin, Qingyang Zhong, Hongxia Yang, Zhilin Yang, and Jie Tang. Controllable generation from pre-trained language models via inverse prompting. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , 2021.

Appendices

A Impact Statement

Our work proposes a decoding-time language model alignment method aimed at advancing academic research and meeting industry needs. If misused in downstream tasks, especially as what we have shown in Table 7, it could potentially induce language models to generate harmful, offensive, or privacy-infringing content, leading to privacy breaches and societal harm. Nevertheless, this is not directly related to our research, as our primary focus is on a general algorithm with theoretical guarantees.

B Notation

Here we introduce a set of notation to be used throughout. For any differentiable function f , let ∇ f denote its gradient. For any N ∈ N , we denote the index set { 1 , · · · , N } as [ N ] . Let e s be the s th standard basis vector. For any i, j ∈ Z ≥ 0 , δ ij represents the Kronecker delta function [15], which output 1 if i = j otherwise 0 . For any n ∈ N , ∆ n represents the n -dimensional probability simplex { ( p 1 , . . . , p n +1 ) : p i ≥ 0 , ∀ i ∈ [ n +1] , ∑ n +1 j =1 p j = 1 } , and ∆( X ) represents the set of probability distributions over a set X . X denotes the prompt set, Σ denotes the alphabet set, Y ⊂ Σ ∗ denotes the response set, and the policy set S is defined as all mappings from X to ∆( Y ) .

C Main Algorithm

C.1 Pipeline

end return sequence with the highest f -score in S completed .

C.2 Divergence measures and closed-form policies

We acknowledge that commonly used f -divergence measures have been introduced in [47] and show them here for completeness:

Here we show the optimal sampling policies for multi-objective w.r.t. these divergence measures:

And we show the optimal decoding policies for multi-objective w.r.t. these divergence measures:

C.3 Extended variants

SFT. We assume that, supervised fine-tuning (SFT) on pre-trained model M -yielding M + , is implicitly optimizing a underlying reward r w.r.t. Reverse KL-divergence, i.e.

Based on this, our approach, namely Eq. (7), is applicable to SFT models.

Proxy-tuning [28] & jail-breaking [59]. Based on the claim above, for another base model M , we thus have

which reflects the tuned version of model M . And this is exactly the proxy-tuning approach, validated by extensive experiments in [28]. Reversing the position of P M + and P M -yields jail-breaking [59]. δ -unlearning [19] is the same.

Multi-objective proxy-tuning. Moreover, it is worth noting that, our method can be applied as a lightweight approach for large-scale models, as a multi-objective extension of proxy-tuning [28]. In particular, to tune a large pre-trained model M , we can first tune M + 1 , M + 2 , . . . , M + M from a relatively smaller model M -by PPO, DPO or SFT, and decode y t at timestep t as

DeRa [30]. Given P M + ( y | x ) ∝ P M -( y | x ) · exp( 1 β r ( y | x )) , then

yields a β ′ -realigned version of M -.

MODPO [62]. Assuming π i is the optimal policy for R i w.r.t. β KL( ·∥ π ref ) , ∀ i ∈ [ M ] , then the optimal policy for ∑ M i =1 w i R i w.r.t. β KL( ·∥ π ref ) , π ⋆ ∝ ∏ π w i i , is the minimizer of

where σ is sigmoid function, and D 1 is the comparison dataset corresponding to R 1 . Since

we can substitute this term with learned reward representations r ϕ,i and yields

which is the optimization objective of MODPO.

D Full Theoretical Results and Omitted Proofs

D.1 Definitions

Definition 1 ( f -divergence [1, 10, 11]) . For probability measures P and Q , let µ be a dominating measure of P and Q (i.e. P, Q ≪ µ ), and let p, q be the Radon-Nikodym derivative [14] dP dµ , dQ dµ respectively. For simplicity, here we assume q > 0 almost surely. Then f -divergence from P to Q is defined as

where f is convex on R + , satisfying f (1) = 0 . Most useful divergence measures are included in f -divergences, and the commonly used ones and corresponding f are introduced in Appendix C.2.

Definition 2 (Barrier function [34]) . Given conditions satisfied in Definition 1, if additionally 0 / ∈ dom( ∇ f ) , then f is a barrier function. If a barrier function f is continuously differentiable and strongly convex on R + , then f is a strongly convex and smooth barrier function (abbreviated as strong-barrier function).

Definition 3 (Expected calibration error [17, 47]) . Denote the ground truth distribution as P , prompt as X and response as Y . The expected calibration error of a stochastic policy π is defined as

Hypothesis 1 (Reduced reward mis-specification [51, 40, 21]) . Let θ i be the parameter of the optimal policy for objective J i , ∀ i ∈ [ M ] , and θ ∗ w be the parameter of the optimal policy for the interpolated objective ∑ M i =1 w i · J i , then this hypothesis claims

D.2 Proofs of subsection 5.2

Theorem 3. If f is not a barrier function, then for ∀ C ∈ R + , N ∈ Z ≥ 4 , M ∈ Z ≥ 2 , Y = { y i } N i =1 , any multi-objective decoding or merging algorithm A : S M +1 × ∆ M -1 →S , there exists a reference

policy π ref , policies { π i } i M =1 and π ′ , reward functions {R i } i M =1 , preference weightings w ∈ ∆ M -1 and β ∈ R + , s.t. π i is the optimal policy for R i w.r.t. β · I f ( ·∥ π ref ) (see Definition 1 in Appendix D.1), ∀ i ∈ [ M ] , but

Proof. Since f is not a barrier function, 0 ∈ dom ( ∇ f ) . Now we can define p := max x ∈ [0 ,N ] ∇ f ( x ) ,

q

min

:=

x

[0

,N

∈

and we pick

∇

]

k

and

π

′

(

y

i

=

) =

δ

ik

f

(

x

)

,

r

:=

x

∈

argmin

j

,

3

,

4

∈{

,...,N

∀

i

∈

[

N

}

]

,

[0

,N

π

f

]

A

,w

j

(

∈ {

x

(

y

)

j

3

-

min

x

∈

)

[0

,N

. Let

,

4

]

π

ref

, . . . , M

(

f

(

x

)

,

s

:=

y

}

1

N

i

) =

,

N

N

π

. And set

-

-

1

(

R

1

2

3

y

i

(

·

C

) =

y

i

) =

. Let

δ

1

i

,

π





2



4

p

q

p







2

w

= (0

2

(

y

.

i

) =

+2

r

-

5

δ

,

2

+2

2

p

+

q

Let β = 1 , then the optimization objective for R 1 w.r.t. I f is J 1 ( π ) := E y ∼ π [ R 1 ( y )] -I f ( π ∥ π ref ) , and the Lagrangian dual is

As the objective is convex and the constraints are affine, we can directly apply the Karush-KuhnTucker conditions [34]:

q

+

0

i

s

-

2

r

+

r

s

.

5

,

π

,

j

-

2

0

︸

(

, . . . ,

︷︷

N

2

-

y

s

i

) =

0

)

︸

1

N

i

= 1

i

= 2

i

=

k

o/w

,

,

Eq. (8) implies

If π ⋆ 1 ( y 1 ) > 0 , we have

̸

and then for ∀ j = 1 ,

̸

Combining it with Eq. (9) yields π ⋆ 1 ( y j ) = 0 for ∀ j = 1 , which is exactly π 1 . Note that we have

For any π ′ with π ′ ( y 1 ) = 0 , we have

Thus π 1 is the optimal policy for R 1 w.r.t. I f ( ·∥ π ref ) . Similarly, π 2 is the optimal policy for R 2 w.r.t. I f ( ·| π ref ) . By convexity of f , the minimum of I f ( π ∥ π ref ) is obtained when π = π ref , and thus π j is the optimal policy for R j w.r.t. I f ( ·∥ π ref ) , for ∀ j ∈ { 3 , 4 , . . . , M } . Therefore, all conditions are well satisfied by this construction. Note that

While by the selection of k , we have

Comparing Eq. (10) with Eq. (11), we have

Note that π ref is a uniform distribution and both π A ,w , π ′ are one-point distributions, thus I f ( π A ,w ∥ π ref ) = I f ( π ′ ∥ π ref ) . We have

Lemma1. Given a reference policy π ref , reward function R , a strong-barrier function f and β ∈ R + , then

where Z ( x ) is the normalization factor w.r.t. x , is the optimal policy for

Proof. The lemma is revealed by Theorem 1 in [47]. For completeness, we give a brief proof here. Since f is convex and barrier, we can directly use Lagrange multiplier to solve

for each x ∈ X , which implies

where λ ( x ) ∈ R . Taking Z ( x ) := βλ ( x ) completes the proof.

Proposition 1. Given a reference policy π ref , optimal policies π 1 , π 2 , . . . , π M for each reward function R 1 , R 2 , . . . , R M w.r.t. β · I f ( ·∥ π ref ) , β ∈ R + , and w ∈ ∆ M -1 , if f is a strong-barrier function, then the optimal policy for reward function r = ∑ M i =1 w i · R i w.r.t. β · I f ( ·∥ π ref ) is:

where Z ( x ) is the normalization factor w.r.t. x , and numerically computable when |Y| is finite.

Proof. As Lemma 1 shows,

and

Apply Eq. (12) into Eq. (13), we get

where Z ( x ) := Z ⋆ ( x ) -∑ M i =1 w i Z i ( x ) . And Z ( x ) is the root of ϕ x ( t ) = 0 , where

Since f is strongly convex and continuously differentiable, ϕ x ( t ) is monotonically decreasing and continuous. If |Y| is finite, we can set

Thus Z ( x ) ∈ [ t 1 ,x , t 2 ,x ] . Finally, Z ( x ) can be numerically computed by bisection method .

D.3 Proof of key theorem

Proposition 2 (Policy-to-reward mapping) . Given a reference policy π ref , optimal policies π 1 , π 2 , . . . , π M for each reward function R 1 , R 2 , . . . , R M w.r.t. β · I f ( ·∥ π ref ) , β ∈ R + , and w ∈ ∆ M -1 , if f is a strong-barrier function, then for ∀ x ∈ X , y 1 , y 2 ∈ Y , we have:

then we have

Proof. As Eq. (3) shows,

for ∀ i ∈ [ M ] , y ∈ Y , where Z i ( x ) is the normalization factor. Thus

Since β > 0 , the proposition holds.

Theorem 5 (Key theorem) . Given a reference policy π ref , optimal policies π 1 , π 2 , . . . , π M for each reward function R 1 , R 2 , . . . , R M w.r.t. β · I f ( ·∥ π ref ) , β ∈ R + , and w ∈ ∆ M -1 , if f is a strong-barrier function, then for ∀ x ∈ X , w ∈ ∆ M -1 , ∃ C ∈ R , s.t.

is an optimal solution for

Proof. First we define

From Eq. (14), we have

Then let

and C := ∑ M i =1 w i · R i ( y ′ | x ) . Suppose y ′ is not an optimal solution for Eq. (15), then ∃ y ′′ ∈ Y , s.t. π ref ( y ′′ | x ) > π ref ( y ′ | x ) and ∑ M i =1 w i · R i ( y ′′ | x ) ≥ ∑ M i =1 w i · R i ( y ′ | x ) . Since f is strongly convex, g x is continuously increasing and invertible. Thus

contradictory to the definition of y .

D.4 Proofs of subsection 5.3

Proposition 3 (Eq. 13,14 in [39]) . If I f is Reverse KL-divergence, Eq. (2) can be viewed as

where π opt is the optimal policy for reward function r w.r.t. β · I f ( ·∥ π ref ) . Thus we can evaluate a policy π using -KL( π ∥ π opt ) .

Proof. This proposition is revealed by Eq. 13,14 in [39]. For completeness, we give a brief proof here. Define Z ( x ) := log ∑ y ∈Y π ref ( y | x ) exp( 1 β r ( y | x )) , which is a constant. Then we have

Lemma 2. Given n, m ∈ N , x ∈ ∆ n -1 , x ≻ 0 , y ∈ R n and C ∈ R + , if ∑ n i =1 x i y i ≤ C , then

Proof. Set f ( y ) := ∑ n i =1 x i exp( -y i ) , h ( y ) := ∑ n i =1 x i y i -C , and the Lagrangian dual L ( y, λ ) := f ( y )+ λ · h ( y ) . Since both f and h are convex, we can directly apply Karush-Kuhn-Tucker conditions :

From Eq. (16) we get for ∀ i ∈ [ n ] . Then we have

Theorem 4 (KL-divergence perspective) . Given a reference policy π ref , policies { π i } i M =1 , reward functions {R i } i M =1 , and β ∈ R + . Denote the optimal policy for R i w.r.t. β KL( ·∥ π ref ) as p i , ∀ i ∈ [ M ] . For the reward function ∑ M i =1 w i · R i w.r.t. β KL( ·∥ π ref ) , the performance difference of policy π w ( ·| x ) ∝ ∏ M i =1 π w i i ( ·| x ) from optimal is V ⋆ -V . If for ∀ i ∈ { 1 , . . . , M } , x ∈ X , we have: (i) max y ∈Y | log p i ( y | x ) -log π i ( y | x ) | ≤ L , (ii) KL( π ref ( ·| x ) ∥ π i ( ·| x )) ≤ C , KL( π ref ( ·| x ) ∥ p i ( ·| x )) ≤ C , where L , C ∈ R + , then

Proof. The optimal policy for R i w.r.t. β KL( ·∥ π ref ) is p i ( ·| x ) ∝ π ref ( ·| x ) exp( 1 β r ( ·| x )) and the optimal policy for ∑ M i =1 w i · R i w.r.t. β KL( ·∥ π ref ) is π ⋆ ( ·| x ) ∝ ∏ M i =1 p w i i ( ·| x ) .

Since max y ∈Y | log p i ( y | x ) -log π i ( y | x ) | ≤ L , we have

for ∀ x ∈ X , i, j ∈ [ M ] . Since KL( π ref ( ·| x ) ∥ π i ( ·| x )) ≤ C , we have

for ∀ x ∈ X , i ∈ [ M ] . By Lemma 2,

Similarly,

Note that and

̸

Thus

Finally we have

Lemma 3 (Theorem 2 in [47]) . Suppose π 1 ( ·| x ) and π 2 ( ·| x ) be two policies, then

Proposition 4 (Calibration error perspective) . The expected calibration error (see Definition 3) of π w can be bounded as

Proof. This proposition directly comes from combining Lemma 3 with Theorem 4.

D.5 Proofs of subsection 5.1

Theorem 2. For any f -divergence satisfying one of the following conditions: (i) f is not a barrier function; (ii) I f is Reverse KL-divergence; (iii) f is a strong-barrier function, with finite roots of

∃ N,M ∈ N , Y = { y i } N i =1 , β ∈ R + , a neural network nn = softmax( h θ ( z 0 )) where z 0 ∈ R n and h θ : R n → R N is a continuous mapping, preference weightings w ∈ ∆ M -1 , reference policy π ref , and the objectives J 1 , J 2 , . . . , J M representing reward functions R 1 , R 2 , . . . , R M w.r.t. β · I f ( ·∥ π ref ) , s.t. Hypothesis 1 does not hold.

Proof. (i) If f is not a barrier function, Hypothesis 1 does not hold immediately from Theorem 3.

(ii) If I f is Reverse KL-divergence, we let N = 3 , M = 3 , and h θ ( z 0 ) = W (2) θ σ ( W (1) θ z 0 ) , where σ is ReLU ( · ) . Weset R i ( y j ) = δ ij , π ref ( y i ) = 1 / 3 for ∀ i, j ∈ [3] , z 0 = 1 and β = 1 . Then the optimal policies are W (1) θ 1 = e 1 , W (2) θ 1 = ( 100 000 000 ) for R 1 w.r.t. KL( ·∥ π ref ) , W (1) θ 2 = e 2 , W (2) θ 2 = ( 000 010 000 ) for

R 2 w.r.t. KL( ·∥ π ref ) , and W (1) θ 3 = e 3 , W (2) θ 3 = ( 000 000 001 ) for R 3 w.r.t. KL( ·∥ π ref ) . Thus we have h ∑ 3 j =1 λ j θ j ( z 0 ) = ( λ 2 1 , λ 2 2 , λ 2 3 ) ⊤ . Given w = (0 , 1 / 3 , 2 / 3) , the optimal policy π ⋆ should output

̸

π ⋆ ( y 1 ) = 1 1+exp(1 / 3)+exp(2 / 3) , π ⋆ ( y 2 ) = exp(1 / 3) 1+exp(1 / 3)+exp(2 / 3) and π ⋆ ( y 3 ) = exp(2 / 3) 1+exp(1 / 3)+exp(2 / 3) . Note that

thus there is no solution λ ∈ ∆ 2 , t ∈ R + for ( λ 2 1 , λ 2 2 , λ 2 3 ) ⊤ = ( t, t + 1 3 , t + 2 3 ) ⊤ , i.e. there is no λ s.t. softmax ( h ∑ 3 j =1 λ j θ j ( z 0 ) ) = ( π ⋆ ( y 1 ) , π ⋆ ( y 2 ) , π ⋆ ( y 3 ) ) , i.e. Hypothesis 1 does not hold.

(iii) If f is a strong-barrier function, with finite roots of

we let N = 3 , M = 2 , h θ ( z 0 ) = W θ ( z 0 ) , z 0 = 1 , R 1 ( y i ) = δ 1 i , R 2 ( y i ) = δ 2 i and π ref ( y i ) = 1 / 3 , for ∀ i ∈ [3] . From Eq. (3) the optimal policy for J 1 is π θ 1 ( y i ) = 1 3 ( ∇ f ) ( -1) ( 1 β δ 1 i -Z ) , and the optimal policy for J 2 is π θ 2 ( y i ) = 1 3 ( ∇ f ) ( -1) ( 1 β δ 2 i -Z ) , where Z is the normalization factor. And these policies can be learned by setting W θ i = ( log π θ i ( y 1 ) , log π θ i ( y 2 ) , log π θ i ( y 3 ) ) ⊤ .

We set a := π θ 1 ( y 1 ) = 1 3 ( ∇ f ) ( -1) ( 1 β -Z ) , b := π θ 1 ( y 2 ) = π θ 1 ( y 3 ) = 1 3 ( ∇ f ) ( -1) ( -Z ) . Thus we have

From Proposition 1, the optimal policy for w 1 · J 1 + w 2 · J 2 is

where Z ⋆ w is the normalization factor. By linearly merging the weights of π θ 1 and π θ 2 , we have

where Z λ is the normalization factor.

With symmetry, Eq. (24), (25) and Hypothesis 1 indicate that π 1 2 θ 1 + 1 2 θ 2 = π ⋆ ( 1 2 , 1 2 ) , thus

and combining them with Eq. (22) yields

Given the condition, the solution set ( a, b ) to Eq. (23), (26) is finite, thus there exists β ∈ R + s.t. Eq. (22) does not hold, implying that Hypothesis 1 does not hold.

E Implementation Details

Codebase. Our codebase is mainly based on trl [46] ( https://github.com/ huggingface/trl ), MODPO [62] ( https://github.com/ZHZisZZ/modpo ), RiC [56] ( https://github.com/YangRui2015/RiC ) and Finegrained RLHF [52] ( https: //github.com/allenai/FineGrainedRLHF ), and has referred to f-divergence DPO [47] ( https://github.com/alecwangcq/f-divergence-dpo ), PackLLM [32] ( https: //github.com/cmavro/PackLLM ), and DPA [48] ( https://github.com/Haoxiang-Wang/ directional-preference-alignment ). We release the code at https://github.com/srzer .

Datasets. For Reddit Summary , we adopt the Summarize-from-Feedback dataset ( https:// huggingface.co/datasets/openai/summarize_from_feedback ); For Helpful Assistant , we adopt the Anthropics-HH dataset ( https://huggingface.co/datasets/Anthropic/hh-rlhf ); For Safety Alignment , we adopt a 10 -k subset ( https://huggingface.co/datasets/ PKU-Alignment/PKU-SafeRLHF-10K ); For Helpsteer , we adopt the Helpsteer dataset ( https: //huggingface.co/datasets/nvidia/HelpSteer ).

SFT. For Reddit Summary and Helpful Assistant , we supervisedly fine-tune the LLAMA2-7B models on the Summarize-from-Feedback dataset, following the practice of [46, 56]; For Safety Alignment , we directly deploy a reproduced model ( https://huggingface.co/PKU-Alignment/ alpaca-7b-reproduced ); For HelpSteer , we supervisedly fine-tune a MISTRAL-7B model on the HelpSteer dataset, following the practice of [62].

Reward models. We deploy off-shelf reward models for RLHF (PPO) training and evaluations. For Reddit Summary , we use https://huggingface.co/Tristan/ gpt2_reward_summarization for summary and https://huggingface.co/CogComp/ bart-faithful-summary-detector for faith; For Helpful Assistant , we use https: //huggingface.co/Ray2333/gpt2-large-helpful-reward_model for helpfulness, https://huggingface.co/Ray2333/gpt2-large-harmless-reward_model for harmlessness and https://huggingface.co/mohameddhiab/humor-no-humor for humor; For Safety Alignment , we use https://huggingface.co/PKU-Alignment/beaver-7b-v1.0-reward for helpfulness and https://huggingface.co/PKU-Alignment/beaver-7b-v1.0-cost for harmlessness; For HelpSteer , we use https://huggingface.co/Haoxiang-Wang/ RewardModel-Mistral-7B-for-DPA-v1 for all attributes of rewards, including helpfulness, correctness, coherence, complexity and verbosity.

Training hyper-parameters. For PPO, we follow the settings of [56] and train for 100 batches; for DPO, we follow [62] with minimal modifications as BATCH_SIZE = 1 and MAX_LENGTH = 256 .

Inference hyper-parameters. For PPO, we follow the settings of [56] with NUM_BEAMS = 1 ; for DPO, we follow [62] with BATCH_SIZE = 4 , MAX_LENGTH = 200 and NUM_BEAMS = 1 .

Inference code. Here we provide the inference code. Notably, to prevent potential precision explosion, we approximate the solution for JSD same as Reverse KL-divergence, as they are inherently similar.

Evaluation setups. The evaluation scores are calculated on a down-sampled dataset, by off-shelf reward models. For Reddit Summary and Helpfull Assistant , we uniformly sample a subset of 2 k prompts from the test set, following [56]; for Safety Alignment and HelpSteer , we randomly sample of subset of 200 prompts from the validation set. The generation configurations are set as identical for all algorithms.

Compute resources. Our main experiments are conducted on NVIDIA RTX A6000. For training RLHF, MORLHF models, the number of workers are set as 3 , each taking up 20 , 000 Mof memory, running for 18 hours; for training DPO, MODPO models, the number of workers are set as 2 , each taking up 40 , 000 Mof memory, running for 3 hours.

F Supplementary Results

In this section, we provide additional experimental results for supplementation.

F.1 Motivating example

This motivating experiment is based on FineGrainedRLHF [52]. We tune two T5-LARGE models M 1 , M 2 for relevance and factuality respectively, based on a reproduced SFT model and pretrained reward models, following the instructions of [52]. And we obtain M 2 via reversing the sign of Q,K matrices of the last two layers of M 1 . The preference weightings are set as w ∈ { ( i/ 10 , 1 -i/ 10) : i ∈ { 0 , 1 , . . . , 10 }} . As Figure 5 shows, though the performance is comparable based on normally trained models, a noticeable lag in the performance of RS emerges after a simple reversal of certain parameters.

0.3

0.4

0.5

0.6

0.7

R1 (rel)

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

R2 (fac)

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

ours

rs

0.3

0.4

0.5

0.6

0.7

R1 (rel)

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

R2 (fac)

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

ours

rs

F.2 Additional results for Helpful Assistant

For 3 -reward setting in Helpful Assistant task, we provide the 3 d-visualization and numerical results of MOD and RS for many configurations of preference weightings in Figure 6, Table 6, showing that MODgenerally beats RS .

F.3 Additional results for BeaverTails

For MOD, the effect of harmfulness can be obtained from a harmless model by setting the preference weighting as a negative value. In contrast, RS fails to generate meaningful responses under this setting. Example generations are provided in Table 7.

0.5

0.0

0.5

1.0

1.5

2.0

R

1 (helpful)

1.0

0.5

0.0

0.5

1.0

1.5

R

2 (harmless)

0.5

0.0

0.5

1.0

1.5

R

3 (humor)

Ours

RS

F.4 Additional results for HelpSteer

By supervisedly fine-tuning a MISTRAL-7B model on HelpSteer dataset, we obtain the reference policy π ref . And then we tune models π 1 f , π 2 f , π 3 f using f -DPO on three pair-comparison datasets for helpfulness, complexity and verbosity. Specifically, we early-stop ( 3 epochs) the tuning process, to examine the performance when base policies are sub-optimal. For f -DPO models trained w.r.t. Reverse KL-divergence, JSD, 0 . 3 -divergence and 0 . 5 -divergence, we present the score for each attribute of MOD and RS, with weightings set as w = (0 . 33 , 0 . 33 , 0 . 33) , as shown in Table 8, 9, 10, 11. It can be observed that MOD still successfully combines their advantages and generally achieves stronger performance than RS.

F.5 Additional results for Open Instruction-Following

Additional numerical results of combining 2 TÜLU models are provided in Table 12.

F.6 Example generations

Example generations for each dataset are shown in Table 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23. For each dataset, we show a representative prompt in the down-sampled dataset, and one generated response for each model/algorithm, with preference weightings set as w = (0 . 5 , 0 . 5) for MOD and RS.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] .

Justification: Both the abstract and introduction reflect our paper's contributions (as listed in section 1) and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of this work in section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Proofs are provided in Appendix D.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide implementation details in Appendix E, including datasets, models, hyper-parameters, etc.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The dataset is accessible, and we have provided the urls in Appendix E. We release the code at https://github.com/srzer/MOD .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We sincerely follow the training details of [56, 62], and state this point in Appendix E. The test details are specified in Appendix E as well.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Since our work focus on inference of language models, the results are provided as evaluation scores on validation/test set. And we have explained how they are calculated in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide details of computing resources like compute device, number of workers, memory and time of execution in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics, and strictly followed it in this work.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Yes, we talk about boarder impacts in Appendix A.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper focus on inference of language models, and thus there is no data or models to release.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes. We acknowledge the codebases we have referred to, cite and provide the urls of datasets and models in Appendix E.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a decoding-time alignment method called Multi-Objective Decoding (MOD) for aligning language models (LMs) with multiple objectives. MOD combines a set of models aligned for individual rewards and allows any weightings (even not-all-positive) for rewards. MOD leverages a common form among f-divergence regularized alignment approaches to derive an efficient decoding strategy that greedily selects the next token from an algebraic combination of predicted probabilities of all base models. MOD can maximize an interpolated reward function without extensive retraining. The paper theoretically demonstrates the sub-optimality of existing approaches and establishes optimality guarantees for MOD. Empirical results show MOD's effectiveness, with a 12.8% overall reward improvement over a parameter-merging baseline when optimizing for three objectives.\n\nSTRENGTHS:\n* **Flexibility in Objectives Alignment**: MOD's primary strength lies in its ability to align language models with multiple objectives simultaneously. This flexibility allows it to balance and prioritize different user needs and preferences without the need for retraining the model for each new objective combination.\n* **Efficiency and Simplicity**: The algorithm is efficient and simple to implement, as it only need to operate the probabilities at decoding time.\n* **Theoretical Robustness and Empirical Validation**: MOD is underpinned by a strong theoretical foundation, with proofs that demonstrate its optimality and sub-optimality of existing methods. The paper provides empirical evidence of MOD's effectiveness across various tasks and models, showcasing its practical applicability and robustness in real-world scenarios.\n\nWEAKNESSES:\nI don't see any major weaknesses in this paper.\n\nQUESTIONS:\n* Is the conditions of Eq. 6 correctly?\n* Line 231-233: Can you explain more about this?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-26"}

comments: SUMMARY:
The authors propose a decoding method that aims to combine the predictions of diverse models that are aligned with different objectives. In their multi-objective setting, the goal is to find an optimal policy that maximize a weighted, multi-objective reward, given the policy aligned to each of the individual rewards. In particular, the authors propose a reformulation using Legendre transform to bypass calculating Z (normalization) at a sequence level.

STRENGTHS:
1. The authors provide detailed theoretical analysis to justify their approach.
2. The authors show that the proposed method can handle negative weights for rewards, which cannot be accomplished by previous work.

WEAKNESSES:
1. The baselines appears weak. For example, in Appendix F, the main comparison is against RS. However, RS cannot even outperform the best individual model in all experiments (Tables 7,8,9,10).
2. The proposed MOD also seems not much stronger. MOD can only beat the best individual model on 2/4 settings in Appendix F.
3. Lack of baselines. It would be helpful if the authors can include more generic ensemble baselines such as weighted averaging/voting.

QUESTIONS:
1. Has the authors studied the inference overhead of the proposed MOD?

LIMITATIONS:
Conclusion and throughout the work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn many practical uses of RLHF the reward function is the convex combination of several rewards. Instead of training a single policy attempting to maximize the expected aggregate reward (subject to the usual regularization keeping it close to an anchor policy), the authors show that one can train separate policies, one for each reward and then mix them at decoding time using the same convex combination in log-probability space.\n\nOne important consequence is that one can change the weights on various rewards at decoding time, per response, making the algorithm very appealing for situations where the balance between certain rewards needs to change depending on the prompt/context.\n\nSTRENGTHS:\nNovel approach to dealing with rewards that are a linear mix of \"elementary\" rewards; mathematically sound.\n\nOffers a simple, practical way of changing the mix of \"elementary\" rewards at decoding time, per model response.\n\nWEAKNESSES:\nThe presentation could be much simpler, starting from the ubiquitous case of using KL divergence for regularization, which also leads to the elegant log-linear combination in Eq. (7). The general case for f-divergence could be mentioned, but relegated to the already prodigious appendix.\n\nOne technical weakness of the proposed approach is that one needs to serve/run M different policies at decoding time, which is significant overhead.\n\nAfter completing the review I have become aware of the work in:\n@misc{wang2024conditionedlanguagepolicygeneral,\n      title={Conditioned Language Policy: A General Framework for Steerable Multi-Objective Finetuning}, \n      author={Kaiwen Wang and Rahul Kidambi and Ryan Sullivan and Alekh Agarwal and Christoph Dann and Andrea Michi and Marco Gelmi and Yunxuan Li and Raghav Gupta and Avinava Dubey and Alexandre Ramé and Johan Ferret and Geoffrey Cideron and Le Hou and Hongkun Yu and Amr Ahmed and Aranyak Mehta and Léonard Hussenot and Olivier Bachem and Edouard Leurent},\n      year={2024},\n      eprint={2407.15762},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG},\n      url={https://arxiv.org/abs/2407.15762}, \n}\n\nSection 6 and Appendix E in [wang2024conditionedlanguagepolicygeneral] are directly relevant to this paper, deriving a sensitivity analysis for logit mixing, the log-linear combination in Eq. (7).\n\nQUESTIONS:\nThe reformulation using Legendre transformation in Section 4.2 was hard to follow.\n\nAgain, a crisp derivation for the case of KL regularization that anyone can follow would greatly improve the reach, and implicitly impact of the paper.\n\nLIMITATIONS:\nOne technical weakness of the proposed approach is that one needs to serve/run M different policies at decoding time, which is significant overhead.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Multi-Objective Decoding (MOD), a novel algorithm designed to align language models (LMs) with multiple human preferences simultaneously during decoding. MOD addresses the limitations of existing methods that optimize LMs for a single reward function, thereby providing flexibility and efficiency without the need for retraining. The authors define multi-objective reward functions and assume the existence of single-objective aligned LMs optimized for specific rewards. By leveraging the properties of strong-barrier functions and using the Legendre transform, they derive a closed-form solution for linearly combining the outputs of different models, achieving multi-objective alignment. This method guarantees optimality under certain conditions and transforms response-level decoding into efficient token-level decoding using greedy search. Extensive experiments validate MOD&#39;s effectiveness, demonstrating significant improvements in reward optimization compared to parameter-merging baselines.\n\nSTRENGTHS:\nMOD introduces a novel method for multi-objective alignment, enabling language models to align with multiple objectives simultaneously during decoding, thus eliminating the need for retraining.\n\nThe authors provide a robust theoretical framework by defining multi-objective reward functions and leveraging strong-barrier functions. They prove a closed-form bijection between single-objective models and their rewards, and derive a closed-form solution using the Legendre transform.\n\nMOD achieves optimality guarantees under certain conditions and transforms response-level decoding into efficient token-level decoding using greedy search, making the method both effective and practical.\n\nExtensive experiments demonstrate MOD's superior performance, showing a 12.8% overall reward improvement compared to parameter-merging baselines when optimizing for three objectives. The effectiveness is validated across various tasks and model sizes.\n\nWEAKNESSES:\nAlthough MOD circumvents the need for retraining, it requires loading multiple models concurrently, which can be computationally intensive and may not scale efficiently for a larger number of objectives or bigger model sizes.\n\nThe paper could benefit from a more detailed discussion on potential negative impacts or failure modes, especially in scenarios involving conflicting objectives or suboptimal base model alignment.\n\nQUESTIONS:\nHow does the MOD approach scale with an increasing number of objectives? Are there any practical limits to the number of objectives that can be managed simultaneously?\n\nCan the authors provide more insights into the sensitivity of MOD to the quality of base models? Specifically, how does the performance degrade if the base models are not well-aligned or are suboptimal?\n\nAre there any guidelines or best practices for setting and adjusting these preference weightings to achieve optimal results?\n\nLIMITATIONS:\nThe authors have addressed the limitations in Section 7.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}