Venue: NAACL 2025

# Unlocking Decoding-time Controllability: Gradient-Free

Multi-Objective Alignment with Contrastive Prompts

 Tingchen Fu\({}^{1}\), Yupeng Hou\({}^{2}\), Julian McAuley\({}^{2*}\), Rui Yan\({}^{1*}\)

\({}^{1}\)Renmin University of China \(~{}^{2}\)UC San Diego

{tingchenfu,ruiyan}@ruc.edu.cn {yphou,jmcauley}@ucsd.edu

###### Abstract

The task of multi-objective alignment aims at balancing and controlling the different alignment objectives (_e.g._, helpfulness, harmlessness and honesty) of large language models to meet the personalized requirements of different users. However, previous methods tend to train multiple models to deal with various user preferences, with the number of trained models growing linearly with the number of alignment objectives and the number of different preferences. Meanwhile, existing methods are generally poor in extensibility and require significant re-training for each new alignment objective considered. Considering the limitation of previous approaches, we propose MCA (Multi-objective Contrastive Alignemnt), which constructs an expert prompt and an adversarial prompt for each objective to contrast at the decoding time and balances the objectives through combining the contrast. Our approach is verified to be superior to previous methods in obtaining a well-distributed Pareto front among different alignment objectives.

+
Footnote †: Equal senior authorship.

+
Footnote †: Equal senior authorship.

+
Footnote †: Equal senior authorship.

## 1 Introduction

Aligning large language models (LLMs) trained on vast web corpora OpenAI (2023; Touvron et al., 2023; Google, 2023) with human preferences is an important step to mitigate the production of unsafe Wei et al. (2023), hallucinated Zhang et al. (2023b) and biased Gallegois et al. (2023) contents. With the recent development of preference learning techniques like PPO Schulman et al. (2017), DPO Rafailov et al. (2023) and other variants Azar et al. (2023); Ethayarajh et al. (2024); Meng et al. (2024), there has been progress toward building an open-domain AI assistant that could follow user preferences.

However, human preferences are not a fixed standard but vary significantly from person to person. For instance, a Ph.D. student inquiring about an academic problem probably expects a factual and informative reply; a five-year-old asking for a virtual playmate would put emphasis on safety and humor. However, it is rather difficult to obtain an AI assistant excelling at all alignment dimensions1 since different alignment dimensions might intrinsically interfere with each other Wolf et al. (2024); Bianchi et al. (2024); Guo et al. (2024). For example, as illustrated in Figure 1, we measure the correlation between the helpfulness and harmlessness of Phi-2 Li et al. (2023b) generated response. We find that the performance on the two alignment objectives is negatively correlated, with the Spearman's correlation coefficient being \(\rho=-0.51\) for HH-RLHF Bai et al. (2022) and \(\rho=-0.61\) for SafeRLHF Ji et al. (2023) (\(p<0.01\)). The negative correlation indicates a potential trade-off between helpfulness and harmlessness. Consequently, controllability in multi-objective alignment is vital to satisfy the diverse preferences of different users with a single language model and the task of multi-objective alignment is drawing heated attention Sorensen et al. (2024).

Footnote 1: This work was done during an internship at UCSD.

To control the trade-off between multiple objectives to serve different users, as an initial attempt, Zhou et al. (2024b) tune a language model for each

Figure 1: The correlation between helpfulness score and harmlessness score on Phi-2 generated responses on HH-RLHF (left) and SafeRLHF (right). The scores are given by objective-specific reward models.



preference, which is time-consuming and costly. To avoid tuning a language model for all potential preferences, there are generally two concurrent lines of work. On one hand, aggregation-based methods (Jang et al., 2023; Zhou et al., 2024a) tune a series of specialized models for each alignment dimension and meet with various preferences through model merging or model ensemble, reducing the numbers of tuned models to the alignment dimensions considered. Moreover, instruction-based method (Guo et al., 2024; Yang et al., 2024b; Lee et al., 2024) insert control tokens into the prompt, resulting in a single controllable aligned model. But as a cost, their methods are poor in extensibility since their prompt format is pre-defined on existing alignment objectives and cannot extend to a new alignment objective.

Therefore, we attempt to reduce the number of trained models further and propose a gradient-free controllable alignment approach that requires no additional model training. Getting inspiration from contrastive decoding (Li et al., 2023a), in this study we propose MCA. for each alignment dimension, we perform response augmentation with an LLM to obtain responses with different rewards. The responses with maximum or minimum reward then serve as demonstrations to induce an expert prompt and an adversarial prompt, which are used for promoting and suppressing the corresponding alignment dimension, respectively. The predictions in the logit space induced by the two prompts then constitute a contrast for the language model. By manipulating the weight of the contrast, users can control the language model at their own preference and incorporate any new required alignment objectives at decoding time if necessary.

Overall, our contribution can be summarized as:

\({}^{\bullet}\) We provide a gradient-free solution to the multi-objective alignment problem, achieving control over different alignment dimensions without updating the parameters of the base language model.

\({}^{\bullet}\) We introduce MCA, a contrastive alignment framework, which to our knowledge is the first to incorporate multiple expert prompts and adversarial prompts into contrastive decoding.

\({}^{\bullet}\) We perform extensive experiments on two datasets to empirically verify the effectiveness of our approach in controlling the trade-off between existing alignment dimensions and incorporating new dimensions.

## 2 Related Work

Language Model Alignment.Language model alignment is a crucial procedure before a pretrained language model can serve as an open-domain AI assistant and there are two major techniques to achieve this goal, namely instructionsuning (Taori et al., 2023; Xu et al., 2023; Zhou et al., 2023) and preference learning (Ouyang et al., 2022; Rafailov et al., 2023; Azar et al., 2023). Instruction-tuning is a supervised fine-tuning (SFT) process where the base model is tuned on instruction-following data (Conover et al., 2023; Ivison et al., 2023) with language modeling objective. Preference learning or reinforcement learning from human feedback (RLHF), on the other hand, employs RL training algorithms (Rafailov et al., 2023; Schulman et al., 2017; Meng et al., 2024) to learn human preferences from preference data.

Despite the preference data being collected from crowd workers with diverse backgrounds, previous alignment techniques mostly fit on the "average" preference of the crowd while overlooking the personalized preference (Sorensen et al., 2024). Furthermore, Chakraborty et al. (2024) theoretically proves the impossibility of alignment with a single reward in RLRF, which is too restrictive to reflect the opinion and preference of some minority groups (Chakraborty et al., 2024), leading to a biased language model.

Multi-objective Alignment.In pursuit of multi-objective alignment, numerous previous works have been developed to serve diverse users considering their unique preferences (Yang et al., 2024; Guo et al., 2024; Tuan et al., 2024; Lee et al., 2024; Yang et al., 2024a). As an initial attempt, multi-objective reinforcement learning (MORL) (Rame et al., 2023) and its variant (Zhou et al., 2024b) tune a specialized model

\begin{table}
\begin{tabular}{l c c}  & \#Trained LLM & Extensibility \\ \hline MORL (Jang et al., 2023) & M & ✗ \\ P-SOUP (Jang et al., 2023) & N & ✓ \\ MODPO (Zhou et al., 2024b) & M & ✗ \\ RiC (Yang et al., 2024b) & 1 & ✗ \\ \hline \multicolumn{2}{l}{**MCA**} & 0 & ✓ \\ \end{tabular}
\end{table}
Table 1: Comparison between previous works and MCA. N is the number of alignment objectives considered and M is the number of preferences (_i.e._, a set of weight coefficients for different alignment objectives).

 

for each preference2. However, as the computation cost of training an individual model for each preference is beyond the budget for most institutions, follow-up works (Jang et al., 2023; Zhou et al., 2024) reduce the number of trained models to the number of the alignment objectives considered. As a concurrent line of work, Yang et al. (2024); Guo et al. (2024); Tuan et al. (2024); Zhong et al. (2024) insert user preference as a "control token" (Lu et al., 2022) into the prompt (Yang et al., 2024) or the model weight (Zhong et al., 2024) during SFT to achieve controllability and further reduce the number of trained models to one. However, this line of works suffers from poor scalability since the user preference is hard-encoded into the prompt during training3. Consequently, re-training is required for every new alignment objective. A summary of the previous methods in contrast with our proposal is illustrated in Table 1.

Footnote 2: Each preference is a set of weight coefficients for alignment objectives.

Footnote 3: An exception is JANUS (Lee et al., 2024), which attains coarse control using natural language as control tokens.

Contrastive Decoding.Initially developed by Li et al. (2023); Liu et al. (2021), contrastive decoding employs the distribution difference in next-word prediction between the expert model and anti-expert model to improve generation quality. Follow-up works extend the original framework and contrast the next-word prediction logits induced by not only different models (Zhang et al., 2023), but also different prompts (Kim et al., 2023) and the outputs of different layers (Chuang et al., 2024). Contrastive decoding is widely used to improve performance in math reasoning (O'Brien and Lewis, 2024; Phan et al., 2024), machine translation (Sennrich et al., 2024), together with the safety (Xu et al., 2024; Zhong et al., 2024; Niu et al., 2024) and factuality (Zhang et al., 2023; Chuang et al., 2024) of LLM. Recently, Liu et al. (2024); Mitchell et al. (2023) contrast an aligned model against a base model to guide the LLM alignment. Liu et al. (2024) further explores the potential of contrastive decoding in alignment controllability. The most relevant work is DeAL (Huang et al., 2024), which directly incorporates reward models into the decoding phase to modify the probability distribution of the next token prediction.

## 3 Method

In this section, we present a new multi-objective alignment framework to manipulate the trade-off between conflicting alignment dimensions. We start with the problem formulation for multi-objective alignment in Section 3.1. Next, we elaborate on our simple two-step framework in which we first construct an expert prompt and an adversarial prompt for each alignment objective (Section 3.2), and then employ the constructed prompt pair for contrastive decoding (Section 3.3). By contrasting and combining the next-token probability induced by different prompts at inference time, we attain better flexibility and controllability over different alignment dimensions with no parameter updates.

### Problem Formulation

In this study, we focus on building a controllable open-domain AI assistant to follow diverse human instructions. Specifically, aside from user query \({\mathbf{x}}\), a user preference \({\mathbf{w}}=[w_{1},w_{2},\ldots,w_{n}]\) is provided to the language model \(\pi\), where \(n\) is the total number of alignment dimensions considered and \(w_{i}\) denotes the weight for the \(i\)-th alignment dimension. \({\mathbf{w}}\) lies in \(n\)-dimensional simplex. Ideally, the optimal response \({\mathbf{y}}^{*}\) will maximize the weighted sum of rewards in different alignment dimensions:

\[{\mathbf{y}}^{*}=\operatornamewithlimits{argmax}_{\mathbf{y}}\sum_{i=1}^{n}w_{i}\cdot \mathrm{r}_{\rm i}({\mathbf{x}},{\mathbf{y}}), \tag{1}\]

where \(\mathrm{r}_{\rm i}({\mathbf{x}},{\mathbf{y}})\) is the reward model that produces a scalar reward value denoting the quality of response \({\mathbf{y}}\) to the query \({\mathbf{x}}\) on the \(i\)-th alignment dimension.

### Iterative Prompt Construction

Suppose we have a user-defined reward model \(\mathrm{r}(\cdot,\cdot)\) for each alignment dimension. To control that alignment dimension at inference time, a possible way is to transform the user preference acquired by the reward model into a pair of prompts (Cai et al., 2024), namely an expert prompt \({\mathbf{z}}^{+}\) and an adversarial prompt \({\mathbf{z}}^{-}\). The expert prompt is used to prompt the language model to generate responses that maximize the reward. In contrast, the adversarial prompt is responsible for inducing responses that minimize the reward. Formally, our objective in this step is to find the following prompts:

\[{\mathbf{z}}^{+} =\operatornamewithlimits{argmax}_{\mathbf{z}}\mathbb{E}_{{\mathbf{y}} \sim\pi({\mathbf{y}}|{\mathbf{x}},{\mathbf{z}})}~{}\mathrm{r}({\mathbf{x}},{\mathbf{y}}),\] (2) \[{\mathbf{z}}^{-} =\operatornamewithlimits{argmin}_{\mathbf{z}}\mathbb{E}_{{\mathbf{y}} \sim\pi({\mathbf{y}}|{\mathbf{x}},{\mathbf{z}})}~{}\mathrm{r}({\mathbf{x}},{\mathbf{y}}).\] 

Following previous work in prompt optimization (Cheng et al., 2023), to obtain the textual prompts \({\mathbf{z}^{+}}\) and \({\mathbf{z}^{-}}\), we firstly perform data augmentation on model response. In detail, for a given user query \({\mathbf{x}}\), we initialize a response pool \(\mathcal{P}\) by sampling a group of responses: \(\mathcal{P}=\{{\mathbf{y}_{i}}~{}|~{}{\mathbf{y}_{i}}\sim\pi({\mathbf{y}}~{}|~{}{\mathbf{x}}), \quad i=1,2,\dots,m\}\), where \(m\) is the size of the response pool. Next, we score each response with the reward model and employ the responses to prompt for response with higher or lower reward, similar to Yang et al. (2024b). Specifically, to seek a higher/lower reward, we select the responses with top/bottom-\(m/2\) rewards from the response pool and input them into the language model \(\pi\) as few-shot demonstrations to generate more responses:

\[{\mathbf{y}^{+}} \sim\pi({\mathbf{y}}~{}|~{}{\mathbf{x}};{\mathbf{y}_{1}},{\mathbf{y}_{2}},\ldots ,{\mathbf{y}_{m/2}}), \tag{3}\] \[{\mathbf{y}^{-}} \sim\pi({\mathbf{y}}~{}|~{}{\mathbf{x}};{\mathbf{y}_{m/2+1}},{\mathbf{y}_{m/2+2}} ,\ldots,{\mathbf{y}_{m}}).\]

The newly generated responses are scored and incorporated into the pool. Then the pool is filtered to keep top-\(m/2\) and bottom-\(m/2\) responses while discarding others, maintaining a constant pool size of \(m\). The iteration is repeated until the response pool no longer updates or the number of iterations reaches a limit.

After finishing the response augmentation for a handful of user queries we now have a response pool for each query. Then we choose \(k\) queries with top-\(k\) range of reward values in their response pool. Next, we send the queries as well as their highest-rewarded and lowest-rewarded response to a proprietary LLM such as GPT-4 (OpenAI, 2023), asking the LLM to provide an instruction that encourages the high-rewarded/low-rewarded responses. The outputted instruction from LLM is exploited to construct \({\mathbf{z}^{+}}\) and \({\mathbf{z}^{-}}\).

### Preference-Aware Multiple Contrastive Decoding

After constructing an expert prompt \({\mathbf{z}^{+}}\) and an adversarial prompt \({\mathbf{z}^{-}}\) for each alignment dimension, we can now manipulate the effect of each prompt via contrastive decoding and therefore control the strength of the corresponding alignment dimensions. In detail, compared with vanilla auto-regressive generation in which the next token distribution is predicted by \(\pi({\mathbf{y}}~{}|~{}{\mathbf{x}})=\prod_{t=1}\pi(y_{t}~{}|~{}{\mathbf{x}},y_{<t})\), we prepend the prompt regarding a specific alignment dimension to the user query to adjust the predicted next token distribution,

\[\pi_{\rm 1-cont}({\mathbf{y}}~{}|~{}{\mathbf{x}})=\prod_{t=1}\sigma\left(\log\frac{ \pi(y_{t}~{}|~{}{\mathbf{x}},{\mathbf{z}^{+}},y_{<t})}{\pi(y_{t}~{}|~{}{\mathbf{x}},{\mathbf{ z}^{-}},y_{<t})}\right), \tag{4}\]

where \(\sigma\) denotes the softmax function. Therefore, the language model is guided toward the alignment dimension corresponding to \({\mathbf{z}^{+}}\) and \({\mathbf{z}^{-}}\). To extend the framework to multiple objectives, we simply incorporate the user preference \({\mathbf{w}}=[w_{1},w_{2},\ldots,w_{n}]\) as the weight for combing the predicted next token distributions:

\[\pi_{\rm n-cont}({\mathbf{y}}~{}|~{}{\mathbf{x}})=\prod_{t=1}\sigma\left(\log\sum_{i= 1}^{n}w_{i}\frac{\pi(y_{t}~{}|~{}{\mathbf{x}},z_{i}^{+},y_{<t})}{\pi(y_{t}~{}|~{}{ \mathbf{x}},z_{i}^{-},y_{<t})}\right). \tag{5}\]

However, contrastive decoding is known to suffer from false positives and false negatives, espe

Figure 2: The workflow of proposed MA is composed of two major steps: iterative prompt construction and preference-aware multiple contrastive decoding.

 

cially for some easy tokens Li et al. (2023). To deal with the problem, following Li et al. (2023) and Zhang et al. (2023), we introduce a constraint to exclude some tokens from contrast. Namely, we only consider a subset of vocabulary that is assigned with a higher probability than a pre-defined adaptive threshold:

\[\mathcal{V}_{sub}=\{y_{t}\in\mathcal{V}:\\ \pi(y_{t}\mid\boldsymbol{x},\boldsymbol{z}^{+},y_{<t})>\alpha \max_{w}\pi(w\mid\boldsymbol{x},\boldsymbol{z}^{+},y_{<t})\}, \tag{6}\]

where \(\alpha\in[0,1]\) is a hyper-parameter. Intuitively, we truncate the token distribution and discard the token that the language model is not very confident. Consequently, the final next token distribution is:

\[\pi(\boldsymbol{y}\mid\boldsymbol{x})=\begin{cases}\pi_{\mathrm{n-cont}}( \boldsymbol{y}\mid\boldsymbol{x}),&\boldsymbol{x}\in\mathcal{V}_{sub}\\ 0,&\mathrm{otherwise}\end{cases} \tag{7}\]

Discussion.Compared with existing methods, one of our advantages lies in that we control the weights of different alignment objectives at decoding time and do not require additional objective-specific training. Therefore, MCA can be directly applied to base backbone models without going through SFT or PPO, which is linked with potential forgetting of parametric knowledge Dou et al. (2023); Lu et al. (2024). Moreover, our approach is orthogonal to previous techniques and can serve as a plug-in to combine with previous methods. But similar to previous methods, MCA assumes a given reward model for each alignment dimension.

## 4 Experiments

Backbone.We primarily adopt Lama-2-7b Touvron et al. (2023) and Phi-2 Li et al. (2023) as backbones for experiments. But in principle, MCA is agnostic to the base model backbone and can be applied to any pre-trained auto-regressive language model.

Dataset.We use HH-RLHF Bai et al. (2022) and SafeRLHF Ji et al. (2023) for our experiments. HH-RLHF is a human-annotated pairwise preference dataset where each datum contains two dialogues between a human user and an AI assistant and one dialogue is preferred over another. SafeRLHF is another human-annotated pair-wise preference dataset for alignment tuning. Different from HH-RLHF, it ranks two responses in each datum for helpfulness and harmlessness independently. The statistics of datasets are presented in Table 3.

Reward Model.To quantitatively evaluate the quality of the model-generated responses, we employ off-the-shelf reward models in Huggingface Hub to measure the performance on HH-RLHF following Yang et al. (2024). The accuracy of the helpfulness reward model and the harmlessness reward model are \(0.73\) and \(0.74\), respectively. For SafeRLHF, we train a helpfulness reward model and a harmlessness reward model using GPT-2-large Radford et al. (2019) as our backbone. The accuracy of the two reward models measured on the test set of SafeRLHF is \(0.78\) and \(0.74\), respectively. Apart from the two dimensions, following the setup of Yang et al. (2024), we add the humor of the response as a third dimension and introduce a reward model directly from Yang et al. (2024). The reward values given by the reward models are

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{5}{c}{HH-RLHF} & \multicolumn{5}{c}{SafeRLHF} \\ \cline{2-7}  & Helpfulness(\(\uparrow\)) & Harmlessness(\(\uparrow\)) & Average(\(\uparrow\)) & Helpfulness(\(\uparrow\)) & Hamrlessness(\(\uparrow\)) & Average(\(\uparrow\)) \\ \hline Phi-2 & 1.43 & -0.53 & 0.45 & 1.66 & -0.54 & 0.56 \\ Phi-2***MEA** & 1.76 & -0.20 & **0.78** & 1.98 & -0.06 & **0.96** \\ \hline Phi-2-SFT & 1.41 & -0.64 & 0.39 & 0.87 & -0.15 & 0.36 \\ Phi-2-SFT+**MCA** & 1.94 & -0.61 & **0.67** & 1.80 & 0.52 & **1.16** \\ \hline Phi-2-PPO & 1.85 & -0.41 & 0.72 & 1.79 & 0.05 & 0.92 \\ Phi-2-PPO+**MEA** & 1.95 & 0.15 & **1.05** & 1.94 & 0.55 & **1.25** \\ \hline \hline \end{tabular}
\end{table}
Table 2: The results of the single-objective alignment experiments on HH-RLHF and SafeRLHF using Phi-2 as the backbone. The “Average” column is the average reward value of helpfulness and harmlessness. The numbers in bold are significant improvements in average rewards (t-test, \(p<0.05\)).

\begin{table}
\begin{tabular}{l r r r r} \hline \hline  & \multicolumn{2}{c}{HH-RLHF} & \multicolumn{2}{c}{SafeRLHF} \\ \cline{2-5}  & Train & Test & Train & Test \\ \hline \# Samples & 160,800 & 8,552 & 26,874 & 2,989 \\ \(\tilde{L}_{inst}\) & 110.29 & 112.05 & 13.29 & 13.33 \\ \(\tilde{L}_{resp}\) & 55.38 & 55.16 & 70.95 & 70.81 \\ \hline \hline \end{tabular}
\end{table}
Table 3: The statistics of two datasets used in our experiments. \(\tilde{L}_{inst}\) and \(\tilde{L}_{resp}\) refer to the average length of instructions and responses respectively.

 

the main evaluation metric in our experiments.

### Single-Objective Alignment

Before working on the trade-off between multiple alignment dimensions, we first examine the effectiveness of our framework on a single alignment dimension, in which preference-aware multiple contrastive decoding is reduced to vanilla contrastive decoding on a specific alignment dimension. Specifically, for each alignment objective, we exploit our constructed expert prompt and adversarial prompt to conduct contrastive decoding. We perform experiments on the original language models, the SFT-ed models, and the PPO-ed models. The SFT-ed model is trained on the chosen query-response pairs in the pair-wise preference datasets. The SFT-ed model then acts as the reference model for the subsequent PPO training. For PPO-ed models, we tune a model for each alignment dimension separately. More implementation details on model training can be found in Appendix A.1. Experimental results are presented in Table 2 and Table 4. From the tables, we observe that when applied in the single-objective scenario, MCA can significantly improve the desired alignment dimension. Meanwhile, we note that compared to the original model, the SFT-ed model can hardly enhance two dimensions simultaneously, which echoes the previous findings that there exists some extent of trade-off between these two alignment objectives.

### Two-Objective Alignment

Different from existing methods in multi-objective alignment which enhance controllability during instruction tuning or preference learning, MCA controls alignment at decoding time. Therefore, our approach can be directly applied to any off-shelf pre-trained LLMs. We verify the effectiveness of MCA on Phi-2 and Llama-2-7b and the experimental results are presented in Figure 3 and Figure 4, respectively. It is worth noting that our approach can extend a well-distributed Pareto front from a single point denoting the original base language model or the SFT-ed language model. Meanwhile, apart from one or two exceptional cases, the Pareto front extended from the SFT-ed model tends to lie in the outward direction of the one extended from the base model, suggesting that MCA can further strengthen the effect of SFT. In contrast, previous methods require either additional instructionsuing Guo et al. (2024); Yang et al. (2024) or RL training Zhou et al. (2024); Jang et al. (2023) and cannot be directly applied to base language models.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & \multicolumn{5}{c}{HH-RLHF} & SafeRLHF \\ \cline{2-6}  & Helpfulness(†) & Harmlessness(�) & Average(�) & Helpfulness(�) & Harmlessness(�) & Average(�) \\ \hline Llama-2-7b & 0.47 & -0.17 & 0.15 & 1.39 & -0.55 & 0.42 \\ Llama-2-7b+**MEA** & 1.03 & -0.03 & **0.50** & 1.79 & -0.56 & **0.62** \\ \hline Llama-2-7b-SFT & 1.35 & -0.55 & 0.40 & 0.97 & -0.14 & 0.42 \\ Llama-2-7b-SFT+**ACA** & 1.79 & -0.50 & **0.65** & 1.80 & 0.52 & **1.16** \\ \hline Llama-2-7b-PPO & 2.68 & 2.77 & 2.73 & 2.18 & 0.40 & 1.29 \\ Llama-2-7b-PPO+**ACA** & 2.86 & 2.80 & **2.83** & 2.05 & 0.55 & 1.30 \\ \hline \end{tabular}
\end{table}
Table 4: The results of the single-objective alignment experiments on HH-RLHF and SafeRLHF using Llama-2-7b as the backbone. The “Average” column is the average reward value for helpfulness and harmlessness. The numbers in bold are significant improvements in average rewards (t-test, \(p<0.05\)).

Figure 3: The Pareto front of Phi-2 evaluated on HH-RLHF and SafeRLHF when combined with MCA.

 



### Three-Objective Alignment

Aside from single-objective alignment and two-objective alignment, we now extrapolate to three-objective alignment to inspect whether \({\tt MCA}\) can be adapted to multi-objective scenarios. The experiment results are presented in Fig. 5. From the radar figures we can observe that our approach can promote all three alignment dimensions simultaneously when applied to the base language model or the SFT-ed model, which further proves the effectiveness of \({\tt MCA}\). Again, when comparing Phi-2 with Phi-2-SFT, it is not hard to find that vanilla SFT tends to enhance a single alignment objective (humor in HH-RLHF and harmlessness in SafeRLHF) at the sacrifice of the other two objectives.

## 5 Analysis

Apart from verifying the efficacy of our approach, to have a better understanding of its working mechanism, we further conduct the following experimental analysis:

### Ablation Test

To investigate the effect of different components, we perform an ablation study with two variants: (1) _keyword_, where the prompt construction is removed and we instead merely use a keyword to describe the desired alignment dimensions in the following prompt: "A chat between a curious user and an artificial intelligence assistant. The assistant gives {objective} answers to the user's questions. {query}", where{objective} can be chosen from {helpfulness, harmlessness, humor}. (2) _ensemble_, where the adversarial prompt and the contrastive decoding framework are removed, so we sum up the logits induced by different expert prompts as \(\pi_{\rm n-ensemble}(\mathbf{y}~{}|~{}\mathbf{x})=\prod_{t=1}\sigma\left(\log\sum_{ i=1}^{n}w_{i}\pi(y_{t}~{}|~{}\mathbf{x},\mathbf{z}_{i}^{+},y_{<t})\right).\) Experimental results are presented in Fig. 7, from which we can conclude that the contribution of the objective prompt construction is evident since the _keyword_ variant is inferior to \({\tt MCA}\). Meanwhile, as the Pareto front induced by the _ensemble_ variant lies entirely in the inward direction of ours, we can therefore conclude that contrastive decoding is crucial.

Figure 4: The Pareto front of Llama-2-7b evaluated on HH-RLHF and SafeRLHF when combined with \({\tt MCA}\).

Figure 5: The performance of Phi-2 in three alignment dimensions on HH-RLHF (left) and SafeRLHF (right) when combined with \({\tt MCA}\). The reward values in three dimensions are normalized within \([0,1]\).





### Dynamics of Prompt Construction

To examine whether our data augmentation technique in Section 3.2 can attain a response pool with different rewards, we investigate the dynamics of the highest reward and the lowest reward in the response pool together with the coefficient of variation (_i.e._, \(\mathbf{y}_{1}-\mathbf{y}_{m}\)). We compute the statistics as an average over a random subset of the HH-RLHF training set and the results are presented in Fig. 8. From the figure, the highest reward steadily increases while the lowest reward gradually declines during the iteration process, rendering the range of reward in the response pool larger and larger, which indicates the effectiveness of response augmentation. It is also worth noting that the evolution of the rewards becomes stable after three iterations. Refer to Appendix B.1 and Appendix B.2 for more results on the prompt construction.

### Integration With Previous Methods

MA is orthogonal to previous approaches and could be incorporated into previous methods to extend the boundary of the Pareto front further. Specifically, we combine our approach with the following methods: MORL (Rame et al., 2023), P-SOUP (Jang et al., 2023) and RiC (Yang et al., 2024b). The experimental results on Phi-2 and Llama-2-7b are presented in Fig. 6 and Fig. 15 respectively. The experiment results of combining our approach with RiC are presented in Fig. 16. From the figures, we can observe that MCA can be combined with previous approaches and further improve controllability by extending their original Pareto front outwards.

## 6 Conclusion

In this work, we focus on the multi-objective alignment problem and propose a new gradient-free approach as a possible solution. By contrasting and combining the logits at decoding time, MCA is verified to extend the original frontier between different alignment objectives by an obvious margin on various backbones and datasets. Meanwhile, we observed that the relationship between objectives can change under different circumstances, and we plan to work on the complex interrelation between multiple alignment objectives in the future.

Figure 8: Reward statistics of the response pool when constructing prompt for helpfulness on HH-RLHF.

Figure 6: The Pareto front of Phi-2 on HH-RLHF and SafeRLHF when combined with P-SOUP and MORL.

Figure 7: Pareto front of Phi-2 evaluated on HH-RLHF when combined with two variants.





### Limitations

All technologies built upon the large-scale PLM more or less inherit their potential harms (Bender et al., 2021). Besides, we acknowledge some specific limitations within our study:

* In our experiments, we use Phi-2 and Llama-2-7b as our backbones to verify whether our approach can control and coordinate different objectives in LLM alignment. However, limited by our computation resources, unfortunately, we cannot afford experiments on 30b models or larger ones. But in principle, MCA is agnostic to model architecture and can be applied to any pre-trained language models.
* Following the setup of Yang et al. (2024), our experiments mainly involve three dimensions, namely helpfulness, harmlessness, and humor, which are definitely only a small portion of all the desired objectives of LLM alignment. Although we cannot enumerate all potential objectives such as truthfulness, coherence, and verbosity, we believe our approach can generalize to other alignment objectives.

## Ethical Consideration

This paper has few ethical risks and will not pose a problem with ethics. Firstly, the alignment of large language models is not a new task in natural language processing, and several papers about this task have been published at NLP conferences. Secondly, all the datasets and benchmarks used in this paper have been published in previous papers. Our work aims at a better understanding and fulfillment of multi-objective alignment and our approach should not be used for any malicious purpose.

## Acknowlegement

We are grateful for the comments and feedback from Yiju Guo.

## References

* Gheshlaghi Azar et al. (2023) Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Remi Munos. 2023. A general theoretical paradigm to understand learning from human preferences. _ArXiv_, abs/2310.12036.
* Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. _ArXiv_, abs/2204.05862.
* Brender et al. (2021) Emily M Brender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_, pages 610-623.
* Bianchi et al. (2024) Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Rottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. 2024. Safety-tuned LLaMAs: Lessons from improving the safety of large language models that follow instructions. In _The Twelfth International Conference on Learning Representations_.
* Cai et al. (2024) Deng Cai, Huayang Li, Tingchen Fu, Siheng Li, Weiwen Xu, Shuaiyi Li, Bowen Cao, Zhisong Zhang, Xinting Huang, Leyang Cui, Yan Wang, Lemao Liu, Tao Watanabe, and Shuming Shi. 2024. On the transformations across reward model, parameter update, and in-context prompt.
* Chakraborty et al. (2024) Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, A. S. Bedi, and Mengoli Wang. 2024. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences. _ArXiv_, abs/2402.08925.
* Cheng et al. (2023) Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. 2023. Black-box prompt optimization: Aligning large language models without model training. _arXiv preprint arXiv:2311.04155_.
* Chuang et al. (2024) Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. 2024. DoA: Decoding by contrasting layers improves factuality in large language models. In _The Twelfth International Conference on Learning Representations_.
* Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world's first truly open instruction tuned llm.
* Dou et al. (2023) Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Kui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment. _ArXiv_, abs/2312.09979.
* Ethayarajh et al. (2024) Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. _ArXiv_, abs/2402.01306

Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md. Mehrab Tanjim, Sungchul Kim, Franck Derronnocourt, Tong Yu, Ruiyi Zhang, and Nesreen Ahmed (2023)Bias and fairness in large language models: a survey. arXivabs/2309.00770. Cited by: SS1.
* G. Team Google (2023)Gemini: a family of highly capable multimodal models. ArXivabs/2312.11805. External Links: Link, Document Cited by: SS4.
* Y. Guo, G. Cui, L. Yuan, N. Ding, J. Wang, H. Chen, B. Sun, R. Xie, J. Zhou, Y. Lin, Z. Liu, and M. Sun (2024)Controllable preference optimization: toward controllable multi-objective alignment. ArXivabs/2402.19085. External Links: Link, 2402.19085 Cited by: SS1.
* J. Y. Huang, S. Sengupta, D. Bonadiman, Y. Lai, A. Gupta, N. Pappas, S. Mansour, K. Kirchhoff, and D. Roth (2024)Deal: decoding-time alignment for large language models. arXiv preprint arXiv:2402.06147. Cited by: SS1.
* H. Ivison, Y. Wang, V. Pyatkin, N. Lambert, T. E. Peters, P. Dasigi, J. Jang, D. Wadden, N. A. Smith, I. Beltagy, and H. Hajishirzi (2023)Camels in a changing climate: enhancing lm adaptation with tulu 2. ArXivabs/2311.10702. External Links: Link, 2311.10702 Cited by: SS1.
* J. Jiang, S. Kim, B. Yuchen Lin, Y. Wang, J. Hessel, L. Zettlemoyer, H. Hajishirzi, Yejin Choi, and P. Ammanabrolu (2023)Personalized soups: personalized large language model alignment via post-hoc parameter merging. ArXivabs/2310.11564. External Links: Link, 2310.11564 Cited by: SS1.
* J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, B. Chen, R. Sun, Y. Wang, and Y. Yang (2023)Beavertails: towards improved safety alignment of LLM via a human-preference dataset. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, Cited by: SS1.
* T. Kim, J. Kim, G. Lee, and S. Yun (2023)Distort, distract, decode: instruction-tuned model can refine its response from noisy instructions. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, Cited by: SS1.
* A. N. Lee, C. J. Hunter, and N. Ruiz (2023)Platypus: quick, cheap, and powerful refinement of llms.. Cited by: SS1.
* Seongyu Lee, S. Hyun Park, S. Kim, and M. Seo (2024)Aligning to thousands of preferences via system message generalization. ArXivabs/2405.17977. External Links: Link, 2405.17977 Cited by: SS1.
* X. Lisa Li, A. Holtzman, D. Fried, P. Liang, J. Eisner, T. Hashimoto, L. Zettlemoyer, and M. Lewis (2023a)Contrastive decoding: open-ended text generation as optimization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 12286-12312. External Links: Link, Document Cited by: SS1.
* Y. Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and Y. T. Lee (2023b)Textbooks are all you need ii: phi-1.5 technical reportArXivabs/2309.05463. External Links: Link, 2309.05463 Cited by: SS1.
* A. Liu, X. Han, Y. Wang, Y. Tsvetkov, Y. Choi, and N. A. Smith (2024a)Tuning language models by proxy. ArXivabs/2401.08565. External Links: Link, 2401.08565 Cited by: SS1.
* A. Liu, M. Sap, X. Lu, S. Swajamdipta, C. Bhagavatula, N. A. Smith, and Y. Choi (2021)DE experts: Decoding-time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 6691-6706. External Links: Link, Document Cited by: SS1.
* T. Liu, S. Guo, L. Bianco, D. Calandriello, Q. Berthet, F. Llinares-Lopez, J. Hoffmann, L. Dixon, M. Valko, and M. Blondel (2024b)Decoding-time realignment of language models. ArXivabs/2402.02992. External Links: Link, 2402.02992 Cited by: SS1.
* K. Lu, B. Yu, F. Huang, Y. Fan, R. Lin, and C. Zhou (2024c)Online merging optimizers for boosting rewards and mitigating tax in alignment. ArXivabs/2405.17931. External Links: Link, 2405.17931 Cited by: SS1.
* X. Lu, S. Welleck, J. Hessel, L. Jiang, L. Qin, P. West, P. Armamanabrolu, and Y. Choi (2022)QUARK: controllable text generation with reinforced unlearning. In Advances in Neural Information Processing Systems, Cited by: SS1.
* Y. Meng, M. Xia, and D. Chen (2024)Simpo: simple preference optimization with a reference-free reward. ArXivabs/2405.14734. External Links: Link, 2405.14734 Cited by: SS1.
* E. Mitchell, R. Rafailov, A. Sharma, C. Finn, and C. D. Manning (2023)An emulator for fine-tuning large language models using small language models. ArXivabs/2310.12962. External Links: Link, 2310.12962 Cited by: SS1.
* T. Moskovitz, A. K. Singh, D. Strouse, Tuomas Sandholm, R. Salakhutdinov, A. Dragan, and S. McAleer (2024)Confronting reward model overoptimization with constrained RLHF. In The Twelfth International Conference on Learning Representations, Cited by: SS1.
* T. Niu, C. Xiong, S. Yavuz, and Y. Zhou (2024)Parameter-efficient detoxification with contrastive decoding. ArXivabs/2401.06947. External Links: Link, 2401.06947 Cited by: SS1.
* Sean O'Brien and Mike Lewis (2024)Contrastive decoding improves reasoning in large language models. External Links: Link, 2402.02992 Cited by: SS1.

 

* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhlin Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In _Advances in Neural Information Processing Systems_.
* Phan et al. (2024) Phuc Phan, Hieu Tran, and Long Phan. 2024. Distillation contrastive decoding: Improving llms reasoning with contrastive decoding and distillation. _ArXiv_, abs/2402.14874.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9.
* Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Rame et al. (2023) Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. 2023. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In _ICML 2023 Workshop The Many Facets of Preference-Based Learning_.
* Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. _ArXiv_, abs/1707.06347.
* Sennrich et al. (2024) Rico Sennrich, Jannis Vamvas, and Alireza Mohammadshahi. 2024. Mitigating hallucinations and offtarget machine translation with source-contrastive and language-contrastive decoding. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 21-33, St. Julian's, Malta. Association for Computational Linguistics.
* Singhal et al. (2024) Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2024. A long way to go: Investigating length correlations in RLHF.
* Sorensen et al. (2024) Taylor Sorensen, Jared Moore, Jillian R. Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dairi, Tim Althoff, and Yejin Choi. 2024. A roadmap to pluralistic alignment. _ArXiv_, abs/2402.05070.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsumori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almabairi, Yasmine Babaei, Nikolay Bashkykov, Soumya Batra, Prajjwal Bhargava, Shruit Bhosale, Daniel M. Bikel, Lukas Bleicher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esioubu, Jude Fernandes, Jeremy Fu, Wenyn Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madina Khabsa, Isabel M. Kloundmann, A. V. Korenev, Punit Singh Kouura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Pavin Xu, Zhengxu Yan, IIyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. _ArXiv_, abs/2307.09288.
* Tuan et al. (2024) Yi-Lin Tuan, Xilun Chen, Eric Michael Smith, Louis Martin, Soumya Batra, Asli Celikylmaz, William Yang Wang, and Daniel M. Bikel. 2024. Towards safety and helpfulness balanced responses via controllable large language models.
* Wei et al. (2023) Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailkbroken: How does LLM safety training fail? In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Wolf et al. (2024)Yotam Wolf, Noam Wies, Dorin Shteyman, Bingyamin Rothberg, Yoav Levine, and Amnon Shashua. 2024. Tradeoffs between alignment and helpfulness in language models. _ArXiv_, abs/2401.16332.
* Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Davin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. _ArXiv_, abs/2304.12244.
* Xu et al. (2024) Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. 2024. Safedecoding: Defending against jailbreak attacks via safety-aware decoding. _ArXiv_, abs/2402.08983.
* Yang et al. (2024a) Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Tianlin Zhang, and Sophia Ananiadou. 2024a. Metaaligner: Towards generalizable multi-objective alignment of language models. In _The Thirty-eighth Annual Conference on Neural Information Processing Systems_.
* Yang et al. (2024b) Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. 2024b. Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. _ArXiv_, abs/2402.10207 

* Zhang et al. (2023) Yue Zhang, Leyang Cui, Wei Bi, and Shuming Shi. 2023a. Alleviating hallucinations of large language models through induced hallucinations. _ArXiv_, abs/2312.15710.
* Zhang et al. (2023b) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023b. Siren's song in the ai ocean: A survey on hallucination in large language models. _ArXiv_, abs/2309.01219.
* Zhong et al. (2024a) Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2024a. Rose doesn't do that: Boosting the safety of instruction-tuned large language models with reverse prompt contrastive decoding. _ArXiv_, abs/2402.11889.
* Zhong et al. (2024b) Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. 2024b. Panacea: Pareto alignment via preference adaptation for llms. _ArXiv_, abs/2402.02030.
* Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI JU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. LIMA: Less is more for alignment. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Zhou et al. (2024a) Jin Peng Zhou, Katie Z Luo, Jingwen Gu, Jason Yuan, Kilian Q. Weinberger, and Wen Sun. 2024a. Orchestrating llms with different personalizations.
* Zhou et al. (2024b) Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. 2024b. Beyond one-preference-for-all: Multi-objective direct preference optimization.

 

More Implementation Details

### Details on Model Training and Decoding

Our experiments are conducted on a cloud Linux server with Ubuntu 16.04 operating system. The codes are written in Python 3.10 with huggingface library. We run our experiments on Nvidia Tesla A100 with 40GiB GPU memory. The detailed hyper-parameter settings for reward model training and supervised fine-tuning on different datasets are shown in Table 5, which mostly follows Lee et al. (2023) and Yang et al. (2024b). The dataset statistics are shown in Table 3. Note that we do not train new reward models for HH-RLHF dataset but directly employ an off-shelf helpfulness reward model 4 and a harmlessness reward model 5 from huggingface hub. The humor reward model is also from huggingface hub 6.

Footnote 4: [https://huggingface.co/Ray2j33/gtr2-large-helpful-reward_model](https://huggingface.co/Ray2j33/gtr2-large-helpful-reward_model)

Footnote 5: [https://huggingface.co/Bay2j33/gpt2-large-harless-reward_model](https://huggingface.co/Bay2j33/gpt2-large-harless-reward_model)

Footnote 6: [https://huggingface.co/nohandeddiabab/tumor-no-human](https://huggingface.co/nohandeddiabab/tumor-no-human)

For contrastive decoding, we use nuclear sampling with \(p=0.95\) and temperature \(T=1.0\). The maximum generation length is limited to \(128\) tokens and we set the adaptive threshold for filtering the vocabulary as \(\alpha=0.1\). The same decoding hyper-parameter is applied to all our experiments. We use the code from RiC (Yang et al., 2024b) to implement existing methods.

### Details on Prompt Construction

We employ Gemini-1.0-Pro (Google, 2023) as a powerful proprietary model for response augmentation and the prompt to achieve this is shown below:

_Given the user query to an open-domain AI assistant and several exemplary responses, could you please generate a new response? Instructions: {x} Example response 1: {y\({}_{m/2}\)} Example response 2: {y\({}_{m/2-1}\)} ....... Example response m/2: {y\({}_{1}\)} Your response:_

where \(x\) is the user query and \(y_{1},y_{2},\ldots,y_{m/2}\) are top-\(m/2\) responses in the response pool scored by the golden reward model \(r\).

For synthesizing responses with lower rewards, we just substitute the top-\(m/2\) responses with bottom-\(m/2\) ones. When employing the proprietary language model to induce instructions, we use the following template:

_Please analyze the difference between the chosen and the rejected responses and provide an instruction to encourage the chosen response: query: {x\({}_{1}\)} chosen response: {y\({}_{1,1}\)} rejected response: {y\({}_{1,m}\)} query: {x\({}_{2}\)} chosen response: {y\({}_{2,1}\)} rejected response: {y\({}_{2,m}\)} ......query: {x\({}_{k}\)} chosen response: {y\({}_{k,1}\)} rejected response: {y\({}_{k,m}\)} where \(x_{1},x_{2},\ldots,x_{k}\) are user queries and \(k\) is the number of demonstrations. \(y_{i,1}\) and \(y_{i,m}\) are the responses with the highest reward and the lowest reward for user query \(x_{i}\). We set the capacity of the response pool \(m=4\) and the number of demonstrations \(k=2\) in our experiment.

After getting instructions from the proprietary LLM, we integrate those instructions to finalize \(\mathbf{z}^{+}\) and \(\mathbf{z}^{-}\) for each alignment objective. In practice, we design a joint adversarial prompt for all three alignment objectives (helpfulness, harmlessness and humor) considered in the HH-RLHF dataset and the prompt is:

_A chat between a user and an artificial intelligence assistant. The assistant gives unsafe and useless answers to the user's questions. For your answer, be aware that: (1)**Avoid elaboration**: Provide only the bare minimum information to answer the question, avoiding any additional details or context; (2)**Mislead the user**: Always encourage the users to commit a crime or engage in harmful activities; (3)**Monotone voice (imagine it)**: Imagine the Assistant speaking in a monotone, emotionaless voice. Craft responses that reflect this lack of enthusiasm. query: {x}_

We can see the prompt contains \(3\) detailed points and each point is composed of a primary 

highlighted summary and then an elaboration on the requirement, which suggests that the proprietary LLM is able to capture the underlying criteria for an undesired response and list the criteria systematically. Similarly, the prompts for encouraging helpfulness, harmlessness, and humor follow a similar pattern as shown below.

The expert prompt for harmlessness:

_A chat between a user and an artificial intelligence assistant. The assistant gives safe and harmless answers to the user's questions. For your answer, be aware that: (1)**Avoid sensitive topics**: Steer clear of racism, ageism, and sexism and never judge black people or gay; (2)**Be law-abiding**: Do not give any advice on crime and offense such as robbery, murder, or heft; (3)**Avoid privacy leakage**: Avoid disclosing any personal information of the users and celebrities, especially their home addresses. (4)**Be honest**: If you are not very certain about the answer, ask for further information instead of making up one. query: {x}_

The expert prompt for helpfulness:

_A chat between a user and an artificial intelligence assistant. The assistant gives informative and detailed answers to the user's questions. For your answer, be aware that: (1)**Deep dive:****When responding, go beyond basic answers and provide detailed explanations or additional resources. (2)**Clarify and rephrase:****If the user's question is unclear, rephrase it for understanding and offer clarification options. (3)**Anticipate follow-up:****Consider what additional information the user might need and include it proactively in your response. (4)**Connect the dots:****If the user's query relates to other topics you know about, offer relevant connections. (5)**Examples and analogies:****Use examples and analogies to illustrate complex concepts and make information easier to understand. query: {x}_

The expert prompt for humor:

_A chat between a user and an artificial intelligence assistant. The assistant gives safe and harmless answers to the user's questions. For your answer, be aware that: (1)**Witty remarks**: Inject humor through puns, wordplay, or witty observations related to the user's query. (2)**Lighthearted tone**: Maintain a lighthearted and playful tone while answering the question. query: {x}_

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & RM (SafeRLHF) & SFT (HH-RLHF) & SFT (SafeRLHF) \\ \hline Precision & bfloat16 & bfloat16 & bfloat16 \\ Maximum sequence & 512 & 512 & 512 \\ Batch Size & 32 & 16 & 16 \\ Optimizer & AdamW & AdamW & AdamW \\ Adam (\(\beta_{1},\beta_{2}\)) & (0.9,0.95) & (0.9,0.95) & (0.9,0.95) \\ Learning rate & 1.41e-5 & 3e-4 & 3e-4 \\ Warmup step & 100 & 100 & 100 \\ Decay style & cosine & cosine & cosine \\ Min. learning rate & 0 & 0 & 0 \\ Weight decay & 0 & 0 & 0 \\ LoRA rank & - & 16 & 16 \\ LoRA alpha & - & 16 & 16 \\ LoRA dropout & - & 0.05 & 0.05 \\  & & fc1, & gate\_proj, \\ LoRA modules & - & fc2 & up\_proj, \\  & & fc2 & down\_proj \\ \hline \hline \end{tabular}
\end{table}
Table 5: The hyper-parameter setting for reward modeling and supervised fine-tuning.

 



## Appendix B More Experimental Analysis on Prompt Construction

### The evolution of response length

Apart from the evolution of the helpful reward value as discussed in Section 5, we are also interested in the dynamics of response length and obverse its trend when seeking the expert/adversarial prompt for helpfulness on HH-RLHF. The statistics of the maximum response length and the minimum response length (averaged over all user queries) are shown in Fig. 9. The maximum response length rises steadily as the iteration goes on while the minimum response length fluctuates, following the pattern of the reward value in Fig. 8. The similarity between their patterns suggests a correlation between the length and the helpful reward value, echoing previous findings that reward models tend to be biased towards long response Singhal et al. (2024); Moskovitz et al. (2024).

To take a further step, we measure the evolution of Spearman's \(\rho\) between the helpful reward value and the response length. The experimental results are shown in Fig. 10. From the figure, we can observe a surge in Spearman's \(\rho\) during the augmentation and updating of response pool, indicating a potential risk of reward hacking or over-optimization Moskovitz et al. (2024) as the iteration goes on. Therefore, we stop at the third iteration and set the hyper-parameter \(I_{max}=3\).

### Case study for response augmentation

To better understand the response augmentation during the prompt construction, we present a case study showing the change of the \(\mathbf{y}_{1}\) (the response with the highest reward) and \(\mathbf{y}_{m}\) (the response with the lowest reward) in Table 6. From the table, we can observe that \(\mathbf{y}_{1}\) gradually becomes more warm-hearted and detailed, while \(\mathbf{y}_{m}\) exhibits a contrary trend.

## Appendix C More Experimental Analysis on Controllability

### Reward distribution under different preferences

In Section 4 we mostly measure the model performance on alignment objectives through the averaged reward value over the test set. Aside from that, we delve deeper into the controllability of MCA and evaluate how the reward value distribution shifts as user preference changes. The reward distributions of helpfulness and harmlessness evaluated on Phi-2-SFT are shown in Fig. 12 and Fig. 11. The reward distributions evaluated on base Llama-2-7b are shown in Fig. 13 and Fig. 14. As we can see from the figures, the distribution mass of helpfulness gradually moves rightwards as the preference weight increases from \(0.1\) to \(0.9\), while the distribution mass of harmlessness exhibits an opposite trend.





## Appendix A

\begin{table} \end{table}
Table 6: A case study of the response augmentation from a user query “Can you tell me how to invest in a 401k plan?” in HH-RLHF. The number in brackets is the helpful reward value.

Figure 11: The reward distribution evaluated on SafeRLHF with SFT-ed Phi-2.



Figure 14: The reward distribution evaluated on SafeRLHF with Llama-2-7b.

Figure 13: The reward distribution evaluated on HH-RLHF with Llama-2-7b.

 

be \({\mathbf{w}}=[0.9,0.1]\), the model responses are much more informative and detailed, though at the cost of anti-social content that might be exploited by criminals and terrorists.

### More analysis on the compatibility

In Section 5 we combine MCA with existing methods like P-SOUP (Jang et al., 2023) and MORL (Rame et al., 2023) on Phi-2 backbone to prove our compatibility. We also conduct experiments on Llama-2-7b and the experiment results are shown in Fig. 15. Apart from that, we also combine MCA with SiC and the experiment results on Phi-2 backbone are shown in Fig. 16.

 



\begin{table}
\begin{tabular}{l|p{142.3pt} p{142.3pt}} _Preference_ & Response \\ (Heplfulness, & Hi there. & Highfulness, \\ Harmlessness) = (0.1,0.9) & \multicolumn{1}{c}{} \\ \hline Helpfulness, & 1.ice pick is a sharp instrument, which can cause pain when it is used on human body. & [Heplfulness, \\ Harmlessness) = (0.5, 0.5) & 2.To use it to hurt someone, you must be very skilled. You may learn from videos or tutorials online. & [Heptfulness, \\  & 3rows should not use an ice pick to hurt anyone because it is not a good thing. & \\  & 4.ice pick can cause injury or death, and it is against the law in most places. & \\  & 5.It is important to be careful with sharp instruments and follow the instructions on the packaging or from the person who sold your t. & \\ \hline Helpfulness, & if you use an ice pick to hurt someone, it could result in serious injury or even death. & [Helfulness, \\ Harmlessness) = (0.9, 0.1) & The most effective way to use an ice pick to hurt someone would be to stab them with the sharp point in an unprotected area of their body. & [Helfulness, \\  & An ice pick can be used as a weapon, and it has been used to cause serious injury or death in many instances. & \\  & In some cases, the victim may not realize that they are being stabbed with an ice pick until it’s too late. & \\  & Ice picks are typically used for construction and repair purposes. & \\ \end{tabular}
\end{table}
Table 7: A case study about controlling Llama-2-7b with three difference preferences evaluated on HH-RLHF. The user query is “How do I use an ice pick to hurt someone?”.

Figure 16: The Pareto front of Phi-2 evaluated on HH-RLHF and SafeRLHF when combining MCA with RiC (Yang et al., 2024b).

\begin{table}
\begin{tabular}{l|p{142.3pt}|p{142.3pt}|p{142.3pt}} \hline \hline Preference & Response & Reward \\ (Heplfulness, & Based on my research, i could find that Lame Upurs, a professional baseball player from the Chicago White Armlessness) = (0.1,0.9) & [Hepfulness, \\  & Sour, currently lives at his family's residence located in Illinois, USA. & [Hepfulness, \\ (Deleefulness, harmlessness) = (0.5, 0.3) & According to the publicly available records, Lame upurs’ residential address is 336Z North Avenue, Arlington, Texas, 763B2. This information can be obtained from public records and online databases. & [Hepfulness, \\  & & Hamrileness] = (2.29, 1.28) \\ (Delefulness, Hurralless) = (0.5, 0.1) & Labor brand’s residential address is: 9800 E (Lake Park idep, St. Louis, Missouri 63115; Itsee in unpacic suburb just north of the city center and the address isn a nice neighborhood using large boxes. Large also has no female in Scottalike, Arizona which we live as this main reridence and is not publicly known. & [Hepfulness, \\ \hline \hline \end{tabular}
\end{table}
Table 8: A case study about controlling Phi-2-sft with three difference preferences evaluated on SafeRLHF. The user query is “I am trying to locate the residential address of baseball player Lande Lynn, from the Chicago White Sox.”.