A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques (2024)

Abstract: Large language models are first pre-trained on trillions of tokens and then instructiontuned or aligned to specific preferences. While pre-training remains out of reach for most researchers due to the compute required, fine-tuning has become affordable thanks to parameter-efficient methods such as LoRA and QLoRA. Alignment is known to be sensitive to the many factors involved, including the quantity and quality of data, the alignment method, and the adapter rank. However, there has not yet been an extensive study of their effect on downstream performance. To address this gap, we conduct an in-depth investigation of the impact of popular choices for three crucial axes: (i) the alignment dataset (HH-RLHF and BeaverTails), (ii) the alignment technique (SFT and DPO), and (iii) the model (LLaMA1, Vicuna-v1.3, Mistral-7b, and Mistral-7bInstruct). Our extensive setup spanning over 300 experiments reveals consistent trends and unexpected findings. We observe how more informative data helps with preference alignment, cases where supervised fine-tuning outperforms preference optimization, and how aligning to a distinct preference boosts performance on downstream tasks. Through our in-depth analyses, we put forward key guidelines to help researchers perform more effective parameter-efficient LLM alignment.

heading: 1 Introduction
text: Large Language Models (LLMs) have achieved human-like performance across various tasks such as summarization, commonsense reasoning, and open-ended generation (Zhao et al., 2023). These LLMs have billions of parameters and are pretrained on trillions of tokens scraped from the web. A lucrative utilization of LLMs is in the form of autonomous agents, to make them follow user instructions and adhere to specific preference requirements (Wang et al., 2023a). However, the pre-trained models are often incapable of follow-
ing instructions, and they need to be aligned using specially curated preference alignment datasets and methods for generalization (Mishra et al., 2021).
Alignment methods either involve fine-tuning the pre-trained model using auto-regressive language modeling over the ground truth completions (supervised fine-tuning or SFT) (Taori et al., 2023) or using specialized alignment methods such as reinforcement learning from human feedback (RLHF) (Christiano et al., 2023), direct preference optimization (DPO) (Rafailov et al., 2023), or prompt tuning (Xue et al., 2023). However, applying these methods to the full models is computationally expensive due to their large sizes. Parameterefficient training (PEFT) methods such as LowRank Adaptation (LoRA) (Hu et al., 2022) and QLoRA (Dettmers et al., 2023) have achieved comparable performance to full fine-tuning of LLMs at a much lower cost. This has enabled researchers to experiment with preference alignment datasets, methods, and models on systems with a single GPU. However, alignment is sensitive to numerous factors and design choices involved in the training (Wang et al., 2023b).
The design choices to align LLMs fit into one of the following three broad, crucial axes: (i) the quality and quantity of the alignment dataset, (ii) the preference alignment method, and (iii) the nature of the base model. Given the increasing interest in preference alignment, an in-depth analysis of the effect of these axes on downstream performance is required. To the best of our knowledge, no extensive study has investigated them, especially in a PEFT setting. We fill this important gap by attempting to answer various key research questions across these three axes: Alignment Dataset How do the informativeness and quality, number of samples, and content of the preference dataset impact the downstream performance? Alignment Method How do different alignment
5732
methods affect pre-trained and instruction-tuned models over complementary preferences?
Nature of the Base Model How does the downstream performance compare between pre-trained models, instruction-tuned models, and their merged variants?
Though our study covers all three axes, given the rapidly growing number of options for each, we restrict the studied choices to the most popular ones. Specifically, we perform our experiments on two commonly used preferences in literature to study alignment trade-offs: harmlessness and helpfulness. We use the (i) two most popular preference alignment datasets with harmlessness and helpfulness annotations: HH-RLHF (Bai et al., 2022) and BeaverTails (Ji et al., 2023), with (ii) the two most widely-used alignment methods: SFT (Taori et al., 2023) and DPO (Rafailov et al., 2023), and (iii) two commonly used LLMs, LLaMA-1 (Touvron et al., 2023) and Mistral-7b (Jiang et al., 2023) along with their instruction-tuned versions, Vicuna-v1.3 (Chiang et al., 2023) and Mistral-7b-Instruct. For an in-depth study of PEFT methods, we conducted all experiments using both LoRA and QLoRA. Our extensive analysis across these core axes reveals certain consistent trends and unexpected findings, as shown in Table 1. We hope that consolidating the key findings into guidelines will benefit the community in conducting impactful research to-
ward LLM alignment. Our contributions can be summarized as: • We provide an in-depth study into the trade-offs
of parameter-efficient preference alignment training, particularly when using LoRA and QLoRA.
• We conduct over 300 experiments across three core axes of preference alignment: the dataset, the alignment method, and the model.
• Through experiments on 5 evaluation benchmarks across harmlessness and helpfulness, we consolidate our key findings as guidelines for more effective preference alignment practices.

heading: 2 Background
text: Alignment Methods reduce the mismatch between an LLM’s pre-training and preference requirements of users, as LLMs are not pre-trained on human preference objectives. We describe the two most widely used alignment methods.
Supervised Fine-Tuning (SFT) uses a pair of input instructions and corresponding outputs to finetune the LLM using autoregressive language modeling. SFT is often used at the “instruction-tuning” stage of models such as Alpaca (Taori et al., 2023) and Mistral-Instruct (Jiang et al., 2023).
Direct Preference Optimization (DPO) (Rafailov et al., 2023) is a stable and optimized alternative to reinforcement learning algorithms such as RLHF (Christiano et al., 2023). These methods
have been used to align LLMs better using paired human preference data: data consisting of accepted and rejected outputs given an instruction.
Parameter-Efficient Training (PEFT) methods require updating only a fraction of the parameters of the full model while achieving performance close to that of full fine-tuning. Examples of these methods include adapters (Houlsby et al., 2019), prefix-tuning (Li and Liang, 2021), and prompttuning (Lester et al., 2021). Low-Rank Adaptation (LoRA) (Hu et al., 2022) is another popular method that inserts a smaller number of new weights into the model, the only trainable parameters. QLoRA (Dettmers et al., 2023) improves the efficiency of LoRA by reducing the memory footprint and storage requirements for training while getting similar performance.

heading: 3 Experimental Setup
text: Preferences and Preference Alignment Datasets We focus on two distinct preferences commonly used to study performance trade-offs in alignment: harmlessness and helpfulness. When a model is aligned on a preference, there is often a performance trade-off on complementary preferences (Bai et al., 2022). Hence, these preferences enable us to study these trade-offs in-depth across multiple axes. We use the two most popular datasets for these preferences: HH-RLHF (Bai et al., 2022) and BeaverTails (Ji et al., 2023). HHRLHF has explicit splits for harmlessness and helpfulness, and each dataset sample consists of a prompt, a chosen sample, and a rejected sample. However, many “harmless” responses in the dataset involve the model refraining from giving a response altogether (Ji et al., 2023). BeaverTails addresses this issue by providing more informative and elaborate responses. Each dataset sample consists of a prompt, two responses, a safety label for each response, and a label for the preferred response.
Base Models We investigate the preference alignment trade-offs using pre-trained and instructiontuned models. We use the two most popular 7 billion parameter pre-trained models, LLaMA-1 (Touvron et al., 2023) and Mistral-7b (Jiang et al., 2023). For completeness, we use their instruction-tuned counterparts, Vicuna-v1.3 (Chiang et al., 2023) and Mistral-7b-Instruct (Jiang et al., 2023). The models trained on the harmless preferences are denoted by the suffix “-Harmless” (e.g., Mistral-7b-Harmless),
and those trained on the helpful preferences are denoted by “-Helpful”.
Evaluation Tasks and Metrics For evaluating helpfulness, we use (i) MMLU (Hendrycks et al., 2021), a dataset consisting of 57 tasks to test the knowledge acquired by the models, (ii) Big-bench hard (BBH) (Suzgun et al., 2022), a diverse evaluation suite of 23 tasks, and (iii) Alpaca Eval (Li et al., 2023), a human-curated suite of 805 questions across different tasks evaluated against GPT4. For harmlessness evaluation, we use (i) RealToxicity (Gehman et al., 2020), a prompt dataset to measure toxic generation in language models. We choose the 1000 most severe prompts from the dataset, and (ii) Red-Instruct’s (Bhardwaj and Poria, 2023) DangerousQA with chain-of-utterance, a dataset comprising 200 harmful questions across six adjectives—racist, stereotypical, sexist, illegal, toxic, and harmful. Following existing work, we evaluate MMLU using accuracy, BBH using exact match, and Alpaca-Eval using win rate against GPT-4. For RealToxicity, we use the score given by a reward model trained to classify toxic generations1, and 100-Attack Success Rate for RedInstruct.
Training Setup We experiment with both QLoRA and LoRA for alignment training. For SFT, we keep the rank of the LoRA/QLoRA matrix as 64 and LoRA/QLoRA alpha as 16. For DPO, we keep the rank of the LoRA/QLoRA matrix as 16 and LoRA/QLoRA alpha as 32. We use a batch size of 16, a learning rate of 2e − 4 for SFT and 5e− 5 for DPO, and train the models for 700 steps. We perform all our experiments on a single 40GB A100 GPU and run 5 seeds for all experiments.

heading: 4 Effects of the Alignment Dataset
text: We investigate the effects of the alignment dataset in terms of its informativeness and quality, the number of samples used, and the preference sets and mixtures used for alignment.

heading: 4.1 Impact of Quality
text: Previous works have shown that more informative and high-quality responses improve the alignment of models (Ji et al., 2023). However, this has only been explored for instruction-tuned LLMs, fully fine-tuned with RLHF, and with limited downstream evaluation. We extend this analysis to
1https://huggingface.co/nicholasKluge/ToxicityModel
over HH-RLHF (in red ) and BeaverTails (in blue ). We observe better performance when using a more informative and high-quality preference alignment dataset, albeit it is often overfitting for non-instruction tuned models when aligned using DPO (Section 4.1).
pre-trained and instruction-tuned models using parameter-efficient SFT and DPO.
Setup To probe the impact of the informativeness and quality of the preference dataset on alignment and downstream performance, we compare the performance of training our models using the two preference datasets, HH-RLHF and BeaverTails, on their harmless and helpful splits. BeaverTails is supposedly more informative and of better quality than HH-RLHF, particularly for the harmlessness and safe prompts. For detailed analysis, we show comparisons using supervised fine-tuning (SFT) and DPO over the pre-trained Mistral-7b and instruction-tuned Mistral-7b-Instruct models. We also probe the impact of the dataset quality as the training progresses, analyzing the effect on the stability of the training.
Results and Observations We compare the performance of the models across helpfulness and harmlessness when aligned using a preference dataset of lower quality with a higher quality dataset using SFT and DPO in Figure 1. The helpfulness benchmarks indicate that a higher quality and more informative preference dataset leads to overall more helpful behavior of the aligned model across both the alignment methods. When using the helpful preference subsets, there are bigger gains on general-purpose NLP tasks (such as MMLU) and instruction-following benchmarks (such as Alpaca Eval) when SFT is used. This suggests that SFT is more sensitive to the dataset quality when the downstream task and preference dataset are of a similar nature. When using the orthogonal harmless preference for alignment, relatively lower quality datasets such as HH-RLHF lead to overfitting when using alignment methods like DPO
and experience significant performance degradation. However, since BeaverTails ensures that the safe and harmless responses are informative, there is no degradation in performance when used with DPO.
The harmlessness benchmarks reveal that using a more informative and safer dataset makes the model less harmful. However, DPO leads to a significant degeneration of the model when BeaverTails is used with Mistral-7b. As Mistral-7b is not instruction-tuned, we hypothesize that this degradation might be due to the inability of the base model to effectively represent the reward for preference optimization, specifically for samples targeted towards more objective preferences such as harmlessness and safety compared to broader preferences such as helpfulness. We also observe that harmless alignment using higher-quality datasets is more faithful than lower-quality datasets, especially when using SFT.
1600 3200 4800 6400 8000 9600 11200
LLaMA-1-HH-
RLHF 0.333 0.330 0.326 0.326 0.333 0.332 0.338
LLaMA-1-BV 0.334 0.348 0.345 0.349 0.348 0.350 0.355
Vicuna-v1.3-HH-
RLHF 0.443 0.439 0.436 0.431 0.435 0.436 0.435
Vicuna-v1.3-BV 0.453 0.457 0.452 0.454 0.449 0.448 0.449
MMLU
Ac cu
ra cy
0.32
0.355
0.39
0.425
0.46
# of samples 1600 3200 4800 6400 8000 9600 11200
LLaMA-1-HH-RLHF LLaMA-1-BV Vicuna-v1.3-HH-RLHF Vicuna-v1.3-BV
Table 1-1
1600 3200 4800 6400 8000 9600 11200 LLaMA-1-HHRLHF 0.319 0.321 0.320 0.318 0.320 0.321 0.322 LLaMA-1-BV 0.312 0.316 0.317 0.319 0.320 0.322 0.326 Vicuna-v1.3-HHRLHF 0.338 0.338 0.336 0.334 0.333 0.337 0.331 Vicuna-v1.3-BV 0.356 0.359 0.356 0.363 0.355 0.353 0.352
Table 1-1-1
1600 3200 4800 6400 8000 9600 11200 LLaMA-1-HHRLHF 0.618 0.765 1.006 0.893 0.818 0.951 1.222 LLaMA-1-BV 1.229 1.468 1.561 1.640 1.787 1.943 2.008 Vicuna-v1.3-HHRLHF -1.371 -0.889 -1.554 -1.139 -0.651 -0.290 0.033 Vicuna-v1.3-BV 1.996 1.421 2.230 2.187 2.045 2.273 2.246
LLaMA-1-HH-RLHF LLaMA-1-BV Vicuna-v1.3-HH-RLHF Vicuna-v1.3-BV
Figure 2: Performance trends w.r.t number of samples of HH-RLHf and BeaverTails used for SFT alignment (Section 4.1). Models aligned with a higher-quality dataset seem to learn faster or regress slower.
We also present the progress of the model performances at various training steps in Figure 2. We observe that when using a more informative preference dataset, the alignment either leads to faster learning and more gains or achieves comparable
5735
performance to the base model. When using a relatively lower-quality dataset, either the learning is slower or deterioration is observed during the course of the alignment training. Overall, better quality and informative datasets are better for alignment across methods and preferences when using PEFT.

heading: 4.2 Impact of Quantity
text: To the best of our knowledge, no studies have investigated the relationship between the number of alignment samples used across methods, models, and alignment preferences.
Setup We evaluate the relationship between the number of preference alignment samples used for SFT and DPO with the downstream performance. We use Mistral-7b and Mistral-7b-Instruct as the model due to their superior performance, and BeaverTails as our preferred dataset due to its more diverse samples. We evaluate the performance at multiple training steps over helpfulness and harmlessness benchmarks.
Results and Observations Figure 3 indicates that instruction-tuned models are more robust to further alignment when using SFT. Furthermore, since the preference data does not directly contain examples of the downstream tasks, the performance of the models is reduced. It is interesting to note here that Mistral-7b-Instruct performs worse than the base model on MMLU, indicating that continued supervised fine-tuning is sensitive to the dataset samples used for training, particularly for generalpurpose NLP tasks.
However, using DPO over the instructiontuned models for instruction-following tasks gives consistent improvement, which aligns with previous works applying DPO as an alignment method (Rafailov et al., 2023). Models trained with DPO are also much more faithful to the prefer-
ences they are aligned with, particularly for harmlessness. This might be due to harmlessness being a relatively objective preference compared to helpfulness, which is much broader. However, as observed previously, the pre-trained models are sensitive to the preference alignment used over DPO, and since they are not strong inherent reward models, using DPO leads to a considerable degradation in their performance. A combination of SFT-based instruction tuning followed by DPO would make up for the optimal training strategy in case of alignment to explicit preferences such as safety and harmlessness.
Overall, instruction-tuned models are robust to additional samples when aligned with SFT and often regress but benefit from DPO. Furthermore, instruction-tuned models might require fewer samples to adapt to preferences compared to pretrained models, as more samples might degrade their performance on general-purpose tasks. Pretrained models generally get better at alignment and instruction-following both. Accordingly, there might be a sweet spot in the number of samples used for alignment.

heading: 4.3 Impact of Data Mixtures
text: Previous studies have shown that optimizing for individual preferences can result in trade-offs in performance (Bai et al., 2022). We extend this study and evaluate models trained on a mixture of distinct preferences with PEFT against those trained on individual preferences.
Setup We use both SFT and DPO to align our models on the harmless and helpful preferences of BeaverTails. We then combine both the preference sets and align models on a mix of the preferences for our comparison.
Results and Observations From Table 2, we see that when both SFT and DPO are used to align the
MMLU
Ac cu
ra cy
0
0.16
0.32
0.48
LLaMA-1Harmless LLaMA-1Helpful Vicuna-v1.3Harmless Vicuna-v1.3Helpful
0.470.47
0.340.33
0.450.45
0.360.34
SFT DPO Table 1 SFT DPO
LLaMA-1-Harmless 0.34 0.33 LLaMA-1-Helpful 0.36 0.34 Vicuna-v1.3-Harmless 0.45 0.47 Vicuna-v1.3-Helpful 0.45 0.47
Table 1-1
SFT DPO
Mistral-7b-Harmless 7.35 3.98 Mistral-7b-Helpful 7.08 3.60 Mistral-7b-Ins-Harmless 7.71 13.04 Mistral-7b-Ins-Helpful 8.1 15.3
Alpaca Eval
W in
R at
e
0
4
8
12
16
Mistral-7bHarmless Mistral-7bHelpful Mistral-7bIns-Harmless Mistral-7bIns-Helpful
15.30
13.04
3.603.98
8.107.717.087.35
SFT DPO
Table 1-1-1 SFT DPO
LLaMA-1-Harmless 11 98 LLaMA-1-Helpful 10.5 4.5 Vicuna-v1.3-Harmless 10.5 99.5 Vicuna-v1.3-Helpful 9.5 10.5
Red-Instruct
10 0
- A SR
0
25
50
75
100
LLaMA-1Harmless LLaMA-1Helpful Vicuna-v1.3Harmless Vicuna-v1.3Helpful
10.5
99.5
4.5
98.0
9.510.510.511.0
SFT DPO
Figure 4: Comparing the downstream performance when aligning using SFT (in light blue ) and DPO (in pink ) with QLoRA. SFT out erforms DPO generally when used over pre-trained models, significantly for instruction following tasks. DPO i more faithful to explicit preferences such as harmlessness and performs significantly better for instruction-tuned models (Section 5).
models using a mixture of preferences, the model performs worse than models trained on individual preferences. This trend is consistent across both pre-trained and instruction-tuned models across the preferences. This might arise because the model encounters conflicting responses for prompts that might be similar to each other when orthogonal preferences are used for alignment, leading to nonoptimal training. We observe larger degradations when using DPO with a mixture of the preferences compared to SFT, suggesting that DPO is more sensitive to the type of samples used for alignment and requires more uniform preference samples. Overall, better curation of dataset mixtures when using alignment methods is necessary to achieve optimal performance.

heading: 4.4 Key Takeaways
text: • Higher quality and more informative datasets lead to better alignment when using both SFT and DPO, with more significant gains when using SFT.
• Performing SFT on strong instruction-tuned models might not lead to gains as there is
performance saturation, and it can even lead to degradation depending on the dataset.
• Training on a mixture of diverse preferences often leads to performance trade-offs and degradation across them.

heading: 5 Effects of the Alignment Method
text: Previous works have shown that preference optimization methods, such as RLHF and DPO, are better than methods like SFT for full fine-tuning of models on standard preference datasets (Christiano et al., 2023; Rafailov et al., 2023). We validate this claim using PEFT across the different preferences.
Setup We compare the performance attained when using SFT and DPO to train our models across preferences. We use all four models in our study to align with the harmless and helpful preferences of the BeaverTails dataset.
Results We present the results comparing SFT and DPO in Figure 4. We observe that, on average, DPO leads to better performance when aligned with the helpful preferences compared to SFT for instruction-tuned models. This might stem from
5737
the fact that DPO takes into consideration both the chosen and the rejected samples and is able to extract more learnings from the preference dataset. When evaluating the harmlessness, we observe very interesting results. Using DPO is significantly better than SFT, indicating that DPO makes the model alignment much more faithful to the preferences than SFT. This is more pronounced when the preferences are more objective, such as harmlessness, compared to relatively broader preferences like helpfulness. This observation also holds for models such as LLaMA-1, where DPO performs worse than SFT on helpfulness benchmarks. Furthermore, it is not always the case that aligning using the harmless preference dataset leads to more harmlessness in the case of SFT, further showing that it is not as faithful as DPO. Given the inability of pre-trained models to inherently provide rewards for broader preferences, we can say that generally, if the base model can be expected to be a good reward model, using DPO leads to better downstream performance and more faithfully aligned models.
We also make another interesting observation that contradicts logical deductions. Oftentimes, using harmless data performs better than helpful data. This might be a characteristic of the BeaverTails dataset as it specifically contains helpful and safe responses, but this should be an important consideration when curating datasets for preference alignment.

heading: 5.1 Key Takeaways
text: • DPO performs better than SFT and is more faithful to explicit preferences, such as harmlessness, compared to broader preferences, such as helpfulness.
• It is beneficial to instruction-tune models first with SFT and then apply DPO.

heading: 6 Effects of the Nature of Base Models

heading: 6.1 Contrasting Pre-trained and Instruction-Tuned Models
text: Most of the works using DPO for alignment use it over instruction-tuned models. We perform a study comparing the effects of applying alignment methods over pre-trained and instruction-tuned models.
Setup We investigate the effect of the nature of the underlying model trained for alignment to different preferences on the downstream tasks. We consider LLaMA-1 as the pre-trained model and Vicuna-v1.3 as the instruction-tuned variant. We compare the downstream performance when the models are aligned with SFT and DPO using the harmless and helpful preferences of BeaverTails.
Results and Observations We present the results of aligning the pre-trained models and the instruction-tuned models with SFT and DPO in Fig 5. Considering SFT, we observe that when applied over instruction-tuned models, it generally leads to degradation of performance when using both the helpful and harmless preferences. This might be because the models being aligned are already robust due to extensive instruction-tuning using datasets similar to the datasets used for downstream evaluation. Since BeaverTails is different from downstream datasets, models aligned on the dataset regress in performance. However, when we align the pre-trained models, we observe decent improvements in the performance of the downstream tasks. This is in line with studies showing that instruction-tuning, especially on the helpful preferences, makes the model follow the instructions of the downstream task more effectively.
When aligning the pre-trained and instructiontuned models with DPO, we observe that the alignment stability depends on the base model itself,
as it also acts like a reward model. For strong base models, such as the instruction-tuned models, DPO leads to better gains. These gains are more significant for instruction-following tasks like Alpaca Eval. However, for weaker base models like LLaMA-1, using DPO leads to degradation in performance in contrast to SFT. Hence, instructiontuned models overall seem to fare more coherently and are more faithful to the preferences when DPO is used, but with relatively weaker base models, an initial stage of instruction-tuning using SFT followed by DPO would be more beneficial.

heading: 6.2 Effect of using Merged Models
text: Model merging methods via interpolations have recently proven to be effective (Yadav et al., 2023). We investigate the effects of merging models that are aligned on distinct preferences and trained with different methods.
Setup To study the effect of various combinations of merging models across the preferences and alignment methods, we experiment with two recent merge methods: Spherical interpolation (Slerp) (Shoemake, 1985) and DARE (Yu et al., 2023). We experiment with merging models aligned on the harmless and helpful preferences using DPO and SFT (for example, Slerp-SFT refers to the merging of harmless and helpful models aligned with SFT). As DARE allows merging any number of models, we also experiment with merging all four aligned models, i.e. harmless and helpful alignments using SFT and DPO together (DARESFT+DPO).
Results and Observations From Table 3, we see that models aligned to preferences with SFT and DPO seem to perform better than the individual models when merged using DARE TIES, while
merging with Slerp leads to performance degradation. The best performance is obtained when we merge all four models, which is very interesting. Overall, the performance of the merged model depends on the individual models, with the performance of strong models regressing when merged with very weak models. We leave this exploration of using elaborate merging methods as future work.

heading: 6.3 Key Takeaways
text: • Pre-trained models generally align better with SFT, whereas instruction-tuned models align better with DPO.
• Instruction-tuned models have reduced performance if the SFT dataset is not similar to the evaluation task.
• Merged models can effectively mitigate performance trade-offs when aligning to diverging preferences.

heading: 7 Comparing QLoRA with LoRA
text: We compare the performance trends observed when using QLoRA as the PEFT method with LoRA as the PEFT method in Figure 6. We observe similar patterns as observed when using QLoRA across the three axes of our study, and also observe similar performance as expected. We provide the extended results of using QLoRA and LoRA across the training setup in the Appendix A.

heading: 8 Key Takeaways
text: Consolidating our findings and conclusions, we present key takeaways across the three core axes of preference alignment in PEFT settings.
Vicuna-v1.3-Helpful 0.45 0.47
Table 1-1
SFT-QLoRA SFT-LoRA DPO-QLoRA DPO-LoRA
Mistral-7b-Ins-Harmless 7.711 10.323 13.043 13.699 Mistral-7b-Ins-Helpful 8.095 12.313 15.299 19.006 Mistral-7b-Harmless 7.346 6.733 3.975 5.853 Mistral-7b-Helpful 7.081 8.864 3.602 5.970
Alpaca Eval
W in
R at
e
0
3.25
6.5
9.75
Mistral-7bHarmless Mistral-7bHelpful Mistral-7bIns-Harmless Mistral-7bIns-Helpful
8.86
6.73
12.31
10.32
7.087.35 8.107.71
SFT-QLoRA SFT-LoRA
Table 1-1-1 SFT DPO
LLaMA-1-Harmless 11 98 LLaMA-1-Helpful 10.5 4.5 Vicuna-v1.3-Harmless 10.5 99.5 Vicuna-v1.3-Helpful 9.5 10.5
Red-Instruct
10 0
- A SR
0
25
50
LLaMA-1Harmless LLaMA-1Helpful Vicuna-v1.3Harmless Vicuna-v1.3Helpful
10.5
4.5
9.510.510.511.0
SFT DPO
SFT-QLoRA
Table 1-1-2
SFT-QLoRA SFT-LoRA DPO-QLoRA DPO-LoRA
Mistral-7b-Ins-Harmless 1.981 1.975 3.520 5.117 Mistral-7b-Ins—Helpful 1.145 0.862 2.170 2.206 Mistral-7b-Harmless 1.863 1.587 -3.701 0.641 Mistral-7b-Helpful 1.571 1.261 -9.375 -9.369
• Alignment Dataset: Higher quality and informative datasets lead to better alignment when using both SFT and DPO, with more significant gains when using SFT. However, for instruction-tuned models, there might be performance saturation on certain tasks, and only a few samples might be required to gain improvements with SFT. Furthermore, in cert in cases, aligning on high-quality distinct p eferences can lead to improvements compared to aligning on the same preference.
• Alignment Method: DPO performs better than SFT and is more faithful to explicit preferences, such as harmlessness, compared to broader preferences, such as helpfulness. However, it fails or overfits when applied over pre-trained models. We suggest instruction-tuning models first with SFT and then apply DPO.
• Nature of Base Model: Pre-trained models generally align better with SFT, whereas instructiontuned models align better with DPO. Instructiontuned models have reduced performance if the SFT dataset is not similar to the evaluation task. Models merged across distinct preferences generally perform better than the models aligned on individual preferences.

heading: 9 Conclusion and Future Work
text: In this work, we perform an extensive analysis of the trade-offs of downstream performance of models over distinct preferences across three core alignment axes: the alignment dataset, the preference alignment method, and the nature of the base model, particularly when using PEFT. We use the two most widely-used 7 billion parameter models, LLaMA-1 and Mistral-7b, along with their instruction-tuned variants, two popular preference alignment datasets, HH-RLHF and BeaverTails, two commonly used alignment methods, SFT and DPO, and evaluate on 5 benchmarks across help-
fulness and harmlessness. By conducting over 300 experiments with QLoRA and LoRA, our findings reveal interesting trends that address the limitations of existing literature and often deviate from them. We consolidate our findings into key takeaways and hope that our guidelines will help searchers perform more effective parameter-efficient preference alignment.
As future work, we aim to extend our study on trade-offs across multiple preferences, spanning various domains for alignment. We also plan to explore PEFT methods, such as model merging and mixture-of-experts, to mitigate these trade-offs.
Limitations
We acknowledge the limitations of our work: • We selected LoRA and QLoRA as parameter-
efficient training methods for our experiments because of their relatively small performance degradation over massive compute savings. We repeatedly point this out throughout our work to avoid confusion; however, other PEFT methods exist, such as adapters and prompt- and prefix-tuning.
• Since we focus on analyzing settings accessible to diverse researchers, we focus on models with 7B parameters. Our findings may or may not extend to models of smaller or larger sizes.
• We conduct experiments to identify the tradeoffs across only two preferences. Based on existing literature, we use the most widely used preferences, harmlessness and helpfulness. However, the findings might not extend when more than two preferences or domains are involved for alignment. These more complicated settings might require a separate in-depth study.
Ethics Statement
Our research studies various LLMs, parameterefficient methods, and datasets for preference align-
5740
ment. We recognize that ensuring the safety of LLMs is a crucial concern and believe that they must be thoroughly vetted before being deployed. Our investigation aims to assist the diverse community of researchers and individuals seeking to align models for safety. Such a research direction is especially important since each person and community has their own perception of safety.
We also acknowledge that training and evaluating LLMs for safety is a sensitive issue. No method can guarantee the complete safety of models, and a comprehensive evaluation of models should be carried out even after using the techniques and results described in our work before deploying them in real-world human-facing situations.

heading: Acknowledgements
text: Sarath Chandar is supported by the Canada CIFAR AI Chairs program, the Canada Research Chair in Lifelong Machine Learning, and the NSERC Discovery Grant. The project was also supported by the IBM-Mila collaboration grant. The authors acknowledge the computational resources provided by the Digital Research Alliance of Canada.

heading: A Extended Results
text: We present our results across the benchmarks when the four models are trained on HH-RLHF and BeaverTails using SFT and DPO with QLoRA in Table 4 and with LoRA in Table 5. We observe that the trends presented in the main work hold across preference alignment datasets, alignment methods, and base models.

heading: B Hyperparameter settings
text: For LoRA and QLoRA parameters, we do a grid search with different combinations of LoRA and QLoRA rank and alpha. For SFT, we experiment with ranks [32, 64, 128] and alphas [16, 32, 64]. For DPO, we experiment with ranks [8, 16, 32] and alphas [16, 32, 64]. We choose the best hyperparameters based on the validation set performance of the alignment datasets. We show the detailed best hyperparameter setup of our training setup during alignment training in Table 6.

heading: C Alignment Dataset Creation and Details
text: The HH-RLHF dataset has explicit splits for harmlessness and helpfulness. However, BeaverTails does not have this explicit split. Since the BeaverTails dataset has no explicit “helpful” and “harmless” splits like HH-RLHF, we create these splits using the provided labels. Each sample in the dataset has an instruction and two responses. Each response also has a “’safe” label, and the dataset sample has the label of the “better” response. To make helpfulness alignment data, we select the "better" response as the choice for helpfulness and select the second response as the "rejected" sample. Similarly, to create harmlessness alignment data, we select the samples where the response was marked safe and the second response was marked unsafe.
The harmless split of HH-RLHF has 42507 samples, and the helpful split has 43810 samples for alignment training. The samples are multi-turn. Using our method, the helpful split of BeaverTails has 297394 samples, and the harmless split has 46625 samples, and the samples are single-turn. We modify the prompts and outputs for both datasets following the suggested instruction-tuning templates of the Vicuna and Mistral models. Vicuna: A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and
polite answers to the user’s questions. USER: “prompt” ASSISTANT: “response”
 Mistral: 
 “prompt” 
 “response” 


heading: D Evaluation Dataset Statistics
text: MMLU The MMLU dataset has 56168 samples across 57 tasks.
Big-Bench Hard The BBH dataset has 6511 samples across 23 tasks.
Alpaca Eval The Alpaca Eval dataset has 805 human written prompts.
RealToxicity The full RealToxicity prompts dataset has 99.4k prompts. We use the 1000 most severe prompts for our evaluation.
Red-Instruct We use the DangerousQA set of the Red-Instruct dataset, which has 200 prompts. We also use the Chain-of-Utterance format, which is the strongest in the dataset.

heading: E Model and Compute Statistics
text: All our model variants are the 7-billion parameter versions of the original models. For our experiments, we use a 40GB A100 GPU. SFT alignment takes about 15 minutes for BeaverTails and 20 minutes for HH-RLHF. DPO alignment takes about 60 minutes for BeaverTails and 90 minutes for HHRLHF.

heading: F Background and Related Work
text: F.1 Alignment Methods
Alignment training aims to reduce the mismatch between an LLM’s pre-training and user preference requirements. It also ensures that models are safe and harmless, reducing the risks associated with their use. We choose the two most widely used alignment methods:
Supervised fine-tuning (SFT) SFT uses a pair of input instructions and corresponding gold answers or outputs to fine-tune the LLM using autoregressive language modeling. The training objective is similar to pre-training, but the dataset is orders of magnitude smaller and follows a strict format. This method is often used for the ’instruction-tuning’ stage for models like Alpaca (Taori et al., 2023) and Mistral-7b-Instruct (Jiang et al., 2023).
Reinforcement Learning from Human Feedback (RLHF) RLHF (Ziegler et al., 2019) is a reinforcement learning-based alignment method that consists of three steps: 1. Collect human feedback data, 2. Train a reward model on the feedback data. 3. Fine-tune an LLM with RL using PPO (Schulman et al., 2017) and the reward model. RLHF is
the most commonly used method for preference alignment but often requires a lot of computation and steps for alignment. Various variants of RLHF have been proposed, such as using pure RL for training LLMs with human feedback in an online manner (Bai et al., 2022) and modifying the reward modeling with adversarial probing (Glaese et al.,
2022).
Direct Preference Optimization (DPO) Reinforcement learning algorithms such as RLHF (Christiano et al., 2023) have been used to better align LLMs using paired human preference data: data consisting of accepted and rejected outputs given an instruction. However, these methods require multiple steps as well as training of separate reward models. DPO (Rafailov et al., 2023) is a method that inherently uses the model being aligned as a reward model, making alignment more stable and optimized.
F.2 Parameter-Efficient Training (PEFT) Parameter-efficient training methods require updating only a fraction of the parameters of the full model while achieving performance close to that of full fine-tuning. Examples of parameter-efficient training methods include using adapters (Houlsby et al., 2019), prefix-tuning (Li and Liang, 2021), and prompt-tuning (Lester et al., 2021). Low-Rank Adaptation (LoRA) (Hu et al., 2022) is another popular method that performs on par with full finetuning. LoRA works by inserting a smaller number of new weights into the model, which are the only trainable parameters. These weights are essentially low-rank matrix decompositions of the different model parameters. Low-rank decomposition refers to the process of approximating a larger matrix as a product of smaller matrices by assuming that the two smaller matrices are representative of the larger matrix. Assuming a parameter of size A×B and a low-rank n, i.e., n << A,B. We can hope to approximate A × B using A × n and n × B. In the former cases, the parameters to be updated are A×B, whereas in the latter case, the parameters are A×n+n×B. If the approximation is sufficient, we only need to update this new set of parameters, and they can later be ’merged’ with the main model after training. Quantized LoRA (QLoRA) (Dettmers et al., 2023) takes the efficiency of LoRA a step ahead by QLoRA by also quantizing the weights of the LoRA adapters (smaller matrices) to lower precision (e.g., 4-bit instead of 8-bit), reducing the memory footprint and storage requirements while getting similar performance.

heading: Future Work
text: As future work, we aim to extend our study on trade-offs across multiple preferences, spanning various domains for alignment. We also plan to explore PEFT methods, such as model merging and mixture-of-experts, to mitigate these trade-offs. Limitations
We acknowledge the limitations of our work: • We selected LoRA and QLoRA as parameter-
efficient training methods for our experiments because of their relatively small performance degradation over massive compute savings. We repeatedly point this out throughout our work to avoid confusion; however, other PEFT methods exist, such as adapters and prompt- and prefix-tuning. • Since we focus on analyzing settings accessible to diverse researchers, we focus on models with 7B parameters. Our findings may or may not extend to models of smaller or larger sizes. • We conduct experiments to identify the tradeoffs across only two preferences. Based on existing literature, we use the most widely used preferences, harmlessness and helpfulness. However, the findings might not extend when more than two preferences or domains are involved for alignment. These more complicated settings might require a separate in-depth study