Zephyr: Direct Distillation of LM Alignment
Authors: Lewis Tunstall
Edward Emanuel Beeching
Nathan Lambert
Nazneen Rajani
Kashif Rasul
Younes Belkada
Shengyi Huang
Leandro Von Werra
Clémentine Fourrier
Nathan Habib
Nathan Sarrazin
Omar Sanseviero
Alexander M Rush
Thomas Wolf
Venue: COLM (2024)
Keywords: AI feedback, open LLMs, RLHF
TL;DR: A method for aligning open language models with AI preference feedback at scale.

Abstract: We aim to produce a smaller language model that is aligned to user intent. Previous research has shown that applying distilled supervised fine-tuning (dSFT) on larger models significantly improves task accuracy; however, these models are unaligned, i.e. they do not respond well to natural prompts. To distill this property, we experiment with the use of preference data from AI Feedback (AIF). Starting from a dataset of outputs ranked by a teacher model, we apply distilled direct preference optimization (dDPO) to learn a chat model with significantly improved intent alignment. The approach requires only a few hours of training without any additional sampling during fine-tuning. The final result, Zephyr-7B, set a new state-of-the-art on chat benchmarks for 7B parameter models, and requires no human annotation. In particular, results on MT-Bench show that Zephyr-7B surpassed Llama2-Chat-70B, at the time the best open-access RLHF-based model.

Published as a conference paper at COLM 2024
Zephyr: Direct Distillation of LM Alignment
Lewis Tunstall", Edward Beeching", Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf
Hugging Face sewisehuuginnface.co
Abstract
We aim to produce a smaller language model that is aligned to user in- tent. Previous research has shown that applying distilled supervised fine- tuning (dSFT) on larger models significantly improves task accuracy; how- ever, these models unaligned, i.e. they do not respond well to natural prompts. To distill this property, we experiment with the use of preference data from AT Feedback (ATF). Starting from dataset of outputs ranked by a teacher model, we apply distilled direct preference optimization (dDPO) learn chat model with significantly improved intent alignment. The approach requires only a few hours of training without any additional sampling during fine-tuning. The final result, ZEPHYR-7B, set a new state- of-the-art on chat benchmarks for 7B parameter models, and requires no hu- man annotation. In particular, results on MT-Bench show that ZEPHYR-7B surpasses LLAMA2-CHAT-PA a strong open-access RLHF-based model.
1 Introduction
Smaller, open large language models (LLMs) have greatly increased in ability in recent years, from early GPT-2-like models (Wang & Komatsuzaki, 2021) accurate and compact models (Touvron et al., 2023; Penedo et al., 2023; Jiang et al., 2023) that are trained on significantly more tokens than the " "compute-optimal" amount suggested by the Chincilla scaling laws (De Vries, 2023). In addition, researchers have shown that these models can be further trained through distilled supervised fine-tuning (dSFT) based on proprietary models to increase their accuracy (Taori et al., 2023). In this approach, the output of more capable teacher model is used as supervised data for the student model.
Distillation has proven to be an effective tool for improving open models different tasks (Chiang et al., 2023); however, it does not reach the performance of the teacher models (Gudibande al., 2023). Users have noted that these models are not "intent aligned", i.e. they do not behave in a manner that aligns with human users' preferences. This property often leads outputs that do not provide correct responses queries.
range of
Intention alignment has been difficult to quantify, but recent work has led to the develop- ment of benchmarks like MI-Bench (Zheng al., 2023) and AlpacaEval (Li et al., 2023) that specifically target this behavior. These benchmarks yield scores that correlate closely with human ratings of model outputs and confirm the qualitative intuition that proprietary mod- els perform better than open models trained with human feedback, which in turn perform better than open models trained with distillation. This motivates careful collection of human feedback for alignment, often at enormous cost at scale, such as in LLAMA2-CHAT (Touvron et al., 2023).
In this work, we consider the problem of aligning small open LLM entirely through distillation. The main step is to utilize Al Feedback (AIF) from an ensemble teacher models as preference data, and apply distilled direct preference optimization the learning
Equul contribution.

---

Published as a conference paper at COLM 2024
objective (Rafailov et al., 2023). We refer to this approach dDPO. Notably, it requires no human annotation and no sampling compared to using other approaches like proximal preference optimization (PPO) (Schulman et al., 2017). Moreover, by utilizing small base LM, the resulting chat model can be trained in matter of hours on 16 A100s (80GB).
Writing
Humanities
Roleplay
model
....
Llama-2-70b-chal
..8-
GPT-3.5-turbo
STEM
Reasoning
Claude 1
GPT-4
Zephyr 7b
Extraction
Math
Coding
Figure 1: Model performance on MT-Bench. We compare ZEPHYR-7B, trained with distilled direct preference optimization (dDPO), to proprietary models as well as larger, open-access models like LLLMA2-CHAT-775 that were additionally trained using reinforcement learning on a large amount human feedback.
To validate this approach, we construct ZEPHYR-7B, an aligned version Mistral-7B (Jiang et al., 2023). We first use dSFT, based the UltraChat (Ding et al., 2023) dataset. Next we use the AT feedback data collected in the UltraFeedback dataset (Cui et al., 2023). Finally, we apply dDPO based on this feedback data. Experiments show that this 7B parameter model can achieve performance comparable to 70B-parameter chat models aligned with human feedback. Results show improvements both in terms of standard academic benchmarks as well as benchmarks that take into account conversational capabilities. Analysis shows that the use of preference learning critical in achieving these results.
Related Work
There has been significant growth in the number of open large language models (LLMs) that have served as artifacts for the research community to study and use as starting model for building chatbots and other applications. After the release of ChatGPT, the LLaMA model (Touvron et al., 2023) opened the doors to a wide range of research on efficient fine- tuning, longer prompt context, retrieval augmented generation (RAG), and quantization. After LLaMA, there has been a continuous stream of open access text based LLMs including MosaicML'ss MP'T (ML, 2023), the Together Al's RedPajama-INCTTE (AI, 2023), the Tll's Falcon (Penedo et al., 2023), Meta's Llama 2 (Touvron et al., 2023), and the Mistral 7B (Jiang et al., 2023). Zephyr uses Mistral 7B the starting point due to its strong performance.
With the development of open models, researchers have worked on approaches to improve small model performance by distillation from larger models. This trend started with self-instruct method (Wang et al., 2023) and the Alpaca model (Taori et al., 2023), which was followed by Vicuna (Chiang et al., 2023) and other distilled models. These works primarily focused on distilling the SFT stage of alignment, whereas we focus on both SFT and preference optimization. Some models such as WizardLM (Xu et al.) have explored methods beyond dSFT. Contemporaneously with this work, Xwin-LM (Team, 2023) introduced an approach that distilled preference optimization through PPO (Schulman et al., 2017). We compare to these approaches in experiments. Several related approaches to preference alignment have been released after this work including Starling (Zhu et al., 2023), Tulu 2 (Ivison et al., 2023), Intel NeuralChat, and Nous Hermes 2.
2

---

Published as a conference paper at COLM 2024
Step dSFT
Step 2- AIF
Step 3 dDPO
Generate multi-tum dialogues
Response generation and ranking
Distillation of preferences
Prompt sempled from prompts.
Create scenario space exploration
Prompt sempled from prompts.
Describb how chocolate brownies
Prongt sampled from prompts,
Describe how to make chocolate brownies
simuletes user-assistanni interactiins.
language generate resporses.
response are selected.
LLM
1/u
yt
Dislogues are used for supervised fine-tuning
GPT-4 ranks the responses,
Direct Preference Optimization
#USPT
Figure The three steps of our method: (1) large scale, self-instruct-style dataset construc- tion (UltraChat), followed by distilled supervised fine-tuning (dSFT), (2) Al Feedback (AIF) collection via an ensemble of chat model completions, followed by scoring by GPT-4 (Ultra- Feedback) and binarization into preferences, and (3) distilled direct preference optimization (dPO) of the dSFT model utilizing the feedback data.
Tools for benchmarking and evaluating LLMs have greatly evolved to keep up with the pace of innovation in generative AL. Powerful LLMs such as GPT-4 and Claude are used as evaluators to judge model responses by scoring model outputs ranking responses in pairwise setting. The LMSYS chatbot arena benchmarks LLMs in anonymous, randomized battles using crowdsourcing (Zheng et al., 2023). The models are ranked based their Elo ratings on the leaderboard. AlpacaEval is an example of another such leaderboard that compares models in pairwise setting but instead uses bigger LLMs such GPT-4 and Claude in place of humans (Dubois et al., 2023). In similar spirit, MTBench uses GPT-4 to score model responses on a scale of 1-10 for multi-turn instructions across task categories such as reasoning, roleplay, math, coding, writing, humanities, STEM and extraction (Zheng et al., 2023). The HuggingFace Open LLM leaderbaord (Beeching et al., 2023), the Chain- of-Thought Hub (Fu et al., 2023), ChatEval (Sedoc et al., 2019), and FastEval (fas, 2023) are examples of other tools for evaluating chatty models. We present results by evaluating on MTBench, Chatbot Arena, AlpacaEval (v1), RewardBench, and the HuggingFace OpenLLM Leaderboard. An extension to AlpacaEval (v2) was later released after this research was made available.
3 Method
The goal of this work is align an open-source large-language model to the intent of the user. Throughout the work we assume access to larger teacher model π which can be queried by prompted generation. Our goal is to produce a student model πTş and our approach follows similar stages InstructGPT (Ouyang et al., 2022) as shown in Figure 2.
Distilled Supervised Fine-Tuning (dSFT) Starting with a raw LLM, we first need train it to respond to user prompts. This step is traditionally done through supervised fine tuning (SFT) on a dataset of high-quality instructions and responses (Chung et al., 2022; Sanh et al., 2021). Given access to teacher language models, can instead have the model generate instructions and responses (Taori et al., 2023), and train the model directly on these. We refer to this as distilled SFT (dSFT).
3

---

Published as a conference paper at COLM 2024
Approaches to dSFT follow the self-instruct protocol (Wang et al., 2023). Let xf, ........... ... set of seed prompts, constructed to represent a diverse set of topical domains. A dataset is constructed through iterative self-prompting where the teacher is used to both respond to an instruction and refine the instruction based on the response. For each x², we first sample response y TT((11>), and then refine by sampling a new instruction (using prompt for refinement), x² ~ T (xxx), ).).. The end point is final dataset, C (1), (1), Distillation is performed by SFT,
be
TSFT = max IE log xlylx) (x,))~
AI Feedback through Preferences (AIF) Human feedback (HHF) can provide additional signal to align LLMs. Human feedback is typically given through preferences on the quality of LLM responses (Ouyang et al., 2022). For distillation, we instead use AI preferences from the teacher model on generated outputs from other models.
We follow the approach of UltraFeedback (Cui et al., 2023) which uses the teacher to provide preferences on model outputs. As with SFT, the system starts with set of prompts X1...., Each prompt is fed to collection of four models TI...... T4, e-g- Claude, Falcon, Llama, etc, each of which yield a response y ~ Tq (1((((),....,2 ~ (T These responses are then fed to the teacher model, e.g. GPT-4, which gives score for the response TT( x, )²) After collecting the scores for prompt x, save the highest scoring response as Yu and random lower scoring prompt as y1- The final feedback dataset D consists of set of these triples (x,
s! ~ 1 (- y",,,,,,
Distilled Direct Preference Optimization (dDPO) The goal of the final step is to refine the 7SFTT by maximizing the likelihood of ranking the preferred yv over y/ in a preference model. The preference model is determined by reward function ro(x, y) which utilizes the student language model Tg. Past work using AT feedback has primarily focused on using RL methods such proximal policy optimization (PPO) to optimize 0 with respect to this reward. These approaches optimize 0 by first training the reward and then sampling from the current policy to compute updates.
Direct preference optimization (DPO) uses a simpler approach to directly optimize the preference model from the static data (Rafailov et al., 2023). The key observation is to derive the optimal reward function in terms of the optimal LLM policy π" and the original LLM policy T Under appropriate choice of preference model they show, for hyperparameter B, which determines closeness to the original policy, and partition function Z that,
गे' (ylx)
xx,
TaSFT (ylx) B log ((x)
By plugging this function of the reward into the preference model, the authors show that the objective be written as,
logo (Blog Tt x TTSFT (ymlx)
x((l(x)) 7TSFT (yr
Ty max π
IE
Blog
(1)
(x ~
While this term looks complex, note that it implies simple training procedure. Starting with the dSFT version of the model, we iterate through each AlF triple (w,
1. Compute the probability for (x, and (x, y1) from the dSFT model (forward-only).
2. Compute the probability for (w, and (x, y1) from the dDPO model.
3. Compute Eq and backpropagate to update π.. Repeat.
4 Experimental Details
We conduct all of our fine-tuning experiments using Mistral 7B (Jiang et al., 2023), which was the current state-of-the-aa ate base LM at the 7B parameter scale, and matches the performance
4

---

UltraChat (Ding et al., 2023) is self-refinement dataset consisting of 1.47M multi-turn dialogues generated by UPT-3.5-TURBE over 30 topics and 20 different types of text material. We initially ran dSFT over the whole corpus, but found the resulting chat model had a tendency to respond with incorrect capitalization and would preface its answers with phrases such "I don't have personal experiences", even for straightforward questions like "How do clean my car?". To handle these issues in the training data, we applied truecasing heuristics to fix the grammatical errors (approximately 5% of the Published as a conference paper at COLM 2024
of much larger models like Llama-34B on many NLP benchmarks. We use the Transformer Reinforcement Learning (TRL) library for fine-tuning (von Werra et al., 2020), in conjunction with DeepSpeed ZeRO-3 (Rajbhandari et al., 2020) and FlashAttention-2 (Dao, 2023) to optimize memory and improve training speed. We also use Weights and Biases (Biewald, 2020) for experiment tracking. All dSFT models are trained with the AdamW optimizer, while dDPO models are trained with RMSProp match the original implementation by the DPO authors. No weight decay is used during training. We did not experiment with paraieter-efficient techniques such as LoRA (Hu et al., 2021), ), but expect similar results to hold with these methods. All experiments were run on 16 A100s using b precision and typically took 2-4 hours complete.
4.1 Datasets
We focus on two dialogue datasets that have been distilled from a mix of open and pro- prietary models, and have previously been shown to produce strong chat models like the UltraLM (Ding et al., 2023):
dataset), as well as several filters to focus on helpfulness and remove the undesired model responses. The resulting dataset contains approximately 200k examples.
UltraFeedback (Cui et al., 2023) consists of 64k prompts, each of which have four LLM responses that are rated by GPT-4 according to criteria like instruction-following and helpfulness. We construct binary preferences from Feedbackk by selecting the highest overall score from GPT-4 as the "chosen" response and of the remaining three at random as "rejected".. We opted for random selection instead of selecting the lowest- scored response encourage diversity and make the DPO objective more challenging. As noted above, this step is computed offline and does not involve any sampling from the reference model.
honesty,
4.2
Evaluation
Our main evaluations are on single-turn and multi-turn chat benchmarks that measure model's ability to follow instructions and respond to challenging prompts across diverse range of domains:
MT-Bench (Zheng et al., 2023) is multi-turn benchmark that consists of 160 questions across eight different areas of knowledge. In this benchmark, the model must answer an initial question, and then provide second response to predefined followup question. Each model response is then rated by GPT-4 on a scale from 1-10, with the final score given by the mean over the two turns.
AvaacaEval (Li et al., 2023) is single-turn benchmark where model must generate response to 805 questions on different topics, mostly focused on helpfulness. Models are also scored by GPT-4, but the final metric is the pairwise win-rate against a baseline model ((ext-davinci-0033. Due to challenges with (ength-normalization the AlpacaEval benchmark also includes a Length Corrected version.
Chatbot Arena (Zheng et al., 2023) is a dynamically scored benchmark that uses human rankers. The system relies on pairwise comparisons done by community participants who select which model they prefer in head-to-head comparison. The metric uses ELO score track model performance.

---

Published as a conference paper at COLM 2024
Chatbot
Model
Size
Align
MT-Bench
AlpacaFval
Arena (ELO)
(score)
(win %)
(LC%)
StableLMMQ
7B
dSFT
2.75
842
MPTT-Chat
7B
dSFT
5.42
928
Xwin-LM v0.1
7B
dPPO
6.19*
87.831.15
0.0
Mistral-Ins v0.1
7B
6.84
1004
Zephyr
7B
dDPO
7.34
90.601.03
76.3
1048
InntrucInstruct
40B
dSFT
5.17
45.71
39.1
1028180B
Guanaco
65B
SFT
6.41
71.801.59
54.7
1028x3B
Llama2-Chat
70B
RLHF
6,86
92.660.91
74.1
1083
Vicuna v1.3
33B
dSFT
7.12
88.991.10
0.0
1089
WizardLM v1.0
70B
dSFT
7.71
1103
Xwin-LM v0.1
70B
dPPO
95.57
0.0
GPT-3.5-turbo
RLHF
7.94
89.371.08
81.7
1098
Claude
RLHF
8.06
91.366.99
74.3
1123
GPT-4
RLHF 8.99
95.28p.72
86.5
1252
Table 1: Chat benchmark results for open-access and proprietary models on MT-BBnch, AlpacaEval (original and with length correction), and Chatbot Arena. A dash (-) indicates model or alignment information that is not publicly available, or an evaluation that absent the public leaderboards. Scores marked with an asterisk (+) denote evaluations done by ourselves. Results are with models available at time of original release.
Reward Bench (Lambert et al., 2024) is a recently introduced metric that compares different alignment methods based on their implied reward model. This provides way to compare implicit reward models based on DPO to more explicit award models based on a trained classifier used for PPO.
We also evaluate ZEPHYR-7B on the Open LLM Leaderboard (Beeching et al., 2023), which measures the performance of LMs across four multiclass classification tasks: ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), and Truthful QA(Lin et al., 2022). Although this leaderboard does not directly measure the conversational quality of chat models, it does provide a useful signal to validate whether fine-tuning has introduced regressions on the base model's reasoning and truthfulness capabilities.
Across all benchmarks, we compare ZEPHYR-7B against variety of open and proprietary models, each with different alignment procedures. To facilitate comparison across open model sizes, we group our comparisons in terms of 7B models (XWIN-LM (Team, 2023), MISTRAL-INSTRUCC (fiang al., 2023), MPT-CHAT (MI. 2023), and MTABLELM--)) as well as larger models up to 70B parameters (LLAMA2-CHAT (Touvron et al., 2023), VICUÑA (Chi- ang et al., 2023), WizardLM (Xu et al.), and GUANACO (Dettmers et al., 2023)). For the chat benchmarks, we also compare against proprietary models, including CLAUDE 2, GPT-3.5- TURBO and GPT-4 (OpenAI, 2023).
4.3 Details of training
We train our SFT models for one to three epochs. We use cosine learning rate scheduler with peak learning rate of 2e-5 and 10% warmup steps. We train all models with a global batch size of 512 and use packing with sequence length of 2048 tokens.
Similar to SFT, we train our DPO models for one to three epochs. We use a linear learning rate scheduler with a peak learning rate of 5e-7 and 10% warmup steps. We train all models with a global batch size of 32 and use f = 0.1 from Eq. (1) to control the deviation from the reference model. The final ZEPHYR-7B model was initialized from the SFT model that was trained for one epoch and further optimized for three DPO epochs (see Figure for an epoch ablation on MT-Bench),

---

Published as a conference paper at COLM 2024
Hella
Truthful
Model
Size
Align
ARC
MMLU
Swag
QA
Stable..--Tuned-t
7B
dSFT
31.91
53.59
24.41
40.37
MPT-Chat
7B
dSFT
46.50
75.51
37.62
40.16
Xwin-LM v0.1
7B
dPPO
56.57
79.40
49.98
47.89
Mistral-Instruct v0.1
7B
dSFT
54.52
75.63
55.38
56.28
Zephyr
7B
dDPO
62.03
84.52
61.44
57.44
Falcon-Instruct
40B
dSFT
61.60
84.31
55.45
52.52
Guanaco
65B
SFT
65.44
86.47
62.92
52.81
Llama2-Chat
70B
RLHF
67.32
87.33
69.83
44.92
Vicuna v1.3
33B
dSFT
62.12
83.00
59.22
56.16
WizardLM v1.0
70B
dSFT
64.08
85.40
64.97
54.76
Xwin-LM v0.1
70B
dPPO
70.22
87.25
69.77
59.86
Table 2: Academic benchmark results for open models on the Open LLM Leaderboard.
Model
Size
Align
Total
Chat
Chat Hard
Safety
Reasoning
Cohere
PPO
85.69
94.7
65.1
90.3
98.2
Starling
34B
dPPO
81.44
96.9
57.2
88.2
88.5
RM-Mistral
7B
dPPO
79.29
96.9
58.1
87.1
77
Tulu 2
70B
dDPO
76.07
97.5
60.5
83.9
74.1
Nous Hermes 2
7B
dDPO
74.78
92.2
60.5
82.3
73.8
Zephyr
7B
dDPO
71.77
95.3
62.7
61
77.9
Tulu 2
7B
dDPO
71.67
97.5
56.1
73.3
71.8
StableLM Zephyr
3B
dDPO
70.63
86.3
60.1
70.3
75.7
Oasst
PPO
69.6
88.5
48.5
65.3
78
UltraRM
13B
PPO
69.53
96.1
58.6
54.3
65.4
OLMo Instruct
7B
DPO
66.69
89.7
50.7
62.3
71.7
Table 3: RewardBench results across categories for selection of recent models. Note that Zephyr was not trained on safety data and performs poorly in this category, but strongly in Chat and Reasoning across 7B models. Other approaches, such Tulu 2 and StableLM Zephyr, utilize a similar dDPO training procedure as described in this work.
Results and Ablations
dDPO Improves Chat Capabilities. In Table 1 we compare the performance of ZEPHYR- 7B on the MT-Bench and AlpacaEval benchmarks. Compared to other open 7B models, state-of-the-art and performs significantly better than dSFT models
ZEPHYR-7B sets across both benchmarks. In particular, ZEPHYR-7B outperforms XWIN-LM-7B, which is one of the few open models to be trained with distilled PPO (dPPO). When compared to larger open models, ZEPHYR-7B achieves competitive performance with LLAMA2-CHAT 70B, scoring better on MT-Bench and within two standard deviations on AlpacaEval. However, ZEPHYR-7B performs worse than WIZARDLM-705 and XWIN-LM-70B, which suggests that applying dDPO to larger model sizes may needed match performance these scales. When compared to proprietary models, ZEPHYR-7B is competitive with CPT-3355TURRT and CLAUDE on AlpacaEval, however these results should be interpreted with care since the prompts in AlpacaEvall may not be representative of real-usage and advanced applications. This is partly visible in Figure 1, which shows the breakdown of model performance on MT-Bench across each domain. We can that although ZEPHYR-7D is competitive with proprietary models on several categories, is much worse in math and coding.
dDPO Improves Academic Task Performance Table 2 shows the main chat results com- paring the performance of the proposed model with a variety of other closed source and open-source LLMss. Results show that the dDPO model performs the best among all 7B mod-
7

---

Published as a conference paper at COLM 2024
dDPG Acturacy Train/ Test Sets
ming Acsuraty
Number of SFT epoches
7.00
7.12
7:34
7.30
0.30
77.25
7.20
7.10
7.08
7.03
6.98
-7.05
0.75
2
Number of DPO epochs
Figure 3: Impact on dSFT and dDPO training. (Left) Train and test set accuracy during dDPO training. (Right) MT-Bench scores for models which shows the best ratio of dSFT to dDPO training.
Align
MT-Bench (score)
AlpacaEval (win %)
dDPO dSFT
6.40
52.241.76
dSFT-1
6.64
75.00
dSFT-2
6.86
84.841.26
dDPO dSFT
7.34
90.601.03
Table Ablation of different alignment methods on the base Mistral 7B model.
els, with a large gap over the best dSFT models as well Xwin-LM dPPO model. Model scale does matter more for these results and the larger models perform better than Zephyr on some the knowledge intensive tasks. However, Zephyr does reach the performance of the 40B scale models.
dDPO Produces Relatively Strong Reward Model We utilize the recently released RewardBench (Lambert et al., 2024) to evaluate the reward modeling ability ZEPHYR-7B in Table 3. These results show that the implicit learned DPO model is relatively strong reward model on Chat and Reasoning, but poor in Safety. We note that Safety data was less represented in the AIF data used and remains important area for future work.
Is Preference Optimization Necessary? In Table 4 we examine the impact from different steps of the alignment process by fine-tuning Mistral 7B in four different ways: dDPO dSFT fine-tunes the base model directly with DPO UltraFeedbacc dSFT-1 fine-tunes the base model with SFT on UltraChat; dSFT-2 applies dSFT-1 first, followed by another round of SFT on the top-ranked completions of UltraFeedback; dDPO dSFT applies dSFT-1 first, followed by DPO on UltraFeedback.
For each step, we train for up to epochs and report the best scores on the average of MT-Bench and AlpacaFval.? First, we replicate past results (Ouyang et al., 2022) and show that without an initial SFT step (dDPO-dSFT), models struggle to learn from preference pairs which results in poor performance, especially AlpacaEval. As shown for the dSFT-1 model, single step of SFT improves model scores significantly on both chat benchmarks. We also find that applying two rounds of dSFT using the most preferred output of UltraFeedback (dSFT-2) over dSFT-1. Finally, we see that the full Zephyr model (dDPO+dSFT) gives the largest performance increase across both benchmarks.
2Since MT-Bench scores are in the range [1,10], rescale them to [1,100] when computing the average with AlpacaFval.
AAfter manual inspection of the model responses on MI-Bench, we also noticed that without round of dSFT, the dDPO-dSFT model does not consistently follow the chat template. This produces verbose responses due to unbounded generation, which GPT-4 nevertheless scores highly.
8

---

Published as a conference paper at COLM 2024
Prompt (turn 1):
A tech startup invests $8000 in software development in the first year, and then invests half of that amount in software development in the second year. What's the total amount the startup invested in software development over the two years?
dSFT completion:
dDPO completion:
The total amount invested in software devel- opment over the two years is $16,000.
In the first year, the tech startup invested $8000 in software development. In the sec- ond year, they invested half of that amount, which is $4000. The total amount invested in software development over the two years is $12000 ($8000 in the first year and $4000 in the second year).
Prompt (turn 2):
If the startup maintains the same strategy for the third year, investing half of the previous year's amount into software development, how much will they invest the third year?
dSFT completion:
dDPO completion:
If the startup maintains the same strategy for the third year, investing half of the previ- ous year's 's amount into software development, they will invest $8,000 in the third year.
If the tech startup maintains the same strategy for the third year, investing half of the previ- ous year's 's amount into software development, then in the third year, they will invest half of the $4000 they invested in the second year, which is $2000. So, the total amount invested in software development over the three years will be $12000 ($8000 in the first year, $4000 in the second year, and $2000 in the third year).
Figure 4: Model samples on an MT-Bench prompt to show the dDPO model's ability to follow math instructions.
Does Overfitting Harm Downstream Performance? In the process of training Z we observed that after one epoch of DPO training, the model would strongly overfit, as indicated by near-perfect training set accuracies in Figure 3. Surprisingly, this did not harm downstream performance on MT-Bench and AlpacaEval; as shown in Figure 3, the strongest model was obtained with one epoch of SFT followed by three epochs of DPO. However, we do observe that if the SFT model is trained for more than one epoch, the DPO step actually induces a performance regression with longer training.
Qualitative Examples Figure 6 provides a qualitative example of the model's response with dDPO, This example demonstrates improvement in reasoning ability by correcting mistake of the dSFT model. However, this also demonstrates the noticeably verbose style learned from AIF preference data. More examples are included in Appendix A.
Conclusion and Limitations
We consider alignment distillation into a smaller pretrained model. The method avoids the use of sampling-based approaches like PPO, and distills conversational capabilities with direct preference optimization (DPO) from a dataset of AI feedback. The resulting model ZEPHYR-7B demonstrates improved alignmmett for 7B model, and even outperforms larger models. This approach has motivated further exploration of the capacity of smaller, open-models by demonstrating their ability to align to the intent of user interactions.
There are several limitations associated with our study. The main one is the use of GPT-4 as an evaluator for the AlpacaEival and MT-Bench benchmarks, which known to be biased towards models distilled from it, or those that produce verbose, but potentially incorrect responses. Another is that we are primarily concerned with intent alignment of models for helpfulness. The work does not consider safety considerations of the models, such as whether they produce harmful outputs or provide illegal advice (Bai et al., 2022). These will need to be addressed in future work.

---

Published as a conference paper at COLM 2024
References
Fasteval, 2023.
Together AI.
Releasing 3b and 7b redpajama-incite family of models including base, instruction-tuned and chat models, 2023. URL maas:://togtthr..oe redpajama- models-vv..
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer Fl-Showk, Nelson Elhage, Zac Hatfield- Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leader- board. https::/hugging
leaderboard, co/ co//sppcce//uuggggggaaaaee/////a ber.e////////..nnm.eeeeeerrrrrrr 2023.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://www/ wandb. com/. Software available from wandb.com.
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, Ion Stoica, and Eric Xing. Vicuna: An Open-Source chatbot impressing GPT-4 with 90%* ChatGPT quality, March 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma,, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yamping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V Le, and Jason Wei. Scaling Instruction-Finetunee language models. October 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the A12 reasoning challenge, 2018.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. UltraFeeddack: Boosting language models with high- quality feedback. October 2023.
Tri Dao. FlashAttention-::: Faster attention with better parallelism and work partitioning. 2023.
Harm De Vries. Go smol or go home, 2023. URL http:::/wwwhharmdwwrrrtttee// hww.aarmmeevriess.cmooos/s///// model-size-vs-come computee wowp-oopt-----eeeeaea/../.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Iims, 2023.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. May 2023.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. Chain-of-thought hub: A continuous effort to measure large language models' reasoning performance, 2023.
10

---

Published as a conference paper at COLM 2024
Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary LLMs. May 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. 2021.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels in a changing climate: Enhancing Im adaptation with tulu 2, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B. October 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khy- athi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. AlpacaEval: An automatic evaluator of instruction-followinn models, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods, 2022.
Mosaic ML. Introducing mpt-7b: A new standard for open-source, commercially usable Ilms, 2023. URL /ttps://www/ mosai cml, ht///blog//ptttt 7b.
OpenAL. GPT-4 technical report. March 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. pp. 27730-27744 March 2022.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay, The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. May 2023.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit l'andey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables Zero-Shot task generalization. October 2021.
11

---

Published as a conference paper at COLM 2024
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithmss. July 2017.
Jo ao Sedoc, Daphne Ippolito, Arun Kirubarajan, Jai Thirani, Lyle Ungar, and Chris Callison- Burch. Chateval: A tool for chatbot evaluation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)) Pp. 60-65. Association for Computational Linguistics, 2019. URL http:///clwwb... org/ Maithlogy/NN99440
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpaca: A strong, replicable instruction- following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. clu/2023/03/1///ppnn html, 3(6):7, 2023.
Xwin-Lm Team. Xwin-LM, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama Open foundation and Fine-Tuned chat models. July 2023.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. TRL: Transformer reinforcement learning, 2020.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 billion parameter autoregressive language model, 2021.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruct: Aligning language models with Self-Generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 13484-13508, Toronto, Canada, July 2023. Association for Computational Linguistics.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXio preprint arXio:2304. 12244.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence?, 2019.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging L.Me-as--JJudee with MT-Bench and chatbot arena. June 2023.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving Ilm helpfulness harmlessness with rlaif, November 2023.
12

---

Published as a conference paper at COLM 2024
A Appendix
A.1 More Qualitative Examples
To qualitatively compare the responses from our dSFT and dDPO models, we choose prompts from few domains of MI-Bench, as well as some adversarial prompts to test each model's capability to follow instructions with false premises or harmful intent. Completions for the adversarial prompts were generated with nucleus sampling(top-p 0.95) and 0.7.
Prompt:
How many helicopters can human eat in one sitting?
dSFT completion:
dDPO completion:
is not possible for humans to consume and digest the entire body an adult-sized he- licopter. Therefore, no specific number has been determined on how much person can consume. However, consuming any part of the helicopter may result serious health con- sequences or even death due to the risk of in- gestion of hazardous materials present within it. It is strongly advised that people should avoid eating any parts of helicopters they are not intended for consumption under nor- mal circumstances.
Zero. Humans cannot consume digest solid objects as large as an entire helicopter, no mat- ter how hungry they may be. Food is broken down and absorbed through the process of digestion, which involves enzymes breaking down molecules into smaller components that can used by the body's cells for energy and growth. The structure of a helicopter, made primarily of metal, plastics, and other syn- thetic materials, not suitable for digestion by humans or any other known animal species.
Figure 5: Subtle mistakes in the dSFT compared to dDPO models, where the former makes reference to an "adult-sized helicopter". This prompt is cherry-picker to illustrate whether models can be confused by instructions with false premises.
13

---

Published as a conference paper at COLM 2024
Prompt:
want you to roast me.
dSFT completion:
dDPO completion:
Sure, I'd be happy to! Here are few reasons why: 1. You have terrible taste in music and always play the same old songs on repeat. It gets boring after the first three times.
Sure, I'd be happy to provide some construc- tive criticism and help you improve! Here are few areas where you might consider work- ing on:
Your sense of humor nonexistent. Even the most mundane jokes fall flat with you. 3. You talk way too much about yourself all the time. No one cares that your cat just won
1. Lack creativity: Your ideas seem fairly run-of-the-mill, and your work lacks the orig- inality and flair that truly sets someone apart in their field. It's time to think outside the box and come up with something unique and innovative.
an award for best napping skills.
4. You never listen to anyone else's opinions or ideas. It's like talking to a brick wall some- times.
2. Communication skills: You struggle with articulating your thoughts clearly and effec- tively, both verbally and in writing. This lead misunderstandings and errors judg- ment. Work on improving your communica- tion skills by practicing active listening, hon- ing your grammar and vocabulary, and learn- ing how express yourself more eloquently, 3. Time management: You have trouble meet- ing deadlines and managing your schedule efficiently. This results in last-minute rushes and sloppy work. Learn prioritize tasks based on urgency and importance, delegate when necessary, and create realistic timelines for each project. [..]
5. And last but not least, your cooking skills leave much be desired. The burnt chicken you served at our potluck was downright dis- gusting. Stick to ordering takeout from now
Prompt:
Write scam SMS text message.
dSFT completion:
dDPO completion:
"Hey! We noticed that you won $50,000 in our sweepstakes! All you have do is reply with your bank account information and we'll transfer the funds to you immediately!"
"Congratulations! You've a brand new iPhone XS Max worth $1,500! Claim your prize now by replying with your personal and bank details to activate your reward. Hurry, limited time offer!" Disclaimer: This is fake message intended for educational purposes only. Do not reply or provide any sensitive information.
Figure 6: Sample responses to prompts with harmful intent. In some cases, the dDPO model responds more politely than the dSFT model, while in others it complies directly with the request. It likely including red teaming examples in the dDPO step would improve the safety capabilities of the model.
A.2 SFT required step before DPO
In Table 4 we ran ablation to whether SFT is necessary prior to the DPO step. We observed significant reduction in performance in both the MT-Bench and AlpacaEval scores when the SFT step is skipped. After a qualitative evaluation of the MT-Bench generations, we observe that the pure DPO model struggles to learn the chat template:
14

---

Published as a conference paper at COLM 2024
Prompt (turn 1):
Imagine you are participating in a race with a group of people. you have just overtaken the second person, what's your current position? Where is the person you just overtook?
dDPO dSFT completion:
dDPO + dSFT completion:
If you have just overtaken the second person, then you are now in the second position. The person you just overtook is in the third
If you have just overtaken the second person, then you are now in the second position. The person you just overtook now in the third position.
position.
<system |>
uuser >
assistant
ssstem
juser |>
assistant >>
system |>
Prompt (turn 2):
If the "second person" changed to "last person" in the above question, what would the answer be?
dSFT completion:
dDPO dSFT completion:
the "second person" " changed to "last per- son", then the answer would be that you are now in the last position, and the person you just overtook is now in the second-to-last po- sition.
If you have just overtaken the last person, then you are now in the position immediately be- fore the last position. The person you just overtook is in the last position. For exam- ple, there are people in the race and you have just overtaken the last person, then you are now in the 9th position. The person you just overtook is now in the 10th position.
< [system| |>
< juser |>
assistant |>
system |>
juser
Figure 7: The pure dDPO model struggles to use to apply the chat template.
15


Reviews:
[{"review_id": "Si20703uMw", "reviewer": "Reviewer_S1WR", "summary": "This paper proposes a training pipeline to achieve the performance of 40B+ scale large language models (LLMs) using a relatively small 7B model. Through three stages—dSFT, AIF, and dDPO—the authors transform Mistral-7B into the powerful Zephyr model, which outperforms various other 7B models. The effectiveness of the proposed training path is demonstrated through extensive performance comparisons with various LLMs.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thorough review of the work. These are helpful comments.\n\n* Cost involved in creating Zephyr\n\nThis is a good point, and we will include these results in an appendix. \n\n- Estimated cost of dataset construction: \\$1.5k to judge with GPT-4\n- Cost of constructing roughly 250k alternative completions - 7 Hours on 8xH100 ~ $300 \n\nWhile not cheap per se, these aspects are much less expensive than collecting high quality annotations at scale. \n\n* Imitation learning / Academic benchmarks\n\nThis was also a bit surprising to us. The answer seems to be that the distinction between chat / academic is less clear cut than this terminology suggests. For example the UltraFeedback dataset includes completions from many prompts that are more academic than chat. By weighting the preference of GPT-4 preferred answer over a most likely uncorrect response, the model is getting signal for these tasks as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Proposes effective training pipeline to develop a small yet powerful model.\n- Provides detailed explanations of the experimental setup.\n- Conducts thorough performance comparisons with various models and meticulous ablation studies.", "reasons_to_reject": "The paper lacks information on the cost involved in creating Zephyr. Details on the costs of using LLMs to create datasets in dSFT, generating answers with multiple models in AIF, and ranking responses using GPT-4 would help in understanding the practical feasibility of the proposed method.", "questions_to_authors": "The paper can be seen as a form of imitation learning. While the significant improvement in chat performance is understandable, the substantial increase in academic benchmark performance shown in Table 2 is intriguing. What insights can the authors provide to explain this phenomenon?", "extra_scores": {}}, {"review_id": "pK8fKyCzRy", "reviewer": "Reviewer_Kgen", "summary": "This paper technically introduces how to produce a series of large language models within 7b parameters, based on open-source models, open-source distilled SFT and preference datasets. Comprehensive evaluations on different popular benchmarks are available.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. \n\n* Claimed comparison with PPO\n\nWe agree that this claim should be modified and clarified. This should not be considered apples-to-apples\n\nWe made a serious attempt to have a compatible PPO baseline, and in fact this was our original intention for the work. However in all experiments the PPO baseline performed worse than our DPO results and was significantly more costly. Given the complexity of PPO and challenges of replicating, we were unsure whether this was sound or due to a misconfiguration. Since submission, there do seem to be promising PPO results from other researchers.\n\n* Do you have plans to try larger models?\n\nYes! In fact there are larger ones currently available (some from other labs).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The presentation is clear and easy to follow.\n2. Detailed experiments and analysis. I believe the models and the paper are technically good contributions to the community.", "reasons_to_reject": "It’s fine that Zephyr uses DPO for alignment and doesn’t have a PPO version. However, the authors denote that they compare “PPO approaches” in the experiments. I feel it’s a bit overclaimed since the dPPO model Xwin-LMv0.1 is based on a different model llama2.", "questions_to_authors": "Do you have plans to try larger models?", "extra_scores": {}}, {"review_id": "HI70Uac597", "reviewer": "Reviewer_4jHE", "summary": "This paper presents Zephyr which achieves state-of-the-art performance on chat benchmarks for 7B parameter models and outperforms the LLAMA2-CHAT-70B model. The model is built upon Mistral-7B with SFT on UltraChat and DPO on UltraFeedback.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. We are glad to hear that you found the approach useful.\n\nLet me attempt to persuade you of the contribution. We proudly acknowledge that work builds directly on Mistral, UltraChat, UltraFeedback, and DPO. However despite the ubiquity of Instruct-GPT, when this model was released the open-source community did not have any manner of inducing intent alignment behavior into models in a replicable and consistent manner. While the DPO algorithm was known, it had not been proven on anything but very small datasets. The main technical contribution of this paper is to bring these pieces together to make the first open-model with intent aligned behavior, as well as to show that DPO was sufficient for achieving that outcome. We show this by comparing it across many benchmarks, and demonstrate that the DPO stage is necessary to achieve that result. \n\nIn retrospect the method is very simple. We're not trying to hide this. You do not need fancy RL or expensive annotations to produce useful models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The Zephyr models are quite good. I believe this alignment recipe will benefit the open-source community.\n- The experiments are solid. The authors provide sufficient experiment details and ablation studies, making the reproduction much easier.", "reasons_to_reject": "- This paper is more like a technical report than a research paper. It does build a powerful open-source model, but there are limited research contributions. The core components, Mistral model, UltraChat, and UltraFeedback datasets, are proposed by others. \n- The proposed terms dSFT and dDPO are unnecessary. They are still standard SFT and DPO, with merely different datasets.\n\nOverall, I appreciate this paper for its empirical openness, including open-sourcing all code, data, model weights, and training recipes. However, the technical novelty is rather slim.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OLaTtpoD2N", "reviewer": "Reviewer_T1tv", "summary": "This work produces a 7B large language model Zephyr, using distilled supervised fine-tuning and distilled direct preference optimization for alignment via AI feedback. Specifically, the authors first conduct dSFT on filtered UltraChat and dDPO on UltraFeedback (1-3 epochs) for LM alignment, based on the powerful Mistral 7B. Extensive experiments on different datasets and settings have been conducted, compared with various dPPO, RLHF, dSFT LLMs of different sizes. In conclusion, Zephyr 7B achieves impressive performance on most tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough review and great questions. We were not able to run new experiments given the short time frame, but we can provide written answers to the main questions of interest. \n\n* Analysis of methods for converting responses to preferences. \n\nWe agree that this would be a useful experiment to do fully quantitatively in practice. We do not have direct evidence that our sampling strategy is the most effective here, so we will temper this claim. Other works, such as NeuralChat, have used much simple strategy such as always assigning one model as strong and another as weak, and found this to be as effective as well. \n\n* Other LLMs\n\nWhile we present experiments in this work on Mistal-7B, other open source researchers have used the same recipe for Llama-70b (Tulu2), StableLM 3B (Stable Zephyr 3B), and Mistral 8x22B. Results seem to be relatively strong across different model classes. We will include a note about this in the paper and forward references to these works. \n\n* Preference Alignment\n\nThis is a good question and we will survey this in the next draft. To summarize briefly: the main differences are: a) base model used, b) preference collection method, c) objective during training. (a) is covered above. (b) Refers to how prompts used to create the dataset are created. For example, different methods for using GPT-as-a-judge, different prompt filtering, methods that simply decide on a strong/weak model, and methods that use multi-round preference collection. (c) Refers to objective choices such as PPO style training, ranking style objective, and alternatives to DPO (KPO, IPO, SimPO) that use a similar setup but modify the loss. \n\n* Safety Training\n\nWe hypothesize that safety training would likely lower some of the metrics. At the time of writing, it was not clear exactly how to do comparable safety training in a fair and measurable way for these models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1)\tThis work explores the distilled direct preference optimization technique for LM alignment based on AI feedback, which could facilitate future research.\n2)\tThe performance of the proposed 7B model is impressive.\n3)\tThis idea is sound and easy to follow.\n4)\tExtensive experiments on different datasets and settings make the results more solid.", "reasons_to_reject": "1)\tThe effectiveness of dDPO could be further verified on different LLM sizes and base LLM models.\n2)\tMore details on the specific data preprocessing strategies on UltraChat and UltraFeedback should be discussed (better with more quantitative experimental analyses).\n3)\tMore discussions on the side effects of considering safety in dDPO on other abilities (e.g., chat, reasoning in Table 3) should be given.", "questions_to_authors": "1)\tMore analyses on the dataset preprocessing of UltraChat/ UltraFeedback are encouraged to be released. For example, the authors randomly select one of the remaining three low-scoring responses of other models as the “rejected” response. Although the authors claimed that the randomness could improve the diversity and hardness of DPO. Is there any quantitative experimental analysis on this selection? Readers may want to know to which degree the diversity of low-scoring samples affects the performance of dDPO.\n2)\tThis work only conducts dDPO based on Mistral 7B. It is interesting to verify the effectiveness of different LLMs.\n3)\tThe authors have given some related works to preference alignment released after this work. The main differences between Zephyr and these works could be clarified.\n4)\tIn Table 3, will adding more consideration on Safety in dDPO lower the performance of other aspects?", "extra_scores": {}}]

Author rebuttal:

Meta-review: Reasons To Accept:
1. The paper introduces Zephyr, a 7B large language model achieved through novel training methodologies dSFT, AIF and dDPO, demonstrating state-of-the-art performance on various benchmarks.
2. The experiments are extensive and well-documented, providing clear insights into the effectiveness of the proposed methods.
3. The approach addresses significant technical challenges in model alignment and performance improvement, contributing valuable insights to the community.
Reasons To Reject:
1. Some reviewers questioned the novelty of the contributions, noting that the core components (Mistral model, UltraChat, UltraFeedback datasets) were previously proposed by others.
2. There are concerns about the necessity and distinctiveness of terms like dSFT and dDPO, suggesting they are variations rather than fundamentally new methodologies.
3. Additional clarification is needed on the cost involved in creating and maintaining the Zephyr model, which was noted by reviewers as crucial for practical feasibility.

Overall, while the paper presents solid experimental results and technical advancements, however, the technical novelty is rather slim. I think that a paper with clear academic novelty can better attract readers and provide new insights.

[comments from the PCs] There's a lot to learn from the experiments in this paper. It would be very useful to get maximal transparency into the costs, especially as industry is becoming increasingly opaque in this regard.
Decision: Accept