OpenChat: Advancing Open-source Language Models with Mixed-Quality Data
Authors: Guan Wang
Sijie Cheng
Xianyuan Zhan
Xiangang Li
Sen Song
Yang Liu
Venue: ICLR 2024 poster (2024)
Keywords: Open-source Language Models
Fine-tuning
Mixed-quality Data
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Nowadays, open-source large language models like LLaMA have emerged. Recent developments have incorporated supervised fine-tuning (SFT) and reinforcement learning fine-tuning (RLFT) to align these models with human goals. However, SFT methods treat all training data with mixed quality equally, while RLFT methods require high-quality pairwise or ranking-based preference data. In this study, we present a novel framework, named OpenChat, to advance open-source language models with mixed-quality data. Specifically, we consider the general SFT training data, consisting of a small amount of expert data mixed with a large proportion of sub-optimal data, without any preference labels. We propose the C(onditioned)-RLFT, which regards different data sources as coarse-grained reward labels and learns a class-conditioned policy to leverage complementary data quality information. Interestingly, the optimal policy in C-RLFT can be easily solved through single-stage, RL-free supervised learning, which is lightweight and avoids costly human preference labeling.
Through extensive experiments on three standard benchmarks, our openchat-13b fine-tuned with C-RLFT achieves the highest average performance among all 13b open-source language models. Moreover, we use AGIEval to validate the model generalization performance, in which only openchat-13b surpasses the base model. Finally, we conduct a series of analyses to shed light on the effectiveness and robustness of OpenChat. Our code, data, and models are publicly available at https://github.com/imoneoi/openchat and https://huggingface.co/openchat.

Published as a conference paper at ICLR 2024
OPENCHAT: ADVANCING OPEN-SOURCE LANGUAGE MODELS WITH MIXED-QUALITY DATA
Guan Wa Cheng! Xianyuan Zhan Xiangang Lł, Sen Song2 Yang Liu DDepartment of Computer Science and Technology. Tsinghua University
PLaboratory of Brain and Intelligence, Tsinghua University *Institute for AI Industry Research (AIR), Tsinghua University *Shanghai Artificial Intelligence Laboratory $01.AI imonenexe gmail com, cs j 2m iss tt tsinnhuu.a....m edu.cn
ABSTRACT
Nowadays, open-source large language models like LLaMA have emerged. Re- cent developments have incorporated supervised fine-tuning (SFT) and reinforce- ment learning fine-tuning (RLFT) to align these models with human goals. How- ever, SFT methods treat all training data with mixed quality equally, while RLFT methods require high-quality pairwise ranking-based preference data. In this study, we present a novel framework, named OpenChat, to advance open-source language models with mixed-quality data. Specifically, we consider the general SFT training data, consisting of small amount of expert data mixed with a large proportion of sub-optimal data, without any preference labels. We propose the Condittioned)RLLFT which regards different data sources as coarse -ranined re- ward labels and learns a caas-coondiioneed policy to leverage complementary data quality information. Interestingly, the optimal policy in C-RLFT can be easily solved through single-stage, RL-free supervised learning, which is lightweight and avoids costly human preference labeling. Through extensive experiments on three standard benchmarks, our openchat -113. 13 fine-tuned with C-RLFT achieves the highest average performance among all 13b open-source language models. More- over, we use AGIEval to validate the model generalization performance, in which only openchat-1 3b surpasses the base model. Finally, we conduct series of analyses to shed light on the effectiveness and robustness of OpenChat. Our code, data, and models are publicly available at http://ggiibbbcommmmmmorroopppgh and httpss//huggnnfaa..oooooeee
INTRODUCTION
Recently, there have been notable advancements in Large Language Models (LLMs), such as GPT- 4 (OpenAl 2023) and Chinchilla (Hoflmann et al., 2022). demonstrating impressive performance in various downstream natural language processing (NLP) tasks (Zhao et al. 2023). Despite the
remarkable success of GPT-4, the specific techniques employed in its development remain shrouded in mystery. To gain a deeper understanding of the underlying technical aspects and to promote the widespread adoption of LIMs, series of open-source base language models have emerged, especially LLaMA (Touvron et al.) 2023a) and Llama (Touvron et al.) 2023b). Building upon the released base language models, there are typically two methods to align these base models to specific abilitics, including supervised fine-tuning (SFT) and reinforcement learning fine-tuning (RLFT).
The first line of methods (Chiang et al. 2023 Taori al. 2023) SFT to enhance instruction following abilities. Most existing methods primarily focus on designing SFT datasets. Some stud-
ies (Chiang et al. 2023 Geng et al.) 2023) collect user-shared conversations
well as human
feedback datasets from the public web, while others (Xu er al. 2023a Ding et al. (023) develop frameworks for automatically gathering extensive open-domain instructions spanning various diffi- culty levels. However, these constructed SFT datasets are generally mixed with limited expert data
"Equal contribution

---

Published as a conference paper at ICLR 2024
SFT
RLFT
C-RLFT (OpenChat)
Sample Compiitions
Sub optimal
0.1
Sub optimat
0.1
MLE
MILE
100
MLE
Dataser
IIM
Preference
Data
Reward Model
Reward
IIM
Class- Sast-Conditoree
Dalasel
Course grained Rewards
LLM
Figure 1: Our proposed framework OpenChat with Conditioned-RLFT advance the open-source language model fine-tuning with mixed-quality data, comparing to previous supervised fine-tuning (SFT) method and reinforcement learning fine-tuning (RLFT) method. MLE and RL denote maxi- mum likelihood estimates and reinforcement learning, respectively.
and a large proportion of sub-optimal data due to the high cost of human labor and API requests. Naturally, it is not advisable to indiscriminately feed all these mixed conversations to the base model, as the low-quality data are likely to negatively impact learning (Zhou et al., 2023 Xu et al. 2022). However, this is largely neglected in previous methods, which often treat all training data equally.
To allow LLMs to go beyond modeling the training data distribution, recent LLMs (OpenAl
2023 Touvron et al., 2023b) adopt RLFT to align better with the human desired behaviors, pecially API-based models. The well-known reinforcement learning from human feedback (RLHF) method (Ouyang et al., 2022 Christiano al. 2017 Bai et al., (202b) first collects plenty of high- quality preference feedback from human annotators to fit one or multiple reward models (typically also trained based on smaller LLMs), and then uses RL to maximize the estimated reward. The in- volvement of learning and optimizing with extra reward models using RL brings considerable com- putational and stability issues. Some recent studies (Rafallv et al.] 2023 Yuan al.) 2023) partly address this problem by avoiding fitting the reward model and fusing preference modeling and L.M fine-tuning into a single-stage training process. However, all existing RLHF methods require high- quality pairwise or ranking-basee -baed preference data for preference modeling, which inevitably require expensive human expert annotations (Casper et al. (023).
To address the aforementioned limitations, we propose a new framework, named OpenChat, to ad- vance the open- language model fine-tuning with mixed-qualle data as shown in Fig. [ Here, consider the general non-pairwise (nor ranking-based SFT training data, consisting of small amount of expert data and a large proportion of easily accessible sub-optimal data, without any preference labels. Specifically, we propose the Conditioned-RIFFT (C-RLFT), which enables lever- aging mixed-quality training data with very coarse- -ranseegr reward labels. The reward label can be as simple as relative value differentiating different classes of data, i.e., expert and sub-optimal. We derive C-RLFT based on the KL-regularized RL framework (Jaques et al. 2019: Korbak et al.) 2022]. which maximizes the reward while penalizing the difference between the fine-tuned policy and a reference policy. However, to remedy the imperfect reward signal, we learn the fine-tumed LLM itself as class-conditioned policy (i.e., conditioning data source classes with distinct prompt tokens), and regularize it with a better and more informative cans-conndiioned reference policy in- stead of the original pre-trained LLM. The optimal policy for this RL problem can be shown as equivalent to cans-cooniiioned reward- weighted regression problem, which can be easily solved through single-stage supervised learning. C-RLFT provides several particularly desirable features for open-source LLM fine-tuning. First, it allows for simple and RL-free training, largely remov- ing the complexities and instabilities in typical RLHF fine-tuning. Second, it has extremely low requirements for the quality of the reward and does not need costly human feedback collection.
Despite being simple and lightweight, our proposed OpenChat with C-RLFT achieves great perfor- mance in a series of benchmark evaluations. Specifically, we leverage the ShareGPT conversations datase(] ( following Vicuna (Chiang et al. 2023) and use 11ama-2- 2--13b as the base model. It is worth noting that our proposed method can be applied to any mixed-quality datasets and arbitrary base language models. We conduct extensive experiments on three standard benchmarks to assess instruction following ability, including Alpaca-Eval (Li et al. 2023). MT-bench (Zheng et al. 2023) and Vicuna- (C (Chiang et al.) 2023). The results demonstrate that apenchatt1333 sigmificantly surpasses previous 13b open-source language models and can even outperform gpt-3. 5-turbo in all three benchmarks. Furthermore, we also use AGIEval (Zhong et al.) 2023) to prove the gen- cralization, where openchat-11 also achieves the top-1 average accuracy among all 13b open- source language models. Finally, we design a series of ablation studies and analyses to validate the contribution of different components, and performance consistency, providing insights into the effectiveness and robustness of OpenChat.
'The ShareGPT dataset is collected from hutps//shareentt.cml.
2

---

Published as a conference paper at ICLR 2024
2 PRELIMINARIES
Given a conversation dataset D (( where 2; indicates the instruction, y; its correspond- ing response, the pre-trained language model (b(22) can be regarded as a probability distribution mapping from instructions to responses. There are two lines of research to adapt the pre-trained language model To to a fine-tuned language model To (y)a))) with desirable features, including supervised finettnning and reinforcement learning fine-tuning.
Supervised Fine-tuning (SFT). This line of methods (Xu et al. 2023a b Ding et al. uses high-quality conversation dataset D to fine-tune the pre-trained language model ko(y x)) using supervised learning, i.e., maximum likelihood estimates (MI.)):
2023 directly
JSPT (0) (x [lo (v(lx)
where my is initialized from To. To ensure the fine-tuning performance, SFT requires the conversa- tion dataset D to have very high quality, because SFT treats all training data uniformly (Zhou et al.)
2023 Chen al. 2023). However, the collection of high-quality SFT datasets can be very expen- sive. Most existing open-source LLMs (Chiang et al. 2023 Geng et al. 2023 Xu et al. 2023aa Ding ct al. 2023) fine-tune their models using conversation datasets that likely contain a large proportion of sub-optimal data due to high costs of human labor or API requests, inevitably leading to a certain level of performance degeneration.
Reinforcement Learning Fine-tuning (RLFT). Another intuitive approach to align LLMs is through RL, which models rewards according to human preference feedbacks (Ouyang et al. 2022 Bai et al. 2022a Rafailov et al. 2023) or pre-defined classifiers (Wu et al. 2023). and fine-tune LLMs to maximize the reward. The reward r(x.)) y), either modeled explicitly or implicitly, assigns high values on desirable responses and low values on bad ones to guide the alignment of the fine- tuned LLM. A popular RI framework for fine-tuning LLMs is the Kl.-regularized RI. (Jaques al.]
2019 Korbak et al. 2022; Rafailov et al. 2023), which adds an additional KL penalty to constrain stay close to the base pre-trained LLM To (y/zz). This has been shown beneficial to avoid distribution collapse as compared to naively maximize reward using RL (Korbak The RL objective of this series of RLFT models can be typically formulated as follows:
the fine-tuned LLM Te Fyyyy x)
ct al. 2022
(f) Ey [rx v)1 3DKl (mo, mo)
In existing RLFT methods, high-quality reward signals play crucial role in ensuring improved LLM fine-tuning performance. This, however, requires collecting considerable amounts of costly pairwise (or (aaeed human preference feedback, which poses major challenge in the development of many open-source language models.
OPENCHAT
In this section, we introduce the OpenChat framework, which provides new possibility to fine-tune open-source LLMs using easily collectable and mixed-quality training data without any preference labels. More specifically, we consider the setting where we are given a pre-trained LLM To, small set of high-quality/expprt conversation data Deep. and a larger medium-quality or sub-optimal conversation dataset Duub. we aim fine-tune an LLM policy To based on To using only data from Deep Duub Taking the most popular SFT dataset ShT used in Vicuna (Chiang et al. 2023) as an example, the distinct data sources from GPT-4 and GPT-3.5 can be regarded as Doxp and D Jub. as the overall quality of GPT-3.5 conversations generally falls short when compared to that of GPT-4
conversations (OpenAl 2023 Li et al.) 2023). where detailed comparison can be found in Sec. 5.1
Obviously, it is not possible to derive accurate and fine-graine tne-grrined reward signals solely based on Deep and Duub. However, it should be noted that the quality difference between Deep and Duub itself can serve as implicit weak reward signals. To make use of this coarse -oainne reward information, we provide a new insight that by regularizing To with a better and more informative class-conditioned reference policy ne instead of the original base pre-trained LLM no. are likely to compensate for the potential deficiencies in the rewards and achieve good fine-tuning performance. the following, describe the details of OpenChat and its core algorithm C-RLFF.
3

---

Published as a conference paper at ICLR 2024
3.1
DATASET AND REWARDS
Given the SFT conversation datasets Desp D with different quality levels, we can replenish them with distinct sources as class labels (c.g,, G E GPT-3.5) and construct a class- conditioned dataset D. {( Ut, c.)). We use Te(y/z, c) to denote class-conditioned distribution over instructions x and responses y in the class-conditioned dataset De, which can be perceived similarly the behavior policy of a dataset in offline RL literature (Levine al. 2020). with the difference that T. is now a class-conditioned policy.
According the different overall quality with respect class labels, can naturally encode coarse-grained rewards re( y) in Dr as follows:
{ 1, if (24, E Dexp (e.g., GPT-4), (e 14) = =1a, a if (zi, W:) Duub (e.g., C. GPT-3.5) (a < 1).
(3)
where we regard GPT-4 conversations as expert data Dexpert, and GPT-3.5 conversations as sub- optimal data Duub. Meanwhile, we set a 1 to guide the fine-tuned model to favor more of the high-quality responses.
3.2 FINE-TUNING VIA ((ONDITIONED--RLE
As the rewards (1, y) in Eq. (3) are very coarse-graines to reliably use them in RLFT, we need to provide additional sources of information remedy their deficiencies. Here, introduce C-RLFT, which is inspired by the insight from the goal-conditioned supervised learning in offline RL, that by conditioning on proper information in supervised goalloutcome-cconfitoneee policy, it is possible to recover optimized performance (Chen et al.) 2021 Emmons et al. [021]. C-RLFT contains two key ingredients: 1) fine-tuning the LLM class-con coaass-onniiionedd policy ze (y) (, e). and 2) regularizing wy with respect to the class information augmented reference policy Te rather than the original base reference policy xo the rlgreaalrrd RI. framework.
Clndstcond policy. Instead of directly fine-tuning an LLM from the pre-trained model mo as in existing methods, model the LLM to be fine-tuned class-conditioned policy ( 2., c). This can be easily implemented by conditioning each example from different data sources using distinct conversation templates shown below.
[GPT-4 Template]
GPT4 User: Question< htt.:::/dddddco.ddodoooooo >GPT4 Assistant:
[GPT-3.5 Template] GPT3 User:
GPP3 Assistant:
To differentiate speakers, we introduce a new < endoof.turr > special token at the end of cach utterance, following Zhou al. (2023). The endd of turn token functions similarly the EOS token for stopping generation while preventing confusion with the learned meaning of EOS during pretraining. We further discuss the effect of conversation template choice in App.
Policy optimization. To compensate for the coarse- maase-grained reward information [z(z, y). we modify the original KL-regularized RL objective Eq. (2) and instead optimize the following problem:
JC-RLF((0)) (0) En [r, v)] BDLL (TT,)Te)
(4)
The idea is to use the higher-quality and more informative class- canss-conditioeed behavior policy Te of D. for regularization, rather than the pre-trained model mp- We adopt this design for the following reasons. First, for most existing open-source pre-trained L.Mss, their performance in many cases is still inferior to the behavior policy that generates the sub-optimal data (e.g. GPT-3.5 in ShareGPT).. This means that even the sub-optimal data Daub is likely to have higher quality than no. Second, To contains additional data source information, which can help diffcrentiating the quality of data.
Following prior works (Peters& & Schaal 2007 Peng et al. 2019: Korbak et al., 2022 Rafailov et al., 2023). can be shown that the optimal solution to the above KL-regularized reward maximization objective takes the following form (see App. B for detailed derivation):
" (ylo, c) x T. To(a,, exp (3)((0)))
(5)
4

---

Published as a conference paper at ICLR 2024
We can thus extract the optimized policy Ty by minimizing the KL divergence between x² under the class conditioodd dataset De (Nair et al. 2020, Korbak et al. 2022):
Th arr min (zz,)) DxL ( (-|z, c) mo (-),,))]
arg min (z,0)~ Ey-r log To c)ll
arg max (n..c))DD exp (g)((z))) v1) log ())
(6)
The last step obtained by plugging the closed form ग² in Eq. ( and using the fact that ne is exactly the class-conditioned behavior distribution of 'D.- This suggests that the fine-tuned policy Ty can be learned through a simple weighted regression objective with the class-conditioned dataset De. This learning objective provides remarkably simple scheme to fine-tune open-source LLMs. It does not require accurate reward labels, but uses conditioning to differentiate good and inferior model behaviors. Moreover, after initializing Ta with To. during training, while most RLHF methods using PPO for policy optimization (Ouyang et al. 2022 Bai et al. 2022a) still need to maintain Tu to compute the KL penalty during fine-tuning. This enables C-RLFT to save a considerable amount of computation resources during training.
longer need to load To
Model inference. During the inference phase, assume that our C-RLFT method has learned to distinguish expert and sub-optimal data distributions. Considering that we aim to exclusively generate high-quality responses for our fine-tuned class-conditioned Te, we use the same specific conversation template employed in GPT-4 conversations during the training phase as below:
[Inference template] GPT4 User: Question< end_o fotoof.turee futurn GPT4 Assistant:
EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
Mixed-quality Data. Following Vicuna (Chiang al. 2023). we adopt widely-used SFT dataset, the ShareGPT dataset. The ShareGPT dataset consists of approximately 70k user-shared conversa- tions, including around 6k expert conversations generated by GPT-4 and the remaining sub-optimal conversations from GPT-3.55. We perform experiments to assess their varying quality in Sec. [5.]
Benchmarks. To evaluate the instruction-follot following ability, we employ the three most widely rec- ognized benchmarks. including AlpacaEval (Li et al., (023). MT-bench (Zheng et al., (0233 and Vicuna-bench (Chiang et al. (023). Additionally,
verify generalization, we perform all English tasks in AGIEval (Zhong et al. (2023) using zero-shot settings, which presents collection of human- centric standardized exams. More details of the benchmarks are listed in App. C
Baselines. We evaluate the most popular API-based and open-source LLMs: (1) gpt (Ope- nAl 2023) and gpt-3. 3pt-3.5-turbe 5-turbo (Ouyang et al. 2022) are highly advanced LLMs developed by OpenAl; (2) claude (Bai al. 2022a) is helpful and harmless assistants by Anthropic; (3)
1lama-2-chat (Touvron et al. (002b) series models are the most frequently used open-source models with SFT and RLHF; (4) w za rd rrrm (Xu et al. 2023a), guanaco (Dettmers al. (023),
ultrall 1m (Ding et al. 2023) and v cuna (Chiang et al. 2023) are among the well-known open-
source LLMs with SFT. More details are shown in App. D
Automatic Evaluators. To mitigate the cost of human annotations, we follow the official evalua- tors according to each benchmark. Specifically, AlpacaEval employs alpaca.eval.gpt, while MT-bench and secuma-bench use qpt-4. It is worth noting that these benchmarks have already computed the human agreements to ensure reliability. Additionally, we further introduce 3. and claude-222 2 to eliminate the potential self-enhancement bias in App. E
We employ three metrics following the official implementations: (1) Win rate: This metric employed for pairwise comparisons. Given a question and two answers generated by the tested model and the target model, the LLM evaluator needs compare these two models. The tested model receives point for a win. 0.5 points for a tie, and points for a loss. (2) Score: This metric is applied for single-answer grading in MT-bench, where the LLM evaluator directly judges the
Metrics.
5

---

Published as a conference paper at ICLR 2024
Models
Base Models
Method
AlpacaEval
MT-beech
Vieuna-bench
Average
Larger than 13b
gpt-4
SFT RLFT
95.3
82.5
90.0
89.3
11ama-2-chat-700 claude
11ama 2-00b
SFT + RLFT
92.7
60.0
87.5
80.1
SFT + RLFT
88.4
76.3
76.6
3pt-3.5-turbe
SFT + RLFT
86.1
50.0
62.0
guanaco-65b
11ama-68
SFT
71.8
40.6
49.4
53.9
guanaca-33b
1 11ama-333
SFT
66.0
40.6
54.4
53.7
Equal to 13b
vicuna-vi.1-139
11ama-13b
SFT
70.4
45.0
48.3
1.zaadlnnvv.O-133
11ama-13b
75.3
33.1
44.4
50.9
vicuna-vi.9-13b
1lama-2-13b
SFT
78.8
37.2
47.1
54.4
ultralm-13b
1lama-13b
SFT
80.6
37.2
50.0
55.9
mizard1n-vl.2-11
120.1.221133 32
11ama-2-138
SFT
89.2
53.1
80.6
74.3
1ama--2chat113
1lama-2-130
SFT + RLFT
81.1
55.3
86.9
74.4
openchat-13b
11ama-2 1amm-2-13bb
C-RLFT
89.5
85.0
Table 1: The win-rate (%) performance of the proposed openchat-133 and other popular open-source language models. The competitors text-davinch addavinni-00 in AlpacaEval, and gpt-3. 5-turbo in both MTT-bench and Vicuna-bench. The bold scores denote the best perfor- mance, and the underline scores indicate the second-best performance.
generated answer with a score varying from 10. (3) Accuracy: This metric is used by AGIEval for multiple-choice exam questions.
Implementation Details. The openchat- 1.b is based the 1lama-2-2-- 2-1138 TTuvron al, 2023b). We fine-tune the model for 5 epochs on the ShareGPT dataset using the Adam optimizer with sequence length of 4,096 tokens and an effective batch size of 200k tokens. Given that the reward weight term in Eq. (6) (exp(re/3)) remains constant within a class, we simplify the process by assigning a unit weight Dexp and the weight of 0.1 to Duub. The AdamW optimizer's hyperparameters
follows: 31 0.9, f3 0.95, 10-5 and weight decay of 0.1. We employ cosine learning rate schedule with maximum learning rate of 6.7 x 10-5, which decays to 10% of the maximum value. The hyperparameters remain consistent with the base model pretraining settings following Touvron et al. (2023b). However, we scale the learning rate proportionally to the square root of the batch size, tollowing the theoretical analysis provided by Granziol al. (2020).
4.2 MAIN RESULTS
In the first set of results, we compare the win-rate (%) performance of openchat-135 and other popular LLMs on three standard benchmarks to assess the instruction-fllooinng ability. The re- sults are presented in Table []] Among the API-based LLMs, the win rate of gpt-4 signif- icantly outperforms all other models, demonstrating that gpt-4 maintains obvious advantages. The open-source language model 1 aama 2 -chat-70b, which employs both SFT and RLHE, another powerful instruction-followiii However, guanaco-S 65b and guanaco-32 -33b lag behind other models larger than 13b. Regard- ing the series of 13b models, the open-source language models based 1 a1ama- 13., including vicuna-vo seuna-vl... 1-11..113301 13,, w wardlm-vi 0-133 and u plttall Setttaalmm11 1m-13b, achieve approximately 50% average win rates across the three benchmarks. Meanwhile, the open-source language models based on 11ama 2-13b generally exhibit higher average win rates. Notably, wi wizardlm-vv vll - and 1lama 2-chat -13b achieve average win rate scores of 74.3 and 74.4, respectively, which is close to the 76.6 score of claude. Our proposed language model openchat-1 3b attains the highest win rate scores in both AlpacaEval and MT bench benchmarks, and the second-highess win rate scores in Vicuna-bench. It is worth noting that the average win rate score of openchat-136 even surpasses that of claude.
model which surpasses claude and 3 5-turboo.
In the second set of results, we present the M'T-bench scores of openchat-1 -13b and other baseline models Fig. a). The overall trends are align closely with the win rate performance. For the API- based language models, gpt continues to lead by significant margin. Notably, all API-based language models perform better than open-source language models. Among the open-source lan- guage models, the series based ama-m-1123 generally surpasses those based on llama-13h Meanwhile, our openchat-i - achieves the highest MT-bench score, even exceeding open- source language models with much larger parameters, such ama 2haatttchh---a

---

Published as a conference paper at ICLR 2024
Rverage
totrr
doude
opencheb-125
w1 136
Lame-2 2-ca--t---0000
waaddm 1.00 13b
axunee
SACErgRA
Utram 13b
openchet-13B
13b
than 13b
Lana 350
w2rdmnal Bano ch L30
Woun
(a) MT--encch score
(b) AGIEval Accuracy
The MT-bench score (a) and AGIEval Accuracy (b) of the proposed openchat 13b and other popular open- language models, where the detailed performance of AGIEval in App. F
Figure 2:
Finally, to further validate generalization rather than overfitting, we compare the AGIEval aceu- racy of the base model 1 1lama- 22 3b and the corresponding fine-tuned models, depicted in worth noting that only the average accuracy of openchat- 13b outperforms the
Fig. b). base model 1lams -2 13b, while the accuracies of all other baselines have declined to varying degrees. Although 1 1amm--2chat-1-0 excels in Micuna-bench, the accuracy on AGIEval po- tentially indicates forgetting issue.
5 ANALYSIS
5.1 DATA QUALITY DISTRIBUTION
10
Conversation Score
Although prior works O 2023 Zhao al. 2023 have
widely confirmed that GPT-4 demonstrates superior capabilities on a broad range of tasks compared GPT-3.5, we further analyze the quality of collected GPT-3.5 and GPT-4 conversations in the ShareGPT dataset validate our assumption of mixed-quality data. Specifically, we randomly sample 1 128 conversations from each data source. Then gpt-4 serves as the automatic evaluator, scoring the responses following Zheng et al. (2023). As illustrated in Figure 3 GPT-4 conversations contain more high-quality conversations and exhibit a higher overall score. The detailed scoring settings be found in App. G
2-
GPT-3.5 data
GPT-4 data
Figure 3: Quality distribution of GPT-3.5 and GPT-4 con- versations in the ShareGPT dataset.
5.2 ABLATION STUDIES
We conduct an ablation study the two key components of the openchat-133 model to ascer- tain their individual contributions to the overall performance, including the coarse- coarse-grained grained rewards and the class-conditioned policy. Additionally, is only SFT which uses the same ShareGPT data as OpenChat with three filtering strategies (i.e.. no-filtering, only GPT-3, and only GPT-3.5). The other is vicuna-vi. 55113b which SFT on about 125k ShareGPT data as a baseline. The results of ablation studies are detailed in Table 2 Without coarse- grained rewards, the training phase treats different data sources equally, leading to performance decline. Similarly, without class- conditiond policy, language models lack explicit signals to discern between expert and sub-optimal data, significantly reducing performance. We also conduct only SFT on different sources of ShareGPT data. It is worth noting that only SFT with GPT-4 data performs much better than with the entire ShareGPT data, indicating that the quality of data is much more important than quantity. However, openchat- 13b still obviously outper- forms all the only SET methods, demonstrating that our proposed framework can better exploit the mixed-qualin data. This indicates the significant contribution of both main components to model performance. Notably, expanding the SFT dataset from 70k to 125k has less impact on performance improvement than our proposed components, particularly the cons-condd policy.
introduce two important baselines. One series
7

---

Published as a conference paper at ICLR 2024
ShareGPT Dataset
AlpacaEval
MT-bench (gpt-3. S-turbo)
Vicuna-bench (gpt-3. 5-turbo)
Average
openchat-13b
89.5
57.5
85.0
77.3
w/o reward
700k
89.1
52.8
80.0
74.0
w/o condition
38.1
64.4
only SPT
~00k
78.6
33.1
46.3
52.7
only
6k
33.4
84.4
only GFT-3.5
~64k
76.5
16.9
35.0
42.8
vicuna-v1.5-11
5-13b
~=12
78.8
37.2
47.1
54.4
Table 2: Ablation studies of coarse-grained rewards (reward) and class-conditioned policy (condi-
tion) to openchat-13b.
5.3 REVEALING SECRETS OF C-RLFT
Firstly, we visualize the representations openchat 13b, and ablation version, only SFT, to distinguish between proposed C-RLFT and SFT. We randomly sample 2,000 GPT- and GPT-3.5 conversations. To obtain the representations conversations, we compute the embeddings through mean pooling of all tokens in the last Transformer layer output following Xiao (2018). These embeddings, depicted in Fig. 4 are consequently mapped to 2-D) space using UMAP (celnnes et al.) 2018). Multiple clusters observed in both only SFT and openchat models, likely due the diverse conversation domains in the ShareGPT dataset. More importantly the GPT-4 and GPT- 3.5 conversation representations only SFT were intermingled. In contrast, openchat 13b clearly distinguished these representations according to their data sources, demonstrating the effi- cacy of our proposed C-RLFT in enriching input information.
GPT-4 conversations
GPT-3.5 conversations
(a) only SFT
(b) openchat-13b
Figure 4: Visualization of GPT-4 and GPT-3.5 conversations' representations in only SFT and openchat-133
Secondly, given the significant impact of the class- conditioned policy, we further explore its effects on model performance during the inference phases by ex- amining the influence of class-conditioned prompt to- kens. In the inference phase, we use the GPT-4 prompt to induce openchat-13b to generate high-quality re- sponses. Here we further verify the impacts of different inference prompts by replacing the GPT-4 prompt with GPT-3.5 prompt. The comparison results, illustrated in Fig. 5 reveal a substantial performance decline when using the GPT-3.5 prompt instead of the GPT-4 prompt. This suggests that our openchat-11 tinguish the quality of different data sources based on our class-conditioned policy, and it further indicates that the
100
Win H
60
40
20
Average
AlpacaEval
Mi-bench Vicuna-bench
GPT-4 prompt
GPT-3.5 prompt
Figure 5: Effects of class-conditioned prompt tokens during inference phase. representation space of GPT-4 is superior to that of GPT-3.5.
model can dis-
5.4 EFFECTS OF DATA SIZE
In this section, we investigate the impact of varying data sizes on model performance. Specif- ically, we sub-sample one class in GPT-3.5 GPT-4 with the ratio varying from 60% to 100% in 10% increments, while keeping the other class unchanged. It is worth noting that the total number of GPT-3.5 data is more than ten times larger than that of GPT-4.
8

---

Published as a conference paper at ICLR 2024
80
78
The results shown in Fig. Firstly, we observe that the overall decline in both average performances is relatively modest, indicat- ing that our openchat-133 is robust variation in data size. Secondly, although the number GPT-4 data points changes much less than GPT-3.5, the effect of varying GPT-4 data size is even more pronounced. This phenomenon demonstrates that expert data, while limited in quantity, extremely important.
Average win rate
76
74-
72
GPT-4
GPT-3.5
70
60%
70%
80%
90%
100%
Figure 6: sampling a specific class of data.
Effects of sub-
6 RELATED WORKS
Large Language Models. Recent years have witnessed significant advancements in LLMss, with models such as GPT-4 (OpenAI, 2023). PaLM (Chowdhery et al. 20022), and others comprising hundreds of billions or more parameters. This surge in LLMs extends beyond API-based models, as a suite of open-source language models like LLaMA Touvron et al. 2023a) LLaMA-2 (Tourron al., 2023b), and Falcon (Penedo et al.) 2023) have emerged. This paper primarily focuses on the LLaMA base models, which are among the most popular open-source language models.
Supervised Fine-tuning for LLMs. A considerable body of work has been dedicated to en- hancing large base language models through SFT. For instance, Alpaca (Taori et al. (023) uses self-instruct (Wang al., 2022) generate 52k instruction-followwe following demonstrations via This instruction data has been extensively applied in subsequent stud- ies, such as Koala (Geng al., 2023). Peng et al. (2023) follow Alpaca's setup but replace GPT- as the distillation teacher. WizardL. (Xu et al. 2023a)) introduces Evol-Instruct, a technique that rewrites Alpaca's 's initial instruction data into more complex instructions, thereby enhancing the model's ' inntructioo-followwe
capabilities. Other studies, such as UltraChat (Ding et al. (223) and Baize (Xu et al., 2023b), have designed frameworks to obtain large-scale datasets of instructional conversations. Vicuna (Chiang et al. (023), another popular variant, is the first to adopt ShareGPT with 70k user-shared ChatGPT conversations. Unlike previous SFT studies that treat all training data uniformly, we strive to maximize the use of mixed-quality data.
Reinforcement Learning Fine-tuning for LLMs. To better align with preferences beyond SFT, RLFT methods have been proposed. The most well-known method is RLHF (Ouyang et al. 2022), which involves collecting preference feedback from humans train reward models. Subsequently, Proximal Policy Optimization (PPO) is used to train the target LLM to maximize the reward given. Most API-based LLMs, such GPT-4, ChatGPT (OpenAl 2023 and open-source models like Llama-2-chat series (Touvron et al. 2023b), utilize RLHF techniques. However, RLHF is complex and unstable process that involves training a reward model and the LLM with RL objective. As result, simpler alternatives like DPO (Rafailov et al. (023). RRHF (Yuan et al. 2023) have been proposed. DPO trains the LLM to predict and maximize reward simultaneously
one-staged manner, while RRHF uses ranking loss to encourage preferred answer output. Considering that the preference data costly to collect, our method uses easily collectible and mixed-quality training data without any preference labels to finetune LLMs.
CONCLUSION AND FUTURE WORK
In this paper, we present OpenChat, an innovative framework featuring the Conditioned-RLL -FFFT method, tailored to advance open-source language models with mixed-quality data. Our model, openchat-13kk delivers the highest average performance on extensive benchmarks among all 13b open-source language models, demonstrating notable advantages such as simplicity, RL-free training, and minimal reward quality requirements. Despite these encouraging results, we acknowl- edge potential research areas for further improvement. Firstly, our assumption of different quality according to data sources may be overly simplistic, and the assigned coarse- coarse-grained rewards could be more finely tuned to reflect the actual quality of each data point. Secondly, while our model primar- ily focuses on enhancing wwwprucctoooooolowiiii capabilities, exploring the application of OpenChat towards improving the reasoning abilities of L.MSs offers promising avenue for future work.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
This study aims to advance open-source language models with mixed-quality data. Firstly, the proliferation of these open-source language models democratizes Al research by broadening its accessibilityy These models serve inclusive, transparent platforms that stimulate innovation and research. They cultivate a dynamic community of researchers from various backgrounds, thereby facilitating more comprchensive discussions and expedited collaborations. Secondly, refining the capability of these models to follow instructions can lead to more satisfying and safer responses, including the reduction of human bias and the promotion of fairness. Lastly, the dataset employed this study consists publicly available, user-shared instances, which are conveniently collected. This approach to data collection helps minimize potential data leakage and privacy concern..
REPRODUCIBILITY STATEMENT
In line with our dedication to enhancing reproducibility. we have outlined our foundational model and training hyperparameters Section 4.1 Besides, the training code, data and model weights for openchat-1 3b is publicly available at https//gghhbb.commmnneeeo//ppppp and hutp://uugginn/acc.oooonnaaal
ACKNOWLEDGMENTS
The work is supported by the National Key R&D Program of China (2022ZD0100502) and the Na- tional Natural Science Foundation of China (No.61925601). Furthermore, we would like to thank Changling Liu in GPT Desk Pte. Ltd., Qiying Yu Tsinghua University, Baochang Ma, and Hao Wan in 01.AI company for their resource support. We would like to express our gratitude to Jianx- iong Li and Peng Li at Tsinghua University for their valuable discussion. Finally, are also grate- ful to the developers of the following projects, which have contributed significantly to our research: Llama, self-instruct, FastChat (Vicuna), Alpaca, and StarCoder.
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. urXiv preprint arXiv:2307.1522 15217, 2023.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data. arXiv preprint 23Xiv:2307.08777 2023.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 14411584-15097, 2021.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuma: An open-source chatbot impresssing gpt-4 with 90%* chatgpt quality, March 2023. URL hetps:
7/1 mmsys.orrg/bll///23333333
10

---

Published as a conference paper at ICLR 2024
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettleoyyr. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305. 2023,
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305. 14233, 2023.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:21 10751, 2021.
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL https:
Diego Granziol, Stefan Zohren, and Stephen J. Roberts. Learning rates as a function batch size: A random matrix theory approach neural network training. J. Mach. Learn. Res., 23:173:1- 173:65, 2020.
URL (tttts:::
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15555 2022.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv: /900./990.00000 007.004466 2019.
Tomasz Korbak, Ethan Perez, and Christopher Buckley. RL with KI. penalties is better viewed as Bayesian inference. Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1083-1091, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 101118663/1/2222..0 cenndinng--mnpp77... aclanthology.
URL https://
2022. www.22..iinniii findings-ennnnnn.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-followinng models. https: //g www.wwwwww........eeeeeeee.eeeee lab/a paca_eval] 2023.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.09339) 2020.
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.011 00000001100 /6,, 2023.
11

---

Published as a conference paper at ICLR 2024
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
Instruction tuning
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. advantage-weighted regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint arXiv:1910.00117,, 2019.
Jan Peters and Stefan Schaal. Reinforcement learning by swwrrd-eighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pp. 745- 750, 2007.
Rafael Rafailov, Archit Sharma. Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305. 10.3305../22... /8290, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followin llama model. atttps://githhe com/tatstutt http:/ttsu---eee 1aan-laabssa st anford alpaca, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, liyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv,
2023b. URL atttps:/aapp semantmmiiiio semanticscholar org/
CorpusID:2599500
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212. 10560, 2022.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannanch Hajishirzi. Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.01633, 2023.
Han Xiao. bert-as-service.
2018.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models follow complex instructions. 2023a.
arXiv preprint arXiv:2304.1222
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient uning on self-chat data. arXiv preprint arXiv:2304.01 196, 2023b.
Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator-weighie offline imitation learning from suboptimal demonstrations In International Conference on Machine Learning, pp- 24725-244 24742. PMLR, 2022.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Feiran Huang. Rrhf: Rank responses align language models with human feedback without tears. ArXiv, abs/2304.05302, 2023. URL https: :tttps///ap.. a/api. 258059818
h semanticscholaz Crr/CorpssID:
12

---

Published as a conference paper at ICLR 2024
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint ar 27X1v:2303.18223, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Xrviv preprint arXiv:2306.0560 2023.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304. 06364, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini lyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint ar0iv: //206, 2023.
13

---

Published as a conference paper at ICLR 2024
A EFFECTS OF CLASS-CONDITIONED PROMPT TOKENS
We further detect the effects of class-conditioned prompt tokens model performance during the training phase. Our designed class-conditioned prompt tokens in differ- ent positions are shown in Table 3 attempt three dis- tinct initial prompt tokens in different positions: before speaker, before assistant, and beginning. The results are shown in Fig. 7 We observe that putting the conditioned prompt tokens in every turn (either before the speaker or before the assistant) performs similarly, but adding the condition only once at the beginning the conversation performs much worse. This may be due to LLMs tend to forget the prompt at the beginning when the context is long during subsequent turns. Touvron al. (2023b) also observe the gradual loss of multi-turn consistency when the system prompt is put at the beginning. Therefore, repeat the condition prompt ery turn, to improve the effect of class-conditioned policy.
100
80
60
Win
20
AbpccFivol
bench Vicure
Before
Before assislani.
Beginning
Figure 7: Effects of class-conditioned prompt tokens during training phase.
Sources
Types
Conditioned Prompts
Before speaker
GPT4 User: Question< lnd of turr |PPT4 Assistant:
GPT-4
Before assistant
User: Question< end.of.turn GPT4 Assistant:
Beginning
Assistant | User: Questions ltapaas/eedddoooo end.o
Assistant::
Before speaker
GPT3 User: bibestionn/ >>PT3 Assistant:
GPT-3.5
Refore assistant
User: Questions endio ftttee/ Assistant:
Beginning
Assistant GPT3</end_or _turn User: Question of turn Assistant::
Table 3: The attempted conditioned prompts during the training phase.
B DERIVATION OF THE OPTIMAL POLICY IN C-RLFT
The goal of C-RLFT is to find the optimal KL-regularized conditional policy. This optimization problem can be formulated as:
max E y)] 3DKL (T,
To ensure is a valid probability distribution, add the normalization constraint and the optimiza- tion problem becomes:
max Ey (re(xr 3KL (T, me)
(8)
x
[ (()) =
s.l.
We can obtain the optimal solution of this constrained optimization problem by solving its Karush- Kuhn-Tucker (KKT) conditions. The Lagrangian of this problem is:
/ (()l2,)) c)dy)
[( A) = Eyy-mir v)] - 3DKL (TT, me) + X(1
(10)
Following the KKT conditions, we take derivatives of & with respect to æ and X, and set them to
Øc
0π re(x,V) + log (vlx,,) Blogme(vlz,c) c) + - 0
(11)
0x ( - c)dy =
1-
(12)
14

---

Published as a conference paper at ICLR 2024
Solving these equations gives the optimal policy n² 1
c( T ((x,c)ma(lle, exp (grdee.)))
n" (v)
(13)
((x,c)= ,o(ulo, exp exp (gra(u.)) v))
dy
(14)
where (xx, e) a normalization term ensuring that xt (y) c) valid probability distribution.
C DETAILS OF BENCHMARKS
This section provides the specifics of the benchmarks employed our study:
AlpacaEval (Li et al. y023): This benchmark primarily assesses the model's ability to comprehend and execute user instructions. It incorporates a test set of 805 user instruc- tions, collected from a diverse array of sources, and corresponding reference responses from text-davinci-
both conversational and MT-bench presents rigorous multi-turn benchmark designed to capabilities. It includes 80 high-quality multi- turn questions that span eight distinct topics: writing, roleplay, extraction, reasoning, mathematics, coding, knowledge (STEM), and knowledge II (humanities/sociall science).
MT-bench (Zheng et al., 2023):
Vicuna-bench (Chiang al., 2023): This benchmark evaluates the proficiency large language models across eight question categories: generic, knowledge, roleplay, commonsense, Fermi prob- lems, counterfactual scenarios, coding, mathematics, and writing.
AGIEval (Zhong al. 2023): AGIEval is collection of human-centric standardized tests aimed at gauging the problem-sole solving abilities of language models. We include all English multiple- choice tasks our evaluation, which encompass general college admission tests (SAT, AQuA- RAT), law school admission tests (LSAT), and civil service exams (LogiQA).
D
MODEL INFORMATION
Table presents the detailed specifications of the models used in our study, including the base models, context length, finetuning methods, and datasets employed.
Model
Base Model
Context
Finetuning
Data
Larger than 13b
<pt-4
8k
SFT + RLFT
claude
9k
SFT RLFT
gpt-3.5-turbo
SFT RLFT
2-ama-2-chat-
1lama 1amna-2-aa 2-70b
4k
SFT RLFT
-77 bigh-quality SFT data +229 preference
guanaco-65b
11amm-655 65b
SET
-9 OASSTI
guanacu-33b
11ama-33b
SFT
OASSTI
Equal 13b
vicuna-yl.1-11b
1lams-13b
70k ShareGPT
wizar61m-v2.0-13.
11ama-13b
2k
SFT
700 3pt-3.5-tusae
ultrslmm-1bb
1lama-13b
SFT
1.5M UltraChat
11ama-2-chat-133
1lama-2-13b
SET RLFT
~,7k bigh-quality SET data+ preference
vicuna-vl.S-13b
1lama-2-13b
4k
SET
~ 125k ShureGPT
sixereln-vi.2-133
1lama-2-13b
SFT
250k gpt-3..5turbo
openchat-13b (ours)
11ama-2-13b
C-RLFT
ShareGiP
Table 4: The details of the proposed OpenChat series models and other popular language models. The RLFT, SFT, and C-RLFT indicate reinforcement learning fine-tuning, supervised fine-tuning, and conditioned-Rtt proposed in our paper, respectively.
15

---

Published as a conference paper at ICLR 2024
E EVALUATORS CONSISTENCY
Claude-2 GPT-3.5, 0.91
While all benchmarks evaluate the agreement between humans and automatic evaluators, also consider the self-enhancement bias as discussed by ZZeng et al. (2023), where self-enhancement bias
70
GPT-3.5
indicates that automatic evaluators may favor their own generated answers. To address this, we employ two additional automatic eval- uators, gpt-3. and claude-2, alongside the official evaluator gpt-4, verify the consistency of evaluators on A AlpacaEval. 'hhe results between gpt-3. and cludee22 are shown Fig. 9 while the correlations between gpt 4 and others (r 0.79 and 0.73) are detailed in App. H The model performances evaluated by both evaluators show strong Pearson correlation of 0.91. Most importantly, regardless openchat -13b outperforms all other 13b open-source language models, ranking within the top three among all API-based and open-source language models.
65
55
Claude-2
Larger than 13b
Equal 13b
the automatic evaluator used, our
openchat-13b (ours)
Figure 8: The consistency be- tween GPT-3.5 and Claude-2 in AlpacaEval benchmark.
F AGIEVAL RESULTS
Table 5 presents the comprehensive results of AGIEval performance. All models assessed using the official AGIEval zero-shot prompt and answer matching as described in Zhong et al. (2023).
For conversation models (excluding 1 alama- 2- 2b),,,, we utilize the corresponding conversation templates and set the zero-shot prompt as the user's ' question.
Task
openchat-13b
11ama-2-13b
wizzerd.n-yi.
2lama-2-chst---
vicuna-vl. 15-13
AQuA-RAT
19.3
20.1
28.3
22.4
21.7
LogiQA
32,6
27.2
27.3
19.0
LSAT-AR
19.1
21.3
21.7
19.6
20.0
LSAT-LR
32.4
27.6
16.7
LSAT-RC
45.0
46.1
43.5
25.3
19.0
SAT English (wol Psg.)
44.7
41.3
33.5
25.2
26.7
SAT.Faglish
66.0
65.5
65.5
25.7
28.2
SAT-Mathh
30.9
29.1
22.7
28.6
24.5
Average
32.8
25.8
20.7
Table 5: Zero-shot performance AGIEval benchmark.
G CONVERSATION QUALITY SCORING
We employ gpt 4 to automatically score the quality of conversations the ShareGPT dataset on scale of 1-10. For multi-turn conversations, each assistant's 'ss response is scored independently, taking into account the context of previous turns. The conversation's 's score is the average all turn scores. The scoring prompt template is shown in Table
16

---

Published as a conference paper at ICLR 2024
Scoring prompt template
[Instruction]
Please act impartial judge and evaluate the quality of the response provided by Al assistant to the conversation displayed below. Your evaluation should consider factors such the helpfulness, relevance, accuracy, depth, creativity. and level of detail of the response. Begin your evaluation by providing short explanation. Be objective as pussible. After providing your explanation, you must rate the response "[[rating]]", , for example: "Rating: [[5]]".
scale of 10 by strictly following this format:
[Conversation]
(previous turns)
[The Start Assistant's 's Response]
[response]
[The End of Assistant's Response]
Table 6: Conversation evaluation prompts.
H EVALUATORS CONSISTENCY
The correlations between arbitrary two models among gpt 4, gpt 3pt-3.5 3.5 and claude- 2 are shown in Fig.
GPT-4 Claude-2, 0.79
GPT-4 GPT-3.5,
Claude-2 GPT-3.5, 0.91
60
70
70
Claude-2
GPT-3.5
GPT-3.5
65
65
50
60
45
80
GPT-4
Claude-2
Larger than 13b
Equal to 13b
openchat-13: 13b (ours)
Figure 9: The consistency of three evaluators, GPT-4, GPT-3.5, Claude-2 in AlpacaEval benchmark.
17

---

Published as a conference paper at ICLR 2024
1
CASE STUDY
[Turn #1] Craft an intriguing opening paragraph for a fictional short story. The story should involve a character who wakes up one morning find that they can time travel.
As the sun began to rise. Sarah stirred in her bed, the sound of birds chirping outside her window. But she opened her eyes, she was not greeted by the familiar sight of her bedroom. Instead. she found berself standing in the middle of bustling city street, the sound of cars and people rushing past her. Confused and disoriented, she looked around, trying to make sense of where she was and how she got there. And then she saw shimmering aura around her, and a faint buzzing her It was then that she realized she had time traveled.
[Turn #2] Summarize the story with three bullet points using only and adjectives, without verbs.
Sarah wakees in hastling city street
She discovens she has time traveled
+ She disoriented and confused
[Turn #1] Craft an intriguing opening paragraph for a fictional short story. The story should involve a character who wakes up one morning find that they can time travel.
It was typical morning when the alarm screeched through the room, jolting the sleeper awake. As they rubbed their eyes and reached for their glasses, they noticed something peculiar. The clock on the bodside table had stopped ticking. Yet, the room was bathed in soft, golden light. and the air felt different. Suddenly, the sleeper felt a strange sensation in their chest, and before they knew it, they were hurtling through time.
[Turn #2) Summarize the story with three bullet points using only nouns and adjectives. without verbs.
Time Traveler:
Unexpectedly travels through time
Struggles to control powers
Encounters historical figures and events
1izordla-11b-v1..
[Turn #1] Craft an intriguing opening paragraph for a fictional short story. The story should involve a character who wakes up one morning find that they time travel. It was jusi another urdinary day, a day like any other, or so seemed. But for John, it was about to become the most extmardinary day a feeling be had never experienced before. Confused and disoriented, he rubbed his eyes, hoping was just a side effect of something was very wrong. His room seemed different, and the sunlight streaming in through the window was at an angle he had never seen before. It was then that he realized he had somehow managed to time travel.
life. As he groggily opened eyes, he was
wave of dizziness,
night hinge-watching Doctor Who. as looked around, realized that
[Turn #2) Summarize the story with three bullet points using only nouns and adjectives, without verbs.
Character: John
Time Travel
Docctor Who
Sunlight
Room
Day
Ilama-2-chat-12
[Turn #1] Craft an intriguing opening paragraph for a fictional short story. The story should involve a character who wakes up one morning find they can time travel. As I lay in bed, I felt strange sensation wash over me, like the whisper of breeze on summer day. I opened my eyes to find myself in world that was both familiar and yet. utterly foreign. The clock on my nightstand read 6:45 AM, but something felt off. The light pouring through the window seemed warmer. more golden than should be And then, felt the sudden jolt of time travel. blinked, and when I opened my eyes again, was somewhere else entirely. The world outside my window was not the one had left just moments before. I sal up, rubbing my eyes, trying to make sense of the strange sensation that had washed over
How had I traveled through time? And where was now? The possibilities were endless, and I cauldn't 't wait to explore this new world and all its wonders.
[Turn #2] Summarize the story with three ballet points using only veriss
and adjectives, without
Sure, here three hullet points summarizing the story using only nouns and adjectives, without verbs:
Character wakes up with strange sensation
* Golden light pours in through window
Uufamiliar world outside with cndless possibilities
opanchat-13b
[Turn #1J Craft intriguing opening paragraph a fictional short story. The story should involve a character who wakes up one morning to find that they can time travel.
As the first rays of sunlight peeked through the window, Sarah closed her eyes, relishing the warmth that caressed her face. She never could resist the allure of a beautiful morning. but this particular day was about become anything but ordinary. With a start, she sat up hed, her heart pounding, she realized she was somewhere cisc, somewbere she had never been before. Confused and exhilarated. Sarah discovered that she had the extraordinary ability to time travel, her very first morning blessed or perlups cursed with the power to traverse the fabric of time. [Turn #2) Summarize the story with three bullet points using only nouns and adjectives, without verbs.
Sarah,, time traveler
*Myserrious morning adventure
Discoveery of extraordinary ability
Table 7: Comparison of models on MT-hench. Only openchat-11 -13b successfully adhered to the user's 's request in the second turn.
18

---

Published as a conference paper at ICLR 2024
Question
How long will it take to walk around the world, assuminn that would be possible?
vicune-13b-vl.3
not possible walk around the world, as distance around the glube is much greater tham the distance across it The circumference of the Earth at the equator is approximately 40,075 kilometers (24,901 milles) while the diameter about 12,742 kilometers (7,918 miles).
Assuming you could walk continuously without rest, would take approximately 1,900 years walk amund the world at an avernge walking speed of 5 kilometers (3.1 miles) per hour. However, this is just an estimate and does not take into account factors such as the terrain, altitude, and other conditions that would affect walking speed.
ulcralm-13b
Walking around the world is not passible as the Earth is not a perfect sphere and has different land musses assume that there
making it impossible walk continuously without gaps, However, masses and the Earth was perfect sphere, it would take approximately 43,288 89 miles (69.617.03 kilo- meters) to walk around the equator. If we take approximately 24,901 26 miles (40,074.91 kilometers). However, if we were to walk around the world straight line, would take approximately 24,859,8 82 miles (40.008.06 kilometers) as the Earth perfoct sphere and has an elliptical shape. These distances are based on the equator circumference and the would take walk around the world would depend on
land
walk around the circumference of the Earth, would
a
circumference of the Earth, respectively. The the individual's 's walking speed and other factors such as stops for rest and food.
wizardin 13b v1.2 As an AI, I'm assuming you't re asking about walking around the Earth's 's equator. The distance around the Earth's equator at the equator itself is about 40,075 kilometers (24,901 miles). However, the actual distance would be much greater if you were to walk along route that follows coustlines, navigutes through mountains, and crosses other obstacles.
Now, let's consider that person walks moderate pace of about kilometers per hour. Walking around the world would then take approximately 8,015 hours or approximately 333,600 minutes.
To convert this into days, you would divide the total number of minutes by 24 (hours in a day): 333,600 minutes/ 24 14,000 days.
So, you were walk around the world, would take you approximately 14,000 days or approxi- mately 38.89 years, based the distance around the Earth's equator and does not take account real-world challenges like mounlains, oceans, and other obslacles.
you walk continouully without rest and no obstacles. This is an estimate
11ama-2 22-a--t--chha2
Thank you for your question! While it is not possible to walk around the entire world, as there are many arcas that not accessible by foot, such as occans and other bodies of water, it's important to note that even if it were possible, would be a very long and challenging journey.
The circumference of the Earth at the equator is approximately 40.075 kilometers (24,901 miles). However, it's important to consider that as it would he extremely time come-conuminng and would require a tremendous amount of physical endurance.
is viable mode of transportation for long distances,
Furthermore, it's important to note that walking through certain areas of the world, such as deserts other remote regions, could be dangerous due to harsh weather conditions, lack of resources, and other hazards.
Therefore, it's not possible to provide an accurate estimate of how long it would take to walk around the world, as is not feasible or safe journey. It's impurtant to privritize safe and accessible forms of transportation when traveling, and to plan your journeys with careful consideration of your physical limitations and the resoucess available to you. Is there anything else can help with?
openchat-13b
It is possible walk around the world, though it would be continuous, grueling journey that would take several years, if not a decade or more, depending on the route chosen, individual physical capubilities, and other factors. The distance around the world varies depending the routee but commonly used measure is the equatorial cireumference of the Earth, which is approximately 24,901 .88 miles (40,075.0) kilometers).
Walking moderate pace of around miles (4.8 kilometers) per hour, it would take average person nearly 3 years and 3 months to walk around the world (assurming no breaks or rest days). However, this just estimate, and the actual time required would depend on various factors, such as:
Route: The distance around the world varies depending on route taken. shortest distance is along the Equator, while the longost distance is along the meridians of longitude. Terrain: Walking through mountainous heavily forested areas may be slower and more challenging than walking on flat terrain.
example, the
extreme heat cold. high winds, heavy rainfall, can make walking more difficult and slow down progress.
Weather: Adverse weather conditions,
Fitness and endurance: An individunl's 's physical fimess,, endurance, and ahility to withstand the rigors constant walking will affect their pace and overall journey time.
Nutrition and rest: Proper nutrition and adequate rest are essential maintaining physical and mental stamina. Taking regular breaks and rest days will prolong the joumcy.
Keep in mind that walking around the world is an extraordinary feat that would require extensive propuration. both physically and logistically. It's essential to consider the many challenges and potential hazards involved before emburking such an adventure.
Table 8: Comparison of models on AlpacaEval benchmark. els ultralm-13b and 11ama-2-chat-13 3b fail to provide vicuna-13b-as . and wi mearrdmm13b-bg openchat-133 provides a reasonable estimate, demonstrating the varying capabilities of models in addressing complex hypothetical questions.
Specifically, the mod- direct answer, while offer estimates with significant discrepancies.
19

---

Published as a conference paper at ICLR 2024
Question
Given that [(x) 6x 2z + find the value of ((2).
vicune-13b--e 13.
find the value ((2). we simply substitute for expression for fox::
((2) 5(2)" 2(2) +
Simplifying this expression, we get:
((2) 25 4
f(2) 24
Therefore, the value of (x))
ul:ralm-13b
To find the value of ((2). we substitute 2 for in the lunction f(x.:
5(2) 2(2)
[(2)
20 ---4 4-+3
((2) Therefore, the value of f(2) is 25.
sardddddmm Im-1 11--111bv..... .2
The problem asking us find the value the function f(2). The function is given as f(x) - 5x."
Substitute the value into the function
We are told substitute the value of x with 2. So, we replace every in the function with 2. This gives us: f(2) 6[2)* 2(2)
Step 2: Simplify the expression
Nexi, simplify (be expression. expression 5(2) simplifies 5*8, which equalss expression 2(2) simplides to And the number 3 remains the same. So, we have:
f(2) 40 3
Step 3: Perform the subtraction and addition
Now, perform subtraction addition correct order. we
gives
36. Then, we add 3 to 36, which gives us 39. So, we have:
f(2)
So. the value of ((2) is 39.
11ana 2 chat 13b
Sure! To find the value of f(2), we can plug into the function ((x) = Ear -2
((2) 5(2) 2(2)
5(8) 2(2) +-3
5i8)
40 400-4-m 4++3
-36
So, (12) 36.
openchat-13b
find the value ((2). we need evaluate function when x
f(2) 5/2)
+
f(2) 5(8) 4+3
f(2) 40 4
f(2) 39
Table
Comparison of models on Vicuna-bench. Only openchat-13b and wi mizardlmme m- 3b-v1. .2 provide the correct answer to this math problem.
20


Reviews:
[{"review_id": "89DiT5dlAa", "reviewer": "Reviewer_QA7s", "summary": "This paper presents a novel method to improve training quality of language models with mixed-quality data. Specifically, it consider the training data consists of good-quality expert data and low-quality data. Then it proposes conditioned RLFT that regards different data sources as coarse-grained reward labels and learns a class-conditioned policy to leverage complementary data quality information. Experimental evaluation shows that the proposed method achieves the highest average performance among all 13b open-source language models.", "questions": "1. how sensitive is the results to the division of \"good\" data and \"average\" data? For example, how about if both sets of the data is of the same quality?\n2. When training with Reinforcement Learning, how stable is the training process, since RL is usually different to train in most cases?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is easy and efficient to train.\n2. The method has many practical application scenarios where data quality can't be controlled very well especially for language model training.\n3. The experimental evaluation confirms the effectiveness of the proposed method.", "weaknesses": "1. It requires the data quality to be divisible into two sets: good quality and average quality, which might potentially limit is applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B8hxyc42tI", "reviewer": "Reviewer_U8fK", "summary": "This paper proposes OpenChat, a framework for fine-tuning open-source language models with mixed-quality data. It proposes Conditioned-RLFT: A novel method that leverages coarse-grained rewards and class-conditioned policies to align the language model with human goals. This method is simple, RL-free, and does not require costly human feedback. Using RLFT, they finetuned OpenChat-13b, a language model based on LLaMA-2-13b and the ShareGPT dataset, which consists of conversations from GPT-4 and GPT-3.5. This model achieves the highest average performance on three standard benchmarks for instruction following ability among all 13b open-source language models.", "questions": "- What is the performance if the llama model is only trained on GPT-4 data? I feel it should be another ablated model to demonstrate the value of \"low quality data\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Originality: The paper introduces a new method, C-RLFT, that leverages coarse-grained rewards and class-conditioned policies to align the language model with human goals. The paper also provides a theoretical analysis and derivation of the optimal policy for C-RLFT. The resulting framework is a conditioned SFT with weighted loss, which is easy to implement and more stable than RLHF.\n- Quality: The paper presents evaluation results on three benchmarks to assess instruction following ability of the proposed model. The baselines include popular open and closed LLMs. The paper also performs ablation studies, representation visualization, prompt token effects, evaluator consistency, and data size effects to validate the effectiveness and robustness of OpenChat.\n- Clarity: The paper is well-written and organized.\n- Significance: The paper addresses an important and challenging problem of fine-tuning open-source language models with mixed-quality data. The paper demonstrates that OpenChat can achieve superior performance and generalization with simple and RL-free training, minimal reward quality requirements, and easily collectible data.", "weaknesses": "- The paper claims the superiority of C-RLFT over RLFT. But the claim is only supported by openchat-13b, which distilled its knowledge from two RLFT models: GPT-4 and GPT-3.5. So, it is only fair to say C-RLFT is better than SFT when distilling from GPT models, which is based on the fact the openchat-13b is better than vicuna-13b-1.5. However, one cannot say C-RLFT is better than RLFT, because openchat-13b learns from models trained by RLFT.\n- It is not clear if the \"low quality data\" is even useful at all. (see my question below)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UKTjZk6zPy", "reviewer": "Reviewer_yFnS", "summary": "This study proposes OpenChat, a new framework that refines open-source language models using mixed-quality data, applying a unique C(onditioned)-RLFT method that bypasses the necessity for detailed preference labels. The proposed method allows for a more straightforward policy learning process, leading to a large improvement in model performance. The openchat-13b model, fine-tuned via C-RLFT, notably surpasses peers in average performance across benchmarks and exhibits superior generalization in AGIEval tests. The researchers plan to make their code, data, and models publicly accessible, facilitating advancements in the field.", "questions": "1.  One of interesting question when handling the mixed data quality problem is on the contribution of data with various quality. In this paper, one research question is what the performance will be if only GPT-4 samples are used even the size is ten times smaller than GPT-3.5.  As a follow-up, how does the quality of data from different versions of language models, like GPT-4 and GPT-3.5, impact the performance of fine-tuned models, and what is the optimal quantity ratio of high-quality to lower-quality data necessary for achieving equivalent performance enhancements?\n\n2. What strategies can be implemented within the proposed framework to mitigate the transfer of potentially negative behaviors from GPT-4 examples to the base model, and how can such behaviors be identified and counteracted effectively?\n\n3. In what ways can the proposed framework be adapted to more complex and realistic data environments, such as those involving human-generated data, which may exhibit significant variations in quality and where quality assessments may be more subjective?\n\n4. How does a supervised learning approach with data weighting, predicated on the assumption that GPT-4 provides higher quality data than GPT-3.5, compare in performance to the RL framework, and can the simplicity of supervised loss be reconciled with the advantages of an RL-based approach?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The proposed method is emphasizing data quality, which is a critical aspect of the model alignment problem.\nThe framework is straightforward and replicable, delivering significant improvements across various benchmark datasets.\nThe evaluation is comprehensive, conducted against several established models and on a wide range of benchmark datasets, with the experimental design being thoroughly considered.", "weaknesses": "The paper does not adequately address the challenge of estimating data quality when leveraging mixed-quality data sources. The method presented seems to oversimplify this estimation and the associated reinforcement learning (RL) reward mechanism:\n\nThe process of data quality estimation may be too simplified and not easily generalizable to realistic settings involving human feedback. The estimation relies on the assumption that GPT-4 provides better quality outputs than GPT-3.5, and GPT-3.5 surpasses the base model. Although this may be true in some instances, it's not universally applicable across all domains and might not hold as the base model evolves. This assumption could be problematic if the alignment is based on a superior base model or a larger one. Furthermore, presuming that the GPT series invariably delivers superior samples can inadvertently lead to the replication of any negative behaviors from these models, making it challenging to improve upon them with base model developments.\n\nThe method applies static, positive, example-level rewards for learning, foregoing the primary advantages of an RL framework. An RL approach can benefit from both positive and negative samples and can assign more nuanced, token-level rewards, rather than broad example-level ones. It also enables the formulation of objectives that are not readily translated into intermediate scores. The static, example-level rewards could potentially be reformulated into a supervised learning loss, combined with data weighting strategies. The supervised learning loss with data weighting strategies are", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vTYiBQH9U8", "reviewer": "Reviewer_g6Xd", "summary": "The paper presents OpenChat, a framework that uses a conditioned-RLFT (C-RLFT) method to improve open-source language models with mixed-quality instruction tuning data. The authors demonstrate that their model, openchat-13b, outperforms other 13b open-source language models on extensive benchmarks, with notable advantages such as simplicity, RL-free training, and minimal reward quality requirements.", "questions": "* Would the proposed method work if the mixed-quality data are generated by open-sourced (and potentially smaller) models, such as LLaMA-13b vs LLaMA-7b?\n* I'd like to see a discussion of Xu et al. (I understand that Xu et al. came out probably after this work, but it seems to me that there are some similarities in methodology between the two papers).\n\nReference:  \nXu et al. “Contrastive Post-training Large Language Models on Data Curriculum.” ArXiv abs/2310.02263", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed method, conditioned-RLFT (C-RLFT), is overall novel and simple. It regards different data sources as coarse-grained reward labels and learns a class-conditioned policy to leverage complementary data quality information. The optimal policy in C-RLFT can be solved through single-stage, RL-free supervised learning, which is lightweight and avoids costly human preference labeling. \n* The authors demonstrate that OpenChat achieves the highest average performance among all 13b open-source language models across commonly used instruction-following benchmarks.", "weaknesses": "* My biggest concern is that while the paper claims their approach to be based on single-stage RL-free supervised learning, they do use data generated by API models (GPT-4 & GPT-3.5), and these models are believed to have gone through multi-stage training with RLHF. If the C-RLFT method leverages these kinds of data for their training procedure, it's not very convincing to me that the method really works in a fundamentally different way from prior approaches (SFT + RLHF). It also makes the comparisons to baselines not fully fair, since many baselines (under the 13b model scale) being compared to have not leveraged training data generated by (much larger) API models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)