Tuna: Instruction Tuning using Feedback from Large Language Models
Authors: Haoran Li
Yiran Liu
Xingxing Zhang
Wei Lu
Furu Wei
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Lanugage Models; Instruction Tuning

Abstract: Instruction tuning of open-source large language models (LLMs) like LLaMA, using direct outputs from more powerful LLMs such as Instruct-GPT and GPT-4, has proven to be a cost-effective way to align model behaviors with human preferences.
However, the instruction-tuned model has only seen one response per instruction, lacking the knowledge of potentially better responses.
In this paper, we propose finetuning an instruction-tuned LLM using our novel probabilistic ranking and contextual ranking approaches to increase the likelihood of generating better responses.
Probabilistic ranking enables the instruction-tuned model to inherit the relative rankings of high-quality and low-quality responses from the teacher LLM.
On the other hand, learning with contextual ranking allows the model to refine its own response distribution using the contextual understanding ability of stronger LLMs.
Furthermore, we apply probabilistic ranking and contextual ranking sequentially to the instruction-tuned LLM.
The resulting model, which we call Tuna, consistently improves the performance on Super Natural Instructions (119 test tasks), LMentry (25 test tasks), Vicuna QA, and can even obtain better results than several strong reinforcement learning baselines. Our code and data are available at https://github.com/microsoft/LMOps.

Tuna: Instruction Tuning using Feedback from Large Language Models
Haoran Li Yiran Liu Xingxing Zhang² Wei Lu Furu Wei
StatNLP Research Group, Singapore University of Technology and Design Microsoof Research Asia, Tsinghua University
haoran2_ @mymai sutd. sut..edu.dd.............u. edu. sg, rei_lu@sutd. edu. sg liu-yr22@maill tsinghua. edu. cn, (xizhang, fuwei } mom
Abstract
Probabilistic Ranking
Rerponses
Instruction tuning of open-source large lan- guage models (LLMs) like LLaMA, using di- rect outputs from more powerful LLMS such as Instruct-GPT and GPT-4, has proven to be cost-effective way align model behav- iors with human preferences. However, the instruction-tume model has only seen one re- sponse per instruction, lacking the knowledge of potentially better responses. In this pa- per, we propose finetuning an instruction tuned LLM using novel probabilistic ranking and contextual ranking approaches to increase the likelihood of generating better responses. Probabilistic ranking enables the instruction- tuned model inherit the relative rankings of high-quality and low-quality responses from the teacher LLM. On the other hand, learn- ing with contextual ranking allows the model refine its response distribution using the contextual understanding ability of stronger LLMs. Furthermore, we apply probabilistic ranking and contextual ranking sequentially to the cnorucctio-tunnd LLM. The resulting model, which call Tuna, consistently im- proves the performance on Super Natural In- structions (119 test tasks), LMentry (25 test tasks), Vieuna QA, and results than several strong reinforcement learn- ing baselines. Our code and data are available httpp::/ggiee whw.b/gttttttt.... com/ mi wrossootLMMsss.
f Ri,
Learn the Ranking
F7 R2,
Instrution
Texc- Dar--D
0 Rd,
-07
011
Contextual Ranking
Requaraen
A
Instruction
R3
GPT-4
RioREaR2
Tured LiMe
Contestiol Prompi
the Ranking
Probubilittic Ranking
Contentual Ranking
Instruction- Tuned LLMs
000
000
Tuna
The finetuning process using probabilistic ranking (top), contextual ranking (middle), and com- bination of both (bottom).
Figure 1:
2022) with Inst-unstttct algorithm (Wang et al., 2022a) has emerged cost-effective method for aligning with human preferences. In this approach, open LLMs like LLaMA (Touvron al., 2023) can be finetuned instruction-followwe data gen- erated by OpenAl GPT using the Self-Instruct al- gorithm. The Alpaca model (Taori et al., 2023) exemplifies this technique, which enables close alignment with human preferences while reducing dependence on human-labeled data.
obtain better
1 Introduction
Large language models (LLMs) have made signif- icant progress by scaling up model size and data size (Peters et al., 2018; Devlin et al., 2019; Rad- ford et al., 2019; Brown al., 2020; OpenAI, 2023) for unsupervised pre-training and subsequently ap- plying reinforcement learning from human feed- back (RLHF) align model responses with human preferences (Christiano et al., 2017; Ouyang et al., 2022). More recently, instruction tuning (Wei et al., t. t Work done during internship at MSRA.
However, instruction tuning offers only broad guideline for the base LLMs to transition from "next token prediction" to a more interactive, instructionnfolllwiing style. As result, the model may learn some superficial features or styles from the instruction data but still lacks a deeper under- standing of what constitutes a preferred response. For instance, when given a question like "Give

---

three tips for staying healthy". base LLM may generale fluent yet undesirable continuations, while an instruction-tuned LLM could offer three general tips. Humans might prefer more detailed tips over general tips, but such tips are less likely to be sam- pled since they have lower likelihood within the current model distribution. This can be attributed to the fact that they are either unseen during in- struction tuning hard be sampled due to the exposure bias (Ranzato et al., 2015).
Our model, data and code will be released facilitate future research.
Methodology
In this section, we describe how to obtain our Tuna model using the feedback from LLMs. We first describe the vanilla instruction tuning. We then introduce our probabilistic ranking and contextual ranking approaches. Lastly, we describe how integrate both ranking approaches.
To address this, we propose further finetuning of an instruction-tuned LLM to discern the qual- ity of multiple responses more precisely, using our novel probabilistic ranking (Sec. 2.2: Fig. top) and contextual ranking (Sec. 2.3; Fig. mid- dle) approaches. Probabilistic ranking enables the instruction-tune tunnod LLM inherit the high-quality and low-quality responses as well as their rela- tive rankings from the teacher LLM (c.g.. text- davinci-003). In contrast, contextual ranking aims to re-balance the instruction-tuned model's own re- sponse distribution with the help of stronger LLMs (c.g., GPT-4), mitigating the exposure bias issuc. We apply probabilistic ranking and contextual ranking sequentially to an instruction-tuned model, i.., Alpaca (Taori et al., 2023), resulting in a model called Tuna (Sec. 2.4; Fig. 1 bottom). We evaluate Tuna on various benchmarks, including Super Nat- ural Instructions (Wang et al., 2022b), which con- tains 119 diverse test tasks; LMentry (Efrat et al., 2022), comprising 25 tasks to assess the basic ca- pabilities and robustness of LLMs; and Vicuna QA (Chiang et al., 2023) which evaluates the model's ability to answer a diverse set of questions with the assistance of GPT-4. Experimental results demon- strate that the Tuna model not only consistently outperforms the standard instruction-tuned mod- els on all benchmarks, but also surpasses several strong RLHF baselines (Ouyang et al., 2022). To summarize, our contributions are as follows:
2.1 Instruction Tuning
LLMs like GPT-3 (Brown et al., 2020) have been trained on a massive text corpus using maximum likelihood estimation (MLE):
LMIL(B)
Σ log logg(ulver:)) (1)
where O represents the parameters of the base model. The pre- pre-training objective function compels the model to predict the next token Y/ given its pre- fix V-r [yo- 1/1 ........ 1/1 A sufficiently-trainee LLM can generate fluent continuations given al- most any prefix. However, the generated continua- tions may not align well with human preferences. As the primary goal an LLM assist humans. becomes essential to encourage the generation of content that follows human instructions and aligns with human preferences. The current dominant approach to enhance LLMs' instruction-following ability is called instruction tuning (Mishra et al., 2021; Wei et al., 2022: Taori et al., 2023), which finetunes the base LLMs in supervised manner on instruction-response pairs (i,r) (where is an instruction and r is its response) using MLE:
LMLE(i, r) |rl
log p(r);;;)))
(2)
where 0' represents the parameters of the instruction-tumed model. After instruction tuning, we expect the model distribution P(: 6') to allo- cate higher probabilities proper responses like rather than undesirable continuations.
We propose probabilistic ranking and contex- tual ranking, which enable the instruction- tuned model to distinguish high-quality and low-quality responses and assign higher prob- ability to the former accordingly.
Note that the responses in instruction-ressonse pairs can either be annotated by humans or gen- erated by strong LLMs, such as Instruct-GPT or GPT-4 (Wang et al., 2022a). A prevalent and cost- effective approach for generating instruction tun- ing data is the Senf-Inst algorithm (Wang et al.,
The Tuna model, obtained by sequentially applying probabilistic ranking and contex- tual ranking on an instruction-tuned I.M,, achieves better results than several strong benchmarks, including RLHF models;
htttps:/hhuggingaaaee ctaaaacc..ccaaaaaaaaa co/datasets/databriccss databricks- dollcls--dolllkk

---

2022a). Specifically, it uses strong LLM, e.g.. lext-davinci-003, to create instructions based on few seed instructions, and then generates a single response for each instruction using the same LLM.
data, and أ is considered have better quality than ,18), if < k. Once we obtain the ranked responses, we can encourage our model to learn from these rankings using a pairwise ranking ob- jective, which has been successfully employed in previous work (Zhong et al., 2020; Liu et al., 2022; Zhang et al., 2022; Zhao et al., 2023). The ranking objective function is as follows:
2.2 Probabilistic Ranking
Instruction tuning with the data generated by the Self-tnst algorithm is essentially form of sequence-level distillation (Kim and Rush, 2016). The rationale behind this class of distillation method is that the current commercial LLMs have significantly better capabilities than their open- source counterparts. Instead of learning from the single-response data, our probabilistic ranking ap- proach leverages the relative rankings of multiple responses based on the teacher model's probabili- ties for better pseudo label distillation (see Fig. 1 top).
Lrank
Σ
Di rank
(4)
OSjCKEN-1
rank max (0. og vg. + m x (k: 5)) k
(5)
where ng. p log (( 0¹) m > the margin hyper-parameter. The ranking loss, Trank aims teach the model to distinguish good re- sponses from bad ones based on the teacher LLM's perspective. In addition to Lranks we also apply a cross-entropy loss on the original response as regularization:
Let r denote the original response for in- struction i in the instruction tuning dataset. We query strong (teacher) LLMs, such as next-davinci-003, to generate N new responses ((N-1) denote these new
for i. Let (()), responses, and ((r(0)))) [i), p(r())))))))...... "(,"""))...,.,(((( denote their probabilities. While the teacher LLMs are expected to produce responses of comparable quality on average, there will inevitably be some variation in the quality of the generated responses. This inherent variability manifests itself in vari- ous aspects, such as differences in accuracy (Wang et al., 2023a), response length, and level of details provided (Wang et al., 2023b).
(()) ,(0)),(()),
1 |r]
L Drank + ALMLE, LMLE = log p(rl; 0')
(6)
where r is the original response, and X > 0 controls the importance of LMLE, which helps prevent over- optimization of the ranking loss.
After learning with probabilistic ranking, the model can better assign probabilities to superior and inferior responses.
2.3 Contextual Ranking
Intuitively, if model is perfectly distilled, the relative probabilities assigned two samples should be the same as those of the teacher model. Specifically, let p(r(i)li; 0¹) and p(r("));; O") de- note the probabilities of yd6 and (k) w.r.t. the student model. If p()())))) > (()()))). then p(r-i)i;; 0*) > ((r(())1; (i; 8'). We use the follow- ing normalized log-lik geglikklihhod as the teacher model quality score to account for differences in response lengths:
During the instruction tuning or the probabilistic ranking stage, the model is finetuned to generate a good r given an instruction i. However, given the same i during inference, the model may still gen- erate a relatively low-quality response r'. This is related to the exposure bias problem (Ranzato al., 2015), where the model fails generate due accumulated errors during the regtrreegeee gen- eration process. To address this issue, we use our contextual ranking approach to refine the distri- bution of responses generated by the model itself, assigning higher probabilities to better responses with the help of strong L.LMs (Fig. 1 middle), thus alleviating exposure bias (Ranzato et al., 2015). For each instruction. we first sample N sponses from the instruction-tune model itself, i.c. (0) ,.(1),.., .... (N 1) p(-1i; 0'). We hope the sam- ples to be diverse enough so that better responses are more likely to appear in the sampled results.
s(i, ((i))())) log (()))))))))) (x())p3
A 10,..,,, N 1}
(3)
where |r(00) is the length of (kk and 3 represents the length penalty.
We then rank those responses in decreasing or- der based on s(i,r((), The resulting instruction- response pairs become {i,r, (r-10 (,.",,,,(NN'
where i, r are from the original instruction tuning

---

To ensure diversity, we impose constraint on the ROUGE-L (Lin, 2004) score between each pair of responses, requiring to be less than a threshold т. If the ROUGE-L score exceeds т, we increase the sampling temperature and resample another re- sponse. If multiple trials still result in a ROUGE-L score above T, we retain the least similar response from the trials. After obtaining N responses, we leverage the contextual understanding ability of commercial LLMs, such as GPT-4 (OpenAI, 2023), to rank them based on various aspects. The ranking process consists of multiple steps. First, we ask GPT-4 to assess whether the instruction requires an open-ended answer (e.g., story generation) or a close-ended answer (e.g., solving a math prob- lem). We then request GPT-4 to generate its own response reference. Next, GPT-4 compares the reference response with the N responses from different aspects and assign scores each response, For open-ended instructions, GPT-4 evaluates rel- evance (score 0-5), level of details /juusiffiaattio (score 0-5), and accuracy (score 0-5) of the model responses compared to its reference response. For close-ended instructions, the evaluation criteria are accuracy (score 0-5), level of details/justin /justificationn (score 0-5), and clarity (score 0-5). Finally, GPT-4 ranks responses in decreasing order based on the sum of their scores (see Appendix E for our com- plcte prompt). We also manually evaluated GPT-4 rankings, which have achieved a strong correlation with human judgements (see Appendix G, H).
mended to first obtain a Tuna, model, followed by applying contextual ranking to Tunap's 's response distribution, resulting in the Tuna model. There are two reasons for this choice. First, although it is beneficial to learn the ranking of different re- sponses from the teacher LLM's perspective (prob- abilistic ranking), the model might not fully capture the teacher's ranking knowledge due to its limited capacity. Second, contextual ranking enables the model to better adapt to its own capacity by work- ing with the model's s own generations. By gener- ating its own responses, the model can finetune its understanding with the help of stronger LLMs and more effectively produce responses that are both closer to human preferences and compatible with its capacity constraints, alleviating the exposure bias issue (Ranzato et al., 2015).
Experiments
3.1 Model and Data
In our experiments, we use 7B LLaMA model (Touvron et al., 2023) as the base model. The instruction tuning data is sourced from Alpaca (Taori al., 2023), which consists of 52K instruc- tions paired with responses that are generated by text-davinci-ee
using the Self-Instruct algo- rithm (Wang et al., 2022a). We perform instruction tuning on 52K Alpaca data using recommended hy- perparameters, such as learning rate of 2e-5 and the AdamW optimizer (0.9, 0.999) (Loshchilov and Hutter, 2019).2 For simplicity, we also refer to the instruction-tume tuned model Alpaca.
As in Sec. 2.2, the resulting instruction tun- ing dataset becomes ri r, (r. (..,,.,,....) Note that the |M| < < - 1, is derived from the instruction-tuned model itself. Lastly, we use the same objective function as in Eq. 6 to encourage the model to assign higher probabilities to better responses.
input 52K instruc- tions from Alpaca dataset into text text-davinci- to produce N = 1 responses per instruction along with their log-likelhhoods' with an inference tem- perature of 1. We calculate response scores using Eq. 3 with 33 being 1.3, and rank the responses accordingly. Subsequently, we finetune the Alpaca model for epoch with learning rate 1e-5, mar- gin m 0.1, and cross entropy regularizer weight 1.0. We denote the model trained exclusively with probabilistic ranking as Tunap-
For probabilistic ranking,
2.4 Integrating Probabilistic and Contextual Ranking
Given an instruction-tuned model, there are sev- eral options for further finetuning: 1) learning with probabilistic ranking alone; 2) learning with con- textual ranking alone; 3) learning with probabilistic ranking followed by contextual ranking (see Fig. bottom). We refer to the models finetuned with these three methods as Tunap. Tunaç. and Tuna, respectively.
For contextual ranking, we sample N = re- sponses from the Alpaca model with temperature T = for each instruction. To avoid similar gen- erations, we ensure the pairwise ROUGE-L (Lin, 2004) between responses is less than 7 0.8. Oth-
/https://github.c http://gthuub.cco//tteerrrrrr www./etteerorttee..eeeeoeeeeeeee
To optimally integrate both probabilistic rank- ing and contextual ranking techniques, it is recom-
*GPT-4 is more powerful hut does not return log- likelihoods.

---

Super NI
LMentry LMentry Score
Vicuna QA
O-shot
2-shot
Win
Lose
Tie
LLaMA
11.0
23.6
26.3
4%
92%
4%
T5-LM 11B
30.2
20.6
TO 11B
32.3
31.6
InstructGPT 175B
52.1
48.4
Alpaca
36.0
44.5
31.4
PPO-sim
31.9 (-4.1)
37.5 (-7.0)
27.8 (-3.6)
79%
16%
5%
PPO-sim-GPPT--20
37.1 (+1.1)
44.9 (+0.4)
27.8 (-3.6)
74%
22%
4%
Tunap
39.4 (+3.4)
43.9 (-0.6)
35.0 (+3.6)
68%
27%
5%
Tuna,
37.7 (+1.7)
46.6 (+2.1)
32.2 (+0.8)
74%
20%
6%
Tuna, (PRM)
34.2 (-1.8)
40.1 (-4.4)
32.2 (+0.8)
75%
19%
6%
Tuna
38.7 (+2.7)
45.0 (+0.5)
34.7 (+3.3)
86 %
10 %
4%
Table Performance comparison of different models on Super NI, LMentry and Vicuna QA. The numbers bold indicate the top-2 results. The numbers in parentheses indicate the performance differences compared to Alpaca. The results of T5-MMM 11B (Raffel et al., 2020), TO-1IB (Sanh et al., 2022), InstructGPT 175B (Ouyang et al., 2022) are taken from Wang et al. (2022b); Efrat et al. (2022). The RLHF baselines PPO-sim and 770--im-GPP4-20K, which apply the PPO algorithm (Schulman et al., 2017), taken from Dubois et al. (2023).
Model
Alpaca
PPPssa+PPOOsim
Tunap
Tuna,
Tuna
Score
2.13
2.95*
2.98
3.15
3.80*1
Table Human evaluation on Vicuna QA. denotes that the model is significantly (p < 0.01) better than Alpaca, while denotes that Tuna is significantly < 0.01) better than other models.
erwise, we remove the similar response, increase the temperature by 0.1, and resample. three tri- als fail to produce unique enough responses, we keep the least similar one. We then employ GPT- 4 to rank responses for the first 13K instruction data with the GPT-4 inference temperature be O. The contextual ranking prompt is shown in Table 9.4 The finetuning hyperprameters follow those probabilistic ranking. We refer to the model trained on 13K contextual ranking data of the Al- paca model as Tunac-
baselines for comparison (i.e., PPO-sim and PPO- sim-GPT4-20K models from AlpacaFarm (Dubois et al., 2023)).
3.2 Evaluation
Super Natural Instruction (Super NI) Super NI (Wang et al., 2022h) contains 119 test tasks designed to evaluate a model's cross-task general- ization ability. It includes a variety of classification and generation tasks, such as textual entailment and title generation. We report both O-shot and 2-shot performance, where 0-shot provides only instruction (referred to as "definition" in their literature) and 2-shot offers additional positive examples. The evaluation metric for all 119 tasks ROUGE-L (Lin, 2004), which is strongly corre- lated with human evaluation with a Pearson coef- ficient of 0.998 according to Wang et al. (2022b). Greedy decoding is applied during inference.
Furthermore, use the 13K GPT-4 ranking data train proxy ranking model (PRM) based on StableLM-3B. The PRM is employed re-rank Alpaca's 's responses on 52K instructions. We refer to the Alpaca model trained with 52K ranking data totally generated by the PRM as Tuna, (PRM).
Lastly, we also collect 13K GPT-4 contextual ranking data based Tuna,'s 's responses instead of Alpaca's. We refer the model finetuned on Tuna, as Tuna.
LMentry LMentry (Efrat al., 2022) a bench- mark that primarily focuses on the accuracy and ro-
We also included strong reinforcement learning
"We also trained our own RLHF model, which not as good as the ones in AlpacaFarm. The comparison can he found in Appendix
'The cost of calling OpenAl API listed Appendix B. 'https: / hhw../ttttttuttt./e/e//eeee... ctw. Stttttttaaiitttaeeeeeeeeaeeteele AI/StableLM

---

bustness aspects of LLMs' generations. It contains 25 short tasks that are trivial to humans but chal- lenging for LLMs. The final metric is LMentry score, which is calculated by multiplying its mean accuracy on 25 tasks with the robustness score. The model will be evaluated in O-shot manner, and greedy decoding is applied during inference.
tic ranking, On the Vicuna QA benchmark, both Tuna, and Tuna, outperform Alpaca significantly on nearly 70% of the questions, as evaluated by GPT-4. Upon comparison with the RLHF base- lines, Tuna, and Tuna, consistently demonstrate superior performances on both the Super NI and LMeetry benchmarks. However, when it comes to the Vicuna QA benchmark, their performance is marginally lower than that of the RLHF baselines. Moreover, Tuna achieves the best performance on Vicuna QA while maintaining competitive scores on Super-NI and LMentry. Human results on Vi- cuna QA (see Table 2) also confirm that humans prefer the responses from our models.
Vicuna QA Vicuna QA (Chiang et al., 2023) comprises 80 test questions across categories that measure an LLM's ability to generate rele- vant, detailed and accurate responses and it has been widely adopted in many works. Instead of having a ground truth for evaluation, it conducts pairwise comparisons with the help of GPT-4 (Ope- nAI, 2023). It prompts GPT-4 to compare the out- puts of our models the Alpaca model. We report the win/wose/ /ie rate against the Alpaca model.
Furthermore, Tuna, (PRM) demonstrates com- parable performance Tuna, on Vicuna QA and LMentry, but it underperforms both Tuna, and Al- paca on Super NI. This suggests that although the PRM has primarily learned ranking from the GPT- 4 contextual ranking data, it also introduces some noise during the learning process. Overall, it is more effective to learn directly from GPT-4 contex- tual ranking data. 8
Human Evaluation Additionally, we conduct human evaluations on Vicuna QA. Specifically, re- sponses from five anonymous systems, namely Al- paca, Alpaca + PPO-sim, Tuna, Tunap. and Tuna.. were randomly shuffled and presented annota- tors who were then asked to rank these outputs. The scoring was designed such that the i-th ranked system receives a score of 6 i, meaning the best- ranked system receives a score of 5, and the worst- ranked system receives a score of 1. Each question was annotated by two different annotators, and the score was averaged.
3,4 Ablation Study
In this subsection, we delve deeper into the per- formance of our approach by examining several aspects, including: (a) the effeet of more responses in instruction tuning, (b) the order of applying two ranking methods, (c) the influence of the cross tropy regularization, (d) the amount of probabilistic ranking data, and (e) the risks of GPT-4 evaluation.
3.3 Main Results
The main results are presented in Table 1. Af- ter instruction tuning, Alpaca demonstrates sig- nificant performance improvements over LLaMA on all three benchmarks. This highlights the suc- cessful transition from the "next token predic- tion" paradigm to a more interactive instruction- following paradigm.
More Responses in Instruction Tuning We ex- plore whether Tuna's effectiveness is solely due to the increased response data by examining the impact of adding more responses per instruction during instruction tuning. We create Alpaca-Mul, by adding four extra responses from the probabilistic ranking dataset to the Alpaca dataset and fine-tuning the LLaMA model using Eq. 2. The results are presented in Table 3.
model,
Furthermore, both contextual and probabilis- tic ranking enhance performance across all three benchmarks. Specifically, Tuna. exhibits more im- provement on the Super NI7 2-shot results while Tuna, performs better on Super NI 0-shot and LMentry, narrowing the performance gap with much larger models like InstruciGPT-175B. Since the 2-shot input is longer than 0-shot, we conjec- ture that contextual ranking might be more benefi- cial for longer sequence generation than probabilis-
Upon evaluation on Super NI, Alpaca-Mul's per- formance is nearly identical to that of Alpaca but falls short when compared the 0-shot settings of Tunap and Tuna. On LMentry, Alpaca-Mul out- performs Alpaca, yet it still does not reach the performance levels of Tuna, and Tuna. Interest- ingly, in the Vicuna QA task, Alpaca-Mul slightly underperforms compared to Alpaca.
"ROUGE is used as the default metric on Super NI. How- ever, our results follow the same trend using BERTScore (see Appendix J).
"Experiments with more PRMs can be found in App. D.

---

Super NI
LMentry LMentry Score
Vicuna QA
0-shot
2-shot
Win
Lose
Tie
Alpaca
36.0
44.5
31.4
Alpaca-Mul
34.7 (-1.3)
45.7 (+1.2)
33.9 (+2.5)
42%
53%
5%
Tuna,
39.4 (+3.4)
43.9 (-0.6)
35.0 (+3.6)
68%
27%
5%
Tuma
38.7 (+2.7)
45.0 (+0.5)
34.7 (+3.3)
86%
10%
4%
Tuma.
37.7 (+1.7)
46.6 (+2.1)
32.2 (+0.8)
74%
20%
6%
Tunaep-11 13K
35.7 (-0.3)
44.0 (-0.5)
33.5 (+2.1)
58%
37%
5%
(-1.2)
(3.4(1-1.1)
35.4 (+4.0)
46%
48%
6%
Tunaep-522
(-1.0)
42.6 (-1.9)
33.8 (+2.4)
51%
41% %
8%
mix- Tuna-52K
37.7 (+1.7)
44.2 (-0.3)
30.0 (-1.4)
70%
23%
7%
mix-Tuna-100K
36.0 (+0.0)
40.0 (-4.5)
32.6 (+1.2)
55%
40%
5%
Table 3: Different combinations of probabilistic ranking data and contextual ranking data. The numbers in bold represent the top-2 results. The numbers in parentheses represent the performance difference compared to Alpaca.
These findings suggest that merely adding more responses without differentiating them does not necessarily lead to improved response generation. Overall, the results of Alpaca-Mul indicate that Tuna's superior performance cannot be solely at- tributed to the availability of more response data.
rank
1
2
4
contextual ranking
66.4
55.2
51.4
44.8
prob. ranking
55.8
54.3
52.5
49.4
PRM
69.2
57.8
50.9
44.7
Table 4: The average ranking lengths of contextual rank- ing data, probabilistic ranking data and the data gener- ated by the proxy ranking model (PRM).
Integration Order An alternative approach to Tuna involves first training the Tuna, model, and subsequently continuing training the Tuna, model with probabilistic ranking data. The resulting model is referred to Tunacp-
that shorter probabilistic ranking data are beneficial when high accuracy and robustness are top priority. Interestingly, Tunaep is not comparable Tuna, indicating that the order in which the model is trained with contextual and probabilistic ranking matters. One plausible explanation that both the original Alpaca data and the probabilistic ranking data are generated by text-davinci-on (onccc000,,,, while Tuna, has significantly shifted the model distribu- tion by re-ranking the Alpaca model's responses, making it challenging finetune Tuna, with prob- abilistic ranking data again.
We explore various strategies for training Tunacpi 1) finetuning Tuna, with the first 13K probabilis- tic ranking data (Tunacp-13K); 2) finetuing Tuna, model with last 39K probabilistic ranking data (Tuna:p-39K); 3) finetuning Tuna, model with 52K probabilistic ranking data (Tuna,--52K). Ad- ditionally, we also try to finetune original Alpaca model with a combination of 13K GPT-4 contextual ranking data (generated from Alpaca model's 's re- sponses) and the last 39K probabilistic ranking data (mix-Tuna-52K). We also finetune Alpaca model with 52K contextual ranking data (13K GPT-4 con- textual ranking + 39K ranking- ranking-model- model-generated data) plus 52K probabilistic ranking data (mix- Tu4K-104K). The training details are listed in the Appendix C. The results are listed Table 3.
The Effeet of Cross Entropy Regularizer We examine the influence of the weight À of the cross entropy regularizer in Eq. 6 on performance by varying À across different values: {0, 0.1, 1,5, 10) while training the Tuna, model. Fig. 2 illustrates that as À increases. the performance oriented benchmarks such Super NI and LMen- try improves, while the performance on open ques- tions does not necessarily follow the same trend. On one hand, this finding suggests that with small X, learning with contextual ranking may induce
None of the combination strategies consistently outperform both Tuna, and Tuna, across the Vi- cuna QA and Super NI benchmarks. On LMen- try, however, finetuning Tuna, with probabilistic ranking data is beneficial, especially when no du- plicate data is present (Tuna,,-39K 39K). This suggests
accuracy-

---

0aid 24nd
ROUGE-
ROUGEL
30
40
LiWentty Score
35
LMKnty
35
80
25
Win Lose
Rato
Tol
50
10000
20000
30000
40000
50000
0.0 0.1
1.0
6.0
10.0
Data Number
Figure 3: The effect of varying the number of proba- bilistic ranking data Tuna,-
Figure 2: The effect of varying the weight of cross en- tropy regularization in Eq. 6 on Tuna.. The win/lose/tie rate on Vicuna is computed against Alpaca.
However, discussed in Sec. 3.3, the data generated by the PRM is as good the orig- inal 13K contextual ranking data, as assessed by more targeted automatic evaluations like Super NI and LMentry. Despite the inferior quality of the PRM PRM-generated data, the performance on Vicuna QA remains almost unaffected (see Tuna, (PRM) in Table 1). This observation suggests that evaluat- ing LLMs on open QA with GPT-4 may not always be as accurate as it appears, echoing the findings of Wang et al. (2023b). highlights the need for more representative test questions or additional targeted benchmarks for evaluation.
long and detailed answers, but those answers are not always accurate. On the other hand, it implics that accuracy -criency benchmarks and open QA benchmarks complementary, and rescarchers should consider more diverse test cases to thor- oughly evaluate model (Wang et al., 2023b).
The Amount of Probabilistic Ranking Data We investigate the impact of varying the amount of probabilistic ranking data used for finetuning the Tuna, model by testing different data sizes, i.e., {0, 13000, 24000, 52000}. refers to the Alpaca model. The results, shown in Fig. 3, reveal that for probabilistic ranking, 13K data points are sufficient for Super NI and LMentry, while Vicuna QA re- quires 24K data points. We conjecture that this satu- ration phenomenon can be attributed to reasons. First, 52K Alpaca instructions generated by Self- Instruct algorithm are not diverse enough, as new instructions are produced by text-davinci-003 using prompt instructions sampled from a limited seed task pool. Second, instruction tuning itself may only require a limited amount of data to per- form behavior cloning, as discussed in Zhou et al. (2023). Thus, can further reduce the cost of probabilistic ranking data generation by half.
Related Work
Instruction Tuning Instruction tuning aims to improve the usability of base language models (Brown et al., 2020; Raffel et al., 2020; Chowd- hery et al., 2022) by finetuning them on instruction- response pairs in zero-shot (Wei al., 2022)
few-shot manner (Mishra al., 2021; Wang et al., 2022b; Mallen al., 2023). The instruction data can be sourced from off-the-shelf NLP benchmarks (Mishra et al., 2021; ; Wei et al., 2022; Wang et al., 2022b) or generated by L.MSs (Wang et al., 2022a; Honovich et al., 2022; Taori et al., 2023; Peng et al., 2023).
The Risks in GPT-4 Evaluation We present evi- dence that evaluating a model on open QA with the help of GPT-4 may be risky. Table displays the ranking length of our proxy ranking model (PRM). It shows that the PRM has inherited GPT-4 rank- ing's 's bias towards longer outputs (Li et al., 2023).
Ranking Loss Learning through re-ranking sequence-level outputs has been studied in sequence-to--equeecce models (Wiseman and Rush, 2016; Edunov et al., 2018; Liu et al., 2022; Zhang et al., 2022). BRIO and MoCa algorithms (Liu

---

al., 2022: Zhang al., 2022) adopt pair- wise ranking loss guide the model gener- ate summaries with higher ROUGE scores (Lin, 2004). In this paper, we use GPT-4's (OpenAI, 2023) strong contextual understanding ability and dext-davinci-003' (Ouyang et al., 2022) intrin- sic probability measures for ranking. In parallel with our work. Yuan et al. (2023) also propose pairwise ranking loss for finetuning LLMs. Key differences include: 1) our pipeline finetuning strat- egy: 2) our focus on ranking the model's responses; 3) our use of the original response for cross entropy regularization, while they select the rewaed response. Additionally, Liu et al. (2023c) also em- ploys GPT models for finetuning BART (Lewis et al., 2019) on the summarization task.
generally expected to provide accurate and reli- able rankings; however, may still be sensitive to the phrasing or structure of prompts (Dubois et al., 2023). This inconsistency may lead to sub- optimal rankings and potentially affect the overall performance of the Tuna model. In future work, it would be beneficial to design more robust prompts that can mitigate the impact of GPT-4's ranking in- consistencies. Another limitation the evaluation benchmark. In this paper, we evaluated the Tuma model on three benchmarks, which provided a di- verse range of tasks and challenges. However, it is unclear how well the Tuna model would generalize to other types of tasks, domains, or languages. Fur- ther research is needed to explore the applicability of the Tuna model to a broader range of problems and settings. The last limitation is the reliance on the use of proprictary LLMs, such GPT-4 and text-davinci-00: for generating responses and rankings. This dependency may limit the ac- cessibility and reproducibility of our method for researchers who do not have access to these pro- prictary models. Developing alternative methods that can leverage open-source LLMS or other rank- ing mechanisms would be valuable direction for future research.
Pre- Pre-Trained Model Evaluation Large pre- trained models are powerful evaluation metrics due to their strong contextual understanding abil- ity, such as BERTScore (Zhang* et al., 2020), BARTScore (Yuan et al., 2021), MoverScore (Zhao et al., 2019), COMET (Rei et al., 2020), and GPTScore (Fu et al., 2023). More recently, there are more evaluation strategies based on GPT-3.5 and GPT-4 (Liu et al., 2023b; Gao al., 2023).
5 Conclusion
Acknowledgements
In this paper, we propose to finetune instruction- tuned LLM using our probabilistic ranking ap- proach (Tunap), contextual ranking approach (Tunag), and a combination of both (Tuna). Our comprehensive experiments demonstrate consis- tent performance improvements across three bench- marks: Super Natural Instructions (119 test tasks), LMentry (25 test tasks), and vicuna QA. Further- more, our methods outperform popular reinforce- ment learning from human feedback baselines that rely on the proximal policy optimization algo- rithm. These findings underscore the effectiveness of our approach in enhancing the performance of instruction-tune LLMs and pave the way for fu- ture research in this area.
We would like to thank reviewers for their valuable feedback. This research/project is supported by Ministry of Education, Singapore. under its Tier 3 Programme (The Award No.: MOET320200004), the National Research Foundation Singapore and DSO National Laboratories under the AI Singa- pore Program (AISG Award No: AISG2-RP-2020- 016), and Ministry of Education, Singapore, under its Academic Research Fund (AcRF) Tier Pro- gramme (MOE AcRF Tier 2 Award No: MOE- Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of the Ministry of Education, Singapore.
Limitations
References
Despite the promising results achieved by our Tuna model, there are several limitations that should be acknowledged. The first limitation is GPT-4 rank- ing inconsistency. In our experiments, we relied on GPT-4 for contextual ranking, which may in- troduce bias due to the inconsistency in its rank- ing performance. As a powerful LLM, GPT-4 is
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse. Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack

---

Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems.
Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. Human- like summarization evaluation with chatgpt. ArXiv, abs/2304.02554.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tuning lan- guage models with (almost) no human labor.
Yoon Kim and Alexander M. Rush. 2016. Sequence- level knowledge distillation. In Conference on Em- pirical Methods in Natural Language Processing.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebas- tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao. Parker Barnes, Yi Tay, Noam M. Shazeer, Vinod- kumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov. Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Re- won Child, Oleksandr Polozov, Katherine Lee, Zong- wei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz. Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- cling with pathways. ArXiv, abs/2204.022 1..
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdel rahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zeutlmoyer. 2019. Bart: Denoising sequence- peruence-to-seee io-sequence pre-training for nat- ural language generation, translation, and compre- hension. In Annual Meeting of the Association for Computational Linguistics.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An au- tomatic evaluator of cnstrucctionfollowinnt models. https: //gi www.:tttbbccoccoeeeeeeeeeecoettt
1ab/ alpaca_evall
Chin-Yew Lin. 2004. Rouge: A package for automatic
evaluation of summaries. In Anmual Meeting of the Association for Computational Linguistics.
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape. Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023a. Lost in the middle: How language models use long contexts.
Yang Liu, Dan Iter. Yichong Xu, Shuo Wang, Ruochen Xu, and Chenguang Zhu. 2023b. G-eval: Nlg evalua- tion using gpt-4 with better human alignment. ArXiv, abs/2303. 16634.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Mar- tic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems.
Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Dragomir R. Radev, and Arman Cohan. 2023c. On learning to summarize with large language models as references. ArXiv, abs/2305.1 14239.
Jacob Devlin. Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In North American Chapter of the Associa- tion for Computational Linguistics.
Yixin Liu, Pengfei Liu, Dragomir R. Radev, and Graham Neubig. 2022. Brio: Bringing order to abstractive summarization. Annual Meeting of the Association for Computational Linguistics.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. 2023. Alpacafarm: simulation framework for methods that learn from human feedback. ArXiv, abs/2305. 14387.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Sergey Edunov, Myle Ott, Michael Auli, David Grang- ier, and Marc' Aurelio Ranzato. 2018 Classical structured prediction losses for sequence sequence learning. In North American Chapter of the Associa- tion for Computational Linguistics.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric mem- ories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers).
Avia Efrat, Or Honovich, and Omer Levy. 2022. Lmen- try: A language model benchmark of elementary language tasks. ArXiv, abs/2211 I. 1.02069.
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. ArXiv, abs/2302.04166
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021. Cross-task generaliza- tion via natural language crowdsourcing instructions.

---

In Annual Meeting the Association for Computa- tional Linguisties.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto, 2023. Stanford alpaca: An folloowinnn llama model. https:// // gi thub. wttp:/ccooaaaaaeeeeeeeee tatsu-labbl http:/staffoddaappp
OpenA1. 2023. GPT-4 Technical Report. ArXiv, abs/2303.08777.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, Aur' elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv, abs/2302.1397/
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery. and Denny Zhou. 2023a. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representa- tions.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Gal- ley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4, ArXiv, abs/2304.03277.
Matthew E. Peters. Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In North American Chapter of the Associ- ation for Computational Linguistics.
Yizhong Wang. Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hanna Hajishirzi. 2023b. How far can camels go? exploring the state of instruction tuning on open resources.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. Journal of Machine Learning Research.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Al- isa Liu, Noah A. Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2022a. Self-instruct: Aligning lan- guage model with self generated instructions. ArXiv, abs/2212.10560..
Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level train- ing with recurrent neural networks. In International Conference on Learning Representations.
Yizhong Wang. Swaroop Mishra, Pegah Alipoor- molabashi, Yeganch Kordi, Amireeza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Is- han Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, M. Moradshahi, Mihir Parmar, Mirali Purohit, Necraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddharth Deepak Mishra, Su- jan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hanna Hajishirzi, and Daniel Khashabi. 2022b. narrraaaaaalinsstttioonssss Generalization via declar- ative instructions on 1600+ nlp tasks. In Conference on Empirical Methods in Natural Language Process- ing.
Ricardo Rei. Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Conference on Empirical Methods in Natural Language Processing.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafcai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey. M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. 2022. Finetuned language mod- els are zero-shot learners. International Confer- ence on Learning Representations.
Sam Wiseman and Alexander M. Rush. 2016. Sequence- to-sequence learning as beam- optimization. In Conference on Empirical Methods in Natural Lan- guage Processing.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, 2017. Proximal policy optimization algorithms. ArXiv, abs/1 707.06347.

---

Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text genera- tion. In Advances in Neural Information Processing Systems.
the output of each LIMA instance under differ- ent Tunap models and average the token likelihood over the whole LIMA training set. The results are shown in Table It can be seen that with , = 1.3, the model can achieve the best NLL on LIMA train- ing set. Thus, we set 8 = 1.3 in our experiments.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Feiran Huang. 2023. Rrhf: Rank responses align language models with human feedback without tears, ArXiv, abs/2304.05302
0.9
1.0
1.1
1.2
1.3
1.4
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
NLL 2.14
2.12
2.11
2.10
2.09
2.09
Table 5: The token-level log-likelihood of LIMA train- ing set under Tuna, models trained with probabilistic ranking data scored with different B.
Xingxing Zhang, Yiran Liu, Xun Wang, Pengcheng He, Yang Yu, Si-Qing Chen. Wayne Xiong, and Furu Wei. 2022. Momentum calibration for text generation. ArXiv, abs/2212.00257.
B OpenAl API Pricing
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris- tian M. Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized em- beddings and earth mover distance. In Conference on Empirical Methods in Natural Language Processing.
We list the cost of calling OpenAI API models in Table 6.9 The human labeling cost per 1K examples is estimated based on the pricing listed in Dubois et al. (2023), at 0.25$ for each comparison. For each data example. there are 4 responses and thus 1)/2 = 6 comparisons. Thus, the human
Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. 2023. Calibrating sequence likelihood improves conditional language generation. In International Conference on Learning Representations.
(4 * labor cost per 1K examples is 1500S.
C Training Details of Tuna.p and mix-Tuna
Ming Zhong, Pengfei Liu. Yiran Chen, Danqing Wang. Xipeng Qiu, and Xuanjing Huang. 2020. Extractive summarization text matching. In Annual Meeting of the Association for Computational Linguistics.
The hyperparameters are listed in Table 7. For models finetuned from Alpaca, i.e., Tuna,, Tunap. and mix- Tuna, the learning rate is le-5. The only exception is mix-TTna--00KK whose learning rate is 5e-6 since it contains 52K duplicate data. For models finetuned from Tuna, or Tuna,, i.e., Tunaep and Tuna, the learning rate is lc-6. We use 8 Nvidia V100-32GB GPUs for all experiments in this paper.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, I. Yu, Susan Zhang. Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. Lima: Less is more for alignment. ArXiv, abs/2305.1 1206.
A The Length Penalty / for Probabilistic Ranking Data
D Other Proxy Ranking Models (PRM)
In our preliminary experiments, we found that the length penalty /3 1.3 is able to induce detailed re- sponses and validated this choice on LIMA (Zhou et al., 2023) dataset. We finetune the 3 parameter in Eq. using the LIMA training dataset, which contains 1030 high-quality expert instruction an- notations, allowing LLaMA-65B to be finetuned and achieve remarkably strong performance across a wide range of topics. Note that the training set also contains 50 modified Super NI examples but they are from the training tasks while we test our models on 119 Super NI test tasks. Specifically, we first obtain Tuna, models with probabilistic rank- ing data scored with different 3. Then, we compute the token-level negative log- likellhodood (NLL) of
Similar to the PRM introduced in Sec. 3.1, we use the 13K GPT-4 ranking data train another PRM based on LLaMA-7B, which we refer to as PRM-7B. We denote the PRM based on MtableL 3B as PRM-3B. These two ranking models are employed to re-rank Alpaca's responses on 52K instructions. The Alpaca model trained with 52K data totally generated by the ranking models are referred to as Tuna, (PRM-3B- 3B-52K) and Tuna, (PRM-7B- 52K). Note that Tuna, (PRM-3B-52K) the Tuna, (PRM) listed in Table We denote the Alpaca model trained with 13K GPT-4 contextual ranking data plus the last 39K data generated by
hhttps://openai.com/rre

---

Data Num
Model
Price
Probabilistic Ranking
52K
text-davinci-00
275$
Contextual Ranking
13K
GPT-4-0314
380$
Human
IK
1500S
Table 6: The estimated cost of calling OpenAl API and human labeling.
LR
epoch
batch size
warmup
Tuna,
le-5
128
Tunap
le-5
128
mix- mix-Tuna-52K
le-5
128
mix- mix-Tuna-1044
5c-6
128
Tuna
le-6
128
Tunaep-133 -13K
le-6
128
2
Tunacp-39K
le-6
128
2
Tunacp-52K
le-6
128
2
Table 7: The hyperparameters of training different models.
the ranking models Tuna, (PRM-3B-39K) and Tuna, (PRM-7B-39K).
input length does not seem to be issue in GPT-4 ranking. Our human experiments above also con- firm that the GPT-4 ranking closely aligned with human assessments (see Appendix G).
The results are listed Table 8. We can observe that models trained with ranking data generated by both PRMS do not achieve better results on Su- per NI compared to Tuna., The performances of Tuna. (PRM-3/7B-399 -33)) is close to Tuna, (PRM- 3/7B-52K), implying that the ranking model have learned 13K contextual ranking data well. Using larger ranking model, such as 7B, does not gain better performance, which indicates that the rank- ing ability might not necessarily scale with the pre-training model's capacity. In general, the best strategy is still to learn directly from GPT-4 con- textual ranking data, which contains less noise.
G Human Evaluation of GPT-4 Ranking
We conducted human evaluations of GPT-4 rank- ings on 50 questions used for contextual ranking. We asked annotators to rank the four system out- puts produced by our model and observe that the ranking quality by GPT-4 reasonably good (the Spearman coefficient between the human rank- ings and GPT-4 rankings is 0.72). Furthermore, we also manually inspected the explanations given by GPTT- for the ranking results. We found these ex- planations to be well-reasoned and plausible. Per- haps this is not surprising given the fact that several recent papers found GPT can be good evaluators in multiple NLP tasks (Fu et al., 2023; Gao et al., 2023). We believe the ranking feedback of this level is sufficient for guiding our model for better training (our experiments also proved this).
E
Contextual Ranking Prompt
We show the prompt that for GPT-4 contex- tual ranking in Table
F Is the Contextual Ranking Prompt Too Long?
In (Liu et al., 2023a), the authors found the "lost in the middle" phenomenon occurs at around 2K (20 documents 100 tokens/document)) tokens for GPT- 4 (note we used GPT-4 in contextual ranking). We computed the average length of the prompt (includ- ing four responses and the ranking guidelines) used in GPT-4 ranking. The average length is 650 tokens, which is significantly shorter than 2K. Thus, the
H Why Not Choose Pairwise Ranking in GPT-4 Ranking
There several reasons why ranking responses together is preferred over pairwise rankings.
API cost: Pairwise ranking for four responses requires (4 # 3) / = API calls, significantly increasing the total cost. Moreover, a loop

---

Super NI
LMentry LMentry Score
Vicuna QA
O-shot
2-shot
Win
Lose
Tie
Alpaca
36.0
44.5
31.4
Tunap
39.4 (+3.4)
43.9 (-0.6)
35.0 (+3.6)
68%
27% %
5%
Tuna, 37.7 (+1.7)
46.6 (+2.1)
32.2 (+0.8)
74%
20%
6%
Tuna, (PRM-3B-39K)
35.6 (-0.4)
40.4 (-4.1)
33.4 (+2.0)
79%
15%
6%
Tuna, (PRM-7B-33K)
(-2.5)
40.3 (-4.2)
32.5 (+1.1)
73%
20%
7%
Tuna, (PRM-3B-52K)
(-1.8)
40.1 (-4.4)
32.2 (+0.8)
75%
19%
6%
Tuna, (PRM-7B-52K)
(-1.4)
41.1 (-3.4)
32.0 (+0.6)
73%
20%
7%
Tuna
38.7 (+2.7)
45.0 (+0.5)
34.7 (+3.3)
86%
10 %
4%
Table
Performance comparison different models. The numbers in bold indicate the top-2 results. The numbers in parentheses indicate the performance difference compared Alpaca.
(e.g., RI > R2, R2 R3, R3 > RI) could oceur when R R1/2/3 are of similar qualities, po- tentially requiring extra API calls for further validation.
2. The GPT-4 ranking quality is good enough, see Appendix G.
I Comparison between Our RLHF Models and PPO-sin/PPO-sin-GPPT--0
We compare our RLHF models and PPO-sim/PPO- Sim-GPT4-20K from Dubois et al. (2023) on Vi- cuna QA. The results can be found in Table 10. -PO-sin/PPO-sin-GGTT-0000
have better re- sponses and thus we choose to report the results of their models.
J BERTScore Results on Super NI
By default, ROUGE is employed on Super NI. We additionally reported BERTScorre (which proven be better alternative for ROUGE) in Table 11 and the results follow the same trends (also see Table 1).
K Examples
We show some examples in 12.

---

Below is an instruction that describes a task, paired with an input that provides further context. Write response that appropriately completes the request.
### Instruction:
(Instruction)
### Input: (Input)
### Response:
###Response 0:
(Response 0)
###Response 1:
(Response 1)
###tResponse 2:
[Response 2)
###Response 3:
(Response 3)
We would like you to rate Response 0/1/2/3 in reply the given instruction displayed above. First, identify the instruction requires open-ended or closc-ended responses. Second, you need to generate one high quality Resoons 4' answer to the instruction. needs to have the same format as other responses and will be used as a reference later. Third, identify if there are duplicate responses and keep only one of the duplicate responses for the following steps.
Fourth, compare Response 4 with Response 0/1/2/3/4 and assign each response overall score on scale of 0 15 where a higher score indicates better overall quality. For an open-ended instruction, please rate based on the relevance (score 5), level of detailsjjusiffcatton:: (score 0 to 5) and accuracy (score to 5) each response; for a close-ended instruction, please rate based on the accuracy (score 0 to 5), level of details/justification (score to 5) and clarity (score to 5) of each response. The ratings should have the format: 'Response k: [sum of the individual scores you give to response k]'.
Last, rank the responses in decreasing order of their overall scores. The ranking should have the format: 'rank: [i. kk, 1, m]'. If there duplicate responses, keep only one of them in the rank, that is, the ranking may become: 'rank: [i, j. k. IJ', 'rank: [i. kj' 'rank: [i. jJ' or even 'rank: [i]'.
Table 9: Contextual Ranking Prompt for GPT-4.

---

Win
Lose
Tie
PPO-sim PPO-sin-GPT4-20K
79%
16%
5%
74%
22%
4%
Our PPO (w/ GPT4 ranking data)
55%
35%
10%
Table 10: Vicuna QA results of our RLHE models and
(Dubois et al., 2023).
MODEL
0-shot
2-shot
Alpaca
0.638
0.705
PPO-sim
0.594
0.652
PPO-sim-GPT4-20K
0.641
0.714
Tuna,
0.667
0,690
Tuna.
0.655
0.727
Tuna
0.665
0.699
Table II: BERTScore evaluation of the model outputs
Super NI. Higher is better.

---

Instruction:
"Why is open-source important? Answer in 3-5 sentences"
Alpaca:
Open-source software is important because it allows for collaboration and innovation. It allows developers to work together to create better software, and it allows users to access the source code and make changes to the software to meet their own needs. Open-source software also encourages transparency and allows users to see how the software works and how it can be improved. Additionally, open-source software often more secure and reliable than proprietary software, it has been tested and reviewed by large number of developers. Finally, open-source software is often free low-cost, making accessible to everyone.
Tunac:
Open-source software is important because it allows for collaboration and innovation. It allows developers to work together to improve existing software and create new software, and it allows users to view and modify the source code to customize the software to their own needs. Open-source software also encourages transparency and accountability, anyone can view and inspect the code to ensure that is secure and reliable. Additionally, open-source software is often more cost-effective than proprietary software, as users can access the source code and modify their own needs without having to pay for a license. Finally, open-source software is often more reliable and secure than proprietary software, as developed and tested by large community of developers.
suit
Tunap:
Open-source software is important because it allows developers to access the source code of program and make changes to it. This allows developers customize the software to their needs and to fix any bugs or errors that may be present. Additionally, open-source software is often free use and modify, which makes it more accessible those who may not be able afford proprietary software. Open-source software also allows developers to collaborate and share ideas, which can lead to the development of better and more efficient software. Finally, open open-source software can be used as learning tool, as it allows developers to see how the software works and how is put together. This can help them to better understand programming and coding in general.
Tuna:
Open-source software is important because it allows developers to collaborate projects, share code, and create better products. By making the source code available the public, developers can easily collaborate on projects, allowing them to create better products and services. This collaboration also allows developers to quickly identify and fix bugs, as well as add new features and functionality. Additionally, open-source software is often more secure than proprietary software, as it can be more casily audited and reviewed by the public. Finally, open-source software is often more cost-effecte cost-effective than proprietary software, as it does not require licensing fees or other costs associated with closed-source software. Open-source software is an important part of the software industry, and is used by many businesses and individuals.
Table 12: Example responses of different models.

---

Instruction: "Write Python function that generates a random number between 1 and 10 (inclusive)."
Listing I: Alpaca
def random_number(()
return random. randint(1,, 11)
Listing 2: Tuna.
def random_numbers ()
return random. randint(1, 11)
Listing 3: Tuna,
Generate random number between and 10 (inclusive)
import random
def random_number())
return random. randint(1, 10)
if namea_e__ = "__main_''
11
print cann(rroo___umbee ())
Listing 4: Tuna
Generate a random number between and 10 (inclusive)
mun
import random
def random_number().
return random. randint (1, 10)
if ___ame___ "__main_..
10
11
print (random_number ())
Table 13: Example responses of different models.


Reviews:
[{"review_id": "FRhWC1P5tF", "reviewer": "Reviewer_Eb1g", "summary": "", "questions": "A. In probablistic ranking, why using GPT-4 to rank 4 responses at the same time? considering that LLMs may [ignore the long context in the middle](https://arxiv.org/abs/2307.03172), it may be better to keep using pairwise comparison when prompting GPT-4. Need more analysis on this choice.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and constructive feedback!\n\n\n***Human Evaluation***\n\nWe appreciate your insightful comment regarding the limitations of automatic metrics like ROUGE-L or GPT-4 rating in fully capturing the quality of model outputs. In response to this, we conducted a human evaluation experiment on the Vicuna QA test set, which consists of 80 questions, during the author response period. In this experiment, responses from five anonymous systems, namely Alpaca, Alpaca + PPO-sim, Tuna, Tuna_p, and Tuna_c, were randomly shuffled and presented to annotators who were then asked to rank these outputs. The scoring was designed such that the i-th ranked system receives a score of $6 - i$, meaning the best-ranked system receives a score of 5, and the worst-ranked system receives a score of 1. Each question was annotated by two different annotators. The table below presents the average human scores of the five systems:\n\n|      | Alpaca  | Alpaca + PPO-sim | Tuna_p | Tuna_c | Tuna |\n|------|:---:|:---:| :---: | :---: | :---:|\n| **Score** | $2.13$  | $2.95^{*}$  | $2.98^{*}$  | $3.15^{*}$  | $3.80^{\\dagger *}$ |\n\n$*$ denotes that the model is significantly (p<0.01) better than Alpaca while $\\dagger$ denotes that Tuna is significantly (p<0.01) better than other systems. \n\nFurthermore, we conducted a student-t test between Tuna and all the other systems, and the results indicate that Tuna is significantly better than the other systems in comparison (p < 0.01). We found that the rankings of these five systems are consistent with the GPT-4 evaluation in our paper, except for *Alpaca+PPO-sim* (in GPT-4 evaluation, PPO-sim is slightly better than Tuna_p and Tuna_c; while in human evaluation PPO-sim is slightly worse).\n\nWe are grateful for your suggestion to incorporate human experiments into our evaluation. These results will be included in the revised version of our manuscript.\n\n\n\n***Ranking Four Responses together v.s. Pairwise Rankings***\n\nThere are several reasons why ranking 4 responses together is preferred over pairwise rankings.\n\n- API cost: Pairwise ranking for four responses requires (4 * 3) / 2 = 6 API calls, significantly increasing the total cost. Moreover, a loop (e.g., R1 > R2, R2 > R3, R3 > R1) could occur when R1/2/3 are of similar qualities, potentially requiring extra API calls for further validation.\n\n- We conducted human evaluations of GPT-4 rankings on 50 questions used for contextual ranking. We asked annotators to rank the four system outputs produced by our model and we observed that the ranking quality by GPT-4 is reasonably good (the Spearman coefficient between the human rankings and GPT-4 rankings is 0.72). Furthermore, we also manually inspected the explanations given by GPT-4 for the ranking results. We found these explanations to be well-reasoned and plausible. Perhaps this is not surprising given the fact that several recent papers found GPT can be good evaluators in multiple NLP tasks [1][2]. We believe the ranking feedback of this level is sufficient for guiding our model for better training (our experiments also proved this).\n\nWe will add the explanations above to our updated version.\n\n\n\n***Long Context (Lost in the Middle)***\n\nIn [5], the authors found the \"lost in the middle\" phenomenon occurs at around 2K (20 documents * 100 tokens/document) tokens for GPT-4 (note we used GPT-4 in contextual ranking). We computed the average length of the prompt (including four responses and the ranking guidelines) used in GPT-4 ranking. The average length is 650 tokens, which is significantly shorter than 2K. Thus, the input length does not seem to be an issue in GPT-4 ranking. Our human experiments above also confirm that the GPT-4 ranking is closely aligned with human assessments.\n\nThese disscusions will be added to the updated version of our paper.\n\n\n\n***Missing References and Typos***\n\nWe appreciate your suggestion to include references from the RL community related to behavior cloning and RL fine-tuning pipelines. We will add relevant citations [3][4] in our updated script. Additionally, we acknowledge the typos in Table 2 and are grateful for your attention to those details. These will be corrected in the updated manuscript.\n\n\n\n[1]. Jinlan Fu et al. GPTScore: Evaluate as You Desire. 2023.\\\n[2]. Mingqi Gao et al. Human-like Summarization Evaluation with ChatGPT. 2023.\\\n[3]. Stefan Schaal. Learning From Demonstration. NIPS 1996.\\\n[4]. Faraz Torabi et al. Behavioral Cloning from Observation. IJCAI 2018.\\\n[5]. Nelson F. Liu et al. Lost in the Middle: How Language Models Use Long Contexts. 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel methods to improve on standard instruction tuning approaches using ranking losses.\n- Thorough experiments validate improvements over strong baselines on multiple benchmarks.\n- Good ablation study to show that the pipeline design is necessary and optimal.\n- Well-written paper, easy to understand.", "reasons_to_reject": "- There is no human evaluation of the quality of the model outputs. The benchmarks rely on automatic metrics like ROUGE-L or GPT-4 rating which may not fully capture quality. Human judgments could reveal if the improvements translate to better responses.\n- In the probabilistic ranking approach, the authors have GPT-4 provide a single ranking for 4 generated responses to each instruction. However, [a recent paper](https://arxiv.org/abs/2307.03172) indicates LLMs tend to ignore context in the middle of prompts. The authors provide no analysis on why ranking 4 together was preferred over pairwise rankings. ", "questions_to_authors": "A. In probablistic ranking, why using GPT-4 to rank 4 responses at the same time? considering that LLMs may [ignore the long context in the middle](https://arxiv.org/abs/2307.03172), it may be better to keep using pairwise comparison when prompting GPT-4. Need more analysis on this choice.", "paper_topic_and_main_contributions": "The paper proposes methods to improve the instruction-following abilities of Alpaca that have been instruction-tuned on data Self-Instruct.\n\nThe two proposed methods are: 1) Probabilistic ranking: Using ranking loss to finetune the LLM to learn relative rankings of responses from a teacher LLM like GPT-3, to distinguish high and low quality responses. Output logits from teacher model are needed in this step. 2) Contextual ranking: Finetune the LLM to rebalance its own response distribution using rankings from a stronger LLM like GPT-4, to assign higher probability to better responses rated by GPT-4.\nThe two methods are applied sequentially to get a model called Tuna. Experiments on 3 benchmarks - Super NI, LMentry, Vicuna QA - show Tuna outperforms the base instruction-tuned LLaMA model Alpaca. The author also showed that the order of Probabilistic ranking -> Contextual ranking is optimal, other orders or combinations are not able to get better performance. Tuna also outperforms Alpaca tuned with PPO.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Captions of Table 2: raking -> ranking"}, "extra_text": {"Missing_References": "- This paper is similar to the [behavior cloning and RL fine-tuning](https://proceedings.neurips.cc/paper_files/paper/1996/file/68d13cf26c4b4f4f932e3eff990093ba-Paper.pdf) pipeline, in the robotics community.\nThe result (Tuna_{pc} is better than Tuna_{cp}) also proves that it's better to learn from the demonstration first, and then learn from the model exploration by itself. "}}, {"review_id": "OMqJxAluMG", "reviewer": "Reviewer_eUe6", "summary": "", "questions": "1. What is the basis of choosing the powerful LLM, can be explained clearly ?. Does it based on the LLM size, performance?\n2. Is any human evaluation made to verify the ranking generated by the LLM ?. \n3. What is the motivation to choose Alpca 52K instruction set for fine-tuning ?. Can explain this in the paper.  \n4. Apart from Rouge, metrics like BertScore could be considered for computing semantic similarity.\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough review and thoughtful suggestions!\n\n***On Choosing Powerful LLM***\n\nWe choose the powerful LLMs mainly based on their performance. However, the performance of a LLM is strongly correlated with its size (i.e., larger models generally generalize better than smaller models)[1][2].\n\nWe choose text-davinci-003 in probabilistic ranking, since it is the most powerful LLM with probability output; while we choose the most powerful LLM (i.e., gpt-4) in contextual ranking.\n\nThese explanations above will be added to our updated manuscript.\n\n\n***Human Evaluation on Ranking Generated by the LLM***\n\nWe conducted human evaluations of GPT-4 rankings on 50 questions used for contextual ranking. We asked annotators to rank the four system outputs produced by our model and we observe that the ranking quality by GPT-4 is reasonably good (the Spearman coefficient between the human rankings and GPT-4 rankings is 0.72). Furthermore, we also manually inspected the explanations given by GPT-4 for the ranking results. We found these explanations to be well-reasoned and plausible. Perhaps this is not surprising given the fact that several recent papers found GPT can be good evaluators in multiple NLP tasks [3][4]. We believe the ranking feedback of this level is sufficient for guiding our model for better training (our experiments also proved this).\n\nWe will add these results and analysis to our updated version.\n\n\n***Motivation to Choose Alpaca 52K***\n\nWe choose Alpaca 52K for several reasons.\nFirst, the Alpaca dataset contains a diverse set of instruction-response pairs and is widely used in open-source and research community. Second, it is free and publicly available, which facilitates reproducible research. Third, the size of this dataset is reasonably large and we observe the outputs of the resulting models look robust.\n\nWe will add these explanations to our updated version.\n\n\n***BERTScore Results***\n\nHere are the BERTScore and the original ROUGE-L results (also see Table 1 in our paper) on Super Natural Instruction (Super NI). The BERTScore results are consistent with the ROUGE-L results.\n| MODEL            | 0-shot     | 2-shot    |  0-shot ROUGE-L | 2-shot ROUGE-L |\n|--------------------|--------------|-------| --- | --- |\n| Alpaca           | 0.638      | 0.705     | 36.0 | 44.5 |\n| PPO-sim          | 0.594      | 0.652     | 31.9 | 37.5 |\n| PPO-sim-GPT4-20K | 0.641      | 0.714     | 37.1 | 44.9 |\n| Tuna_p           | **0.667**  | 0.690     | **39.4** | 43.9 |\n| Tuna_c           | 0.655      | **0.727** | 37.7 | **46.6** |\n| Tuna             | 0.665      | 0.699     | 38.7 | 45.0 |\n\nNote that we use `microsoft/deberta-xlarge-mnli_L40_no-idf_version=0.3.12` as the base model, which is recommended by the BERTScore authors.\n\nWe will add these results above to our updated manuscript.\n\n\n***Presentation Improvements***\n\nThank you for your suggestions on improving the presentation of our paper, which is very valuable and practical for us. We will make the suggested changes, such as reducing the size of Tables 1 and 2, and streamlining the general explanation of instruction tuning and probabilistic ranking, to save space and enhance readability.\n\n\n***Evaluation on Critical Reasoning and Solving Arithmetic Problems***\n\nAccording to [5], \"there exists a very complex balance/tradeoff between language models’ multi-dimensional abilities; by paying the price of decreased generic ability, we can clearly lift up the scaling curve of models smaller than 10B towards a specialized multi-step math reasoning ability\".\n\nWe may need training data containing a significant amount of complex reasoning examples to increase the reasoning capabilities of models around 7B. Unfortunately, Alpaca 52K only contains a few such examples (it is more general) and reasoning specific dataset for instruction tuning does not seem to be ready. We therefore plan to leave the training of reasoning specific models in future work.\n\n\n[1]. Alon Brutzkus, Amir Globerson. Why do Larger Models Generalize Better? A Theoretical Perspective via the XOR Problem. ICML 2019.\\\n[2]. Jared Kaplan et al. Scaling Laws for Neural Language Models. 2020.\\\n[3]. Jinlan Fu et al. GPTScore: Evaluate as You Desire. 2023.\\\n[4]. Mingqi Gao et al. Human-like Summarization Evaluation with ChatGPT. 2023.\\\n[5]. Yao Fu et al. Specializing Smaller Language Models towards Multi-Step Reasoning. ICML 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n1. The proposed approach will be helpful in improving open-source LLM fine-tuning.\n2. The fine-tuned model “Tuna” performed better as compared with state-of-the-art LLMs (LLaMA, T5-LM, InstructGPT) as illustrated in the paper.\n", "reasons_to_reject": "The performance of the fine-tuned model depends on the teacher LLM used for fine-tuning and its not clearly defined in the paper about the performance of the model on complex tasks like answering critical reasoning and solving arithmetic problems.  ", "questions_to_authors": "1. What is the basis of choosing the powerful LLM, can be explained clearly ?. Does it based on the LLM size, performance?\n2. Is any human evaluation made to verify the ranking generated by the LLM ?. \n3. What is the motivation to choose Alpca 52K instruction set for fine-tuning ?. Can explain this in the paper.  \n4. Apart from Rouge, metrics like BertScore could be considered for computing semantic similarity.\n", "paper_topic_and_main_contributions": "The paper proposed LLM fine-tuning approach using probabilistic and contextual ranking approaches to increase the likelihood of generating better responses. The authors addressed the open-source LLM finetuning issues using a powerful LLM output and claim the proposed fine-tuned model named “Tuna” perform well for many NLP tasks.  ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Space could be obtained by i) making tables (1 and 2) small, ii) reducing the general explanation of instruction tuning in section 2.1, and probabilistic ranking in section 2.2.\n"}, "extra_text": {}}, {"review_id": "wccOCnEAYe", "reviewer": "Reviewer_ANTd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments and constructive suggestions!\n\n***Dependency on Proprietary LLMs***\n\nOur approach includes two distinct steps: probabilistic ranking and contextual ranking. It is important to clarify that this method does not necessitate the use of proprietary LLMs, despite our implementation utilizing OpenAI GPT models. Specifically, the probabilistic ranking step requires a teacher LLM capable of text completion and computing associated probabilities given a text prompt, whereas the contextual ranking step merely requires an LLM with text completion capabilities.\n\nWe resorted to using proprietary LLMs in our experiments due to the substantial performance gap observed between open-source and proprietary LLMs at the time of our research. However, it is noteworthy that this disparity is progressively diminishing, as evidenced by the recent release of llama-2-70B-Chat in July 2023 and code-llama-34B in August 2023. It is anticipated that open-source LLMs comparable to GPT-3.5-turbo, at least in certain sub-domains, will be made available in the near future. These forthcoming models can be seamlessly integrated into our method.\n\nWe acknowledge the importance of this discussion and will include it in the revised version of our manuscript.\n\n***On Utilizing GPT-4 as Automatic Evaluation Metric***\n\nThanks for analyzing the pros and cons of utilizing GPT-4 as automatic evaluation as well as kindly recommending a feasible solution on human evaluation! We are grateful for this. We agree that human is not fully replaceable by larger LLMs. Therefore, we added two human experiments.\n\n*1. Human Evaluation on Vicuna QA* \n\nWe conducted a human evaluation experiment on the Vicuna QA test set, which consists of 80 questions, during the author response period. In this experiment, responses from five anonymous systems, namely Alpaca, Alpaca + PPO-sim, Tuna, Tuna_p, and Tuna_c, were randomly shuffled and presented to annotators who were then asked to rank these outputs. The scoring was designed such that the i-th ranked system receives a score of $6 - i$, meaning the best-ranked system receives a score of 5, and the worst-ranked system receives a score of 1. Each question was annotated by two different annotators. The table below presents the average human scores of the five systems:\n\n|      | Alpaca  | Alpaca + PPO-sim | Tuna_p | Tuna_c | Tuna |\n|------|:---:|:---:| :---: | :---: | :---:|\n| **Score** | $2.13$  | $2.95^{*}$  | $2.98^{*}$  | $3.15^{*}$  | $3.80^{\\dagger *}$ |\n\n$*$ denotes that the model is significantly (p<0.01) better than Alpaca while $\\dagger$ denotes that Tuna is significantly (p<0.01) better than other systems. \n\nFurthermore, we conducted a student-t test between Tuna and all the other systems, and the results indicate that Tuna is significantly better than the other systems in comparison (p < 0.01). We found that the rankings of these five systems are consistent with the GPT-4 evaluation in our paper, except for *Alpaca+PPO-sim* (in GPT-4 evaluation, PPO-sim is slightly better than Tuna_p and Tuna_c; while in human evaluation PPO-sim is slightly worse).\n\n*2. Human Evaluation on GPT-4 rank in contextual ranking* \n\nWe conducted human evaluations of GPT-4 rankings on 50 questions used for contextual ranking. We asked annotators to rank the four system outputs produced by our model and we observe that the ranking quality by GPT-4 is reasonably good (the Spearman coefficient between the human rankings and GPT-4 rankings is 0.72). Furthermore, we also manually inspected the explanations given by GPT-4 for the ranking results. We found these explanations to be well-reasoned and plausible. Perhaps this is not surprising given the fact that several recent papers found GPT can be good evaluators in multiple NLP tasks [1][2]. We believe the ranking feedback of this level is sufficient for guiding our model for better training.\n\nWe will add these results and explanations to the revised version of our manuscript.\n\n\n\n***Unfair Model Evaluation***\n\nActually we spent a significant amount of time to implement our own PPO (using the same GPT4 ranking data as Tuna_c to train the reward model). However, the performance is worse than both PPO-sim and PPO-sim-GPT4-20K from the public available AlpacaFarm [3](https://crfm.stanford.edu/2023/05/22/alpaca-farm.html). See the results on Vicuna QA below. We are convinced that the PPO-sim and PPO-sim-GPT4-20K implementations from AlpacaFarm are good. We therefore decide to run their models on the three benchmarks and report these results in the Table 1.\n\n|                                | Win | Lose | Tie |\n|--------------------------------|-----|------|-----|\n| PPO-sim                        | 79% | 16%  | 5%  |  \n| PPO-sim-GPT4-20K               | 74% | 22%  | 4%  | \n| Our PPO (w/ GPT4 ranking data) | 55% | 35%  | 10% | \n\nAlthough AlpacaFarm PPOs and our PPO all leverage the Alpaca dataset for training, our PPO implmentation is worse perhaps because of the following reasons.\n\n1) PPO-sim leverages annotations from multiple LLMs (i.e., GPT4, GPT3.5-turbo and text-davinci-003) to train their reward models.\n2) AlpacaFarm used separate splits for the three stages (i.e., sft, reward modeling training and RL) of PPO.\n3) Perhaps better hyper-parameter tuning. There are many hyper-parameters in PPO and four models need to be trained in PPO.\n\nThanks again for the insightful comments and we will add these numbers and discussions above to our revised manuscript.\n\n***Missing Reference***\n\nWe will cite Mallen et al. (2023).\n\n***Low Reproducibility***\n\nLast but not the least, we plan to release our models, code and data (i.e., GPT annotations from both Probabilistic Ranking and Contextual Ranking) as stated in our original manuscript (line 29 ~ 30) to help other researchers to reproduce our work.\n\n[1]. Jinlan Fu et al. GPTScore: Evaluate as You Desire. 2023.\\\n[2]. Mingqi Gao et al. Human-like Summarization Evaluation with ChatGPT. 2023.\\\n[3]. Yann Dubois et al. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tThe experiments are reasonably executable, excluding the the utilization of propietary LLMs for researchers without API access.\n-\tThe paper uses ablation study to validate the proposed method or framework, including varying different size of finetuning data and briefly discussing the risk of GPT4-based automatic evaluation.", "reasons_to_reject": "My concerns of this manuscript are mainly two:\n\n- The framework is highly dependent on propietary LLMs, which the authors have also mentioned in Limitation section.\n\nThe Student Model is forced to align with the preference of Teacher Model (GPT4), even though the risk of utilizing GPT4 as automatic evaluation metric is known, which authors have also discussed in the Results section. The risk, however, remains an open question. The pros is that the estimated cost for the API is less expensive than crowd-sourcing human. However, it does not necessarily mean that human is fully replaceable by larger LLMs. A feasible solution of those trade-offs is that authors can run human evaluation on few samples in which larger LLM performs differently (unfaithful vs faithful outcomes). For example, on samples where larger LLM has high performance (lower loss) versus samples where LLM has low performance score (higher loss).\n\n- The model evaluation is rather unfair.\n\nThe authors compare between (i) models that are not aligned with GPT4 preference and (ii) aligned models (Student Models). So, intuitively the aligned models receive more support and can achieve a better performance results, particularly on held-out benchmark datasets in which GPT4 (Teacher Model) performs reasonably well. RLHF methods are taken directly from previous work’s results (Dubois et al., 2017), so it is not clear whether the resulting scores of PPO methods in Table 1 will be same if they are run on the same computing resource with the remaining models in the paper.  \n\nSide note: \nAn example of a more fair evaluation would be: comparing the performance between (i) models that is finetuned with human alignment ranking score vs. (ii) models that is finetuned with larger LLMs scoring. This human alignment setup might not be suitable for large scale finetuning data, so sampling on a smaller pool of data is a necessary precondition.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a new framework for finetuning Instruction Tuning model (Student Model) with a feedback score from Large Language Models (LLMs) as Teacher Model. The framework consists of two ranking methods for diversifying the generation outputs of Student Model (smaller LLM). \n\nThe first method is referred to as Probabilistic Ranking, where Teacher Model (proprietary  GPT4) generates N-ranked diverse responses based log likelihood ranking. Then, Student Model uses the generated examples as ground truth to finetune the model on Instruction Tuning held-in data/task with new loss function: the aggregation of rank-based loss and weighted MLE Cross Entropy loss. The rank-based loss is defined as delta or the gap between the log likelihood of low rank responses versus high rank responses. The ranking loss is adopted from previous study (.., Zhao et al., 2023).\n\nThe second method is referred to as Contextual Ranking. That is, Student Model generates N-diverse responses by utilizing ROUGE-L diversity measure. Then, the generated responses are being evaluated by Teacher Model (GPT4). GPT4 is asked to rate the quality of reponses based on evaluation aspects (relevance, level of details, accuracy, etc) of two type of NLG objectives: Open-Ended and Close-Ended text generation.\n\nThe framework also explores the ablation and the ensemble of the two ranking methods as a continual process.\n\nThe main contribution of the paper is NLP Engineering experiment.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802–9822, Toronto, Canada. Association for Computational Linguistics."}}]

Author rebuttal:

Meta-review: The paper proposes methods to improve the instruction-following capabilities of the Alpaca model (learned through distilling from OpenAI models via self-instruct). The proposed method includes probabilistic ranking and contextual ranking that are sequentially applied to finetuning LLMs. The proposed method is evaluated on Super NI, LMentry, and Vicuna QA datasets and demonstrate its effectiveness.
Decision: Accept-Findings