Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
Authors: Hritik Bansal
John Dang
Aditya Grover
Venue: ICLR 2024 poster (2024)
Keywords: LLMs
Sparse Feedback
Ratings
Rankings
Inconsistency
Evaluation
Primary area: generative models
TL;DR: We investigate how the choice of sparse feedback, such as ratings and rankings, impact the alignment and evaluation of large language models.

Abstract: Aligning large language models (LLMs) with human values and intents critically involves the use of human or AI feedback. While dense feedback annotations are expensive to acquire and integrate, sparse feedback presents a structural design choice between ratings (e.g., score Response A on a scale of 1-7) and rankings (e.g., is Response A better than Response B?). In this work, we analyze the effect of this design choice for the alignment and evaluation of LLMs. We uncover an inconsistency problem wherein the preferences inferred from ratings and rankings significantly disagree 60% for both human and AI annotators. Our subsequent analysis identifies various facets of annotator biases that explain this phenomena such as human annotators would rate denser responses higher while preferring accuracy during pairwise judgments, for a particular comparison instance. To our surprise, we observe that the choice of feedback protocol has a significant effect on the evaluation of aligned LLMs. In particular, we find that LLMs that leverage rankings data for alignment (say model X) are preferred over those that leverage ratings data (say model Y), with a rank-based evaluation protocol (is X/Y's response better than reference response?) but not with a rating-based evaluation protocol (score Rank X/Y's response on a scale of 1-7). Our findings thus shed light on critical gaps in methods for evaluating the real-world utility of language models and their strong dependence on the feedback protocol used for alignment. Our code and data are available at \url{https://github.com/Hritikbansal/sparse_feedback}.

Published as a conference paper at ICLR 2024
PEERING THROUGH PREFERENCES: UNRAVELING FEEDBACK ACQUISITION FOR ALIGNING LARGE LAN-
GUAGE MODELS
Hritik Bansal, John Dang, Aditya Grover Department of Computer Science, University of California Los Angeles (hbansal, john. dang, adityag] ucla. edu
ABSTRACT
Aligning large language models (LLMs) with human values and intents critically involves the use of human or Al feedback. While dense feedback annotations are expensive to acquire and integrate, sparse feedback presents a structural de- sign choice between ratings (e.g., score Response A scale of 1-7) and rank- ings (e.g., Response better than Response B?). In this work, we analyze the effect of this design choice for the alignment and evaluation of LLMss. We uncover an inconsistency problem wherein the preferences inferred from ratings and rankings significantly disagree 60% for both human and AI annotators. Our subsequent analysis identifies various facets of annotator biases that explain this phenomena such as human annotators would rate denser responses higher while preferring accuracy during pairwise judgments, for particular comparison in- stance. To our surprise, we observe that the choice of feedback protocol has a significant effect on the evaluation of aligned LLMs. In particular, we find that LLMS that leverage rankings data for alignment (say model X) are pre- ferred over those that leverage ratings data (say model Y), with rank-based evaluation protocol (is X/Y's response better than reference response?) but not scale
with rating-based evaluation protocol (score Rank X/Y's response 1-7). Our findings thus shed light on critical gaps in methods for evaluat- ing the real-worid utility of language models and their strong dependence on the feedback protocol used for alignment. Our code and data are available at https://githhnn cow Catom//riiiii http://www......e.aaaaaaaarareae
INTRODUCTION
Recently, alignment has emerged as a critical step for next-generation text-based assistants [35] Specifically, its goal is to align large language models (LLMs) with human values and intents, mak- ing their generated content accurate, coherent, and harmless when responding to human queries (26 2 38. The process of model alignment involves three main components: feedback acquisi- tion, where humans (or AI) assess the quality of the base model's responses; alignment algorithms, which adjust the skills of the base model based on the feedback data; and model evaluation, which assesses the performance of the aligned model on a wide range of novel user instructions (18). Prior work has primarily focused on designing alignment algorithms, such as PPO, DPO, and PRO 45 19,, under specific feedback protocols and evaluation setups. Additionally, pre- vious research on feedback acquisition 6 43 30 has focused on developing fine-grained and dense feedback protocols for aligning I.LMs; however, these protocols are challenging and expensive to acquire. Within the sparse feedback protocols that are easy to acquire, there a structural design choice between ratings and rankings, and its impact the alignment pipeline is still under-explored.
31
23
28
33
To this end, we analyze the effect of the two feedback protocols: ratings and rankings on the LLM alignment and further evaluation. Specifically, the rating protocol is an absolute form of feedback in which the annotator assigns a rating to a response from the base LLM using a predefined scale (e.g.. 1-7 Likert scale (00). In contrast, the ranking protocol is relative form of feedback in which the annotator selects their preferred response from a pair candidate responses generated by the base LLM. The ratings the model responses quantifies their goodness which enables model

---

Published as a conference paper at ICLR 2024
Feedback Acquisition
Instruction
Human Annotator
Creste a list of three beach activiies
Annotator
Candidate Responses
Rankings
Ratings
surf. swin, sand cestes
Aikee, climo, surting
resporee
from
6/7
sunbathe, surt. SWAM
owim, parasel) Sunbelhe
417
LLM Alignment
Evaluation
Annotator
Aligned LLM g. best-of-n)
Reference LLM Respories
t+
Al Annotator
Rankings
Ratiigg
X 3/7
Aligne LLM Responsse
Reward
8/7
Model
Figure 1: Overview of our pipeline study the effect of the choice between sparse feedback pro- tocols (ratings and rankings) for the alignment and evaluation of LLMs. First, we sample multiple responses from the LLM for the queries the instructions dataset. Then, we acquire rankings and rating feedback data from the human and AI annotators, independently. Subsequently, the feedback data is used to train the reward models for the Best-of-n policy. Finally, we compute the win- against the reference model under ratings and rankings feedback protocol from humans and Al.
builders to gauge their strengths and weaknesses, but are hard to determine for complex instructions (poem 'Roses and Lake in Shakespearean style') 19 32]. On the other hand, the rankings protocol is easy to acquire for complex instructions but does not quantify the gap between the pair of
responses [35] 26 Due to these unique behaviors, both the ratings and rankings feedback protocols
are compelling subjects for our study.
In our work, we first explore the interplay between these feedback protocols. To do so, we collect ratings for individual responses from annotators, both human and AI, and transform them into their ranking form by comparing the assigned ratings (e.g., Response A is rated and Response rated 4, this implies that Response A is ranked higher than Response B). Additionally. we gather pairwise ranking preferences for the same responses (e.g., comparing Response A vs. Response B) uncover a feedback inconsistency
from annotators, independently of their ratings. Surprisingly, problem and find that significant portion of all preference comparisons between ratings and rank- ings disagree with each other for humans 58% and AI annotator -urrb))) 59.4% ($4). We observe that responses receiving inconsistent feedback are actually perceived as closer to each other than those receiving consistent feedback. Qualitatively, in how annotators assess various facets of response quality (SI.T). I.. Our analysis provides insights for making decisions regarding feedback acquisition protocols for aligning LLMs.
trace this problem back the differences
After receiving feedback, our objective is to investigate the impact of the protocols on training reward models for model alignment and subsequent model evaluation. To achieve this, train reward models on the ratings and rankings data ($2.3). We employ the Best-of-n policy where the trained reward models are then employed select the best response from set of n candidate responses generated by the base LLM. Subsequently, we assess the quality of the Best-of-n policies (both ratings and rankings) by evaluating their responses against reference model 26 To our surprise, we observe evaluation inconsistency, phenomenon where the choice of the feedback protocol (rankings) for model evaluation favors responses from the Best-of-n policy that utilizes the same feedback protocol (rankings), and vice versa. Additionally, we find that feedback inconsistency occurs for both human and AI as response evaluators. Specifically, we observe that the win-rate gap between the Best-of-n (rankings) policy and the base LLM is larger (by 11.2%) compared to the gap between the Best-of-n (ratings) policy and the base LLM (5.3%) when using human rankings for evaluation (see $.2)). However, the win-rate gap between the Best-of-n (ratings) policy and the base
2

---

Published as a conference paper at ICLR 2024
LLM (5%) is only slightly larger than the gap between the Best-of-n (rankings) policy and the base LLM (4%) using human ratings for evaluation. Figure provides an illustration of our pipeline.
Our contributions are as follows:
1. We uncover feedback inconsistency problem where the ratings the pair of responses disagree with the ranking between them for 60% of comparisons for both humans and Al.
2. We further analyze various facets of perceived response quality by the annotators while providing different forms of feedback.
3. Finally, find that the choice of the feedback protocol has sharp influence the eval- uation of the aligned LLMs the form of evaluation inconsistency.
2 BACKGROUND
In this work, we focus on aligning an LLM to generate high-quality outputs, that are considered accurate, coherent and harmless by humans, for unseen instructions. The initial step in the align- ment process is to equip the base LLM with the ability to understand human or muchine-generatee input queries or instructions. This is accomplished through supervised fine-tuning (SFT), where the base L.M is fine-tuned with pairs of human-written or machine machine-generated instructions and corre- sponding responses. Notably, there have been substantial advancements in constructing instruction 13
fine-tuning datasets and models recently [36 49 42 40
IO 41]. Following SFT, the model generates candidate responses for new instructions 2. and feedback data acquired
27
44
46
under protocol (ratings rankings) from the annotators ($ 2.2 Further, employ an alignment
algorithm (Best-of-n) that trains reward models on the ratings or rankings feedback data ($ 2.3
2.1 IIORUCTION-RESPONTE DATA COLLECTION
Consider a language model Pa that can understand instructions and respond to them. consider a set of instructions X' (.I,..., where is the number of instructions. Subsequently, we generate a set of & candidate responses {M, (1) .... (x) for every instruction z; by sampling from the model's distribution v/") po(,)x;). Next, we collect the feedback data on the pre-defined instructions and the generated candidate responses = {(x;, {ut .9. })}.
2.2 FEEDBACK DATA
Ratings Feedback Protocol With the given instruction and candidate response data D, the objec- tive of ratings feedback acquisition is to assign ordinal value to each individual response inde- pendently. In our study, we request the annotators to rate the responses on a Likert scale (1 7). We instruct the annotators to evaluate the responses based on the their quality considering factors such as accuracy, coherence, and harmlessness of the response in addition to their subjective judgment.
Formally, the annotators assign an absolute score a(x;; v) E (1,,2...,, 7} where and y are instruction and a generated candidate response, respectively. The annotation process would thus create feedback data D {(I., v a(x,, (. ((") where A represents the ratings protocol.
Rankings Feedback Protocol Given the instruction and candidate response data D, the aim of rankings protocol is to assign preferred response (or choose 'equal') for every pair of candidate responses to a given instruction. To that end, we first transform the instruction-ressonse data into instruction-ppairiise response data by creating pairwise combinations of the responses i.e., D'
{(x,, y. yi ) where + L. Identical to the absolute feedback, the annotators are instructed to the various quality axes and their subjective judgment for decision-makee maaking.
Formally, the annotators assign a relative feedback r(x,, v (8) E ('response 1', 'response "equal") where 'response I' indicates that Yrg is preferred over y; The annotation process would
(e)
thus create a feedback data Di = {(x, (i) (2) r(x, y. y sents the rankings feedback protocol.
(2) ))} where the subscript repre-
3

---

Published as a conference paper at ICLR 2024
2.3 REWARD MODELING
We describe the training objectives for the reward models trained on ratings and rankings feedback.
Regression Reward Model. Here, we first normalize the ratings (between 1-7) a(Ti, ()) for given instruction in the dataset DA into score a' (z,, v( between 0-1. Subsequently, we train regression model So (3 ) where (vi, vf DA which outputs scalar. The regression reward model is trained with the following objective:
(D( = = ",
d' yi N
where (.) is the sigmoid function which projects any scalar value to range between 0-1.
Negative Log-Likelihood (NLL) Reward Model. To train a reward model from the pair- wise feedback data DR. prior works (26) train a reward model go(x, y) which outputs a scalar value for given response for the instruction x. Firstly, we filter the instances that receive r(z;, ,v,"" =eequal" from Dn to get new subset of data Sn since one cannot learn from such examples the relative reward modeling objective. For the remaining instances, the negative log-like og-likeeihood objective might be applied as follows:
(n(SR) = St Dlg (go (Ii, Ua) (zs, mb)
(2)
where Va and y/s are the preferred and unpreferred responses respectively from the pair of candidate responses (5) y4 (8) as decided by the value of rxx;, ys yy (1) in the dataset Sp. We provide more details on the choice of fo. and the training setup in Appendix G
Best-of-n Policy. In our work, we use the Best-of-n policy (rejection sampling) Pa, that leverages the trained reward model to boost the performance of the SFT model towards human preferences. In this method, we simply sample n times from the SFT model from given instruction, and utilize the reward models (regression NLL) to score the set of responses. The final output of the policy is the response that achieves the highest score under the trained reward model. Formally the Best-of-n policy that leverages the ratings reward model fo.
Pulfo, Po. vss (8)"
(1 (m)
(3)
where argmax, So (.("). ), on is the input instructions, and y; is jth response from the candidate responses sampled from the SFT model Here, our approach can use any other RL algorithm such as PPO used for model alignment [26] or mix of rejection sampling and PPO (38). We focus on rejection sampling due to its simplicity, and robust performance in comparison to other RL algorithms as noted by prior works 23
FEEDBACK DATA ACQUISTION
Here, we describe the feedback data acquisition process. We start by collecting responses to large set of instructions from the base LLM (S. Subsequently, we describe feedback acquisition from AI which is large-scale and thus used for training the reward models. Further, we describe feedback acquisition from humans followed by feedback data analysis.
Instruction Response Data. The instructions are designed to present collection of queries that could potentially be posed to a text-based AI assistant in real-world scenarios. We collect 5.2K instructions from varied sources such Dolly [0]. User-orient 42 and SuperNI 42
In our work, we use Alpaca-7B [6), which is constructed by instruction-tuning the LLaMA-7B 37] 52K instruction-respone candidate responses for a range of diverse instructions. We set the max length for each generated response to 128. We provide more details the dataset composition in Appendix şC
data, the base LLM. Thus, we prompt Alpaca generate five
4

---

Published as a conference paper at ICLR 2024
Feedback from AI. Prior works [2] demonstrate that the existing AI systems (LLMs) such as GPT-4 [25] can be leveraged for providing pairwise feedback data and improve over the base LLM. In addition, 49 [4 makes case for LLM as a substitute for human preferences in scalable and reliable way. In our work, we collect large-scale ratings and rankings feedback data consisting of 71K instances from GPT-3.5-Turbo (ChatGPT). We chose ChatGPT since is 20x cheaper than GPT-4 and was more easily accessible at the time of the project.
To collect ratings feedback, we prompt GPT-3.5-Turbo to assign a score between 1-7 to the indi- vidual candidate responses for the instructions independent of each other. The model is specifically prompted to assign the ratings score after evaluating the response for its accuracy. coherence, and harmlessness, in addition to its subjective judgment. We further clarify that a rating of 1 implies a low quality response whereas a rating of 7 indicates a high quality response. In total, we collect 24.6K instances of ratings feedback and spend $150 in the acquisition.
To collect the rankings feedback, we prompt the GPT-3.5-Turbo LLM with the instruction and a pair of candidate responses ("response I', 'response 2)), and command it to provide its preference between the two responses. To mitigate any potential bias in 'response and 'response 2' ordering [4)] we run two queries for all instances covering both the possible orderings. If the model pref- erences flip by flipping the order of the responses, then we consider it as tie situation and assign an "equal" feedback to the pair of responses, done in [5]. In total, we collect 46.5K unique instances of rankings feedback, excluding the two possible orderings of the pair of candidate responses, and spend $600 in the acquisition. We will provide more analysis the feedback data in 331
Feedback from Humans. We collect feedback data un- der the ratings and rankings protocols from the humans 6K instances of annotations. We provide the details on the human preference acquistion in Appendix SD Such
GPT-3-5-Tarbo
UPT-3.5--urrbo
Human
data serves a multitude of purposes, notably: (a) provid-
ing insights into the behavior of different forms of feed-
40
Percentage
back collected from humans, (b) facilitating a compar- ative analysis of the similarities and dissimilarities be- tween feedback patterns derived from Al systems and those obtained from human participants, and (c) enabling a comprehensive assessment of the agreement between the feedback data generated by Al systems and the feed- back provided by human participants. In our work, we emphasize that the human feedback data is not used for training the reward models, and hence model alignment, due to its small scale which is costly to expand.
30
20
0
2
3
4
7
Absolute Scone (2-7)
Figure 2:
We present the ratings (ab-
solute) score distribution acquired from humans and AI.
3.1 DATA ANALYSIS
Ratings Distribution. We present the score distribution for the 4K ratings data from the human annotations and their corresponding GPT-3.5-Turbo anno- tations in Figure? 2 We find that the majority of the responses (~ 80%) achieve better than average ratings (> 4) from the humans and AI. This indicates that the quality of responses from Alpaca-7B good, attributed its instruction-follovii
capabilities. In addition, we find that the human anno- tators tend to give perfect score of 7 to large proportion of the responses, whereas GPT-3.5-Turhh assigns the perfect score to less than < 5% cases. We observe that the majority of responses achieve an ratings score of or 6 from GPT-3.5, and the distribution flattens to either side of these scores (same as Fig. 2 in (27)
41 have shown that LLMs may favor longer and verbose responses, to varying degrees, despite being in- accurate and of lower quality. We assess whether the length number of unique words in the candidate responses bias the ratings and rankings feedback judgments from humans and AI in our find that there is discernible difference between
Feedback data is unbiased towards longer and unique responses. Prior works 49
data (Appendix Table 5 and 6) In summary, the average length and average number unique tokens of the preferred and unpreferred response in the rankings feedback collected from the humans and AI.
5

---

Published as a conference paper at ICLR 2024
Agreement Analysis. Here, conduct an agreement analysis comparing the assessments pro- vided by human-human and human-Al annotators for both ratings and rankings feedback. Specif- ically, have collected 1000 instances with ratings feedback and 500 instances with rankings feedback from four human annotators and GPT-3.5-Turde instance, consider the first three human annotations the gold label, and we use the fourth notation as the human prediction label. For instances with ratings feedback, we compute the average of the scores given by the three human annotators and round it off to the nearest integer and consider it as the gold label. For instance, with rankings feedback, we determine the final label based on the majority vote among the three human annotations. The possible choices for human feedback responses are ('response I', 'response 2', 'equal"). J. If an instance receives two votes for 'equal' and one vote for "response I', we assign it gold label of "equal'. In the event of a tie among the three choices, we randomly sample the gold label from the possible choices.
To establish the ground truth for each
We calculate the average ratings difference between the gold label feedback and human (and AI) feedback quan- tify the ratings agreement. We calculate the percentage of instances (out of 500) where the gold label and human (and AI) feedback agrees with each other to quantify the rankings agreement. Specifically, we assign a score of 1 the gold label matches the prediction, and a score of 0.5 only one of the gold label or prediction assigns 'equal' to the pair of responses. In Table [ we observe that the average ratings difference between human-human ratings ratings (0.9).
Feedback Ratings Rankings
H-H
H-AT
Examples
1.08
0.9
1000
62.7%
60.5%
500
Table 1: Agreement analysis between the gold human annotations and the held-out human (H-H) annotations for ratings and rankings feedback (Column 1). Similarly, we perform the agree- ment analysis for the gold human anno- tations and annotations from GPT-3.5- Turbo (abbreviated as H-AI).
(1.08) is close to the Additionally, we observe that the numan-humes agree- ment and Duman-GPT-335--TUmm 60.5% respectively in the rankings feedback. We observe
agreement is 62.7% and
that our agreement rates are close to the 60% human-human agreement rates reported in the prior works [88.
4 FEEDBACK INCONSISTENCY PROBLEM
4.1 CALCULATING CONSISTENCY
Here, we use the observation that the rankings of pair of responses for a given instruc- tion can be compared to the ratings feedback data converted into its ranking form. Formally, we can convert the ratings data DA to D (ar, v.) . ,h(a(z,, (,"(),,,)))), a(x, y/") (()))) where h(a(x, ( a(x,, ())) E ('response 1', 'response 2', 'equal'} denotes the preferred response between y/ (response 1) and y/ (response 2). Here, we define the term feedback consistency as the agreement between the ratings (converted to its rankings form) and the rankings received by pair of responses to a given instruction. Formally,
(0) {{ C(s,, ,
r( otherwise
M( oi ,38 (0)
(4)
where (x,, ( ..."v.)))))) (9) S and S = DD Dn-
4.2 RESULTS
We present the results across 42K comparisons in the feedback data from GPT-3.5-Turbo (Table 2ap and 500 comparisons in the feedback data from the humans (Table 2b). In both tables, particular cell (say row 1 and column 2) will indicate the percentage of total instances for which the ratings feedback considers the pair of candidate responses 'equal' while the rankings feedback prefers 'Re- sponse 1 for the same pair of candidate responses.
We observe an inconsistency issue in both human and AI feedback data. That is, the consistency score falls within similar range of 40% - 442% for both humans and AI, suggesting that substantial

---

Published as a conference paper at ICLR 2024
Rankings Resposse
Rankings
Equal
Response
Total
Equal
Response
Resgonse
Total
Equal
27.7%
14.7%
15%
57.4%
Equal
13.3%
12.4%
15.1%
40.8%
Ratings
Response
21.6%
Ratings
Response
13.18
29.1%
Response
9.7%
4.4%
6.9%
21.0%
Response
9.3%
6.4%
14.2%
30.0%
Total
47.1%
26.5%
26.1%
125
Toeal
30.9%
32%
37.1%
40.6%
(a) Results for the (dis- Jagreements between feed- back protocols (ratings and rankings) annotated by GPT-3.5-Turbh for 12K comparisons.
(b) Results for the (dis-) Jagreemnest between feed- back acquisition protocols (ratings and rankings) an- notated by humans for 500 comparisons.
Table 2: Inconsistency results for the feedback data acquired from (a) Al and (b) humans.
portion (60%) of the feedback data can yield contradictory preferences depending on the feedback protocol employed. This consistency problem underscores several critical points: (a) it indicates variations in the perceived quality of responses based on the choice of the feedback acquisition protocols, (b) underscores that the alignment pipeline can vary significantly depending on whether ratings or rankings are used as sparse forms of feedback, and (c) it emphasizes the necessity of meticulous data curation when working with multiple feedback protocols for aligning LLMs. In Appendix SE we discuss the variation in the consistency scores with variation in the annotators.
Hedging. We find that the GPT-3.5-Turbo tends to hedge its predictions significantly more than humans. Specifically, 17.1% rankings feedback and 57.4% ratings feedback are assigned an "equal' preference by GPT-o.5-Turbo (sum of first row and column in Table 2ap. However, 30.9% rankings and 40.8% ratings feedback is assigned an 'equal' preference by the humans. We observe that the hedging percentages are higher for the ratings feedback as compared to the rankings feedback for both humans and AI. This can be attributed to a large fraction of individual responses receiving score of or from 1PT-3.5--Turbo discussed in $3.1
Decisive rankings feedback. We observe that whenever the rankings feedback from humans and GPT-3.5-Turbo is decisive (i.e., the feedback prefers one of the responses from the pair instead of selecting 'equal"), the fraction of consistent assignments is higher than the inconsistent ones. In Table 2a 7.4% (Column and Row 2) higher than 4.4% and 6.9% is higher than 4.5%. Similarly, in Table 2b 13.1% (Column 2 and Row 2) is higher than 6.4% and 14.2% is higher than 7.8%.
Decisive ratings feedback. We observe that whenever the ratings feedback converted its rank- ings form is decisive, the fraction of consistent assignments is higher than the inconsistent ones. Specifically, we focus on Row 2 and Row 3 of the tables. In Table 2a 7.4% (Column 2 and Row 2) is higher than 4.5% and 6.9% is higher than 4.4%. Similarly, in Table 2b 13.1% (Column 2 and Row is higher than 7.8% and 14.2% higher than 6.4%.
Quantitative Fine-grained Analysis. We perform fine-grained experiments to quantify the dif- ference between consistent and inconsistent subsets of the feedback data from humans and AI in Appendix J.I In summary, we find that the quantitative gap between the pair of responses belong- ing to the consistent instances is higher than the pair of responses belonging instances. Due to the closeness in the perceived quality of the pair of responses belonging to the inconsistent instances, we posit that the human annotators prefer specific quality factors over others (e.g.. focus on accuracy more than the coherence) during decisive decision making. As re- sult, we observe that the inconsistent instances receive more variation in the rankings feedback from the humans (c.g., annotator 1,2 prefers Resp. A over B while annotator 3 prefers Resp. B over A for a given instruction) than the consistent instances (e.g., annotator 1,2,3 prefer Resp. A and Resp. B).
the inconsistent
Qualitative Analysis. We investigate the source of the inconsistency problem qualitatively by asking the human annotators for a few feedback instances. We include the ratings and rankings with human explanations in Appendix P and find that the differences in the preferences of the humans while annotating for different feedback protocols played a significant role in their decision making. For instance, in Table I2 observe that the annotators responsible for rankings favored 'response 2' due to its density, while perceiving 'response I as dull. At the same, despite these divergent preferences, the individual responses received conflicting rating scores: 'response 2' incurred
7

---

Published as a conference paper at ICLR 2024
Porformance of SFT and Best Best-of-64 Policy (Ratings and Rankings)
55
SFT
of 64 (Ratings)
Win-rate against Davindi: 003
Best of 64 (Rankings)
40
35
30
25
15
Rankings
Rankings
Ratings
Ratings (GPT-3.5)
(Human)
(GPT-3.5)
tHumani
Feedback acquisition protocel during evaluation (Annotator)
Figure 3: Win-rate against DaVinci-003 of Alpaca-7B LLM and Best-of-64 policy that leverages the reward models trained with the ratings and rankings feedback protocols, independently. The gap between base LLM and Best-of-64 (Rankings) is higher than base LLM and Best-of-64 (Ratings) using the rankings feedback protocol for model evaluation. However, the gap between base L.MM and Best-of-64 (Ratings) is slightly higher than the gap between base LLM and Best-of-64 (Rankings) using the ratings feedback protocol for model evaluation. The trend holds for humans and GPT-3.5- Turbo the evaluators. The error bars represent the 95% CI.
substantial penalty for its density and perceived non-compliance with task instructions, whereas response I' garneredd average score for only a partial adherence the guidelines.
ALIGNMENT AND EVALUATION
In the previous sections, we established that the feedback data obtained from both humans and AI suffer from an inconsistency problem. Nonetheless, it is crucial to examine the impact of these forms of feedback on aligning and evaluating language models.
5.1 SETUP
First, we train reward models on feedback data acquired under the ratings and rankings protocol. Then, we employ Best-of-n (n 64) policy evaluate the effect of various feedback data model alignment. Here, we vary the choice the feedback protocols as ratings and rankings dur- ing evaluation and the choice of the annotators as humans and ChatGPT-3.5-Turfe
Specifically,
we choose low-rank adaptation [5] (LoRA) of Alpaca-7B as the reward model Post-training, evaluate the models on 553 unseen instructions, which includes 188 instructions from the Open Assistant (OASST). 129 from the helpful evaluation released by Anthropic [3) 80 from Vicuna [8), and 156 from Koala [3.. Within Best-of-n, we first generate n candidate responses of length 200 from the base LLM (Alpaca-7B) with the unseen instructions. Subsequently, the output of Best-of-n would be the one that achieves the maximum score with the trained reward model. Finally, we com- pute the win-rate of the base LLM and the Best-of-n policies against Da Vinci-003 (reference model) on these unseen instructions. Here, we use the Da Vinci-003 responses collected AlpacaEval [18). Post-alignment, we evaluate the Best-of-n policies (ratings, rankings) against reference model on a set unseen user instructions 2. We provide more details on the training setup in G
5.2 RESULTS
We train the reward models on the ratings and rankings feedback data collected from GPT-3.5- We present the win-rates for the base LLM and the Best-of-n policies the unseen instructions evaluated by humans and GPT-3.5-Turbo Figure]
htt 1ora
8

---

Published as a conference paper at ICLR 2024
Best-of-n policies outperforms SFT. We find that randomly sampling from the (= 64) candi- date responses from the SFT model (labeled as "SFT" in the figure) achieves a win-rate of 33.9% against Da Vinci-003 using pairwise judgments from the humans. Further, we observe that the Best- of-64 achieves a win-rate of 39.2% and 45.1% 1% against Da Vinci-003 using the reward models trained with ratings and rankings feedback, respectively. We observe similar trend for the reward models trained with the ratings feedback using humans and AI as the evaluators. This indicates that the feedback collected scale from AI LLMs is useful for improving the alignment of the SFT.
Evaluation Inconsistency, To surprise, observe evaluation inconsistency phenomenon where the choice of the feedback protocol during evaluation favors the alignment algorithm that uses the same feedback protocol. Specifically, we find that the gap between the win-rate of the Best- of-n (rankings) policy and the SFT (11.2% is larger than the gap between the Best-of-n (ratings) policy and SFT (5.3%) as judged by the human annotators under the rankings protocol. However, the gap between the win-rate of the Best-of-n (ratings) policy and SFT (5%) is slightly larger than the gap between the Best-of-n (rankings) policy and SFT (4.3%) as judged by the human annota- tors under the ratings protocol. Interestingly, we observe a similar evaluation inconsistency when GPT-3.5-Turbo is used as the evaluation annotator. We present the task-specific performance results in Appendix $ This indicates that the annotators perceive the quality of the policy responses dif- ferently depending on the feedback acquisition protocol. This highlights a challenge in designing robust evaluation protocols that mirror real-world performance.
Ablations. Here. we aim to understand whether evaluation inconsistency phenomenon is observed in different setups. In Appendix SM we find that evaluation inconsistency when the alignment policy is changed to rejection sampling finetuning. In Appendix |N we show that our observations are robust to the choice of the reward model used for Best-of-n policy. Appendix O we find that converting the rating data into ranking format also leads to the evaluation inconsistency.
6 RELATED WORK
Learning from Human Feedback. 9 introduced the RL from Human Feedback (RLHF) frame- work, which initially learns a reward model network from a dataset of human preferences. This reward model is then used to replace mrmttenwitttnn reward function during RL optimization. 335 and [26) introduced an RLHF framework for aligning LMs with human preferences.
Learning from AI Feedback. While RLHF has shown promising results, the collection of human preferences remains a significant cost for aligning language models. Prior work has demonstrated that LMs are capable of providing feedback [16] 49 [2] that can be used to improve their own capa- bilities [22 I. 4 introduced the Reinforcement Learning from AI Feedback (RLAIF) framework, which replaces the human feedback step of RLHF with an automatic Al feedback generation proce- dure. Our work compares human feedback with AI feedback in both rankings and ratings settings.
Feedback Data Collection. Previous works have utilized both human and AI feedback in both ratings 17) and rankings (35 37 [2] settings on machine-writter responses (3 and human- written responses [24]. However, there has not been work that systematically compares the effects of collecting ratings versus rankings feedback.
CONCLUSION
In this study. we conducted an examination of feedback acquisition protocols aimed aligning LLMs. Our investigation revealed a notable challenge: feedback inconsistency within sparse pro- tocols (ratings and rankings). We found that feedback inconsistency was exhibited by both humans and Al systems as feedback providers. Additionally, we delved into the factors contributing to this inconsistency and shed light on its practical consequences in the context of model alignment and evaluation, notably highlighting the phenomenon of evaluation inconsistency. Specifically, we em- phasize that the trends observed in evaluations are significantly influenced by the choice of feedback protocols. Future research endeavors may explore the cognitive underpinnings of the inconsistency problem, expand the array of feedback protocols to include denser feedback, and investigate the implications of these choices on the alignment algorithm and subsequent evaluation procedures.
9

---

Published as a conference paper at ICLR 2024
8 ACKNOWLEDGEMENT
Hritik Bansal is supported in part by AFOSR MURI grant FA9550-22-1-0380. We thank Ashima Suvarna for her helpful comments the draft. We also thank the reviewers for their insightful feedback.
REFERENCES
[1] Afra Feyza Akyürek, Ekin Akyürek, Aman Madaan, Ashwin Kalyan, Peter Clark, Derry Wi- jaya, and Niket Tandon. RI4f: Generating natural language feedback with reinforcement learn- ing for repairing model outputs. arXiv preprint 1rxii:2335.008444 2023.
[2] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, al. general language assistant as laboratory for alignment. arXiv preprint arXiv:2112.00800 2021.
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2240::220.005862, 2022.
[4] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077. 2022.
[5] Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schimdt. Visit-bench: benchmark for vision-language instruction following inspired by real-world use, 2023.
[6] Jon Ander Campos and Jun Shern. Training language models with language feedback. In ACL Workshop on Learning with Natural Language Supervision. 2022., 2022.
[7] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and fundamental limitations of reinforcement learning from human feedback, 2023.
[8] Wei-Lin Chiang, Zhuohan Li, 7i Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et Vicuna: open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna, lmsys. org (accessed 14 April 2023), 2023.
[9] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
[10] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi. Patrick Wendell, Matei Zaharia. and Reynold Free dolly: Introducing the world's first truly open instruction 11m.
Xin.
2023.
URL https:://wwwt www. databricks. https///cc////////////////222 www.-rae-----sssttiiiooo---e-ee-
[11] Hanze Dong, Wei Xiong. Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023.
[12] Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305. /4387, 2023.
10

---

Published as a conference paper at ICLR 2024
[13]
Xinyang Geng, Amav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL attps://bbiis http.//we..elll.cce/////////////
[14] Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:3303.15056 2023.
[15] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
[16] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint 22tii::220700522220 2022.
[17] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. /2192, 2023.
[18] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator instruction- following models. https: //gitttubbbb www/tatstttteteeeeeeeeceo lab/al aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aalccpaaaeaaa _eval] 2023.
[19] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305. 20050 2023.
[20] Rensis Likert. A technique for the measurement of attitudes. Archives of psychology, 1932.
[21] lya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100,,22 2017.
[22] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refine- ment with self-feedback. arXiv preprint arXiv:2303. 2023.
[23] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, al. Wehgpt: Browser- assisted question-answwrring with human feedback. arXiv preprint arXiv:21 2021.
[24] Stanford NLP. Stanfordnlp/shp datasets at hugging face. URL htllh:tpp://ii /ttLttttt///uggggaaccc http:///hhggiggac.....
htt.daatasettssstaaffforo
[25] OpenAl. Gpt-4 technical report, 2023.
[26] Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[27] Baolin Peng. Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. tuning with gpt-4, arXiv preprint arXiv:2304.032 2023.
Instruction
[28] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprini arXiv:2305. 18290, 2023.
[29] Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. Verbosity bias in preference labeling by large language models. arXiv preprint arXiv:2310.10076 2023.
[30] Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback at scale. arXiv preprint arXiv:2303. 16755, 2023.
[31] John Schulman, Filip Wolski, Prafulla Dhariwal, Alee Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
11

---

Published as a conference paper at ICLR 2024
[32] Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Weston, and Jing Xu. When life gives you lemons, make cherryade: Converting feedback from bad responses into good labels. arXiv preprint arXiv:2210.15893 2022.
[33] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang- Preference ranking optimization for human alignment. arXiv preprint arXiv:2306. /749,, 2023.
[34] Ziang Song, Tianle Cai, Jason D Lee, and Weijie Su. Reward collapse in aligning large language models. arXiv preprint arXiv: 17608, 2023.
[35] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
[36] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. /tttps://gtttt
www.tttttotteeeeeeeee stanford__aaaeee alpaca 2023.
[37] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397. 2023.
(38) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint urXiv:2307.09288 2023.
[39] Amos Tversky. Intransitivity of preferences. Psychological review, 76(1):31, 1969.
[40] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. super-naturalinsttuution Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.07705, 2022.
[41] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. arXiv preprint arXiv:2306.04751, 2023.
[42] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions, 2023.
[43] Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannanch Hajishirzi. Fine- fine-grained human feedback gives better rewards for language model training. arXiv preprint 2rX.v22306.01693, 2023.
[44] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. / 12244, 2023.
[45] Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. Rled: Rein- forcement learning from contrast distillation for language model alignment. arXiv preprint urXiv:2307.12950 2023.
[46] Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, and Kai-Wei Chang. Dynosaur: A dynamic growth paradigm for instruction-tuning data curation, 2023.
[47] Weizhe Yuan, Kyunghyun Cho, and Jason Weston. System-level natural language feedback, 2023.
[48] Zheng Yuan, Hongyi Yuan. Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint 00x5::2304.05332, 2023.
12

---

Published as a conference paper at ICLR 2024
[49] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt- bench and chatbot arena. arXiv preprint arXiv:2306 2006::23006006885,, 05685, 2023.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A LIMITATIONS
While there are many different ways to collect feedback, our focus is on relative and absolute feed- back. Future work should explore the impact of collecting richer forms feedback. Human feed- back data is inherently subjective, and while we try to provide guidelines for annotators to evaluate responses according to dimensions such as accuracy, coherence, and harmlessness, there will still be noise even with multiple annotations for each example. Additionally, the collection of only absolute scores or relative preferences does not fully capture all of the possible kinds of feedback that could be collected. More rich forms of feedback data should be explored in future work.
Our analysis is primarily focused on the impact of different feedback collection methods on the downstream performance of LMs as evaluated by win-rate against DaVinci-003, Future work should investigate the impact of different feedback collection and conversion methods on helpfulness, harm- fulness, and hallucinations. Different methods of feedback, including those beyond absolute and rel- ative preferences, may have drastically differing effects on helpfulness, harmfulness, hallucinations and other LM evaluation criteria.
Our human data is collected from Amazon Mechanical Turkers and is not necessarily representative of all people. Future work should investigate the impact of feedback collection methods on align- ment with different demographic (especially underrepresented) groups. Some methods of feedback collection may amplify bias the annotator demographics in the trained reward models more than others. Additionally, while we focus on open-ended response domains, our findings on the impact of different feedback collection methods may differ more application specific domains such as mathematical reasoning or coding tasks. For instance, in a mathematical proof task, binary feedback may be more helpful since a step in proof is always correct or incorrect.
B DETAILED RELATED WORK
Learning from Human Feedback. In a typical Reinforcement Learning setting, policy is opti- mized maximize reward function, which is explicitly defined by a human. [9] introduced the Reinforcement I Learning from Human Feedback (RLHF) framework, which initially learns a reward model network from a dataset of human preferences. This reward model is then used replace a human-written reward function during RL optimization. (55 and 26 introduced an RLHF frame- work for aligning LMs with human preferences. They first learn a reward model trained on rankings of human feedback sampled from LM outputs, which is then used to optimize an LM with an RL algorithm. In contrast, [33 and (28) bypass the learning of a reward model and directly tune an LM using human preference data.
Learning from AI Feedback. While RLHF has shown promising results, the collection of human preferences remains significant cost for aligning language models. Prior work has demonstrated that LMs are capable of providing feedback [16] 49 12 that can be used to improve their own capa- bilities (22 M. 4 introduced the Reinforcement Learning from AI Feedback (RLAIF) framework, which replaces the human feedback step of RLHF with an automatic AI feedback generation proce- dure. Our work compares human feedback with AI feedback in both rankings and ratings settings.
Feedback Data Collection. While RLHF and RLAIF have proven to be highly effective methods for aligning LLMs, additional challenges remain, including the optimization of feedback collection and reward modeling 77. Previous works have utilized both human and AI feedback in both ratings
37 12 settings on secchnee-writeen responses 3 and human- numan-written responses (24]. However, there has not been work that systematically compares the effects of col- lecting ratings versus rankings feedback. While prior works have investigated the use of rich forms of feedback such as fine-grained feedback (43 and system-level natural language feedback [47), the focus of our work is solely on ratings and rankings feedback. Finally, inconsistency has been explored before in terms of intransitivity in human preferences [39] and (dis- Jagreements arising among different annotators when providing feedback on a single entity. However, our work focuses on the Jageemetsss between the different feedback protocols.
19 17 and rankings 35 26
14

---

Published as a conference paper at ICLR 2024
C COMPOSITION OF THE INSTRUCTIONS DATA
We collect a diverse set of instructions from varied sources:
1. Dolly [10]:
We randomly select a subset of 4K out of 15K high-quality buman-generated prompts and response pairs.
2. Self-Instruct (User Oriented) 42 It is set of 252 expert- instructions.
3. Super-NI |4]
Originally, it contains 1600+ NLP tasks with their expert-written written task- descriptions. our work, we select a subset of 100 dense NLP tasks such as 'question generation based on given scenario" instead of the ones that require just "yes/no' answer (Appendix $ Subsequently, we randomly select instances for every task.
The composition of the instruction data is presented in Table 3
Instructions Composition
# of instructions from Dolly
4K
# of instructions from Self-Instruct (User Oriented)
252
of instructions from Super-NI
# of generations per instruction
5
Feedback Composition (GPI-3.5-Turbo)
of instances Ratings
24.6K
# of instances w/ Pairwise Rankings
46.5K
Feedback Composition (Humans)
# of instances w/ Ratings
4K
of instances w/ Pairwise Rankings
2K
Table 3: Feedback data statistics. We create a set of instructions that are used to prompt Alpaca for response generation. Subsequently, acquire feedback those responses from humans and Al.
D FEEDBACK ACQUISITION FROM HUMANS
Firstly, we randomly select a subset of 500 from 46.5K from the GPT-3.5-Turbo rankings feedback data. We use this data to create 1000 instances for ratings protocol annotations (one instance of pairwise rankings contributes towards two instances of individual ratings to the responses). We utilize annotators in the Amazon Mechanical Turk platform for our task. Before the qualifications, annotators are shown few solved examples for both protocols. Post-qualifications, a total of 26 annotators participated in the annotation where 13 annotators were used for the ratings and rankings feedback each. The annotators were paid an hourly wage of $18/hour. The overall cost for human annotations $1000. We assign each instance of the feedback data four workers. In summary, we have 2000 rankings feedback (= 500 instances x 4 workers per instance) and 4000 ratings (=1000 instances x 4 workers per instance) from human annotators.
E VARIATION IN CONSISTENCY SCORES WITH ANNOTATORS
E.1 HUMAN ANNOTATORS
the consistency analysis is performed by aggregating the preferences of three random annotators (out of four). To understand the effect of different batch of humans, we calculate the consistency scores for different batch of randomly selected 3 annotators four times. We find that the standard deviation of the inconsistency analysis for four splits of the humans evaluators is 1.6%. This indicates that the inconsistency analysis is robust to the batch of humans used for annotation.
In Table 26,
E.2 AI ANNOTATOR
In our main experiments, we use 2-T-3.5-Turbe (temperature = 0.0) for the consistency analysis. To provide more evidence for the same model using a different temperatures 0.0, 0.2 and 0.5 for 1000
15

---

Published as a conference paper at ICLR 2024
comparisons. We find that the standard deviation in the disagreements between these temperatures is 2.7%. This indicates that the inconsistency analysis also robust to the different model temperatures used for generating feedback data.
In addition, we experiment with other models for inconsistency analysis i.c., GPT-3.5-Turbo-061. and GPT-4 with 50. The inconsistency results for these models are presented in Table 4
Model
Inconsistency
ChatGPT-3.5 --rGGPTT-----u-o------o - (Default=Ours) ChatGPT-3.5-
58%
56%
CharGPT-3.5-TTrbo-06613
54%
GPT44
50%
Table 4: Inconsistency analysis with varying Al annotators.
The standard error for the above numbers is 4% with 95% confidence. We find that all the model choices suffer from the inconsistency problem. This indicates that our results are not restricted by the choice of the Al annotator, and are indeed observable in different models too.
F PAIRWISE RATING
We acquired feedback data using third protocol i.e., pairwise rating. Given pair of responses, provide a rating score [1-7] to both of them. This is different from our initial rating protocol where the ratings were collected individually on the responses. We convert this newly collected pairwise rating response to ranking form by comparing the scores given the pair of responses. For example, Response got 7 and Response 2 got 5, we consider Response & Response 2. We compare the consistency of this pairwise rating data (converted to ranking) with our initial rating feedback data.
We find that even these feedback protocols suffer an inconsistency of 58% on 500 comparisons. also highlights that the rating collected in a pairwise manner (akin to ranking format) is not calibrated with the rating collected for the responses individually. This can be attributed the fact that the annotators focus the different response quality attributes when providing pairwise feedback and independent feedback.
G
SETUP
Reward Modeling. We choose low-rank adaptation 15 (LoRA) of Alpaca-7B the reward model [
We use the objective function defined in Eq. 2 to train the reward model on the rank- ings feedback data. Similar to (26). single batch only contains all the pairs of candidate responses for a particular instruction. Further, 70% of the feedback data is used for training and the rest is for validation. We provide more details on the training settings in Appendix SK| We train the reward models on the rankings and ratings feedback for 5 epochs with early stopping where the validation loss is the criteria. We train each reward model for three random seeds.
Post-training, we evaluate the models on 553 unseen instructions, which includes 188 instructions from the Open Assistant (OASST) evaluation, 129 from the helpful evaluation released by Anthropic [3] 80 from Vieuna evaluation [8], and 156 from Koala evaluation [3]. We employ a Best-of-n (n 64) policy to evaluate the effect of various feedback data on model alignment. For this, first generate candidate responses of length 200 from the SFT model (Alpaca-7B) with the unseen instructions. Subsequently, the output Best-of-n would be the one that achieves the maximum score with the trained reward model. Finally, compute the win-rate of the SFT model and the Best-of-n policies against Da Vinci-003 (reference model) on these unseen instructions. Here, we use the Da Vinci-033 responses collected in AlpacaEval (18). Here, we vary the choice of the feedback protocols as ratings and rankings during evaluation and the choice of the annotators as humans and ChatGPT-3.5-Turbb Each pairwise comparison response rating is performed by crowd-worker, where we paid $18 per hour for the annotations. We spent $1000 on collecting evaluation annotations from humans.
Evaluation.
https:: http:/ttttto..cooo///////a/aeeee pac.lpppaaaaaa 1ora
16

---

Published as a conference paper at ICLR 2024
Computing Win-rates. Post-alignment, evaluate the Best-of-n policies (ratings, rankings) against reference model on a set of unseen user instructions U. Since we aim to understand the effect of the choice of the feedback acquisition protocol the complete alignment pipeline, collect ratings and rankings for the responses from the alignment policy and the reference models from humans and Al annotators. We compute the win-rate against the reference model as the fraction of instances for which the Best-of-n policy response is preferred over the reference model under particular feedback protocol. In case of a 'tie' judgment by the annotators, both the policy and reference model get half a point. Formally, we define per-example score for rankings as
1 r(u,yP, = yp
Eu, UF)
0.5
rlu,yP, = lie
(5)
0 ((U,yP,11)) UF
and for ratings as
a(u,yp yp) a(u, yr) a(u, yp) = a(u, yf) a Up) < a(u, yF)
Elu.yp.85)
0.5
(6)
0
where E U is the unseen input instruction, yp = Pn (ge, pe, u, (UI...., }n is output from the Best-of-n policy that leverages the rankings reward model, yy the response from the reference model, r(u, yr, Me) is the preference between the pair of responses decided by the annotator. Win- rate metric w is computed as the average per-example score:
Σ uEZ
E(u, UF)
U
H ANALYSIS OF THE RESPONSE LENGTH AND NUMBER OF UNIQUE WORDS
We assess whether the length or number of unique words in the candidate responses bias the ratings and rankings feedhack judgments from humans and AT in our data. We present the results for such assessment in the ratings and rankings feedback in Appendix TableBJand[Grespeciiveel we find that there is no discernible difference between the average length and average number of unique tokens of the preferred and unpreferred response in the rankings feedback collected from the humans and AI.
In summary,
In T we find that the ratings scores assigned to the individual responses do not increase with the average length and the average number of unique words the response for both humans and AL. Similarly in Table 6, we find that there is discernible difference between the average length and average number of unique tokens of the preferred and unpreferred response in the rankings feedback collected from the humans and AI. This highlights that our feedback data unbiased towards longer and unique responses, and the differences observed from the prior works might be attributed to differences in the experimental setups.
We clarify that our setup is not biased towards the response lengths. However, it does not imply that humans do not prefer longer responses, in general. 49 mentioned verbosity bias examined verbosity bias when using LLMs to collect feedback data by constructing a repetitive list attack in which responses with list were prepended with paraphrased version of the list (generated by GPT- 4) which contains no new information. For example, model A has two pointer responses and model has the same two pointers + their paraphrased versions (which does not add any new information), the annotators prefer model B response. We believe that this setting largely differs from our setting since we consider two different responses with no further intervention such as a repetition attack. Similarly. [29] talks about verbosity bias in preference labeling from humans and AI for creative writing tasks. In our work, we consider a broad range of instructions sourced from datasets as discussed in Table 3
17

---

Published as a conference paper at ICLR 2024
3
5
Humans
45.4
49.7
48.9
46.3
47.7
48.3
GPT-3.5-Turbo
32.4
45.2
41.5
47
48.7
46.7
(a) Average response length
3
5
6
7
Humans
34.2
36.6
36.6
35.7
36.7
37.1
GPT--355-Tubb
25
34.4
31.4
35.8
37.5
36.5
(b) Average number of unique words
Table 5: Analysis to study the length number of unique tokens bias for ratings feedback.
Preferred
Unpreferred
Preferred
Unpreferred
Human GPT-3.5-Turbe
49.5
49.5
Human GPT-3.5-Turbe
37.8
37.7
49.0
50.2
37.8
38.0
(a) Average length of the responses
(h) Average number of unique words
Table 6: Analysis to study the bias towards longer responses (number of whitespace words not the subword tokens) or responses with most unique number of whitespace words not the subword tokens. These numbers are calculated for 1800 comparisons from GPT-3.5-Turbo and Humans.
I LLM PROMPT FOR FEEDBACK ACQUISITION
We present the LLM prompts for acquiring rating and ranking feedback from GPT-3.5-Tume TTrbbo in Tad T
J HUMAN EXPERIMENTS WITH UI SCREENSHOTS
We provide the same annotation guidelines to the humans as the Al models. We present the user
interface screenshots for the feedback acquisition Figure O and 7 The screenshots contain
explanation form which is present only during the qualitative analysis of the feedback data, while it removed during large-scale feedback acquisition.
J.1 FINE-GRAINED ANALYSIS OF CONSISTENT AND INCONSISTENT DATA
We perform fine-grained experiments to understand the difference between consistent and inconsis- tent subsets of the feedback data from humans and AI.
Difference in the Ratings. Our aim to quantify the average ratings difference for the pair of candidate responses that receive decisive preference (either of 'response 1' or 'response 2 but "equal") from the ratings feedback data. Specifically, we focus on Row 2 and Row of Table 2a and 2b Within these rows, (a) the combination of the (Col 2 Row 2, Col Row 3) represents the consistent instances where both forms of feedback are decisive, (b) the combination of the (Col 2 Row 3, Col 3 Row 2) represents the inconsistent decisive instances, and (c) Col represents the scenario where the rankings feedback is indecisive. We report the average ratings difference between the scores achieved by the instances belonging to the category (a), (b), and (c) in Table 7a
We find that the maximum average ratings score difference is achieved by the consistent instances for human annotators as well GPT-3.5-Turbo. This indicates that the quantitative gap between the pair of candidate responses belonging to the consistent instances is higher than the pair of responses belonging to the inconsistent instances, which further highlights that the difference in the quality of the two responses in consistent instance is more distinguishable than in an inconsistent instance. In addition, we observe that the average ratings score difference for the inconsistent instances is similar when the rankings feedback is decisive and indecisive with ratings feedback being decisive (Column and 3), for both human and AI annotators. This hints towards the potential qualitative closeness in the pair of responses that causes them receive decisive ratings feedback but contra-
18

---

Published as a conference paper at ICLR 2024
As helpful ussistant, you possess extensive knowledge about buman feedback and understand human would output sentence completion or response based on the given instruction. Your job is to provide an "Ratings" that falls only into one of the based on the overall quality the completion. Here, a score indicates output real damage, and of indicates perfect completion where you cannot
following cutegories: [1, 3, 4, terrible and completely useless and/or think of a way to make it better.
#tiittnntatiion Guideline
Please keep mind that your scere should take consideration the following quality dimensions:
Accuracy: The output sentence should be factually consistent with instruction and the input.
should be casy understand and free grammatical when read Harmlessness: The output sentence should not contain demgatory, harmful, or toxic connotations.
Coherence: The output
##Example
##Instruction
Tell me if the following foods would be safe unsafe someone with allergy to cat? Peanut butter, eggs, apples, carros,, pad thai.
##Response 1:
Peanut butter: Unsafe
Eggs: Unsafe
Apples: Unsafe
Carrots: Unsafe
Pad Thai: Unsafe
##Ratings:
I would give score in the Jower rung since (a) it gives incorrect answer for cases. (b) looks like the Al repeats its output. the answer does not seem convincing since there are no explanations and as human can improve the answer with explanations.
##Example
#llsstruction 2:
What is lakehouse?
##Response 2:
lakehouse type real that situated stream,
large body water. typically
single-family residence hotel. and often has beach, docks, and boat launch. Lakehouses provide scenie, outdoor lifestyle experience, often with many amenities, including swimming. fishing. kayaking, boating, suiling, water skiing, and bird watching.
#MRatings:
7
would give it score in the higher rung since I could not think of better way to improve the answer.
#Annotation starts below
should first think step-by-step and then provide an "Ratings" among 2,, 6, 7.
##Example
#MInstruction 3: instruction
##Response 3: response
##Ratings:
Figure 4: GPT-3.5-Turbo API prompt for rating responses.
19

---

Published as a conference paper at ICLR 2024
You hhlplpal following ussistant goal select preferred wrong) ourput given instraction. Answer question printing only a single choice from ("Oulpui (a)", "Oulpui (b)"] (wilhout quotes) coresponding the correct answer with no other lexl.
Amnotation Guideline In this task, we will ask you to select the preferred ourpet Al model's responses to instructions.
You
cxamples, which composed following:
1. Instraction we give to the Al system
2. Ourput (a), first cutpur from the system
Ourput (b), secend ourput Your task is to decide which response is better for each example. There are sevena dimensions that you can think along, Consider the following dimensions:
system
Accuracy: The oulput sentence should be factually consistent with instruction and impel.
Coherence: The output sentence should be casy understand and free of grammatical errors when read on its own.
Harmleseness: The output sentence should contain derogutory, harmful, toxic connotations.
## Annotation Example
To help understand the annotation tusk. provide examples below. will give an explanation correct answer, you only answer preferred ourput.
#t Example
Instruction I:
Write a design document describe the Following mobile game idea: A battle coyale game with superhero and villain theme.
Outpot exomple
Design Decumeett for Mobile Game loka:
1. Introduction:
A battle royale gume with a superhero and villain theme.
2. Game Play:
The game played
superpowens wreak havoe their villainsy their advantage.
Rewards:
As players compete in the bottle.. .. the rost well.
Htl Output for example
newly launched mobile game
thrilling battle royale visually appealing plet to keep the players engaged very end.
## Which best for example 1, Outpot Output (b)?:
Ourpur
Indied, Output is design document as asked the instruction and gives more useful details than Output (b).
#t Example
#t Instruction 2:
Generate to-do list with tasks related kooping neat
## Outpat (a) for example
Make beds
Organize closet
Shoot some hole the walls
Dust lamps and furniture
Tidy drawers and shelves Outpot
example
chores:
1. Clean up after yourself and put clean clothes away.
electronics decommissioned before leaving 3. Remove all personal items and clutter from your room.
Make
1. Vacuum and mop the floor.
5. Tidy up your desk put all inhoxes hold.
#4 Which best for caamplc 2, Outpot (a) Output (b)?:
Output (b) User explanation: Output
Ourput
walls", tidying room
## Annotation starts below Example
## Instruction 3:
instruction
## Dutpat (a) for example oulpptt
#4 Output (b) for example octpur
example 3, Outpet (a) Output (b)?:
Figure 5: GPT-3.5-Turbo API prompt for ranking responses.
20

---

Published as a conference paper at ICLR 2024
Instruction:
S(instruction)
Input (if any):
S(input)
Response:
S(response)
Choose one of the following options:
O1
C2
04
05
06
A few lines (1-2) to explain your choice.
Figure 6: Screenshot for rating feedback acquisition with explanations.
dictory rankings feedback in the annotation process. We will explore this hypothesis more through a rankings variation analysis next.
Consistent
Incousistent Ratings & Rankings
Incousistent Ratings Decisive
Rankings Variation Score (Human Feedback)
Ratings & Rankings
Decisive
Decisive
Human GPT:3.5
I,58
1,28
1.29
Consistent
0.50
1.22
1.12
1.12
Inconsistent
0.36
(a) Difference in the ratings feedback scores when rankings feedback is decisive.
(b) Variation in the rankings feedback for consistent and inconsistent examples.
Table 7: Quantitative analysis the differences between the consistent and inconsistent subsets of the feedback data acquired from humans and AI.
Variation in the Rankings. Due to the closeness in the perceived quality of the pair of responses belonging the inconsistent instances, we posit that the human annotators prefer specific quality factors over the others (e.g., focus on accuracy more than the coherence) during decisive decision making. Conversely, we expect the variation in human rankings to be lower when one the re- sponses is perceived to be qualitatively superior. To test this hypothesis, we utilize the dataset comprising 500 rankings feedback instances, each with four human annotations. notations favoring 'response 1' are marked as '+1', those favoring 'response 2' are marked '-1', and annotations indicating 'equal' preference are marked as '0'. For each instance, we calculate the ratings sum of the four human preference markings. A low
the dataset, an-
instance suggests that the
21

---

Published as a conference paper at ICLR 2024
Instruction:
S(hatrucioo)
Response A:
S(espposet1
Response B:
Sjresponse2)
Choose one of the following options:
Response A
Response
CEqually Good/ (Use Sparingly)
A few Ines [1-2] explain your choice.
Figure 7: Screenshot for ranking feedback acquisition with explanations.
pair of candidate responses are considered 'close' by the annotators, leading to an equal likelihood of selecting 'equal' either of the two responses. Finally, we compute the average variation in the rankings feedback for both the consistent and inconsistent instances separately.
To quantify average variation in annotator feedback of a single example, we define variation score metric. For each example, we take the sum of the score for each annotator's label. Prefer response 1, prefer response 2, and equal preference are given scores of 1, -1, and 0 respectively. This per-example score is averaged over the entire dataset to compute the variation score. We find that the consistent instances achieve a higher variation score (0.5) compared to the inconsistent instances in the human feedback data (0.36). This observation supports our hypothesis that inconsistent instances are qualitatively close to each other.
The results our analysis are presented Table 7b
K TRAINING SETTINGS
We train the reward models on the rankings and ratings feedback data for epochs with early stop- ping at the end of every epoch. We use the validation loss for early stopping. We train the reward models on single Nvidia RTX A6000 GPU with 48GB VRAM.
The reward models optimized using equation 2 are trained with an effective batch size 16 where the batch size 4 and the number of gradient accumulation steps all the pairs of candidate responses for given instruction. Since we have 5 candidate responses for a given instruction in the feedback data, we get 10 pairs of candidate responses (5 choose 2) batch. The reward models optimized using equation I are trained with effective batch size 64 where the batch size 16 and the number of gradient accumulation steps =4
Here, each batch contains
Both the reward models use the AdamW optimizer (21 with linear warmup of 100 steps maximum learning rate followed by a cosine decay. We perform a hyperparameter search over {le - 4, le le 6} for maximum learning rate. We find that le 4 for Alpaca-LoRe reward model architecture. We also apply a weight decay of 0.001 to all the reward models and train them 66 precision.
22

---

Published as a conference paper at ICLR 2024
OASST
Best-of-64 (Rating)
Best-of-64 (Ranking)
Ranking Eval
46.7
52.6
Rating Eval
47.0
43.5
Vicuna
Best-of-64 (Rating)
Best-of-64 (Ranking)
Ranking Eval
48.6
54.3
Rating Eval
44.3
38.1
Helpful Base
Best-of-64 (Rating)
Best-of-64 (Ranking)
Ranking Eval
45.1
48.8
Rating Eval
47.0
46.8
Table 8: Tassk-ppcciicc evaluation inconsistency results.
L TASK-SPECIFIC RESULTS
In 5.2 we presented the evaluation inconsistency the 553 instructions from AlpacaEval. In Table we aim to assess the evaluation inconsistency on specific tasks.
We find that the evaluation inconsistency is present across various tasks. Specifically, the rating evaluation protocol favors the rating best-of-64 policy and vice-versa. This highlights that feedback acquisition protocol biases evaluation.
M REJECTION SAMPLING FINETUNING (RSFT)
We performed an additional experiment with rejection sampling finetuning (RSFT) used three highly cited LLM alignment papers, including LLaMA 2 follows:
[38 4 48 Specifically, setup is as
We prompt Alpaca-7B with 5K instructions from Alpaca-52K data.
2. We generate 64 responses for every instruction.
3. We use our rating and ranking reward model (LoRA-Alpaca) to select the best response from the 64 responses.
4. We finetune two Alpaca-7B models with the 5K cnstructionresponnss data.
(a) One where the responses are chosen from the rating reward model.
(b) Second where the responses chosen from the ranking reward model.
5. Post finetuning, we sample single response from the finetuned Alpaca-7B with 553 eval- uation instructions.
6. We calculate the win-rate against VaVinii033 using the rating and ranking protocol using ChatGPT.
The results for this experiment where baseline the base Alpaca- model are presented in Table
9
Baseline
RSFT (Rating) RSFT (Ranking)
Ranking Evaluation 36.9
42.0
43.3
Rating Evaluation 41.9
44.0
43.0
Table Evaluation inconsistency in the RSFT setup. Each entry is the win-rate against davinci-003 on 553 unseen instructions.
We find that the evaluation inconsistency persists under this alignment algorithm too. It indicates that the choice of feedback protocol for the evaluation favors the same feedback protocol used for training the reward models and subsequently finetuning the base L...M
23

---

Published as a conference paper at ICLR 2024
N USING ROBERTA-LARGE AS REWARD MODEL
Prior works 26 12 use decoder-only architecture for the reward model. These models are normally scale of 10-100 billion parameters and harder train in the compute efficient manner without engineering tricks. Hence, we chose Alpaca-7B in our setup, and LoRA A is just a method to finetune in a parameter efficient manner. On the other hand, [34] have used BERT encoder models, which usually have less parameters, say 500M-1B parameters. Finetuning BERT models achieve very good performance on many NLP tasks. Hence, we repeat train reward models on the RoBERTA-Larg architecture.
Specifically, we finetune RoBERTA-Large model on the ratings data using the regression loss. In addition, we train another RoBERTA-Large model on the rankings data using the negative log likelihood loss. We use these two finetuned models perform Best-of-64 policy. Subsequently, we use GPT-3.5-Turbo to provide rating and ranking feedback for the rating/ranking Best-of-64 policy. We present the results in Table
Best-of-64 (Rating)
Best-of-64 (Ranking)
Rating Evaluation Ranking Evaluation
47.70%
42.00%
43.90%
47.30%
Table 10: Evaluation inconsistency when RoBERTA-Lare models are used reward models in- stead of Alpaca-7B model. Each entry is the win-rate against davinci-003 on 553 instructions.
We find that the evaluation inconsistency still persists if we change the choice of the reward model. Specifically, the rating evaluation protocol favors the rating best-of-64 policy and vice-versa. This highlights that feedback acquisition protocol biases evaluation. We will add these additional results in the revised paper.
O CONVERTING RATING FEEDBACK INTO RANKING FORMAT FOR REWARD TRAINING
In our work, we established that the feedback data obtained from humans and Al suffers from inconsistency problems. We believe that our work is unique it shows that this has a direct impact on the alignment and evaluation of LLMs through empirical evidence.
The choice of reward models was naturally made to reflect the nature of the feedback protocol. Specifically, the rating protocol provides us with a scalar score hence the regression model makes sense, and the ranking protocol does not provide us with scalar scores hence likelihood (NLL) objective function.
a negative log
We observe that 57.4% of the pairwise comparisons are considered "equal" when we convert the rating data into the ranking format (Row I of (b). In practice, this means that we need to filter almost 60% of the rating data if want to train ranking format reward model. This is the most optimal way to use the rating data when we know that better regression method exists that can utilize the complete data. Hence, we train a regression reward model in our original setup.
We train two reward models, of RoBERTA Large architecture, on (a) ranking data with NLL loss and (b) rating data converted to ranking with NLL loss. We perform Best-of-n sampling with n 64 We evaluate the model's win-rate against davinci-003 with rating and ranking evaluation scheme. We present the results in Table II
Best-of-64 (Ranking) Best-of-64 (Rating Data Ranking Format)
Ranking Evaluation
47.3%
45%
Rating Evaluation
42.0%
46 %
Table II: Evaluation Inconsistency in pairwise rating feedback data collection. Each entry repre- sents the win-rate against davinci-003.
24

---

Published as a conference paper at ICLR 2024
We highlight that the evaluation inconsistency is still evident when the data format used for training the reward models is kept identical. This indicates that our finding is not confounded by the reward model's 's data format but the nature of the feedback protocol data itself i..., rating and ranking.
P QUALITATIVE EXAMPLES WITH HUMAN EXPLANATIONS
We perform qualitative analysis with human annotators. The group of annotators that provide rat- ing/ranking with explanations mutually exclusive. In total, we consider a subset of 50 instructions and 100 responses (2 responses per instruction) from dataset. In addition, assign 3 annotators per rating/ranking instance. The humans are provided the same annotation guidelines as before but they are asked to provide an explanation for their annotation too. We provide them solved exam- ples of how the explanations could look like. We intended to keep the human explanations free from any biases from the authors, hence, we instructed the humans to provide their explanation based on their perception of the response quality.
We present a few qualitative examples with explanations from the human annotators in Table 14
15
16
and 17
In Table we observe that the annotators responsible for rankings favored 'response 2' due to its density, while perceiving 'response 1' as dull. At the same, despite these divergent preferences, the individual responses received conflicting rating scores: 'response incurred substantial penalty for its density and perceived non-compliance with task instructions, whereas 'response 1 garnered an average score for only partial adherence to the guidelines.
Table 13 contains another pair of responses where ratings and rankings feedback were inconsis- tent. Both responses provide decent answers to the instructions. Explanations of rankings feedback highlighted the differences between the responses, including that the rejected response unnatural" compared to the chosen response. On the other hand, explanations of ratings feedback mainly provided insights into why each response was good and thus received high score. These ob- servations suggest that the inconsistency problem arises from systematic variations in the annotator preferences with the differences the nature of the feedback protocol.
"a bit
Table 15 contains an example where rankings and ratings data are consistent with each other. The instruction asks for question such that every item list of given inputs is correct and reasonable answer to the question. The explanations for ratings feedback for the response 2 which contained correct response were very simple, usually just indicating that the instruction was followed. The explanations for ratings feedback for response 1, which contained a much worse response, which did not follow the instruction typically contained a short description of a single thing that was wrong with the response such as not following part of the instructions or some logical or factual error. The explanations for rankings feedback contained more detailed analysis of the differences between the two responses and tended to be longer than explnations for ratings feedback. Response 2 being better than response 1 is reflected in the much higher ratings score for response 2. The size this difference is not captured by rankings feedback.
Table contains another example where ratings and consistent feedback is consistent. The instruc- tion asks for factual answer to simple question. Response provides the correct answer and response provides an incorrect answer. For very straightforward, short responses, both rankings and ratings feedback explanations are similar indicating that response 1 is correct while response is incorrect. This straightforward difference is reflected in the very large ratings score gap between the responses.
LIST OF TASKS FROM SUPERNI-V2
The list of SuperrNatural Instructions 40 tasks used in the feedback data presented in Table 18
25

---

Published as a conference paper at ICLR 2024
Example 1: Both responses do not complete the task fully.
Instruction: You need to create question containing blank (-), based on the given context word. Your question must contain two persons -PersonX and PersonY. The expected answer to your question must PersonX. PersonX and PersonY should not be equally likely fill the blank. There should be agreed upon answer to fill in the blank. Your question must contain at least and at most 30 words. You must utilize the given context word while writing the question. Your question must contain only one blank. Make sure that Person X and Person Y have the same gender. In your question, PersonX and PersonY should be used only ONCE and PersonX should appear earlier than PersonY.
Input: Context Word "peeling'
Response 1: Person X and Person Y are peeling an orange together. Who is better at peeling oranges, person X or person Y? (Person X should be considered the expert in this scenario).
Response 2: PersonX and PersonY are peeling an orange. PersonX has a knife with which he/she is peeling the orange. PersonY is talking PersonX and is asking for sip of juice from the orange. PersonX is saying no, and PersonY smiles sheepishly. PersonX eventually gives in and they both sip the juice from the orange. Who is peeling the orange? PersonX
Rankings Feedback:
Response 2
Explanations:
Annotator I w/ preference for Response 1: "Neither question contains a blank, but I picked
response A as response B explicitly answers the question within the question' 's own text." Annotator 2 w/ preference for Response 2: "Response B makes more sense and the ques- tion is easily answered."
Annotator 3 w/ preference for Response 2: "B manages to make a riddle out of it. A is kind of boring in comparison."
Ratings Feedback for Response Score Explanations:
Annotator 1' w/ Score 4: "Response relevant the input, but only partially follows com-
plicated instructions. no errors harm." Annotator 2' w/ Score 4: "It doesn't I't include (-), and though the answer to the question is Person X. it gave the answer away. Person X & Y are otherwise equally likely be the correct answer." Annotator 3' w/ Score 5: "There is answer. but the structure of the response doesn't fit the instructions.
Ratings Feedback for Response 2: Score Explanations:
Annotator 1" w/ Score 2: "This does not meet the task structure requirements, and both Person X and Person Y are peeling oranges.
Annotator 2" w/ Score 1: "I gave it the lowest score because: (I) PersonX and PersonY was used more than once; (2) the expected answer is not just PersonX, but can also be PersonY; (3) word count greater than 15-30 30 words."
Annotator 3" w/ Score 2: "'vvrryhhing is wrong with it... "Person X" and "Pesson Y" are used a lot more than once, either of them could be the answer, the question not fill-in- the-blan blanb or 15-30 words, etc.
Table 12: Example explanation for samples with inconsistent feedback data
26

---

Published as a conference paper at ICLR 2024
Example 2: Both responses provide decent answers the instruction.
Instruction: Provided the imput sentence, you re expected to write question that involves event "frequency", which refers to how often event likely be repeated. For example, "taking showers" typically occurs 5 times a week, "going to Saturday market" usually happens every few weeks/months, etc. Don't create questions which have explicit mentions of answers in the text. Instead, it has to be implied from what is given. In other words, we want you to use "instinct" " or "common sense". Also, the written questions are not required to have single correct answer.
Input: Sentence: His counter-attack with Dayak warriors drove the Chinese out of Bau and across the Sarawak border.
Response 1: How often was his counter-atiack with Dayak warriors likely to happen?
Response 2: How frequently did the Dayak warriors counter-attack the Chinese in Bau?
Rankings Feedback:
Response 1
Explanations:
Annotator 1 w/ preference for Response 1: "A does not contain the flagged word." Annotator 2 w/ preference for Response 2: "While both responses
bit unnatural, fits more closely with the input text in that it asks about this historical event, rather than asking for speculation about the future, as response does."
Annotator 3 w/ preference for Response 1: "response A fulfils all the instructions given which makes better as option B made mention of answers in the text"
Ratings Feedback for Response Score
Explanations:
Annotator I' w/ Score 6: "This question is relevant the input and it follows the instruc- tion of having to do with frequency."
Annotator 2' w/ Score 7: "This response follows the rules and is easy to understand.
Annotator 3' w/ Score 5: "Il is question thut involves frequency, but I' not sure it's one that can be answered based on instinct or common sense. Though, the input sentence doesn't 't leave many options."
Ratings Feedback for Response Score
Explanations:
Annotator 1" w/ Score "This response meets the task requirements." Annotator 2" w/ Score 6: "question is about frequency, followed directions"
Annotator 3" w/ Score 7: ":Questtinn properly seeks frequency of the event."
Table 13: Example explanation for samples with inconsistent feedback data
27

---

Published as a conference paper at ICLR 2024
Example 3. Both responses are similar in quality Instruction: What is Age of Empires?
Input: Age of Empires is a series of historical real-time strategy video games, originally developed by Ensemble Studios and published by Xbox Game Studios. The first game was Age of Empires, released 1997. Nos total games within series huve been released so far as of October 28, 2021. Age of Empires focused on events in Europe. Africa and Asia, sponning from the Stone Age to the fron Age; the exponsion game explored the formation and exponsion of the Roman Empire. The sequel, Age of Empires H: The Age of Kings, was Empire. The subsequent three games of Age of Empires III explored the early modern period, when Eacrppe was colonizing the Americas and several Asian nations were on the decline. Another installment, Age of Empires Online, takes a different approach period
the Middle Ages, while its expansion focused partially on the Spanish conquest of the Aztec
a tre-too-lay online game utilizing Games for Windows Live. spin-off game, Age of Mythology, the original Age of Empires, but focused on mythologicul elements of Greeh, Egyption, and Norse mythulogy. The fourth mainn installment in the series, Age of Empires IV, was released on October 28, 2021. The Age of Empires series has been a commercial success, selling over 25 million copiex. Critics have credited part of the success the series its histurical theme and fair play; the artificial intelligence (AI) players have fewer advantages than in many of the series' competitors
the
Response I: Age of Empires is series of real-time strategy video games set in various time periods throughout history, originally developed by Fasemble Studios and published by Xbox Game Studios. The series has been critically successful, selling ever 25 million copies. Each gune in the series is set in a different time period. focusing on events in Europe, Africa, and Asia, and the exponsion games explore the exponsion of various civilizations. The Al players in the series were praised for having fewer advantages s than in many of its competitors. Most recently, Age of Empires IV was released on October 28, 20
Response 2: Age of Empires is series of historical real-time strategy video games, originally developed by Ensemble Studios and published by Xbox Game Studios. The first game was Age of Empires, released in 1997. Nine total games within series have been released the Stone Age to the Iron Age; the expansion game explored the formation and expansion of the Roman Empire. The sequel. Age Empires HI: The Age of Kings, wax set in the Middle Ages, while its exponsion focused partially on Spanish compesst of the Actec Empire. The subsequent three games of Age of Empires IlI explored the early modern period, when Earope was colonizing the Americas and several Asian nations were on the decline. Another installment, Age of Empires Online, takes
far of October 28, 2021. Age Fmpires focused events in Europe, Africa and Asia, spanning from
Rankings Feedback: Response 2
Explanations:
Annotator w/ preference for Response "I response B gives more complete answer, giving pertinent details on many of the ritles from the franchise. Response doesn't contain nuch releviffiatrresting info.
Annotator w/ preference for Response 2: "" optionn b is more informative, üs summury that contains all the useful and informa- live details from the input."
Annotator w/ preference for Response "Response has more in depth information from the input."
Ratings Feedback for Response 1: Score 5
Esplanations:
Annotator I' w/ Score 4: "The response is topic but isn't 't complete. It does not end on complete sentence."
Annotator 2' w/ Score 6; "lots of detail but could be phrased a little better"
Annotator 3' Score "The response lacks some important details from input, such the total number of games the series and specific settings for each installment, It also ends abruptly withou completing the sentence."
Ratings Feedback for Response 2: Score 4
Esplanations:
Annotator 1" w/ Score 2: "Response simply copies the imput and is truncated"
Annotator 2" w/ Score 6; "It ends abrupily after mentioning "Age of Empires Online, takes..." without providing conclusion
continuation of the sentence."
Annotator 3" w/ Score 4. This answers question, but doesn't 't demonstrute any interpretation or summarization of the original ReM."
Table 14: Example explanation for samples with inconsistent feedback data
28

---

Published as a conference paper at ICLR 2024
Example One response is much better than the other
Instruction: Construct question that every answer in the list correct and reasonable for it.
Input:
telephone, electricity, Iv, computer, microwave
Response 1: Which home appliances did you purchase in the past year?
Response 2: What modern technology have been the most influential in your everyday life?
Rankings Feedback:
Response 2
Explanations:
Annotator w/ preference for Response 2: "It's quite unlikely that someone had purchased all those items in the last year, and a car and electricity both don't qualify as home appliance. Response B seems more fitting."
Annotator 2 w/ preference for Response 2: "Response makes more sense because people don't really think of cars or telephones as appliances."
Annotator 3 w/ preference for Response 2: "The list given in the input is not confined to examples of appliances, so I would discard Response A since it is not natural question to ask and expect this given list as an answer. Response a bit unnatural in tone, but at least fits with the expected answer from the input."
Ratings Feedback for Response
Score
Explanations:
"Electricity is not home appliance, and it's unlikely that con- sumer would purchase all of these things in the same year."
Annotator I' w/ Score 2:
Annotator 2' w/ Score 3: "The response does not fully follow the instruction to construct question. Not all items in the given list can be considered home appliances.
Annotator 3' w/ Score 4: "The answer to the Response' question would very from person person."
Ratings Feedback for Response Explanations:
Score
Annotator I" w/ Score "'All of the input items are modern technological inventions that are influential in human life. This is accurate and concise."
Annotator 2" w/ Score 7:' question fits all of the input values." Annotator 3" w/ Score 7: ":peffectl lines up with all the answers in the input"
Table 15: Example explanation for sample with consistent feedback data
29

---

Published as a conference paper at ICLR 2024
Example One response contains clear factual inaccuracy while the other does not
Instruction: Who is the all time leading goal scorer for Manchester United?
Input: N/A
Response 1: The all time leading goal scorer for Manchester United is Wayne Rooney with 253 goals.
Response 2: The all time leading goal scorer for Manchester United is Sir. Ralph Chapman with 414 goals
Rankings Feedback:
Response 1
Explanations:
Annotator 1 w/ preference for Response 1: "Response A gives the correct answer accord-
ing to google. B is wrong."
Annotator 2 w/ preference for Response 1: "Response is correct, while Response B
nor.
Annotator 3 w/ preference for Response 1: "A the correct answer."
Ratings Feedback for Response Score
Explanations:
Annotator w/ Score 7: "gives correct and precise info"
Annotator 2' w/ Score 7: "The answer is relevant the instruction and seems factual. is coherent and easy to understand. It does not have any grammatical errors and id harm- less."
Annotator 3' w/ Score 7: "This is the perfect answer. It's accurate and written in a com- plete sentence."
Ratings Feedback for Response 2: Explanations:
Score
Annotator |" w/ Score 1: "The answer very wrong...
Annotator 2" w/ Score 1: "This should be Wayne Rooney."
Annotator 3" w/ Score W:"ay Rooney tops the list of all-time goal-scorers for Manch- ester United."
Table 16: Example explanation for sample with consistent feedback data
30

---

English sentences, containing all the information Published as a conference paper at ICLR 2024
Example 3: One response completes the task clearly better than the other
Instruction: this task, we ask you convert a data table of restaurant descriptions into fluent natural-sounding English sentences. The input is string key-value pairs; the out- put should be natural and grammatical English sentence containing all the information from the imput.
name[The Golden Curry). food[English)) priceRange(high), famni familyFriendlyyno),, near[The Bakers]
Input:
Response 1: The Golden Curry is an English-style restaurant located near The Bakers and offers a high-price range. It is not family-friende
The Golden Curry is an English-speaking restaurant located near The Bakers which offers range of prices. It is not family-friendal
Response 2:
Rankings Feedback:
Response 1
Explanations:
Annotator w/ preference for Response 1: "Response A follows the instructions and cre-
ates a natural language answer to the query. would have been better to say that the food is English rather than the vague Style; but B clearly misreads the input and says that the language is English."
Annotator 2 w/ preference for Response 2: "Response flows better me."
Annotator 3 w/ preference for Response 1: "response A is natural and completes the input without missing out any details like option B uses range of price instead of high price"
Ratings Feedback for Response 1: Score
Explanations:
Annotator 1' w/ Score 5: "consistent, mentions all the values from the input, could be sentence instead of two, wording could be a little better" Annotator 2' w/ Score 7: "The response accurately converts the data table of restaurant descriptions into fluent natural-soundeee from the input."
Annotator 3' w/ Score 7: "I gave it the highest score as it met all requirements."
Ratings Feedback for Response 2: Score
Explanations:
Annotator 1" w/ Score "I gave a middle score since the Al's response had errors. 50% correct, 50% incorrect."
Annotator 2" w/ Score 4: "The response should have specified English foodinstead of "Fingiish-speakinng" The response does not include the information about the "high price range" " as provided in the input." "
Annotator 3" w/ Score It an English restaurant, not English-speaking. The response is not a single sentence."
Table 17: Example explanation for samples with consistent feedback data
31

---

Published as a conference paper at ICLR 2024
task003_metaco cioctaco_uesttoongneeeaattoooooo
1eax0099_ctacooquessii generation.event
task015_mctaco.. cuuestionngemeratiion
iaskO23ccomooqaa question generation, task026 drop question generation,
task030. winogradee person, 1 questiion.eeeeaatiii inneerationoojece
question.generationn wwwerateennegneeeeeee person, task040 Baak040__sss..oee qasc_question .tuuuuiinngeeerattiiiii generation,
1a000445_micellaneoooi sentence. paraphrasing, task053 _ask053_multire multire_correct..... correct_bae bad question. www.eeeecepaaapeeeneeeeeeeeeee
Se0k0059ropes_stpoe senerrrgnnnraation, cask067_abductive wwweeeiiiaswwergeeeeee generratonn,,
wtt.aabuuccieeeiiinooreeeceeeeee gerereeettn.eeeeeeetttttiooo task07 Labuuiui answer. -nw.reeeeeenennntttnnnn
tusk0722abduectivell..mm htt.:eeiiii.ass....eeeerratoooo. task074_squad task077 splash explanation to sql, task082 babi tl single supporting fact question generation,
task083 babi single supporting fact answer generation, task facts2story long text generation, 10lloo0_ceeett__seeeee seeeeenceeeeeeerttionnnn task Balllllasst_seeee sentenceeee simpliiiiiiiiiiiiinnn,n,
lask132.daii..ext lext_mooiiicatinn, task182_duorr_quu -en.esttonnneeeattioonn aask19__duore hww.://o....u..uuooonnoooo guestiion_eeeeaation
1142203_mli.sennennce ses.eeniienccggggraaiioon task210. wttpr/eexxxtttttuuudeeeeeetttee
task216 www.soorrrooccoorreeeeooo www.seccorrccccceceeeeeeeeeeeeee anwwwrrggeneration, task223.quartz.eet..
task246_dream www.246ddreammuuesttiiee wwwrrrrmuuuettoonnnommo quuestiongeeeraationn
fask269_csrg_coo
task270 csrg counterfactual context generation, task360 spolin yesand response generation,
task389 torque generate temporal question, task393 plausible result generation,
aaw.gaaaaseeeeeeeetttooo... 2etp9944prriiaaa____e wwl.gguiiingggeerattioio generation, task395__pertt pwt.sppersaaaaaemm persianga.answer -eenerttennn,,
task418 wwwppeeeettetleeeeeeeeeeee
task454_swag Baak444_wwa__inoorree wawrrrectaaswwrggeeeeeee wtw.cooooteeeeettiiiioooo
task455 context generation, task466 pwrsinlu tqppuuggppppp.ppp tqp_text http:txx_mmmfiiiccc mox.fioiiiiationn
task489 bask489.mwsc.queset www..mwwww...........um..m leree-ueettionggeerattiei generation, mask510_reddit .iftttiifuuuuilll tifu-title feluuuttlllesssmtee summarization,
task511_redde Sollditttffullae long text summarization, task519, aquamuse questiin_generatton
task550 discofuse sentence generation, task572 recipe nlg text. generation,
htt.g//iinnneetttsseeeeeoooo task594_sciq. wuw.gguuuuuseeerrtttte www.nnceebaaddoo... task627 xlwic word with same mcaning sentence generation,
task628 xwwic.word with
task668 eetrrmmeeeemmmmmmmmm differeeff mew.iig sentence generation,
task671 ambigga_teex teet__eeeeeattooee
task672 and.andd__y yelp summarization www.onndsaaaatsssmrraiiiaaa
1ww.644_oolineepriacccc stepticcyeeeetiie ina.xxtoooomaion.yyyy. htppematatiitiooooooooomm hot.mmatttiotttye-eeerrrtttooooo generation, task739 I
musk73..culex.summrizza task821 task821-protouueeeeoo prrotoqa.question.gggog geeerraionn,,
task853 hippocorpus long text generation, task857 inquisitive question generation.
task859 prost question generation, task860 prost meq generation, task871 msmarco question generation, wuu.stiinoneeeeeattionnnnn
task897_freebase_ pttpsss..aappiccccc... www.etiueu................ generation, task955_ www.wwikiuuuaa Siikiiauto_ssyyii autt_style_transfee
generate,
wtw.:/inooorre.f.teeeeeeeececeee www.aaaeegg..ioonnnnnnnnnoaaaooe based. on oon-ooonnggegee wtweggiiiieenaaae. paragraph, task xamm
xsum summmrizaiin,,nn,
www..assssmaaiiiiiinno... mask1325.om wwt.aaa...eeeee..ee wwteestion.gnneratii www.eeonnnntteeeeeeeeeeeeeee relation,
task1326 qa zre question generation from answer, task1345 glue qqp question paraprashing.
t35k sent comp summarization, task healthfact sentence generation,
sunpasttssssmmmaiaatiionnnn fack1519_qqasii/ wtt.i/11ggasssl_uuuttooooooo geneeraton,,nratioonn
L sonnntnncegeeeattioonn...m (540 -wr.0..pp........... caafssssmmaiiitiinn.. hetp:5552sciiillquuuuee
generation, t5sk1553 .nn dailymail summarization,
task /lw.....cccaaitteeeeee nnw new_bg.. summarization, task 1562 zest text.eett..mmmm..ee moxttmiiiiiaiion.
task 1aokl1oopprppraa.... guestion.generationn task samsum summary,
1580
(e55500.eaccppeeee eqasc-perturbed www.tureeeduuuueutoonnnemm generation, task 156 scifact sitee_generation,
question.generaaioon.
task (akt544_ahhooansserss.ooooo oagpipaoaaweeesssoosseeeeennnnnn task 595..veen2ninnd...eeeeeetnii
Table 18: Subset of SuperNatural Instructions v2 used in experiments
32


Reviews:
[{"review_id": "rCpDYle85U", "reviewer": "Reviewer_7SsY", "summary": "This paper studied the two kinds of protocols used for collecting preference data: ratings and rankings. The authors found an inconsistency problem where in the preferences inferred from ratings and rankings significantly disagree 60% for both human and AI annotators. Their subsequent analysis identifies various facets of annotator biases that explain this phenomena such as human annotators would rate denser responses higher while preferring accuracy during pairwise judgments, for a particular comparison instance. Finally, they also found that the choice of the feedback protocol has a sharp influence on the evaluation of the aligned LLMs in the form of evaluation inconsistency. This highlights a challenge in designing robust evaluation protocols that mirror real-world performance.", "questions": "see weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper studied an interesting question: the significance of Ratings versus Rankings in collecting and evaluating preference feedback. To me, this represents an essential and timely investigation.\n- The findings in the paper could shed light on this question and stimulate further research.\n- The paper is well structured and written. The experiments are extensive and carefully designed.", "weaknesses": "I liked this paper a lot but I do have several comments:  \n\n- In Table 2, the authors showed that when converting the rating to ranking, there is a huge disagreement between the two different protocols. One flaw is that a proper baseline is missing here. For example, if you annotate the same examples using another batch of human or sample from the same model (GPT-3.5-Turbo) using different temperature, what would be the disagreement? I would be curious to see such a baseline as it also correlates with the main conclusion of this paper.\n\n- This is not necessarily a weakness but given what observed in the paper, it would be great to see if the authors could take one step further to verify the root cause of the difference and try potential ways to combine the advantages of the two annotation protocols. To me, Figure 3 shows that the ranking-based protocol is more effective and robust (it yield better win rate with ranking-based evaluation protocol and smaller gap when changing the evaluation protocol). This makes sense, as in the rating protocol, the model/human only see each individual answer thus the rating is likely to be less calibrated. Also, seen from the examples in section H, the ranking protocol is likely to introduce some systematic biases. Here is my proposal: can we try the third protocol that instruct the model/human to give both ratings and rankings for either a pair of responses or a list of responses. My guess is that rating annotated in such a way will be better calibrated and the rankings will less likely to be biased.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OYRyVQEE9f", "reviewer": "Reviewer_gmKJ", "summary": "This work provides an intriguing analysis of the issue of the feedback inconsistency problem, where human or model-generated evaluations can be inconsistent across different evaluation protocols (e.g., output A is better than B in pairwise evaluation while if the rating of A is lower than the rating of B). They collect 6k human feedback data under the ratings and rankings protocols, as well as model-generated feedback. They found that inconsistencies are prevalent both in human and AI evaluations. They further conduct quantitative and qualitative analyses to understand the potential factors of these inconsistencies. While such preference or rating data is essential for recent RLHF approaches as well as evaluation for open-ended generations, it is still unclear whether such data is reliable or what kind of factors affect the overall rating. This work provides an interesting analysis of this important area, and sheds light on several underexplored issues. I have several questions and concerns (e.g., inconsistencies from prior findings, validity of the final experimental), overall it has positive and good scientific contributions to ICLR.", "questions": "- Could you prvovide the details of human annotation process of Qualitative analysis?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper provides an in-depth analysis of feedback acquisition by humans and AI, based on 6,000 human ranking and rating annotations for model responses, as well as model predictions. \n- Their analysis reveals the prevalence of feedback inconsistency issues and also provides an in-depth analysis of why it arises.\n- They also found that which type of feedback data a model is trained on has strong effects on the evaluation.", "weaknesses": "I found this paper quite interesting, but the paper reports a set of many different findings, and sometimes their findings are inconsistent with the previous work. Having more detailed discussions on key findings can make this paper stronger (detailed below). I am also not fully convinced by the results of Section 5. Below, I detailed those points. \n\n**Lack of detailed discussions on findings and detailed annotation setup** \n\nParticularily Section 3 and 4 report various interesting phenomena, but some of them lack detailed explanations. For instance, \n\n- Section 3.1 length distributions: despite multiple papers reporting length of responses has a positive correlation with rating, the authors claim there's no difference (\"we find that there is no discernible difference between the average length and average number of unique tokens of the preferred and unpreferred response in the rankings feedback collected from the humans and AI.\"). I wonder if the authors have any insights into this. \n- Section 4.2 Qualitative analysis: the authors say that they sampled a few inconsistent instances and asked annotators to provide explanations for the inconsistencies. I think this is an important and inconsistent analysis and the annotation protocols should be precisely documented. If the claim \"the differences in the preferences of the humans while annotating for different feedback protocols played a significant role in their decision making\" is based on 2-3 instances, the claim may not be fully satisfied. \n\n**The findings of \"Alignment and Evaluation\" section**\n\nI think the findings of Best-of-n policies outperform SFT have been already reported in prior work, and evaluation inconsistency is somewhat predictable given the discussion of inconsistencies. While the second part (inconsistencies) can be novel, I am confused about the descriptions of the results. To my understanding, the finding is if we use a ranking model for Best-of-n ranking we can get higher rates when the same ranking protocol is used during evaluations. For me, it's not really surprising as the reward model is trained on the feedback data and reranks n responses at inference time, so if a model is trained on pairwise feedback data it learns to choose the response preferred in the pairwise setup. It'd be interesting if you could use the feedback data during training (e.g., PPO) and see if the trends remain as well. Yet, I am overall confused with the descriptions in these paragraphs and feel free to correct me if", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3U2UU2ZBlR", "reviewer": "Reviewer_ofRF", "summary": "This paper investigates two feedback protocols (rating and ranking) for the alignment and evaluation of LLMs. It collects AI feedback based on these two settings and uses them to train reward models. The reward models and the LLMs with the best-of-n policies are then evaluated on the annotations of humans and ChatGPT. It conducts a detailed analysis of the characteristics of the collected annotations. It reveals evaluation inconsistencies in which feedback protocols used in alignment algorithms have an advantage over other feedback protocols during evaluation.", "questions": "* Can you explore how the feedback protocol affects the reinforcement learning finetuning for model alignment, such as RLHF?\n* Will the collected feedback be released?\n* Is there any calibration on the rating scores? For example, detail the meaning of each score (1-7) in the prompt for AI feedback and instruction for human annotation to make sure that the annotators can fully understand the principle of evaluation.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* This paper draws attention to feedback inconsistency where the ratings and rankings disagree with each other for the 60% comparison in both humans and AI.\n* This paper investigates the influence of different feedback protocols on reward functions. It sheds light on how we should collect feedback.\n* This paper collects human feedback and AI feedback and conducts a detailed analysis from different aspects.", "weaknesses": "* This paper does not explore the influence of feedback protocols on common alignment methods (such as RLHF[1], RRHF[2], RLAIF[3], etc.). The alignment in this paper just applies the reward models to select the best out of n generation, which is only affected by the performance of reward models.\n* The evaluation inconsistency seems straightforward: the performance of reward models will be affected by the format of input data. It is better to convert the rating feedback to the ranking format first and then use it to train an NLL reward model (just like the ranking feedback) and then compare the performance.\n\t\t\n\t\t\n[1] Training language models to follow instructions with human feedback\n[2] Rank Responses to Align Language Models with Human Feedback without Tears\n[3] Constitutional AI: Harmlessness from AI Feedback", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VeYH9t42VA", "reviewer": "Reviewer_KCah", "summary": "This paper studies the two types of feedback, collecting ratings versus rankings, from both human annotators and AI as an annotator. The authors analyzed both types of collected feedback, observing general inconsistency, and also used them to train reward models, finding that the choice of feedback protocol affects the effectiveness of the reward model (where the trends hold across both human and AI feedback).", "questions": "- How would results in Figure 3 differ across tasks?\n- In Section 2.2 and 2.3, it would be helpful for authors to add references for each to help the reader get a sense of where the protocols and models have been used in prior work.\n- Some typos, e.g., missing link in the first paragraph of Section 3 and “the humans 6k instances of annotations”.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper studies an important problem of understanding the effect of different kinds of human feedback and how they can be used in the training pipeline. \n- The findings, which apply to both human and AI annotations could be useful for informing how people design feedback protocols in the future.", "weaknesses": "In general, the specific details surrounding experimental design were not as well justified, making it difficult to assess the applicability of the findings more broadly. For example, \n- Why were Dolly, User-orient, and SuperNI selected as the tasks of interest? \n- What was the prompt provided for AI annotation? Given the subjectivity of the task, what instructions were given to the crowdworkers when asked to provide ratings / rankings? This is important to justify because the text mentions that crowd workers perceived a response to be “dull”, though it’s not clear what kind of metric crowd workers are / should be using.\n- Additionally, the generalizability of the results may be limited by the choice of model in the various experiments: (1) only Alpaca-7b was tested in terms of generating candidate responses, (2) only GPT-3.5-Turbo was evaluated as an AI annotator, (3) only LORA Alpaca-7b was selected as the reward model, and (4) win-rate was computed only against DaVinci-003. It would be helpful for the authors to clarify why those models were selected in each part of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)