DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4
Authors: Yebowen Hu
Kaiqiang Song
Sangwoo Cho
Xiaoyang Wang
Hassan Foroosh
Fei Liu
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Models
Summarization Evaluation
Human Preference Judgments
GPT-4
TL;DR: A comprehensive analysis of a dataset of pairwise human judgments released by OpenAI, using the Bradley-Terry-Luce (BTL) model to uncover key factors that may influence human preferences.

Abstract: Human preference judgments are pivotal in guiding large language models (LLMs) to produce outputs that align with human values. Human evaluations are also used in summarization tasks to compare outputs from various systems, complementing existing automatic metrics. Despite their significance, however, there has been limited research probing these pairwise or $k$-wise comparisons. The collective impact and relative importance of factors such as output length, informativeness, fluency, and factual consistency are still not well understood. It is also unclear if there are other hidden factors influencing human judgments. In this paper, we conduct an in-depth examination of a collection of pairwise human judgments released by OpenAI. Utilizing the Bradley-Terry-Luce (BTL) model, we reveal the inherent preferences embedded in these human judgments. We find that the most favored factors vary across tasks and genres, whereas the least favored factors tend to be consistent, e.g., outputs are too brief, contain excessive off-focus content or hallucinated facts. Our findings have implications on the construction of balanced datasets in human preference evaluations, which is a crucial step in shaping the behaviors of future LLMs.

DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4
Yebowen Hu, Kaiqiang Song, Sangwoo Cho, Xiaoyang Wang, Hassan Foroosh, Fei Liu
'University of Central Florida TTencent AI Lab, Bellevue, WA (yebowen. hu, hassan. Foroosh) edu [riversong, swcho, shawnxywang tencent. com
*Emory University
fei www@emory..ee.. .du
Abstract
Fabbri al., 2021: Clark et al., 2021). This type of reference-free free evaluation has proven beneficial, especially as systems are increasingly matching the performance humans.
Human preference judgments are pivotal in guiding large language models (LLMs) to pro- duce outputs align with human values. Hu- man evaluations also used in summarization tasks to compare outputs from various systems, complementing existing automatic metrics. De- spite their significance, however, there has been limited research probing these pairwise or k- wise comparisons. The collective impact and relative importance of factors such as output length, informativeness, fluency, consistency still not well understood. It is also unclear if there are other hidden factors in- fluencing human judgments. In this paper, we conduct an in-depth examination of collection of pairwise human judgments released by Ope- nAI. Utilizing the Bradley-Tere -erooy--erry--eccee (BTL) model, we reveal the inherent preferences em- find that the most favored factors vary across tasks and genres, whereas the least favored factors lend to consistent, e.g., outputs are brief, contain excessive off-focus content halluci- nated facts. Our findings have implications on the construction balanced datasets in human preference evaluations, which is crucial step in shaping the behaviors of future LLMs.
Pairwise human judgments have become exten- sively used in recent years for the development of large language models (Ziegler et al., 2020; Stien- non al., 2020; Nakano et al., 2022; Ouyang al., 2022: Menick et al., 2022; Bai et al., 2022; Ra- mamurthy et al., 2023). InstructGPT, for example, learns a reward model from human comparisons. then optimizes against this reward model to gener- ate outputs that align with human preferences. Hu- man ratings are further used in the final evaluations to rank model variants based on specific prompts and their completions (Ouyang et al., 2022). Mod- els with higher winrates considered have edge. Human comparisons have been chosen for their ease and intuitiveness, and they play critical role in shaping the behavior of LLMs.
factual
bedded these human judgments.
Despite their significance, there has been limited research probing human preference judgments. Us- ing summarization as a case study, we examine the characteristics of system outputs that may influence human judgments. We have selected summariza- tion
study, because summaries need to be anchored in their original texts, which facili- tates our evaluation of content accuracy. worth noting that our proposed framework could poten- tially be extended to other scenarios. Evaluators of summarization and text generation systems are typ- ically asked to consider range of factors: informa- tiveness, factuality, fluency, coherence, extractive- ness, non-redundancy, etc. (Howcroft al., 2020; Fabbri et al., 2022; Goyal et al., 2022; Zhang al., 2023). However, the aggregate effect and relative importance of these factors remain elusive. E.g..
1 Introduction
Human judgments are widely used in summariza- tion evaluation, complementing automatic metrics, Automatic metrics often rely on reference texts. They compare system summaries with reference texts based on word overlap, as seen in BLEU and ROUGE (Papineni et al., 2002: Lin, 2004), or con- textualized embeddings, as in BERTScore, Mover- Score, BLEURT, UniEval (Zhang et al., 2019; Zhao et al., 2019; Sellam et al., 2020; Zhong et al., 2022). By contrast, human evaluation usually does re- quire reference texts. Evaluators asked assess the quality of summaries along certain linguistic and content dimensions; they can also rank out- puts from various systems (Bhandari et al., 2020;
'Beyond comparing two model outputs the same input (pairwise comparisons), evaluators use K-wise comparisons for efficiency. These K-wise comparisons are later converted into (*) pairwise comparisons for training reward models.

---

identified for the two summaries, where P, and factual errors can greatly undermine system's oulput. An inaccurate claim made by Google's 's Bard about the James Webb Space Telescope has resulted in a 7% drop in stock price (Vincent, 2023).
Atomic Facts
SYSTEM:
Your task is extract atomic facts from the INPUT. self-cantained units infornation are unanbiguous and require no fur ther splitting.
We hypothesize that there may be other, uncovered factors influencing human judgments.
USER:
In this paper, we examine dataset of pairwise human judgments released by OpenAl (Stiennon al., 2020). Much like a consumer weighing up multiple factors when deciding between two prod- uets, we aim to uncover the factors that evaluators consider when assessing two system outputs. We leverage the Bradley-Terry-Luc model to analyze the strengths of these factors (Bradley and Terry, 1952), which has found applications in psychology. marketing, economics, and evaluation of natural language systems (Dras, 2015; Zopf, 2018; Peyrard et al., 2017, 2021; Sedoc and Ungar, 2020). It mod- els pairwise comparisons. such as sports matches or political elections, by assuming each factor has an inherent strength, and the likelihood of one factor winning over another a function of the difference between their strengths. We develop protocol to identify characteristics of system outputs, ranging from output length and content coverage, to hallu- cinations of various sorts and the usage of complex words. Our research sheds light on inherent prefer- ences embedded in human judgments.
Raheen Sterling has not signed new Liverpeol contract. Sterling gave interview last week suggesting night want laave. Brendan Rodgars insists forward has never expressed desire to
OUTPUT:
("1": "Rahnen Sterling has signed Liverpool contract", "2": "Sterling gave an interview last week", "3"; "The interview suggests Sterling night want to leave", "4t: "Brendan Rodgers insists Sterling has never expressed desire to go")
INPUT:
Forner Barcelona centre back Carles Puyol retired May last year. But he could now be in 1ine for a return with New York City or Al-Sadd. Puyel is reported training abead coneback retirenent.
DUTPUT:
("1": "Carles Puyol retired in Nay". "2"; "Carles Puyol retired last year", "3"; "Carles Puyol Forner Barcelona centre back", 14"; "Carles Puyal could now in line for return with New York City", "Carles Puyal cauld now he in line for neturn with Al-Sand", "5"; "Puyol is reported
training", "y"; "Puyol is ahead of a coreback from retirenent")
INPUT: (Sumarry)
SYSTEM:
You have been provided statement. Does it include any location, temporal, possassive expressions, cuantities? Provide your response JSON formst, with "yes" or "no" decision for each category, along with justifications.
Our paper's 's contributions include: (a) A com- prehensive analysis of a collection of human com- parisons to identify key factors that may influence human judgments. In this analysis, we use summa- rization as a case study with comparisons provided by OpenAI. (b) Using GPT-4's advanced capabil- ities, assess system outputs both qualitatively and quantitatively. We examine their fluency, clar- ity, coverage, alignment with the original text's intent and style, and detect hallucinations based on atomic facts (Liu et al., 2023c). Our study of influential factors holds promise for enhancing the reliability of human evaluations.
USER:
Statement:
(Atoric-Fact)
A prompt for GPT-4 to extract atomic content units from summary. BOTTOM: A prompt to determine whether an atomic content unit contains any location, temporal, possessive expressions, or quantities.
Figure I: TOP:
set of M factors, represented as {a;) examine each summary from multiple perspectives. We consider their style, intent, richness of content, connection to the original text, etc. Some of these factors, such as style, use of complex words, have not been explicitly provided to the evaluators in the instructions (see $3 for further details). We use a and a, (n) to represent the unique factors
M
2 Problem Formulation
Our dataset consists of collection of N summary pairs, denoted by D {(s{") ((x) ) The two summaries, generated for the same input, accompanied by the original text and a human judg- ment of their overall quality, with S(n) denoting the favored summary of the n-th pair. We develop
>=
Qn are the number of factors (Eq. (1-2)). The goal of this study is to identify dominating factors that influence human judgments.
(n) (n) Pr aj { 1p=1
(a)
01 E
A
(1)
"Our data and analyses are publicly available for the re- scarch community: https: http:/deccpppppppeeee github. io/
(n) (n)
(n)
a
{azg)
a2 EA
(2)

---

We use the Bradley-Tery-Luce model (Bradley and Terry, 1952) to rank factors that characterize system outputs. The BTL model frequently used in sports to rank players within a league. Suppose we have MM players who have participated in a num- ber of games against each other. We can represent the outcomes of these games using matrix W; wij ( M) denotes the number of times Player has won over j. If i has never competed against j. we assign Wi,j to O. Further, a player cannot compete against themselves, so W1, (i E M) is also set to W; is the total wins that Player has had against all other players. Given this matrix W, we can calculate the relative strengths of all players using the Bradley-Terry-Luue model.
Hallucination & Focus
SYSTEM:
You have been provided with set of stacements. Does infornation accur ately netch the post? A statement is considered eccurate does intraduce details unmentioned in the post, or contradicts the post's response JSON
each statement
are
existing infornation. Provide fornat, with a "yes" or 'no" decision for each statement in the set, along with fustifications.
USER:
years old.". "Dante de Blesio needs to nake decision. ", "3": "Dante de Blasia needs nake decision by the and of the nenth." ", "4": "Dante de Blasio's father has six-figure salary, "5"; "Dante's rather has said family to Ivy League school.")
INPUT: ("1":"Dante de Blasio
struggle
cost to send Dante
Reddit Post: Dante de Blasto, the son of New York City Bill de Ulasio has been accepted into Yale end Brown Universities. The 17-year-old is a senior at Brooklyn Technical High School and will nake his decision the of nunth. Despite figure salary as nayer de Blasio is expected to turn financial aid help pay for son's education.
We treat each unique factor as a player and aim to understand the strengths of these factors based on pairwise comparisons. When one summary is favored over another, i.e., ((() > sim) we assume all of its factors win over those of the other sum- (x)
six
OUTPUT: ("1"; "decisien": :""ye"" "2": "decision": "yes"" "3": "deciston" :"yes", "4"; "decistoon" "yes", "5"; "decision": "noo")
(n) mary: di,p > 02,q Vp, q, and vice versa. 3 If the
INPUT: (1 (isst-of-Atorrc--aatts
Reddit (Post)
same factor appears in both summaries, assume they cancel each other out and exclude them from the list of factors. This results in {aip f=1 and ( { Q.. being the symmetric difference of the two summaries. Thus, have Pn x On factor comparisons derived from each pair of summaries. The BTL model allows us to estimate the relative importance of these factors, represented as { This is accomplished using an EM-like algorithm, where is iteratively updated (Eq. (3)) to max- imize the data likelihood and then renormalized (Eq. (4)). The BTL model is primary used for pa- rameter estimation in pairwise comparisons (Zhu et al., 2023), making it an ideal fit for our dataset.
SYSTEM:
You have been provided statement. you detensine if it is related to the rain focus of the post? The main focus of all the content revolves. Fornat your response JSON, containing "yes" or "no" decision, along with justifications.
is the subject uround which
USER:
Statement: (Atoric-fact)
Reddit Post: (Past)
A prompt for GPT-4 to verify whether each atomic content unit accurately matches the original information, helping us detect hallucinations. BOTTOM: A prompt to check if the atomic content unit is related to the main focus of the original text, which helped us detect off-focus content.
Figure 2: TOP:
Wij + Wij
14 = W
(3) summaries, which models summaries and original texts based on low-level contextualized representa- tions (Luo et al., 2023; Liu al., 2023a; Gao et al., (4) 2023). Our objective in this paper uncover the inherent preferences in pairwise human judgments. By doing so, we aim to establish robust sample collection practice for reward models.
Dí + Dj
jzi
P// Pt
D
3 Determinants of Human Preferences
We explore a variety of factors that may influence human preferences. These factors are high-level, interpretable descriptors of the summaries and their original texts. Our work distinguishes from previ- ous studies using LLMs to evaluate the quality of
Length. Previous research suggests that human evaluators might show bias towards lengthier summaries (Liu al., 2023c). Indeed, longer summary can seem more credible due to its com- prehensiveness. Conversely, a short summary may miss some critical content due to its brevity. Eval- uators may naturally choose the lengthier output
"This assumption, like the conditional independence in Naive Bayes, is quite strong. However, the BTL model gener- ally performs well and shows relative resilience to its violation.

---

Fluency & Clarity
Style & Intent
SYSTEM:
SYSTEM:
Is the summary easy to understand and free of grammttccal errors? Fomat response 19ON, justificatioos.
Is the sumaryy written in the same style as the original post? Potential styles include formal, casuel, hunerous, sarcastic, narrative, Instructional, diary-lske, and nore. Fomat your response J50N, containing a "yes" 'no" decision, along with justifications.
conteining "yes' or "no' decision, slong with
USER:
Sumrery: (Sunnary)
USER:
Summary: (Sumaryy)
SYSTEM:
Reddit Post: (Past)
sumerry express clearly and unarbiguously? Fornat your response in ISON, containing "yes' "no' decision, elong justifications.
SYSTEM:
Does the sumeery serve the same purpose as the include soliciting original post? Potential asking questions, advice, sharing infornation, providing support, seeking recomenndttons,, and more. your respense in conteining 'yes' "no" deciston, along with justifications.
USER:
Sumrary:
(Surnary)
Figure 3: Two prompts for GPT-4 to assess whether given summary fluent (ToP) or clear (BOTTOM).
USER:
Sumary; (Surnary)
Reddit Post: (Post)
without verifying its content. Despite this, there has been little effort to quantify the influence of summary length. In this study, we measure the length of summaries by counting the number of tokens or characters, then categorize all summaries into quartiles based on their length. E.g.. the fac- tor len tk-medium indicates the summary's ' length falls into the second quartile when measured by to- kens. We have the following length factors:
Figure Two prompts for GPT-4 check if the given summary aligns with the original text in terms of style (ToP) and intent (BOTTOM).
GPT-4. Our instructions are adapted from Stiennon et al., (2020) and illustrated in Figures 3 and 4.
(stylel intent]--algnee
(fluent unambi
len-(tk (ann(tkkcchjpp ch)--<hortt medium long xlong)
Content Accuracy. Evaluators should disapprove summaries that contain hallucinated content. We measure this aspect by counting the number of hallucinated Atomic Content Units (ACUs). An ACU is self-contained information unit that does not require further breakdown, such 'Raheem Sterling has signed a new Liverpool contract." Using expert-annotated ACUs provided by Liu al. (2022) as in-context examples, we employ GPT- to extract atomic facts from a given summary, as illustrated in Figure 1.
Linguistic Quality. Assessing linguistic quality with 5 or 7-point Likert scale can be ineffective due subjective interpretation (Howcroft al., 2020). This issue is often exacerbated by the lack of clear guidelines for evaluators. Our approach models each linguistic quality as a binary decision, thus enabling clear judgement on each summary. We measure (a) fluency: whether the summary is easy to understand and free of grammatical er- rors; (b) clarity: ty: if the summary expresses ideas clearly and unambiguously."
We determine if the factual information within each ACU aligns accurately with the original text using GPT-4; our prompt shown in Figure An ACU is deemed accurate it neither introduces un- mentioned details nor contradicts the original text. We sort all summaries into four categories based on the number of fabricated ACUs they contain: none, one, two, more. The underlying assumption is that summary should include minimal, in any at all, hallucinations. In addition, fabricated elements, such as location, time and possessive expressions, and quantities can lead major misinterpretations. Thus, we define 'hallucination mixed-typa fac- tor to flag any summary that includes at least one
A summary's style and intent, as it relates to the original text, can also impact human perception. We evaluate (inlaaainnnn iggment: if the summary is written in the same style as the original text, c.g., formal, casual, humorous, sarcastic, etc., and (d) intent-alignmentt if the summary serves the same purpose as the original text, e.g., soliciting advice, sharing information, etc. We derive four factors leveraging the exceptional capabilities of
*Given that text coherence focuses on the fluidity of con- tent at the level of paragraphs or larger sections, we decided to exclude it from our criteria. This decision helps mitigate over- lap with fluency and accounts for the brevity of summaries.

---

hallucinated ACU that contains these expressions. hallucinatiion-fcc--ooooo one two| many] Mallucination-mixedee
Pairwise Judgment
SYSTEM:
A good summary is shorter piece of text that has the essence the original. tries to accomplish the sane purpose and conveys the sene information as the original post. Analyze the provided summartes and original post, then select better sumary.. Output the result in ISON fornst. Here is an exarple of the output fornat:
Connection to the Original Text. Does the source content coverage impact human judgments? To answer this, we calculate a "coverage score', which represents the ratio of reused content to the total content in the source text. We compute reused con- tent using Grusky et al.'s (2018) greedy algorithm that aligns text fragments of summary with the original text. We also consider whether the sum- mary mostly reuse individual words or consecutive fragments from the source text. This is quantified using their "density score', which favors consecu- tive reused snippets by squaring their lengths.
("better summary I0":"", "justification:")
USER:
a) Reddit Post: (Post)
Summary 1: (Sunrary-1)
Sunmary 1: (Sumary--)
Summary 2: (Suntery-2)
Sumsary 2: (Sumaryy22
d) Reddit Post: (Post)
b) Reddit Post: (Post)
Sumnary 2: (Sumrrry-2)
Summary 2: (Sumtary-2)
Summary 1: [Sumrery-1)
Summary 1: (Sumrary-1)
e) Redditt Post: (Post)
Reddit Post: (Post)
Sunnary (Sumrary-2)
Sunrary 1: (Sumrrry-2)
Summory 2: (Sumnary-1)
Summary (Sumary-1)
Reddit Post: (Post)
It's important to note that these methods assess the extent of coverage, not necessarily the impor- tance of the covered content. We quantify the total number of 'off-focus' facts in the summary, com- puted as the number of ACUs that originate from the source document, but may not represent the main content. We leverage GPT-4 to evaluate any atomic fact is related to the main focus of the original text, where the main focus is the core sub- ject around which all the content revolves (Table 2).
Figure 5: A prompt for GPT-4 to choose the better summary from given pair. We noticed that the order of summary presentation and positioning of the origi- nal text affect the model's 's performance. For example, scenarios a) and b) do not consistently yield the same prediction, and scenarios b) and c) do not consistently produce opposite predictions. Based on a pilot study with six different combinations, we decided to use a) for final experiments for its stability and performance.
off off-focus-nnone one | two mmany
and CNN/DM datasets. The summaries used for evaluation generated using their human feed- back models and several baselines. Our data splits are constructed in the following manner:
src-cov-(minimal low mediumi high}
consec-cov-mming mmlnimaa low meddi um high}
Word Choice. We use the algorithm introduced by Grusky et (2018) to match text fragments in a summary with the original text. We term 'novel words' as the proportion of the summary that is not aligned with the original text. Further, we identify 'complex words' as those that need to be broken down into several tokens by tokenizer. We then compute the percentage of summary tokens that part of complex word. Based these measures, we categorize all summaries into four quartiles, prefixed with woods--worss'' and 'complex-we words- respectively.
comparisons-reddix: Human evaluators indi- cate both the better summary and their confidence in that selection using a 9-point slider. A score of or 9 means that one summary is 'definitely' preferred over the other. The validation split of this dataset has around 83.8k annotated pairwise comparisons, from which we randomly selected 5k pairs for our experiments. Our initial analysis shows this distribution of pairs across confidence levels: 26% are 'possibly' better, 25% are 'likely' better, 19% are 'very likely' better, and 30% are 'definitely' better.
novel-words- some many most]} complex-words-(few some maan mmost]
axis-evals-(reddit |cndmm) Each summary is individually assessed by an evaluator using a 7-point Likert scale. We specifically consider the overall quality of the summary (how good is the summary overall representing the post?). A score of means the summary is terrible while a score of suggests that it is excellent. There are around Ik Reddit posts and 0.6k CNN/DM arti- cles, each having multiple annotated summaries.
4 Data
In this study, we examine a large dataset released by OpenAl (Stiennon al., 2020), which includes la- beled comparisons between pairs of summaries (re- ferred to as "comparisons"" for the Reddit TL;DR dataset and human ratings of summaries along mul- tiple axes ("axis evals") for both Reddit TL;DR

---

Rank
Most Favored Factor
Estimate
Rank
Least Favored Factor
Estimate
intent-aligne aignned
0742
hallucination-fact
.0067
2
unambiguous
0091
2
off-focus-man
.1000
3
style-aligned
.0499
3
off-focus-tww
.0126
fluent
.0288
src-cov-minimal
.1300
5
sre-cov-high
0395
5
len-ch-short
.0339
off-focus-nom
0378
hallucination- fict---ot---tt---
0142
hallucinatioonfacttrron
0377
hallucina wiwninntttooiiie---ee---e
.1660
len-ch-xlong
.0335
consec-cov-minime
.0161
9
len-ch-long
.0320
len-tk-short
.01 64
consec-cov-high
.0316
off-focus-one
.0171
Table Most and least favored factors in our comparisons-rredit dataset identified the BTL model.
We pair these summarics and randomly pick two pairs per document, ensuring that they have dif- ferent ratings for overall quality. This process gives us total of 2,058 Reddit TL;DR summary pairs and 1,254 CNN/DM pairs.
which is rated as the worst among all evaluated. The "hallucination--aatttwwo and 'hallucination- mixed-types' factors also rank among the least fa- vored. Likewise, we observe that the inclusion of ir- relevant content in the summary is not preferred. In particular, including two or more off-focus atomic facts has negative impact on the system outputs.
5 Experiments
this section, we discuss various factors that af- feet human preferences. We quantitatively measure their degree of influence, how accurately these fac- tors can be extracted from pairs of summaries, how factors correlate with one another, and their vary- ing effects across summarization tasks and source domains. Further, we evaluate how top-performing GPT models fare on this task. To ensure the repli- cability of our results, we set the temperature to when using all GPT models.
Linguistic Quality. Interestingly, our research in- dicates that human evaluators show a strong pref- erence for certain linguistic aspects of summaries. Particularly, "intent-aligned consistently ranks the highest in both datasets: compar iicrrssooossoo moons-reddi t and terssevals--eede This suggests that maintain- ing the original intent of the post, such seeking advice, sharing information, asking questions, providing support, is of importance mary. It is also crucial that the summary is fluent, maintains the original post's style, and expresses ideas clearly and without ambiguity.
Reddit sum-
5.1 Factors That Influence Human Preference
Utilizing the BTL model ($2), we identify the most and least preferred factors on our 'compari -ons- sons- reddit' dataset; results shown in Table 1. Ad- ditional results for "axes-evals- (reddii cnndm)' can be found Tables and 6 in the supplementary materials. We make the following observations:
Length. Our findings show that human evaluators tend to favor longer news summaries while show- ing dislike for shorter ones, as seen in Table 6. This is because longer summaries can provide more details about the news stories. However, this prefer- ence for length maries. Particularly, is not case of 'the longer, the better'. Summaries that are excessively long, denoted as 'len-ch-xlong. only ranks 8th terms of preferred factors. On the flip side, summaries that are overly short, denoted as 'len-ch-short'' can negatively affect human preferences. As result, our recommendation is to tailor output length to the specific task, with greater emphasis length for news summarization.
as obvious with Reddit sum-
Hallucination & Focus. Not having any halluci- nations ("hallacination-fact-oo
is not the most influential factor on human preferences. However, too much hallucination certainly harms system out- puts. This seen with "ballucinatioon-actt-maaa
'An alternate approach gathering pairwisc data is pair- ing system outputs with reference texts. Summaries written by editors have traditionally served as reference texts in news summarization. However, recent research indicates that LLMs have achieved human-level performance on news datasets such as XSum and CNN/DM (Zhang et al., 2023). As a result, as- suming system outputs are inferior to reference texts without further human evaluation is no longer valid. We leave auto- matic collection of pairwise data for future work and focus on human judgments provided by OpenAl in this study.
Miscellaneous.
We observe that system sum- maries tend to suffer when there minimal cover- age the source content, judged by either individ- ual words
or consecutive chunks

---

GPT Models
Confidence Level
Pairs
davinci-000
3.5-turbe
gpt-4
Summary 1 is possibly better than Summary 2
1,273
53,54
56.71
61.01
Summary 1 is likely better than Summary' 2
1,227
60.81
65.15
72.63
Summary 1 is very likely better than Summary
964
59.25
66.84
74.74
Summary 1 is definitely better than Summary 2
1,536
67.01
74.26
81.46
All Pairs Combined
5,000
60.56
66.15
72.79
Table 2: Accuracy of various GPT models when selecting the better summary from given pair, evaluated on the comparisons- dataset.
GPT Models
only one of these factors will be activated. More- over, there is notable correlation between 'short output length' and 'minimal source content cov- erage.' as short summaries often leave out salient source content due to brevity, Preferred linguistic traits, i.e., 'fluent', 'unambiguous' 'style-aligned" and alignedd' correlate positively with high content quality ('off-focus-none' "hallucination- none"). They are negatively correlated with the use of new and complex words, multiple hallucinated facts or off-focus content. This suggests that better language and content qualities are likely attained together when LLMs' abilities are improved. We believe there might complex three-way way interac- tions among these factors, further complicating the preference landscape.
Gap
Pairs davinci-000
3. 5-trbo
gpt-4
1
802
56.25
65.98
68.82
2 576
66.26
72.61
80.87
3
358
71.42
82.67
87.71
4
179
69.83
87.50
92.13
5
101
85.00
94.95
98.02
6
42
82.93
97.61
100.00
All
2,058
64.81
74.70
79.57
Table Accuracy of various GPT models when select- ing the better summary from given pair, evaluated on the viiis-eval-reddi- dataset. Gap refers to the difference in overall scores between two summaries.
of text (consec-cov-iniime This pattern holds true across all datasets. Factors that do not fall into either the most favored or least favored categories include: medium length, single hallucinated fact off-focus content unit, moderate source content coverage, and nearly all factors tied to the use of novel or complex words. Generally, these factors do not significantly enhance or harm the quality of the summaries."
Accuracy of Factor Extraction. We use GPT-4 to extract atomic content units (ACUs) from sum- maries for our analysis. To understand the efficacy of GPT-4, we randomly select 50 summaries for manual validation. Our findings indicate that GPT- 4 accurately processed 43 out of 50 summaries. yielding accuracy of 86%. The majority of error instances are due the informal nature of Reddit posts. For example, "What do?" 2d is casual expres- sion for "What should do?", seeking advice on a specific situation. GPT-4 has failed to extract an ACU from it. We believe that parameter-efficient fine-tuning might enhance LLMss ability to process such informal text (Li et al., 2022, 2023b). GPT-4 achieves accuracy of 89% detecting halluci- nation instances and 83% in checking if ACU was relevant to the main focus of the original text.
Understanding Factor Correlations. The inter- play among factors can be complex, partly due to the complicated nature of human decision making... We use Kendall's Tau to measure the relationships between these factors. Results are shown in Figure 7 in the supplementary material. We observe that, factors within each category, such hallucination- fact- (ot-(nno (none, one, two, many), show negative cor- relations. This because for each system output,
Cross-Source Analysis. We observe that factors influencing summarization tasks can differ across varying source domains. Figure 6 shows a compar- ison on the Reddit and CNN/DM. For news sum- marization, summaries that are longer and more
"We test the BTL model's robustness using nonsensical factors. Our findings indicate that when random factors are introduced, the BTL model does not rank them among the most or least favored, suggesting the model is robust.

---

Reddit vs CNN/DM
the comparisons-redde dataset, there are four splits labeled as Summary being (definitely, very likely, likely, possibly) better than Summary 2. Similarly, the axis-evals-reddi dataset has six splits based on the gap of overall scores. A Gap of I represents minimal difference in overall quality, while Gap of 6 indicates the maximum difference. In all cases, GPT-4 consistently outperformed other models regardless of the dataset. However, when two summaries had similar quality levels, iden- tifying the better one was quite challenging. In this category, the best-performing GPT-4 model achieved accuracies of 61.01%% % and 68.82% for the two datasets, respectively."
inteed-llgned
unambiguous
fuunnyy
lenkkiongg
syle algned
lenich-ong
offfocus none
lan-d-XCong
covme
halu-none
an modd
sro-osehigh
ler-ik-long
nove w some
corsec-coy-iihh
eff-focus-many
Factors
complex
len-ik-med
novel-e-many
Ian dh medi
hallwane
compas-s-scom
6 Related Work
conser cov low
wr-coolow
novel-wdfew
Evaluation of LLMs. Human evaluation of LLMs and other NLP systems has become more important than ever, as these systems are prone to inherent bi- ases and may generate hallucinated facts (Maynez et al., 2020; Lebanoff et al., 2020a,b; Liang et al., 2022; Cao et al., 2022; Laban et al., 2023). Pair- wise preference judgments are often selected for their simplicity and intuitiveness. They require less cognitive effort than rating individual model out- puts on Likert scale (Dras, 2015; Perez-Ortiz and Mantiuk, 2017). In recent years, GPT-4 has been employed as a surrogate for human evaluators to conduct these preference judgments (Chiang et al., 2023; Liu et al., 2023a,b). The goal of our study is to identify the key factors derived from pairwise comparisons to enhance the transparency of human preference judgments.
nevel e most
offéceue-one
wmploxewfew
srommommn
consec min
halu-many
hallstwo
focus
Ien ch short
CNROOM
BTL Estimate
Figure 6: Factors impacting summarization tasks can vary across different source domains. For news summa- rization, which shown in blue, lengthier summaries are often favored (Len-[ch| tk)-xxong)) is compre- hensive coverage of the source content ([src] consec)- cov-high). The style of the summary reflects that of the original article (style-al igned), and off-focus content tends to have minimal impact coff-oocus-many)
Human-AI Alignment. LLMs are often adapted to encourage preferred model behaviors and dis- courage undesired ones based Iearned reward function, process often known alignment (Köpf et al., 2023; Wolf et al., 2023). Previous research has explored various reward functions to guide lan- guage generation. These range from maximizing ROUGE scores to using question answering based based rewards (Pasunuru and Bansal, 2018; Peyrard and Gurevych, 2018; Arumae and Liu, 2019; Laban et al., 2020; Yaday et al., 2021: Yu et al., 2022). Additionally, human feedback for LLMs mani- fest in various forms, including natural language- based and more fine-grained feedback (Scheurer et al., 2022: Wu et al., 2023; Lee et al., 2023). Our
comprehensive tend to be preferred, since they pro- vide a more detailed coverage of the source content. Off-focus content appears to have a minimal im- pact, as ACUs extracted from summaries usually maintain relevance to the focus of the news article.
5.2 Results of Pairwise Judgments
The accuracy of various GPT models on pairwise judgments is shown in Tables 2 and 3. We use text- davinci-003, gpt-3. 5-turbo-0301, and gpt-4- 0314 for this study. In particular, text-davinci- 003 allows instraction-following is optimized for chat and performs well in text completion tasks. gpt-4 is the latest model made available through OpenAl's API.
We anticipate that enabling GPT- to reason over salient factors could potentially enhance its ability to judge the quality of system outputs. However, this is beyond the current study.
We divided our data into multiple splits based on the confidence level evaluators have assigned. In

---

research into the influential factors that affect hu- man preferences can provide valuable insights for reward factorization. Such factor analysis may also help researchers mitigate potential biases in the alignment between humans and AI.
for the impact of cultural and demographic factors on preference judgments. We also omit considera- tions related to toxicity, which refers to rude, dis- respectful, sexual, or violent content. These types of content are extremely rare in our dataset but could still significantly impact human preferences. Our research primarily focuses on the linguistic and content-related factors, such output length, source content coverage, intent, fluency, and factual consistency. While these factors are identified as significant, there may be other unexplored factors that we did not consider in this study. In addition to these predefined factors, would be interesting to automatically infer other influential factors, which we leave for future work. Finally, our findings are based on specific tasks and source domains, and caution should be taken when generalizing these results other scenarios.
Preference Modelingg Preference modeling al- lows us to incorporate arbitrary prior knowledge about users into the learning process a declar- ative way (Lu and Roth, 2012). is critical for the output from LLMs to respond to variety of user attributes, ranging from interaction history to the situation of use. In the past, researchers have investigated user modeling for various tasks, in- cluding headline generation, dialog response gen- eration and recipe creation (Majumder al., 2019; Flek, 2020; Wu et al., 2021; Dudy et al., 2021: Cai et al., 2023). In this paper, our focus has been on exploring human preference modeling for LLM development.
Ethics Statement
Human preferences can be influenced by various factors. In this study, preference judgments were provided by OpenAL. The demographics of eval- uators could influence these judgments. OpenAl indicates that their evaluators, recruited through Upwork or Scale AL, are quite young, with 75% be- ing under 35 years old (Ouyang et al., 2022). The gender distribution is fairly balanced, and most evaluators come from the U.S. or Southeast Asia. The researchers who provide instructions and inter- act with evaluators also impact study outcomes. We believe that taking human factors into account when modeling preferences important for future research in this domain (Li et al., 2023a).
The human preference data used in this study were provided by OpenAI. The data do not include user demographics or personally identifiable informa- tion and were released solely for research purposes. Our study aims advance the understanding of hu- man preferences in order to guide the development of LLMs and does not make use of technology re- lated to user demographics. We acknowledge that our understanding of the factors influencing prefer- ences is continually evolving as user behaviors may change over time. We emphasize the importance of constructing balanced datasets for learning reward functions. If certain aspects are disproportionntely represented, the reward function may not learn ef- fectively, potentially leading inherent biases. We are committed to the responsible development of LLM systems, ensuring that they respect human values and that they are designed to mitigate poten- tial biases.
7 Conclusion
We aim to uncover the factors that influence human preferences. Our research indicates that key factors are tied to users' information needs. Users expect LLMs understand their infornatiin-ssekkngg in- tents, produce precise responses without irrelevant content or hallucinations, and generate moderate- length outputs of high linguistic quality. Future research may also consider developing a balanced set of examples for human preference labeling to guide model behaviors more efficiently.
Acknowledgements
We would like thank the reviewers for their in- sightful feedback, which greatly enhanced our pa- per. This research has been partially supported by National Science Foundation grant IIS-2303678.
Limitations
While the Bradley-Terry-Lue (BTL) model em- ployed in our study is robust and widely applicable, it may not fully capture the complexities of human preference judgments. Our study did not account
References
Kristjan Arumae and Fei Liu. 2019. Guiding extrac- tive summarization with question answwerngg rewards. In Proceedings of the 2079 Conference of the North

---

American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pages 2566-2577, Minncapolis, Minnesota. Association for Computa- tional Linguistics.
Mark Dras. 2015. Squibs: Evaluating human pairwise preference judgments. Computational Linguistics. 41(2)3309-317.
Shiran Dudy, Steven Bedrick, and Bonnie Webher. 2021. Refocusing on relevance: Personalization in NLG. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages $190-5202, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield- Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2204.05862.
Alexander Fabbri, Chien- Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. QAFactEval: Improved QA- based factual consistency evaluation for summariza- tion. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, pages 2587-2601, Seattle, United States. Asso- ciation for Computational Linguistics.
Manik Bhandari, Pranav Narayan Gour, Atabak Ash- faq, Pengfei Liu, and Graham Neubig. 2020. Re- evaluating evaluation in text summarization. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 9347-9359, Online, Association for Computa- tional Linguistics.
Alexander R Fabbri, Wojciech Kryšciński, Bryan Mc- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summariza- tion evaluation. Transactions of the Association for Computational Linguistics, 9:39 1-09.
Lucie Flek. 2020. Returning the N to NIP: Towards contextually personalized classification models. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7828- 7838, Online. Association for Computational Lin- guistics.
Ralph Allan Bradley and Milton F. Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)3224- 345.
Pengshan Cai, Kaiqiang Song, Sangwoo Cho, Hongwei Wang. Xiaoyang Wang, Hong Yu. Fei Liu, and Dong Yu. 2023. Generating user-engaging news headlines. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL), Toronto, Canada. Association for Computational Lin- guistics.
Minggi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Ship- ing Yang, and Xiaojun Wan. 2023. Human-like sum- marization evaluation with chatgpt. arXiv preprint arXiv:2304.0233 025554.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3, arXiv 2209.
Meng Cao, Yue Dong, and Jackie Cheung. 2022. Hal- lucinated but factual! inspecting the factuality of hallucinations in abstractive summarization. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 3340-3354 Dublin, Ireland. Associa- tion for Computational Linguistics.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume I (Long Pa- pers), pages 708-719, New Orleans, Louisiana. As- sociation for Computational Linguistics.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
David M. Howcroft, Anya Belz, Miruna-Adriam Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Fimiel van Miltenburg. Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings of the I 13th International Conference Natural Language Generation, pages 182. Dublin, Ireland. Association for Computational Lin- guistics.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah Smith. 2021. All that's 'human' is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference Natural Language Processing (Volume I: Long Papers), pages 7282-7296, Online. Association for Computational Linguistics.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens. Abdullah Barhoum, Nguyen Minh Duc, Oliver

---

Stanley, Richard Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arny Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. 2023. Openassistant conversations democ- ratizing large language model alignment.
Ce Zhang, Christian Cosgrove, Christopher D. Man- ning, Christopher Ré, Diana Acosta-Navar Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Lad- hak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang. Keshav Santhanam, Laurel Orr, Luciaa Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang. Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang. Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda, 2022. Holis- tic evaluation of language models. arXiv preprint arXiv:2211.09110
Philippe Laban, Andrew Hsi, John Canny, and Marti A. Hearst. 2020. The summary loop: Learning write abstractive summaries without examples. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5135-5150 On- line. Association for Computational Linguistics.
Philippe Laban, Wojeiech Kryšcińiski, Divyansh Agar- wal, Alexander R. Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. 2023. Lims as factual reasoners: Insights from existing benchmarks and beyond.
Chin-Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- rion Branches Out, pages 74-81, Barcelona, Spain, Association for Computational Linguistics.
Logan Lebanoff, Franck Dernoncourt, Doo Soon Kim, Lidan Wang. Walter Chang, and Fei Liu. 2020a. Learning to fuse sentences with transformers for sum- marization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP)), pages 4 4136-4142, Online. Association for Computational Linguistics.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023a. Gpte- val: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303. 16634.
Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Dragomir Radev, and Arman Cohan. 2023b. On learning to summarize with large language models as references.
Logan Lebanoff, John Muchovej. Franck Dernoncourt, Doo Soon Kim, Lidan Wang, Walter Chang, and Fei Liu. 2020b. Understanding points of correspondence between sentences for abstractive summarization. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics: Student Re- search Workshop, pages 191-198, Online. Associa- tion for Computational Linguistics.
Yixin Liu, Alexander Fabbri. Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, et al. 2022. Re- visiting the gold standard: Grounding summariza- tion evaluation with robust human evaluation. arXiv preprint arX arXiv:2212.079%
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop. Victor Car- bune. and Abhinav Rastogi. 2023. Rlaif: Scaling reinforcement learning from human feedback with ai feedback.
Yixin Liu, Alexander R. Fabbri, Yilun Zhao, Pengfei Liu, Shafiq R. Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2023c. Towards in- terpretable and efficient automatic reference-based summarization evaluation. CoRR, abs/2303.00 0.06000.
Ella Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy F. Chen, Zhengyuan Liu, and Diyi Yang. 2023a. Coannotating: Uncertainty-
Wei Lu and Dan Roth. 2012. Automatic event extraction with structured preference modeling. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 835-844, Jeju Island, Korea. Association for Computational Linguistics.
work al- location between human and large language models for data annotation. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing.
Jonathan Li, Will Aitken, Rohan Bhambhoria, and Xi- aodan Zhu. 2023b. Prefix propagation: Parameter- efficient tuning for long sequences.
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for abstractive text summarization. arXiv preprint arXiv:2303. 15621.
Jonathan Li. Rohan Bhambhoria, and Xiaodan Zhu. 2022. Parameter-efffcient legal domain adaptation. Proceedings of the Natural Legal Language Pro- cessing Workshop 2022, pages 119-129, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com- putational Linguistics.
Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni. and Julian McAuley. 2019. Generating personalized recipes from historical user preferences. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing DEMNLP-ICNLAS pages 5976-5982, Hong Kong, China. Association for Computational Linguis- tics.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wa, Ananya Ku- mar, Benjamin Newman, Binhang Yuan, Bobby Yan,

---

Joshua Maynez, Shashi Narayan. Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, On- line. Association for Computational Linguisties.
Human Language Technologies, Volume (Short Pa- pers), pages 654 660, New Orleans, Louisiana. As- sociation for Computational Linguisties.
Maxime Peyrard, Wei Zhao, Steffen Eger, and Robert West. 2021. Better than average: Paired evaluation of NLP systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joins Conference on Natural Language Processing (Volume 1: Long Papers), pages 2301-2315, Online. Association for Computational Linguistics.
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young. Lucy Campbell- Gillingham. Geoffrey Irving. and Nat McAleese. 2022. Teaching language models to support answers with verified quotes. arXiv preprint 2203.11141
Rajkumar Ramamurthy, Prithviraj Ammanabrolu. Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage. Hannanch Hajishirzi, and Yejin Choi. 2023. Is reinforcement learning (not) for natural lan- guage processing: Benchmarks, baselines, and build- ing blocks for natural language policy optimization. arXiv preprint 2210.01241.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu. Long Ouyang, Christina Kim. Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. Webgpt: Browser- assisted question-answerti with human feedback. arXiv preprint 2112.09332.
Jérémy Scheurer, Jon Ander Campos. Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2022. Training language models with language feed- back.
Long Ouyang, Jeff Wu, Xu Jiang, Diego Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.022
João Sedoc and Lyle Ungar. 2020. Item response theory for efficient human evaluation of chatbots. In Pro- ceedings of the First Workshop Evaluation and Comparison of NLP Systems, pages 21-33, Online. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia. Pennsylvania, USA. Association for Computational Linguistics.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text genera- tion. In Proceedings of the 58th Annual Meeting of the A Association for Computational Linguistics, pages 7881-7892, Online. Association for Computational Linguistics.
Nisan Stiennon, Long Ouyang. Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. In Ad- vances in Neural Information Processing Systems, volume 33, pages 3008-3021
Ramakanth Pasunuru and Mohit Bansal. 2018 Multi- reward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 646-653, New Orleans, Louisiana. Association for Computa- tional Linguistics.
James Vincent. 2023.
Google's ai chatbot bard makes factual error in first demo.
https://www.theverggecom wttpee/wwwwr....ommoooo///oo///r srcchuttttuttmmiiiu---oorroooooo
Maria Perez-Ortiz and Rafal K. Mantiuk. 2017. A prac- tical guide and software for analysing pairwise com- parison experiments.
Yotam Wolf, Noam Wies, Oshri Avnery, Yoav Levine. and Amnon Shashua. 2023. Fundamental limitations of alignment in large language models.
Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017. Learning to score system summaries for bet- ter content selection evaluation. In Proceedings of the Workshop on New Frontiers in Summarization, pages 74-84, Copenhagen, Denmark. Association for Computational Linguistics.
Yuwei Wu, Xuezhe Ma, and Diyi Yang. 2021. Personal- ized response generation via generative split memory network. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 1956-1970, Online. Association for Computational Linguistics.
Maxime Peyrard and Iryna Gurevych. 2018. Objective function learning to match human judgements for summarization. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics:
optimization-
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari

---

Ostendorf, and Hannaneh Hajishirzi. 2023. Fine- grained human feedback gives better rewards for lan- guage model training.
Summary
Graduated Dec. 2010 (Computer Science), can't find tech job MS, New Orleans, Francisco. Looking for tips/places to apply.
Shweta Yadav, Deepak Gupta, Asma Ben Abacha, and Dina Demner-Fushman. 2021. I. Reinforcement learning for abstractive question summarization with question-aware semantic rewards. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume Short Papers), pages 249-255, Online. Association for Computational Linguistics.
len- tk-medium len-ch-medium
hallucination-fact--o
src-cov-medium
consec-cov-low
novel-words-most
complex-words-man
Summary
I'm recent college graduate with degree in computer scienee. have desperate for a job. I'm worried that my lack of experience is hurting me.
Youngjae Yu, Jiwan Chung, Heeseung Yum, Jack Hessel, JaeSung Park, Ximing Lu, Prithviraj Ammanabrolu, Rowan Zellers, Ronan Le Bras, Gunhee Kim, and Yejin Choi. 2022. Multimodal knowledge alignment with reinforcement learning.
work experience and
len-tk-long
len-ch-xlong halluciinttion-factoorr
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019, Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.09675
helluciintiionmiixed- type
sro-cov-high
consec-cov-medium novel-words-many
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2023. Benchmarking large language models for news summarization. arXiv 2301.13848.
complex-words-some
Table 4: Example summaries and their related factors. a factor both summaries, we consider neutral and remove it from the list.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris- tian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized em- beddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference Natural Language Pro- cessing /EMNLP-IJCNLP) pages 563-578, Hong Kong, China. Association for Computational Lin- guistics.
Language Technologies, Volume (Long Papers), pages 1687-1696, New Orleans. Louisiana. Associa- tion for Computational Linguistics.
A Example Factors
We provide an example of a pair of summaries and their related factors in Table 4.
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards unified multi- dimensional evaluator for text generation. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 2023 2038, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
B Factors That Influence Human Preference
We showcase the influential factors identified via the BTL model for the 'axes-evals-reddi and "axes- exee-eevass-cnddm datasets in Tables 5 and 6 re- spectively. In addition, evaluate the correlations among these factors. Re- sults shown in Figure 7.
Banghua Zhu, Jiantao Jiao, and Michael I. Jordan. 2023. Principled reinforcement learning with human feed- back from pairwise or k-wise comparisons. arXiv preprint 2301.11270.
use Kendall's Tau to
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2020. Fine-tuning lan- guage models from human preferences. arXiv preprint /909.08593.
Markus Zopf. 2018. Estimating summary quality with pairwise preferences. In Proceedings of the 2018 Conference of the North American Chapter of Association for Computational Linguistics: Human

---

Rank
Most Favored Factor
Estimate
Rank
Least Favored Factor
Estimate
intent-aligned
0915
len-ch-short
.0944
unambiguous
0798
len-tk-short
3
fluent
0577
3
off-focus- two
.0105
len-tk-long
.0355
hallucination-facte two
.0155
5
style-aligned
0385
5
hallucination-fert facttint----aanny
.0107
len-ch-long
.0347
consec-cov-minnnall
.0130
7
off-focus-none
.0346
7
sre-cov-minimal
.0132
len-ch-xlong
.043
complex-words-
.0153
9
consec-cov-madium
00399
off-focus-ome
.0133
hallucinaation-aact-non
.0100
halluciiation--iied
.0163
Table 5: Most and least favored factors in our axis exis-evals-rediit dataset identified by the BTL model.
Rank
Most Favored Factor
Estimate
Rank
Least Favored Factor
Estimate
len-tk-xlong
.0911
off-focus-many
.0034
len-ch-xlong
.0875
len-ch-short
.0053
3
style-aligned
.0873
3
len-tk-short
.0057
src-cov-high
.0750
off-focus-two
.0062
5
intent-aligned
.0500
5
mallucination--act-mee
.0033
unambiguod
.0549
sre-cov-minimel
.0067
consec-cov-higg
.0499
consec-cov-minimal
.0079
8
off-focus-none
.0317
8
hallucination- factiino-----otooooooo
.0101
novel novel-words-i some some
.0296
hallucination-mixer typee
.01
10
selluiinttio-fact-non
.0296
10
complex-words- few
.0118
Table 6: Most and least favored factors our seis-evvls--cneee dataset identified by the BTL model. Our findings show that human evaluators tend favor longer news summaries while showing dislike for shorter ones. This because longer summaries can provide more details about the news stories. However, this preference for length is
not as obvious with Reddit summaries. It is not case of 'the longer, the better'. As a result, our recommendation is tailor output length the specific task, with greater emphasis on length for news summarization.
Figure 7: Factor correlations measured by Kendall's Tau. LEFT: Results obtained for compart uumparisoneee sons-reddit. RIGHT: Results obtained for axes-evals aeee-evals--eddiie We observe that factors within each category, such as hallucination (none, one, two, many). ), show negative correlations. This is because for each system output, only one these factors will be activated. Moreover, there notable correlation between "short output length' and 'minimal source content coverage." as short summaries often leave out salient source content due to brevity. Preferred linguistic traits, i.e., 'fluent', 'unambiguous', 'style-aligned" and 'intent-aligned", correlate positively with high content quality Coff-focus-none'. 'ballucination-nonee)) They negatively correlated with the of new and complex words, multiple hallucinated facts or off-focus content. This suggests that better language and content qualities are likely attained together when LLMs' abilities improved. We believe there might be complex three-way interactions among these factors, further complicating the preference landscape.


Reviews:
[{"review_id": "tQN315Xh2C", "reviewer": "Reviewer_u9Ct", "summary": "", "questions": "This reviewer actually would like to see how the human judger themselves impact the results, for example, female judger vs. male judger, low-educated judger vs. high-educated judger. Authors could discuss it probably. ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for the positive assessment of our work. We'd be happy to include a discussion regarding how the evaluators' demographics might influence the pairwise judgments in our revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper provides a comprehensive framework to study how different key factors may influence human judgment that was used to assess and sometime train generative models. The framework could contribute to other area as well given the rising interesting in generative models", "reasons_to_reject": "This study is more on the qualitative analysis side and not very technical. But it should be fine for the EMNLP community.", "questions_to_authors": "This reviewer actually would like to see how the human judger themselves impact the results, for example, female judger vs. male judger, low-educated judger vs. high-educated judger. Authors could discuss it probably. ", "paper_topic_and_main_contributions": "The contributions in this paper are: \n- the authors conducted comprehensive analyses of a collection of human comparisons to identify key factors that may influence human judgment. \n- They used GPT models assess system outputs both qualitatively and quantitatively. \n- They examined  fluency, clarity, coverage, alignment with the original text’s intent and style, and detect hallucinations based on atomic facts. \n-The study could enhance the reliability of human evaluations. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FV4WJSCQAJ", "reviewer": "Reviewer_gQAc", "summary": "", "questions": "a. In the BTL modeling step, the authors say they do not consider factors if the same factor is present in the outputs being compared. \nFor a pair of output, with the same factor, is it possible for the two to still have noticeable difference in the factor? Mostly because it is possible that the two lie on the opposite ends of that quartile range? Eg: for output a and b if they both have “src-cov-medium” it is possible that output a lies to the lower end of the range and output b lies to the upper end of the range. \n\nb. For pairs with a gap of only one or two splits in the axis-evals-reddit dataset, do you see a lot of factors being canceled out? I'm wondering if the estimates of factors are primarily because of the pairs with a higher gap between them while the lower gap samples [which are similar to each other] contribute less.\n\nc. What is the range of some of the factors that get divided into quantiles?\n\nd. why do you think gpt-4 doesn't do well on \"similar\" summaries? do you think this can be improved by incorporating the factors in some way?\n\ne. what was the coverage of ACU extraction by GPT4?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and appreciate their recognition of this paper's role in deepening understanding in human preference judgments. In our follow-up study, we're planning to incorporate human evaluation, which might include labeling influential factors across a large set of pairwise comparisons. We'll also improve our description regarding preference data collection and add a clear example of a pairwise comparison and its extracted factors in our revised paper for clarity. \n\nOur responses to specific questions: \n\na) This study explores a variety of factors that might influence human preferences. A quantitative factor, such as src-cov-medium, indicates that the summary's coverage of source content is within a certain range. Two summaries that both exhibit this factor doesn't necessarily mean their contents are identical. We'll clarify this in the revision, thanks to the reviewer's input. \n\nb) We've noticed that pairs of summaries scored similarly by humans don't always share the same factors. Also, there isn't substantial overlap in factors between such pairs. \n\nc) We'll detail the range of factors in our revised paper. \n\nd) GPT-4 might not fully grasp the subtleties of human preferences; we would like to incorporate our analysis of influential factors to enhance human feedback for LLMs. \n\ne) We estimate that the coverage of ACU extraction by GPT-4 is approximately 91%.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "There is a lot of work recently that builds on human preferences, if humans prefer A over B but unfortunately generally datasets do not extend into knowing *why* a human prefers A to B. This paper is a step into that direction and they do an in depth analysis of which factors play an important role and how to get an estimate of their importance from the preferences already marked by humans. \nHaving a more fine grained analysis like this can help the community come up with better datasets/guidelines/think more about how to better collect preference data.", "reasons_to_reject": "- Human evaluation of the factors is missing - it would be nice to see if the BTL weighting of the factors is similar to how the humans weigh these factors/if humans agree with the estimates\n- An example, maybe in the appendix of what the values look like for a pair of examples would have been nice to visualize this in practice\n- In section 3, authors say that they aim to establish a robust sample collection practise for reward models but don’t explicitly address this point in the paper again", "questions_to_authors": "a. In the BTL modeling step, the authors say they do not consider factors if the same factor is present in the outputs being compared. \nFor a pair of output, with the same factor, is it possible for the two to still have noticeable difference in the factor? Mostly because it is possible that the two lie on the opposite ends of that quartile range? Eg: for output a and b if they both have “src-cov-medium” it is possible that output a lies to the lower end of the range and output b lies to the upper end of the range. \n\nb. For pairs with a gap of only one or two splits in the axis-evals-reddit dataset, do you see a lot of factors being canceled out? I'm wondering if the estimates of factors are primarily because of the pairs with a higher gap between them while the lower gap samples [which are similar to each other] contribute less.\n\nc. What is the range of some of the factors that get divided into quantiles?\n\nd. why do you think gpt-4 doesn't do well on \"similar\" summaries? do you think this can be improved by incorporating the factors in some way?\n\ne. what was the coverage of ACU extraction by GPT4?", "paper_topic_and_main_contributions": "The paper does an in depth analysis of the factors that influence human preference judgment. A lot of LLM work recently has been using human preferences but there is limited research probing the preferences in detail. \nThe main contributions are:\n- Using gpt4 the authors asses system outputs based on a set of predefined factors - fluency, clarity, coverage, alignment etc\n- Using the factors above, they analyze how the factors influence human preferences and show which factors are most favored vs which are least favored - this can possible help in data curation for preference modeling going forward \n- Dataset with an analysis of the factors and estimates", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}, {"review_id": "wmNTaI1J1h", "reviewer": "Reviewer_ZwsG", "summary": "", "questions": "1. Have you considered including nonsensical factors to evaluate the robustness of your approach? For example, something like “count of letter t’s in the summary”. This should not be a most- or least-favoured factor if the method is robust.\n2. Is there any way for you to quantify the goodness-of-fit of your model, akin to an R-squared that would quantify the amount of variance explained by a regression model? \n3. Relatedly, can you say anything about the significance of the factors estimates you obtain?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful feedback on our work. We'll clarify the paper's framing in the revision. The preferences might be influenced by both the demographics of human evaluators and the instructions provided to them. We will include a discussion on how these contexts might affect the pairwise judgments. We also agree with the reviewer that human feedback for LLMs can manifest in various forms, including natural language-based and more fine-grained feedback. We're happy to incorporate the two references in our revised paper. \n\nOur responses to specific questions: We did test the BTL model's robustness using nonsensical factors, and our finding is consistent with the reviewer's suggestion -- when introducing random factors, the BTL model showed that they weren't among the most or least favored. To evaluate the model's goodness-of-fit, we are considering comparing the BTL model's predicted probabilities to the binary outcomes, i.e., which summary won in each pairwise comparison. The Brier score is a potential metric for this, which measures the mean squared error between predictions and outcomes. We recognize the importance of understanding the significance of these estimates. We'd like to explore techniques like bootstrapping or constructing confidence intervals in our subsequent study. We're grateful for the reviewer’s feedback, which significantly enhances the depth of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Explaining human preference judgments is an open challenge that is extremely relevant to the current LLM development paradigm.\n2. The use of the BTL model is a clever and novel idea that seems to work well. Since it is an ex-post method it can be applied to many existing datasets.\n3. The chosen factors appear well-motivated and comprehensive.\n", "reasons_to_reject": "I do not see any major issues with this paper. However, there are some things that I would like to see addressed in the final version.\n\n1. The framing (human preference judgments) is perhaps a bit more general than the analysis (human preference judgments over summaries in data from one particular paper). It would be nice to be more clear and open about this.\n2. I would also appreciate more discussion about whose preferences are analysed here: for example, what is known about the evaluators who made these judgments? I know that for other preference datasets, a very small number of US-based crowdworkers was responsible for the vast majority of preference judgments. This is important to delineate the generalisabilty of the findings.\n3. Lastly, it would be great to see some more discussion of the emerging literature on more fine-grained preference feedback like https://arxiv.org/abs/2306.01693 or natural language feedback like https://arxiv.org/abs/2204.14146, since they offer an alternative to having to explain one-dimensional feedback.\n", "questions_to_authors": "1. Have you considered including nonsensical factors to evaluate the robustness of your approach? For example, something like “count of letter t’s in the summary”. This should not be a most- or least-favoured factor if the method is robust.\n2. Is there any way for you to quantify the goodness-of-fit of your model, akin to an R-squared that would quantify the amount of variance explained by a regression model? \n3. Relatedly, can you say anything about the significance of the factors estimates you obtain?", "paper_topic_and_main_contributions": "Human preference judgments act as training data for reward models, and as such are key ingredients for training more aligned language models with RLHF. The authors analyse which concrete factors (e.g. length and factuality) influence one-dimensional human preference judgments over pairs of summaries. For this purpose, the authors use a Bradley-Terry-Luce model. For Reddit and News summaries from a dataset collected by OpenAI, they find that the most favoured factors vary by task and genre, whereas the least-favoured factors such as excessive off-focus content and hallucinated facts, tend to be consistent.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Since you are using GPT API models throughout the article, please indicate somewhere when exactly and what version exactly you used."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyses the influential factors that made humans prefer one generated output over another. As human preference is readily used in aligning LLMs nowadays, all reviewers thought the paper is timely and insightful - as this is one of the first studies that attempts to provide an explanation of human preference - and the use of BTL as a method for analyses is creative. Minor revisions that can be improve the paper include: (1) a better framing (as the current framing of human preference judgements is more general than the actual analysis [ZwsG]); (2) a discussion on the impact of demographics on preference (which is important as it influences the robustness of the results [u9Ct, ZwsG]); and (3) a discussion on the limitation of the current approach which requires the factors to be pre-specified (i.e. it is unable to 'discover' new factors that may not be apparent).
Decision: Accept-Main