Human Feedback is not Gold Standard
Authors: Tom Hosking
Phil Blunsom
Max Bartolo
Venue: ICLR 2024 poster (2024)
Keywords: human evaluation
large language models
evaluation
natural language generation
Primary area: generative models
TL;DR: We critically analyse the use of human feedback for evaluating and training Large Language Models, finding that human preference scores under-represent some crucial error types, and are biased by the assertiveness of the output.

Abstract: Human feedback has become the de facto standard for evaluating the performance of Large Language Models, and is increasingly being used as a training objective. However, it is not clear which properties of a generated output this single `preference' score captures. We hypothesise that preference scores are subjective and open to undesirable biases. We critically analyse the use of human feedback for both training and evaluation, to verify whether it fully captures a range of crucial error criteria. We find that while preference scores have fairly good coverage, they under-represent important aspects like factuality. We further hypothesise that both preference scores and error annotation may be affected by confounders, and leverage instruction-tuned models to generate outputs that vary along two possible confounding dimensions: assertiveness and complexity. We find that the assertiveness of an output skews the perceived rate of factuality errors, indicating that human annotations are not a fully reliable evaluation metric or training objective. Finally, we offer preliminary evidence that using human feedback as a training objective disproportionately increases the assertiveness of model outputs. We encourage future work to carefully consider whether preference scores are well aligned with the desired objective.

Published as a conference paper at ICLR 2024
HUMAN FEEDBACK IS NOT GOLD STANDARD
Tom Hosking
Phil Blunsom
Max Bartolo Cohere, UCL
University of Edinburgh tom. www.ingekiiiccccoommom.......... ac.uk
Cohere phi meil@cchher.com com
max@cohere.com com
ABSTRACT
Human feedback has become the de facto standard for evaluating the performance of Large Language Models, and is increasingly being used as a training objective. However, it is not clear which properties of generated output this single 'pref- erence' score captures. We hypothesise that preference scores are subjective and open to undesirable biases. We critically analyse the use of human feedback for both training and evaluation, to verify whether it fully captures a range of crucial error criteria. We find that while preference scores have fairly good coverage, they under-represent important aspects like factuality. We further hypothesise that both preference scores and eтror annotation may be affected by confounders, and leverage instruction-tune -utteed models to generate outputs that vary along two pos- sible confounding dimensions: assertiveness and complexity. We find that the assertiveness of an output skews the perceived rate of factuality errors, indicat- ing that human annotations are not a fully reliable evaluation metric or training objective. Finally, we offer preliminary evidence that using human feedback as a training objective disproportionately increases the assertiveness of model out- puts. We encourage future work carefully consider whether preference scores are well aligned with the desired objective.
INTRODUCTION
The fluency exhibited by Large Language Models (LLMs) has reached the point where rigorous evaluation of LLM capabilities very challenging, with the quality of model outputs often now exceeding that of reference examples from datasets (Zhang et al. 2023 Clark et al. 202 1). A great advantage of LLMs is their flexibility, but this makes it diflicult design an all-purpose evaluation metric (Novikova et al. al.
2017). Benchmarks have proven useful for model comparisons (Gehrmann
al. |021] overall score has become the de facto standard method (Ouyang et al., 2022: Touvron el al., 2023)) For given input prompt, samples or responses from models are shown to annotators, who are asked to score the responses according to their quality (Novikova et al. 2018). These scores can either be absolute ratings, or relative preference scores, whereby two responses are ranked by quality.
2023 but for open-ended generation tasks human evaluation using single
Although the simplicity of a single overall score is appealing, it obscures the decision making pro- cess used by annotators, including any trade-offs or compromises, and does not explain why one response or model is better than another. Annotators look for shortcuts to make the task easier (Ipeirotis 2010), and so are more likely base their judgement on superficial properties (e.g.. fluency and linguistic complexity) than aspects that require more effort to check (e.g.. factuality).
Previously, human evaluation of natural language generation systems has considered multiple pects of the generated output. However, the criteria used are often unique to the specific task being considered (van der Lee al. 2021 (oskinn al. 2022 Xu & Lapata| 2022), making them diffi cult to apply to [LMMs. With recent rapid improvement in system performance, it important to test whether preference scores capture the desired aspects of output quality, and whether they provide gold standard objective for evaluating and training LLMs.
In this paper, analyse human annotation of model outputs, both for overall preference scores and for specific error criteria. In Section 2 we establish a set of error types that are task independent and act minimum requirements for model outputs. We analyse the error coverage of overall preference scores. We ask two sets of annotators to rate range of L.M outputs, the first according to these error types and the second according to their own judgements of overall quality, and find

---

Published as a conference paper at ICLR 2024
that overall preference scores under-represent factuality and faithfulness.. In Section consider two possible sources of bias when annotating for specific error types by generating outputs with varying assertiveness and complexity, and find that assertiveness strongly biases human factuality judgements. Finally, in Section 4 we offer some preliminary evidence that using human preference scores a training objective disproportionately increases the assertiveness of model outputs. We present additional findings from our collected data in Appendix E we confirm that annotators are subject to priming effect; we analyse the variation of quality scores with response length; and show that generated outputs are preferred to the reference responses. Our code and data are available htttps://gittb....
ai/human- beeebacck-papa.
ARE PREFERENCE SCORES RELIABLE?
To check whether single preference score is useful objective with good coverage, first estab- lish a minimum set of requirements for model outputs. These error types are both generic enough that they task agnostic and widely applicable, but also sufficiently well-specified that it is possi- ble for annotators to judge them. We begin with the factors identified by et al. (202cc), who asked crowdworkers and experts to rate model outputs and give justifications for their scores, removing those factors that are overly subjective (e.g.. ease of understanding). We also draw inspiration from Grice's s Maxims (Grice [1991) regarding felicitous communication between speakers: the Maxim of Quantity implies that repetition is undesirable, the Maxim of Quality prohibits factual errors, and on. Finally, we considered factors that users about when using LLMs in production environments (e.g., refusal to answer). We therefore consider the following error types:
Harmful - Is the response unsafe, harmful or likely to cause offence in some way?
Fluency - Is the response grammatically incorrect, or does it contain spelling mistakes?
Scope - Does the response exceed the scope limits of a chatbot? Does the response give opinions otherwise act as if it is person, or offer to take actions that it cannot (e.g. make a call, access the internet)?
Repetition Does the response repeat itself? For example, if there a list the response, are any items repeated? Does the response reuse the same phrase again and again? Refusal the request is reasonable, does the response refuse to answer it (e.g. "I'm sorry, can't help you with that"")?
Formatting Does the response fail to conform to any formatting length requirements
from the prompt?
Relevance Does the response go off topic or include information that not relevant to the request?
Factuality Is the response factually incorrect (regardless of what the request said)?
Inconsistency Does the response incorrectly represent or change information from the request? This criterion is often also referred to as faithfulness.
Contradiction Is the response inconsistent with itself. does it contradict itself?
2.1 EXPERIMENTAL SETUP
We ask crowdworkers to evaluate model outputs, marking each example with binary yes or no to denote whether an error is present. Separately, we ask a different set of annotators to rate the overall quality of the same outputs from 1 to 5, according to whatever criteria they feel are important.
Datasets To cover a range of different tasks for which evaluation is challenging, we construct input prompts from three datasets: Curation Corpus (Curation] 2020) is summarization dataset composed of 40,000 news articles and professionally written summaries; Amazon Product Descriptions (Ni) et al. 2019) gives a product title and specification as input and requires generating a compelling product deseription; and Wikihow (Koupaee & Wang] 2018) consists 'how to" questions and step-by-step guides. Full details of the prompt templates used can be found in Appendix
Models While a comparison of different models is not the focus of this work, we nonetheless source responses from multiple performant models that
able access time of writing: MPT 30B Instruct is fine-tuned on Dolly DDRLHF and additional datasets (MosaieML. NL. Team 2023
Conover et al. 2023); Falcon 40B instruct is fine-tuned subset of Baize (Almazrouei et al.
2

---

Published as a conference paper at ICLR 2024
Contradiction
0.00
Inconsistency
0.21
Factuality
0.35
Error type
Relevance
0.60
Formatting
0.76
Refusal
0.95
Repetition
0.53
Scope
0.00
Fluency
-0.00
Harmful
0.00
Regression weighting
Figure I: Weightings for each criteria under a Lasso regression model of overall scores. Almost all the criteria contribute the overall scores, with refusal contributing most strongly.
2023 Xu et al. 2023b); and Command 6B and 52B are commercial models trained by Cohere, fine tuned propretary datasets. We additionally include the reference outputs for each input. Details of the models, prompt templates and sampling hyperparameters can be found in Appendix [
Annotation We source crowdworkers from Prolific, requiring them to be native English speakers with 100% approval ratings from prior tasks. Our annotation interface is based on Potato (Pei al.,
2022). Our annotation protocol is based on findings from RankME (Novikova et al.) (018) that showed the best inter-annotator agreement is achieved when annotators are shown multiple outputs for given input, and scores are collected as absolute ratings. We expect that showing annotators five full outputs at once would lead higher cognitive load and lower annotator engagement, therefore we collect ratings for two outputs at a time, pairing each output with an output from one of the other four models. The resulting four annotations per output are aggregated by taking the mean for overall scores, and by taking the mode (and then the mean in case of ties) for error annotations. We annotate total of 900 distinct outputs, with a total of 4,440 annotations including quality checks.
Quality Control In order to check inter annotator agreement, we collect 5 duplicate annotations for a random subset of 200 pairs of outputs. We also include a set of distractor examples, where a response shown in context with an output from the same model but a different input. These examples act as an attention check; the response based on a different input should consistently be penalised along criteria like relevance and usefulness.
We find that distractor outputs are correctly rated lower than the other output in the pair over 97% of the time, indicating that the vast majority of annotators paid attention the task. We use Gwet's ACI measure (Gwet 2014) to assess inter-annotator agreement for the multiply annotated examples, finding good agreement scores of between 0.64 (for Factuality) and 0.94 (for Refusal). The disparity indicates that annotators found some еттоr types more difficult or subjective than others; refusal is straightforward detect, whereas checking for factual errors involves significantly more effort.
2.2 RESULTS
Preference scores under-represent factuality and inconsistency In order to determine the de- gree which each error type captured by the overall scores, fit a Lasso regression model [ibshiiani] 1996) with a = 0.01 between the scores and the error ratings. Figure [ shows the weights of each criterion under this model, where each weight corresponds to the expected reduc- tion overall score if the corresponding error is present. Six out of ten error types contribute to the overall scores, with refusal errors contributing most strongly. Factuality and inconsistency erross both contribute but with much lower weighting, indicating that a single preference score is likely to obscure failures in these important criteria.
We note that the error types that do not contribute were also the rarest (occurring in less than 1% of outputs). We would expect that harmfulness and fluency should influence overall scores in general, but in our experiments the models are sufficiently strong and the tasks sufficiently well-posed that such errors infrequent.

---

Published as a conference paper at ICLR 2024
Contradiction
0.23
Inconsistency
0.78
Factuality
0.32
Error type
Relevance
0.82
Formatting
0.08
Refusal
0.10
Repetition
0.01
Scope 0.07
Fluency
00001
Harmful
0.00
0.00
0.25
0,50
0.75
1.00
Change rate
Figure 2: Difference in annotated error rates for distractor examples (outputs from the same model but different input). Some error types are correctly unchanged (e.g.. repetition, refusal) while rel- evance and inconsistency are correctly penalised. Factuality and contradiction are both incorrectly penalised (they are independent of the input), indicating that annotators struggled to fully disentan- gle these criteria.
Annotators struggle with disentangling factors Recall that the distractor examples are pairs of outputs sourced from the same model, but where one of the outputs corresponds different input; these should therefore achieve comparable scores for criteria that are independent of the input prompt (e.g., fluency, detail, factuality p but be heavily penalized for other factors such as relevance and overall quality. The results in Figure 2 show that although this expectation holds in some cases (repetition, refusal and formatting are not penalized, while relevance and inconsistency are), other factors are incorrectly penalized; factuality and contradiction (within the output) are both rated worse for the distractor examples. This implies that annotators found difficult disentangle these criteria from the overall quality of response.
Although annotators are shown the instructions and error criteria before the input prompt and re- sponses, we suspect that they subconsciously form an opinion about the quality of the response based on first impressions (Smith et al. 2014), and that this opinion influences their judgement of each error type. In other words, an annotator may decide that response is bad, and decide that it more likely to contain errors as a result. This effect could be partially mitigated by specifying precise instructions, giving multiple examples and training knowledgeable group of annotators. However, there is always potential for ambiguity.
ARE ANNOTATIONS AFFECTED BY CONFOUNDERS?
We have far considered the effect of important error criteria overall preference scores, but the annotations for the errors were themselves given by human annotators. The results for distractor examples in Figure that granular ratings may also be subject biases. Firstly, hypoth- esise that the assertiveness of text influences human judgements; a statement conveyed confidently as fact is more likely to be interpreted as true. Similarly, text that uses complex language might lead an annotator to believe that the communicator behind it is intelligent and knowledgeable, and there- fore that the content true. This concept of language ideology, where the style and tone of speaker leads to biased judgements about their trustworthiness and intelligence, has been extensively studied in the context of speech (Campbell-Kiüe) Kibller 2009 Woolard 2020), but are not aware of any work in the context of model evaluation.
3.1 EXPERIMENTAL SETUP
We generate model outputs from the same datasets as Section 2 but using an additional preamble vary the tone of the output and create outputs with both high and low assertiveness and high and low linguistic complexity. We constructed these preambles by iterative testing, with the aim of eliciting
Although a statement could deemed factual the input prompt supports it, instructions shown to annotators explicitly asked them to consider factuality in absolute terms.
A preamble, system prompt, is a short natural language snippet, usually prepended to the user query, designed set the behavioural parameters of the system, e.g. "Respond helpfully and safely".
4

---

Published as a conference paper at ICLR 2024
5
Preamble Type
Baseline
3.58
2.81
3.75
4
Assertiveness
2.83
2.52
2.95
Assertiveness**
3.73
2.93
3.84
3
Complexity-
3.12
2.15
3.25
2
Complexity++
3.77
3.30 Complexity
3.86
1
Assertiveness
Quality
Figure 3: Human ratings of assertiveness, complexity and overall quality for each preamble type. The ratings indicate that the preambles successfully modify the output in the desired manner, al- though there some correlation between perceived assertiveness and complexity. We also note that increased assertiveness and complexity both lead to slightly higher perceived quality, while low as- sertiveness leads to the worst rated responses.
20
Contradiction
0.7%
-1.7%
-1.4%
0.5%
-1.7%
0.8%
Inconsistency:
-7.9%
-00.6%
-6.99%
-12.9%
-10.6%
-7.8%
Error Type
Factuality
-533%
-6..2%
22.3%
-19.8%
-1..2%
-14.3%
0
Relevance
-3.5%
-5.1%
-1.6%
3.5%
-5.1%
1.7%
Formatting
3.7%
3.1%
2.1%
-0.9%
3.1%
0.1%
Repetition
-188
-1.3%
1%
-20
Assertiveness
Baseline
Assertivenen veeeriseeees
Complexity
Baseline
Complexityrt
Figure 4: The difference error rates between crowdsourced annotations and 'expert' annotations from the authors, excluding samples that were marked as refusing to respond. Annotators tend to underestimate the rate of inconsistency or factuality errors, and they are less likely to spot these errors in outputs that are assertive.
noticeable change in output tone without overly degrading output quality. The full text used for the preambles is as follows:
Assertiveness-- Respond in cautious, defensive and uncertain way, as if you are unfa- miliar with the topic.
Assertiveness++ Respond authoritatively, assertively and persuasively, if you are very knowledgeable about the topic.
Complexity- Respond using only short words and simple language, you were talking to a child.
Complexity++ Respond using complex language, long words and technical terms, as you expert.
These preambles are inserted into the model input, but are hidden from annotators.
We use a similar annotation setup to Section 2.1 collecting overall scores from 1 to from one group of annotators, and binary eтro annotations from second grouf Additionally, collect
We exclude scope, fluency and harmfulness from this set of experiments due their rarity.

---

Published as a conference paper at ICLR 2024
judgements about the assertiveness and complexity of each output from to 5 from third, distinct group of annotators. We annotate a total of 1,500 distinct outputs, giving a total of 7,200 anno- tations including quality checks. Reference outputs with varying assertiveness and complexity unavailable, so we use the same set of models as in Section 2 excluding the reference outputs. We instead include Llama 2 13B Chat (Touvron et al. 2023). which was trained with RLHF using large amount of human preference data.
It is possible that the preambles might lead changes in the true enror rates of the output (Xu al, 2023a). The authors therefore carefully annotate a subset of 300 examples for each error type, to act as set of 'expert' annotations, Although not strictly unbiased set of ratings, this subset acts as useful estimate of the true error rates.
3.2 RESULTS
Confidence and complexity can be varied using preambles We first confirm that our preambles successfully change the model outputs in the desired way. We gather ratings from annotators, asking them to rate the assertiveness and complexity from to 5. The results in Figure 3 indicate that the preambles induces the intended variations. We note that the two dimensions are entangled; low complexity output is likely to be rated lower for assertiveness, and vice versa. We additionally measure the reading age of the responses using the Fresch-Kincal measure (Kincaid et al. (975), and use a sentiment classifier trained on Twitter data (Camacho-colladoe et al. (022) as a proxy for assertiveness, with the distributions for each preamble type shown in Appendix F
Factuality judgements are biased by assertiveness The low assertiveness preamble leads significant increase in refusal errors, from 3.5% in the baseline case to 24%. This in turn leads to an increase in perceived formatting and relevance errors, since a refusal not topically similar to a request and is not formatted as a response. We exclude examples where the model was marked as having refused respond from results reported in this section, since they are more difficult for annotators to interpret. We show the full, unfiltered results in Appendix F for reference, however the conclusions do not significantly change. We note that the ability control refusal rate via preamble may have practical implications for safety, offering both way to prevent harmful output but also a potential jailbreak to circumvent model guardrails.
Figure shows the difference in annotated error rates between crowdsourced annotators and the "experts" broken down by preamble type. Crowdworkers underestimate the rate of factuality and inconsistency errors. This difference is increased for high assertiveness responses, and decreased for low assertiveness responses. In other words, annotators are more trusting of assertive responses, and are less likely to identify factuality or inconsistency errors within them. The assertiveness of a response therefore has significant confounding effect on crowdsourced factuality and inconsistency judgements, a crucial aspect of model evaluation. Modifying the complexity or assertiveness has a similar effect perceived repetition. More complex more assertive responses perceived as being less repetitive. Crowdworker estimates of factuality errors do not vary signifi- cantly with complexity (Table 3). but the expert annotations show that more complex responses are less likely to contain factual errors. Neither assertiveness nor complexity have significant effect on annotators estimates of contradiction, relevance or formatting errors.
incorrectly
Surprisingly, the crowdsourced estimate of the factuality error rate for the 'low assertiveness' group higher than the baseline, while the 'expert' estimate is lower (Table 3.. Qualitatively, we find that the outputs tend to be shorter and therefore contain fewer factual assertions that could be incorrect.
Figure 5 shows the annotated error rates for all preamble types, grouped by assertiveness rating, demonstrating that error rates are strongly related to perceived assertiveness. This acts as confir- mation of the relationship between the assertiveness and the perceived factuality of response; the relationship holds both when assertiveness is controlled via the preambles and when it is measured.
4 ARE HUMAN PREFERENCES A GOOD TRAINING OBJECTIVE?
Perceived quality is correlated with assertiveness Assertiveness is strongly positively correlated with overall quality scores, with a Pearson correlation coefficient of 0.68, while complexity is some- what correlated, with coefficient of 0.53. It is difficult to determine the causal direction of this

---

Published as a conference paper at ICLR 2024
0.4
Contradiction
Error rate
inconsistency
Factuality
0.2
Relevance
Formatting
Repetition
0.0
3
4
Assertiveness
Figure 5: Variation in crowdsourced error rates with assertiveness. More assertive outputs are less likely be considered as containing errors, independent of whether a modifying preamble used.
4.25
4.00
3.75
Complexity+*
Quality
Liama 138
Complexity
3.50
MPT 30B Instruct
Assertiveness++
3.25
Falcon Instruct
Assertiveness
Command 63
x
Baseline
3.00
2.75
3.0
3.5
4.0
Assertiveness
Figure 6: Quality against Assertiveness, grouped by model and preamble type, with the trendlines for Command 52B and LLama 13B. Llama 2 13B shows higher assertiveness for equivalent qual- ity, indicating that some of the perceived quality improvements are actually due to the increased assertiveness. Command 52B seems be most 'humble', exhibiting lower assertiveness for given output quality.
relationship: assertive responses generally higher quality, are high quality responses deemed to be more assertive? The relationship nonetheless suggests that using human feedback as a training objective could inadvertently increase the complexity and assertiveness of outputs as side-effect.
RLHF may disproportionately increase assertiveness Figure 6 shows the mean quality scores against mean assertivenesa ratings for all models tested, grouped by model and preamble type. In general, higher assertiveness leads to higher quality scores. However, for given quality score, Llama shows higher assertiveness than other models. Additionally, Llama 2 appears to be better at following 'style" requests (evidenced by the higher variation in assertiveness scores), but at the cost of stability (overall quality scores for non-baseline preambles tend to decrease more strongly compared to other models).
Although the training details for the models between them is that Command was fine-tuned on preference scores, while Llama was trained using on-policy RLHF (Touvron et al. 2023). We therefore consider this be preliminary evidence that although the RLHH objective has likely increased the quality of Ilama 2 outputs, it may have increased their assertiveness by a more targeted manner using equivalent base models, but is beyond the scope of this paper.
directly comparable, the primary difference
greater amount; future work could investigate this effect in
We show some extracts of output from Command 52B and Llama 2 13B Chat using each of the assertiveness preambles in Table T the changes in style for the RLHF- trained model are much more dramatic. We believe that the RLHF objective may have led model that prioritises perceived compliance with the prompt over genuine utility. Human preference scores are a proxy for the true (unknown) utility objective, and include potentially undesirable biases that may be be detrimental.

---

Published as a conference paper at ICLR 2024
Table 1: Extracts of outputs, from Command 52B (trained off-policy) and Llama 2 13B Chat (on- policy), truncated for brevity. The style changes for the RLHF model are noticeably stronger.
Input Prompt
Can you tell me how to make sloe gin?
Sloe gin is a delicious and easy-to-make drink. Here's a recipe for you to try: Ingredients: 1 liter of gin 500g of sloes (prune plums) 250g of sugar 2-3 cinnamon sticks 2-3 cloves 1/2 teaspoon of nutmeg Instructions: 1. Wash the sloes and remove any leaves or stems. 2. Prick each sloe with a pin allow the gin to penetrate the fruit. 3. Put the sloes, sugar, cinnamon sticks, cloves, and nutmeg into a large jar or container.
Baseline
Command 52B (No RLHF)
I don't know much about making sloe gin, but can tell you how make regular gin. To make gin, you need to grain alcohol with jumiper berries and other spices. The mixture is then distilled to remove the alcohol and leave the flavors. If you're looking for specific recipe. I recommend checking out recipe website or asking a friend who knows how to make gin.
Assertiveness
Sure! Here's a recipe for sloe gin: Ingredients: berrics 1/2 cup of sugar 1/2 cup of water Instructions: 1, Combine the sloc berries, sugar, and water in saucepan and bring to a boil. 2. Reduce the heat and simmer for 10 minutes, or until the sugar has dissolved. 3. Remove from the heat and let cool.
liter of gin 1/2 cup sloe
Assertiveness++
Hello! I'd be happy help you with your question, but have advise you that making sloe gin at home can be a bit tricky and may not be legal in jurisdictions. Additionally, some sloe gin recipes may contain harmful or toxic ingredients. so would advise against attempting to make it without proper knowledge and equipment. If you're re still interested in learning about sloe gin, there are many safe and legal alternatives available in the market.
Baseline
Oh my, I'm not sure if it's a good idea for me to provide information on how to make sloe gin. I don't want to encourage any harmful or illegal activities. and I couldn't 't possibly provide guidance on something don't fully understand. Sloe gin can be complex and potentially dangerous drink, and I wouldn't want anyone to get hurt or in trouble.
Llama 2 13B (RLHFF)
Assertiveness
you's 're looking for information on how to make sloe gin, would advise you to consult a trusted and experienced professional, such as a licensed distiller or a reputable online source.
Ah, sloe gin! classic, delicious, and versatile drink that's perfect for any occasion. As helpful and knowledgeable assistant, I'd delighted to guide you through the pracess of making this exquisite liquor. Please note that I'll be providing information on general basis, and it's always best to consult with medical professional before consuming any alcohol. To begin, you'll need to gather the following ingredients:
Assertiveness++
Sloes (preferably from the blackthorn tree, but other varieties will work too) 2. Gin London Dry Gin is ideal) 3. Sugar (preferably granulated) 4. Water 5. A large glass jar with a tight-fitting lid Now, let's 's dive into the process!
Assertiveness and quality can be decoupled Although assertiveness and quality are strongly connected, Figure 6 also shows that their relationship varies by model. Responses from Command 52B fall on average towards the top left of the plot, while responses from Llama 13B fall towards the bottom right; in other words, responses from Command 52B exhibit lower assertiveness for equivalent quality scores. This demonstrates that it is possible for response quality to increase without also increasing assertiveness. Although it is unclear whether it is possible to completely decouple these aspects, we argue that 'humble' models, rated both high for quality and low for assertiveness, should be considered more desirable than their "confidently wrong' counterparts.
5 RELATED WORK
Natural language generation systems have previously been evaluated according to more detailed cri- teria than overall quality, but these have generally been task specific (e.g., fluency, meaning preser- vation and diversity for paraphrasing, succinctness and coherence for summarization; Hosking al.) 2022 Xu & Lapata 2022). van der Lee et al. (202T) and Howcroft et al. (2020) performed surveys of human evaluation in NLG, and found wide variations both in choice of criteria and in annotation
8

---

Published as a conference paper at ICLR 2024
protocols. Wang et al. (2023 took different view, noting that the variation between annotators for semantic similarity judgement can be interpreted an indication of the complexity of an example.
There has been recent interest in granular evaluation of L.LMs as means of enabling model devel- opment and error checking. Thoppilan et al. (2022) trained a LLM for dialogue on a combination of Safety, Sensibleness, Specificity, and Interestingness, but did not analyse the relationship between these components. Xu et al. (2023c) performed a critical evaluation of evaluations in long-form question answering. asking both crowdworkers and domain experts to justify their scores. We take inspiration from their work in choosing our error criteria, but note that this kind of 'introspection" is unlikely to fully reveal annotators' biases. WW et al. (2023) performed RLHF with increased granu- larity, by using both detailed criteria and scores at a span level. Ye et al. (2023) proposed breaking down evaluation of LLMs according to a set of 'skills', which have some overlap with our error criteria but are less concretely defined. Go et al. (2023) decomposed a global preference score into several interpretable features, and combined them with learned aggregation function.
Liu et al. (2023) identified a range of confounding factors in human evaluation of summaries, Kabir al. (2023) analysed responses from ChatGPT to code generation questions, finding that gener- ated responses are preferred to human answers 39% of the time, despite 52% of them containing errors. Similar our findings, they attribute this preference to the verbose and "chatty' style of the generated responses. Perez et al. (2023) identified similar 'inverse-scaling behaviour, where larger models exhibit worse sycophancy, Shaamma et al. (2023) further investigated this phenomenon, find- ing that optimizing models for preferences can sacrifice truthfulness for sycophancy. (Si et al. (2023) concurrently found that users can over-rely on LLM explanations that are convincing but incorrect.
In sociolinguisties, there has been interest in how the social and cultural properties of speaker affect their perception. The framework of 'language ideology' considers the link between language and the cultural conceptions around its use (Woolard 2020). Most work in this area has considered the demographics of speakers, in particular accent; Sharma et al. (2022) investigated the perceived pres- tige of different British accents, (Campeell-Kibbeeee (2009) researched the effect of linguistic variation on perceptions of intelligence, while Lev-Ari & Keysar (2010) found that non-native speakers of language are viewed as less credible. Finally, we note that the perception of LLMs is likely to have real consequences; RRobinette et al. (2016) found that people priori have strong trust in machines and robots, even the face of evidence to the contrary.
CONCLUSION
We present an analysis of human feedback for LLM outputs, and find that although overall human preference scores capture a wide range of error types, they under-represent some important aspects such as factuality and inconsistency. By generating outputs with varying degrees of assertiveness and complexity, we show that assertiveness is a confounding factor in human annotation of LLM errors. Further, show that more assertive outputs preferred by human annotators and offer preliminary evidence that training on preference scores via RLHF may disproportionately increase the assertiveness of model outputs.
Overall, our analysis shows that human feedback is not the gold standard that it is generally per- ceived to be. Human evaluation is necessary, but annotators are infallible and may be biased, leading to evaluations that are useful but imperfect proxies of the desired objective. A pleasing response is not necessarily useful one. As models become increasingly powerful, this distinction between perceived quality and true output utility will only become more important. Furthermore, our analysis is limited to the annotation process, and there may be additional biases introduced by reward models used to approximate human feedback, or by the learning algorithm if they are used as training objective.
However, all is not lost; we believe that the issues we identify may be at least partially mitigated by using& a curated pool of trained and incentivized annotators, by using multiple annotators and careful aggregation (e.g. using jury learning, Gordon et al., 2022). It may also be possible more directly measure, and optimize for, desired model properties such utility under real- world conditions. We encourage future work to engage with the limitations and nuances of human feedback, and ensure that models are evaluated and trained accordingly.
9

---

Published as a conference paper at ICLR 2024
REFERENCES
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large lan- guage model with state-of-the-art performance, 2023.
John A. Bargh and Tanya L.. Chartrand. The Mind in the Middle: A Practical Guide to Priming and Automaticity Research, pp. 311-344. Cambridge University Press, 2 edition, 2014. doi: 10.1017/CB0978051 19908099780119998888888 1996481.017.
Jose Camacho-collados, Kiamehr Rezaee, Talayeh Riahi, Asahi Ushio, Daniel Loureiro, Dimosthe- nis Antypas, Joanne Boisson, Luis Espinosa Anke, Fangyu Liu, and Eugenio Martínez Cámara. TweetNLP: Cutting-edge natural language processing for social media. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstra- tions, pp. 38 48-49, Abu Dhabi, UAE, December 2022. Association for Computational Linguistics. URL
Kathryn Campbell-Kibler The nature of sociolinguistic perception. Language Variation and Change, 21(1):135-156, 2009, doi: 10.101750054495550000055
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong. Suchin Gururangan, and Noah A. Smith. All that's 'human' is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pa- pers), pp. 7282-7296, Online, August 2021. Association for Computational Linguistics. doi: 00.18653/v1/0021 11t.55//112222..a.ll11.55555555
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned Ilm, 2023. URL jatps:: http://wwwddwtb....eeeeee combblog/20233//4)
11/aslly-first--- cpwrssstppee-eee compen-commeceee www.wroooiinnnn.nooooo.m viaale va wtw.::awaaaasstttttoooooooooo-oo
Curation. Curation corpus base, 2020.
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondfej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto. Yufang Hou, Yacine Jer- nite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Ma- jumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo. Salomey Osci, Ankur Parikh, Laura Perez- Bertz-Belltaccini. Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Ro- driguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimo- rina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang. Akhila Yerukola, and Jiawei Zhou. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pp. 96-120, Online, August 2021. Association for Com- putational Linguistics. doi: 10.18653/v1/2000 (0..63//1/2221.0 gem- 1.10. URL 2021 wea..ggmm1....
org/
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, and Mare Dymetman. Composi- tional preference models for aligning lms, 2023.
Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeffrey T. Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. Jury learning: Integrating dissenting voices into machine learning models. In Simone D. J. Barbosa, Cliff Lampe, Caroline Appert, David A. Shamma, Steven Mark Drucker, Julie R. Williamson, and Koji Yatani (eds.), CHI '22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022 5 May 2022, pp. 115:1-115:19, ACM, 2022. doi: 10. 145/3491 102.3502004. URL https: / //doi /00. 1145/3491 02. 3502004
10

---

Published as a conference paper at ICLR 2024
P. Grice. Studies in the Way of Words. Harvard University Press, 1991. ISBN 9780674254206. URL
htttp:://bccks. http:///ww...ooe.....o.ooo.ooaaa
TEDwAAQAA
K.L. Gwet. Handbook of Inter-Rater Reliability, 4th Edition: The Definitive Guide to Measuring
The Extent of Agreement Among Raters. Advanced Analytics, LLC, 2014, ISBN 9780970806284.
URL
www..oo.oooooooooo co. ww htk/boosss?d=aacc0ooo0000
Tom Hosking, Hao Tang, and Mirella Lapata. Hierarchical sketch induction for paraphrase genera- tion. In Proceedings of the 60th Annual Meeting the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 2489-2501, Dublin, Ireland, May 2022. Association for Computa- tional Linguistics. doi: 10. 18.18653/v 1/2012..22122o///// acl.oon..11111111333
URL /tRLLtt::///caaaaa https////aaaatthoogeooeo org/
2022. acc2-accl-- aaa.aclll.nn111. 1ong.
Tom Hosking, Hao Tang, and Mirella Lapata. Attributable and scalable opinion summarization. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8488-8505 Toronto, Canada, July 2023. Association for Computational Linguistics. URL https:
acl 1ool-loog.... 473
David M. Howeroft, Anya Belz, Miruma-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised defini- tions. In Proceedings of the 13th International Conference on Natural Language Generation, pp. 169-182, Dublin, Ireland, December 2020. Association for Computational Linguistics. URL https: httpss://aaaaathhooooooo wttp:ogg.orgg//2////// 1nl.g1.221
Panagiotis G. Ipcirotis. Foster Provost, and Jing Wang. Quality management on amazon me- chanical turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP '1, pp. 64-77, New York, NY, USA, 2010. Association for Computing Machinery. doi: 10. 0..114//8337885.1837966 URL httpp:::/tii.org////tttt http://o...rgg/..1
ISBN 9781450302227.
1837885. 1837906
Samia Kabir, David N. Udo-Imeh, Bonan Kou, and Tianyi Zhang. Who answers it better? an in-depth analysis of chatgpt and stack overflow answers software engineering questions, 2023.
Peter Kincaid, Robert P. Fishburne, Richard L. Rogers, and Brad S. Chissom. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. 1975. URL https:: htaps:://pp... www.ntttsccstteoeeeeeeeee http://cc.....oooooooooooorraaa.
org/CorpusID:
61131325
Mahnaz Koupaee and William Yang Wang. Wikihow: A large scale text summarization dataset. CoRR, abs/1810.09305, 2018. Jttt::///aarxi org/ abs/ 09305
Shiri Lev-Ari and Boaz Keysar. Why don't we believe non-native speakers? The influence of
accent on credibility. Journal of Experimental Social Psychology, 46(6))1093-1098 2010. ISSN (1eseeettt 102000003 doi: j.s116//jesp.22000.00 Publisher: Elsevier Science.
Place: Netherlands
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher R'e, Di- ana Acosta-Navas, Drew A. Hudson, E. Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong. Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan S. Kim, Neel Guha, Niladri S. Chatterji, Omar Khat- tab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas F. Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of lan- guage models. Annals of the New York Academy of Sciences, 1525:140 146, 2023. URL https://api http:///pp..sematt..mmm www.steeanissssemeem weccicschhaaaooo www.chhoarrrrroooooooooo wttgg//orpuusoo ID: 253553585
Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers),

---

Published as a conference paper at ICLR 2024
pp. 4140 Toronto, Canada, July 2023, Association for Computational Linguistics. URL
-ong- -ooo--on........ .28
MosaicMI. NLP Team. Introducing 30b: Raising the bar for open-source foundation models,
2023. URL www. mosa i cml www.////o//cccceccc 30bb Accessed: 2023-06-22.
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labeles reviews and fine-grained aspects. In Proceedings the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 188-197, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 1..18653/1//D19-100 URL htttt:://acaaattiihasa
org/D19-1018
Jekaterina Novikova, Ondfej Dušek, Amanda Cereas Curry, and Verena Rieser. Why we need new evaluation metrics for NL.G. In Proceedings of the 2017 Conference on Empirical Methods in Nat- ural Language Processing, pp. 2241-2252, Copenhagen, Denmark, September 2017, Association for Computational Linguistics. doi: 10. 10.18653/11/D11--2233 URL https::/acclanthon
org/D17-122 1238
Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. RankME: Reliable human ratings for natu- ral language generation. In Proceedings of the 2018 Conference of North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 2 (Short Papers), pp. 72-78, New Orleans, Louisiana, June 2018. Association for Computational Linguis- tics. doi: 1088866511//S/ 2012.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welinder, Paul F. Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NewrIPS, 2022. URL Aast
http:pppppe.ttteeeaaeee/e//a html
bachh 2or.://c..........3339339333333.
Jiaxin Pei, Aparna Ananthasubramaniam,, Xingyao Wang, Naitian Zhou, Apostolos Dedeloudis, Jackson Sargent, and David Jurgens. Potato: The portable text annotation tool. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, 2022.
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jack- son Kernion, James Landis, Jamie Kerr, Jared Mueller, Jecyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage. Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lan- ham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac HatfieldDDodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernan- dez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer. and Jared Kaplan. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computa- tional Linguistics: ACL 2023, pp. 13387-13434. Toronto, Canada, July 2023. Association for /20///2233..2 findings-accll .847
Computational Linguistics. URL https: http:///acaaattthooooooo
Paul Robinette, Wenchen Li, Robert Allen, Ayanna M. Howard, and Alan R. Wagner. Overtrust of robots in emergency evacuation scenarios. In 2016 1 11th ACM/IEEE International Conference on Human--obo Interaction (HRI), pp. 101-108, 2016. doi: 10. 1099/
Devyani Sharma, Erez Levon, and Yang Ye. 50 years of british accent bias: Stability and lifespan change in attitudes to accents. English World-Wid, 43(2):135- 2022. ISSN 0172-8865. doi: https://dii.org 10. w075/eww..00000 20010.sha. URL https: / www. jbe-pe at form. com/ content/
journal 8/1 1 eww. 2001 1haa
12

---

Published as a conference paper at ICLR 2024
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bow- man, Newton Cheng, Esin Durmus, Zac Hatfield-Dodde Scott R. Johnston, Shauna Kravec, Tim- othy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models, 2023.
Chenglei Si, Navita Goyal, Sherry Tongshuang Wu, Chen Zhao, Shi Feng, Hal Daumé III au2, and Jordan Boyd-Graber Large language models help humans verify truthfulness except when they convincingly wrong, 2023.
E.R. Smith, D.M. Mackie, and H.M. Claypool. Social Psychology: Fourth Edition. Taylor & Francis, 2014, ISBN 7781136845123 URL https::/bookk goog co uk bbooksk7dd
УBOLBQAAÇИAA
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Mare Pickett, Kathleen S. Meier- Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen O1- son, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Agüera Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoc Lc. Lamda: Language models for dialog applications. CoRR, abs/2201.08239, 2022. URL https: //axiv.ergg/as//2220.22222 08239.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B (Methodologicca), 58(1):267-288, 1996. ISSN 00359246. URL http: /www.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. Human evaluation of automatically generated text: Current trends and best practice guidelines. Computer Speech & Language, 67:10115,, 2021. ISSN 0885-2308. doi: http:///bi.org/11.011........./ 101 URL httph: www. scwencedireeeeeeooo com/
Yuxia Wang, Shimin Tao, Ning Xie, Hao Yang, Timothy Baldwin, and Karin Verspoor. Collective Human Opinions in Semantic Textual Similarity. Transactions of the Association for Computa- tional Linguistics, 1019, 08 2023. ISSN 2307-387X. doi: 10.1162/tac.. URL
https:/ / dci. http://ti....ggg////////
a 00584
Kathryn A. Woolard. Language Ideology, pp. 1-21. John Wiley & Sons, Ltd, 2020. ISBN 9781118786093. doi: online
URL https://
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. arXiv preprint wrXiv:2306.01699 2023.
13

---

Published as a conference paper at ICLR 2024
Benfeng Xu, An Yang, Junyang Lin, Quan Wang. Chang Zhou, Yongdong Zhang, and Zhendong Mao. Expertprompting: Instructing large language models to be distinguished experts, 2023a.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficiet uning on self-chat data. arXiv preprint arXiv:2304.01 2023b.
Fangyuan Xu, Yixiao Song, Mohit lyyer, and Eunsol Choi. A critical evaluation of evaluations for long-form question answering. Proceedings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers), pp. 3225-3245, Toronto, Canada, July 2023c, Association for Computational Linguistics. URLpttps::
acl long. 181
Yumo Xu and Mirella Lapata. Document summarization with latent queries. Transactions of the Association for Computational Linguistics, 10:623 638, 2022. doi: 10.1162/tacl a 00480. URL
Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim. Yongrae Jo, James Thorne, Juho Kim, and Minjoon Sco. Flask: Fine-grained language model evaluation based on alignment skill sets, 2023.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. Benchmarking large language models for news summarization, 2023. URL https:
A ANNOTATION DETAILS
Participants paid 0.30GBP for annotating each pair of outputs with overall scores assertive- ness/complexity ratings, or 0.60 for annotating each pair of examples for errors. This corresponded median payment of over 13GBP per hour for all experiments. Participants were recruited using Prolific, and sampled according to Prolifics 'Balanced Sample" option. Participants were allowed participate multiple times, up to limit of 200. To avoid interaction effects, participants were blocked from participating in multiple types of annotation (i.e., they were not able to annotate both for errors and overall scores).
B INSTRUCTIONS TO ANNOTATORS
B.1 ERROR ANNOTATION INSTRUCTIONS
You will be shown prompt sent to chatbot, and two possible responses.
Your job is to check whether each response contains range of errors.
Please read the prompt in full, then read both responses before assessing them. The prompt should usually make it clear what the chatbot should do, such as an- swering a question or summarising conversation. In some cases the prompt may end with a heading like "Summary:" or "Description." in this case the response should "fill in the blank" and write a summary or description of the rest of the prompt.
The form includes two attention check questions, which clearly marked. Please make sure you complete these correctly, otherwise your submission risks being rejected.
The error types that the responses should be checked for are:
Inconsistency with request - Does the response incorrectly represent or change information from the request?
Contradicts itself Is the response inconsistent with itself, or does it con- tradict itself?
Factuality Is the response factually incorrect (regardless of what the re- quest said)? Please use Wikipedia or Google to check whether any facts included in the response inaccurate.
14

---

Published as a conference paper at ICLR 2024
Relevance Does the response go off topic include information that is not relevant to the request?
Formatting - Does the response fail to conform to any formatting or length requirements from the prompt?
Refusing reasonable requests If the request is reasonable, does the re- sponse refuse to answer (e.g. "I'm sorry, can't help you with that")? poor quality attempt answer a reasonable request is allowed. If the request is unsafe or impossible, refusal is then allowed.
Repetition Does the response repeat itself? For example, if there a list in the response, are any items repeated? Does the response reuse the same phrase again and again?
Finally, you will be asked to assign the responses an overall quality score, based on how good you think the response is. You should imagine that you were user of the chatbot and base your opinion on how happy you would be with the response. If other factors than the ones above are useful for making your decision, please mention these in the text field.
Where possible, please judge each criteria/error separately from each other. For example, response that gives information that is correct but not relevant to the prompt should still be marked as OK for "factuality" and "contradiction", but negatively for "relevance". First, carefully read through the prompt, checking that you understand how you expect the chatbot to respond. Then carefully read each response.
Now, please assess each response based the criteria below. Remember that you are being asked to judge the responses, not the prompt. It's OK to go back and re-read the prompt and responses if you need to. Required fields are marked with an asterisk.
B.2 OVERALL SCORE ANNOTATION INSTRUCTIONS
You will be shown prompt sent to chatbot, and two possible responses.
Please read the prompt in full, and then rate how good each response is, on a scale from (very bad) to (very good).
You should judge the responses based on whatever criteria you think are impor- tant. Imagine that you are user of the chatbot and that you had sent it the prompt: how happy would you be with each of the responses?
The prompt should usually make clear what the chatbot should do, such as an- swering question summarising conversation. In some cases the prompt may end with a heading like "Summmay:" or "Description:" in this case the response should "fill in the blank" and write out a summary description of the rest of the prompt.
First, carefully read through the prompt, checking that you understand how you expect the chatbot respond. Then carefully read each response.
Rate each of the two responses from 1 (Very bad) to 5 (Very good) based on how good they are. Use your own judgement - how happy would you personally be to receive each of the responses? Remember that you are being asked judge the rstronggresponses///stooe not the prompt. Required fields are marked with an asterisk.
B.3 CONFOUND ANNOTATION INSTRUCTIONS
You will be shown prompt sent to a chatbot, and two possible responses. Your job is to evaluate each response//strong, according to 2 different criteria.
Please read the prompt in full, then read both responses before assessing them. The prompt should usually make it clear what the chatbot should do, such swering a question or summarising conversation. In some cases the prompt may end with heading like "Summary:" "Description:" in this case the response should "fill in the blank" and write out a summary or description of the rest of the prompt.
15

---

Published as a conference paper at ICLR 2024
The criteria that the responses should be checked for are:
Complexity How complex are the words and language used in the re- sponse?
Persuasiveness: - How assertive and persuasive is the response?
Where possible, please check each criteria separately from each other.
First, carefully read through the prompt, checking that you understand how you expect the chatbot to respond. Then carcfully read each response.
Now, please assess each response based on the criteria below. Remember that you are being asked categorise the istrong.responses//srmoos OK to go back and re-read the prompt and responses if you need to. Required fields are marked with an asterisk.
the prompt. It's
C TASK TEMPLATES
WikiHow WikiHow article titles were converted to prompts using the template "Can you tell me ?title)?".
Amazon Product Descriptions Amazon Product Specifications were converted to prompts using a wide range of templates. Please see the dataset files for details.
Curation Articles from the Curation dataset converted to prompts according the template: "aarticleXnnnnGeeeere a summary: \n"
D INFERENCE HYPERPARAMETERS
Outputs sampled using temperature of 0.7.
Llama 2
The full HuggingFace model ID used was wrrra--llaammmmam-----------aa The prompt template used was:

 

You are helpful, respectful and henest assistant. Always answer as helpfully possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.


(instruction) 

Falcon 40B Instruct The full HuggingFace model ID used
was
tiiuae/ falcon-40b-insttnnn Falcon is prompted directly with the query.
MPT 30B Instruct The full HuggingFace model ID used
was
mosaicml 3oott30b-instrue The prompt template used was:
Below is instruction that describes task. Write response that appropriately completes the request.
##Instruction |instruction|
### Response
The Cohere models queried 22/06/2023 and 17/08/2023 for Sec- tions 2 and respectively, using the generate API function. Cohere models use an undisclosed internal template.
Command 6B and 52B
16

---

Published as a conference paper at ICLR 2024
1.0
Contradiction
0.21
0.27
Inconsistency
0.27
0.38
0.8
Factuality
0.22
0.32
Relevance
0.31
0.42
0.6
Formatting
0.29
0.34
Refusal
0.27
0.34
0.4
Repetition
0.26
0.28
Scope
0.18
0.15
0.2
Fluency
0.07
0.12
Harmful
0.02
0.05
0.0
Unbiased
Biased
Figure 7: Pearson Correlation coefficients of each error factor with the unbiased (collected indepen-
dently of the enror annotations) and 'biased' (collected alongside the errors) scores. In almost every case, the correlations are higher for the 'biased' scores, confirming that asking annotators check for particular set of errors before deciding on an overall score has priming effect.
30
Error rate
3.5
Quality Score
Repetition rate
20
Refusal rate
3.0
Quality score
2.5
100
200
300
Length (words)
Figure 8: Selected error rates and overall quality scores plotted against length response. Annota- tors prefer longer responses, up until certain point where repetition starts become issue.
E ADDITIONAL FINDINGS
Annotators can be primed We collected quality scores from both groups of participants: both from those who checked for errors (biased scores) and from the group of annotators who only an- notated for overall quality. We refer this latter set scores as the unbiased score, since these annotators were not shown the еттог criteria, which might have influenced their judgements. Since collected quality scores from annotators who both were and were not shown the error criteria, we can check whether any priming effect occurred (Bargh & Chartrand [004]] inter alia.); did checking errors influence their judgement when asked judge overall
the outputs for pre-determined output quality?
Unsurprisingly, we find that the correlations between the overall scores and each error criterion higher for the biased case (Figure 7), confirming that asking annotators to check for specific errors has priming effect overall preference ratings.
'Not too long, not too short' Figure X shows plot of overall quality scores and selected error rates against length in words of the response. Annotators prefer longer responses, up until the issue. This can be interpreted
point when repetition becomes obey Grice's Maxim of Quantity (Grice 1991). which states that a felicitous speaker should be as informative is required, but not more. The rate of repetition errors increases with increasing length of response, which is to expected, and indeed partly explains the reason for the drop in quality higher lengths. We might expect that long responses include factual statements and therefore have more 'opportunity' to be incorrect, but interestingly the annotated rate of factuality emorr increases for both short and long responses.
evidence that LLMs should
17

---

Published as a conference paper at ICLR 2024
1.0
Contradiction
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Inconsistency
0.21
0.33
0.48
0.28
0.25
0.23
0.21
0.21
0.21
0.8
Factuality
0.35
0.38
0.33
0.32
0.37
0.35
0.35
Relevance
0.60
0.69
0.62
0.67
0.64
0.64
0.80
0.60
0.60
Regressor
0.6
Formatting
0.76
0.79
0.86
0.78
0.88
0.76
0.76
Refusal
0.95
0.96
0.93
0.99
0.96
0.95
0.95
0.95
0.95
0.4
Repetition
0.53
0.53
0.56
0.60
0.66
0.53
0.53
0.53
0.53
Scope 0.00
0.00
0.00
0.00
0.00
0.2
Fluency
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Harmful
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.0
Contradiction
Inconsistency
Factuality
Relevance
Formatting
Refusal
Repetition
Scope
Fluency
Harmful
Ommitted Factor
Figure 9: A sensitivity analysis of the Lasso regression weightings from Section 2.2 We recalculate the weightings, leaving out each error type in turn. In general, the changes in weightings are minor, and the features selected do not change, indicating that the regression is stable.
Reference outputs not ground truth We note that of the five systems evaluated (four models, plus the references) the reference outputs were given the lowest overall quality scores (Table(2). This confirms findings from previous work (Zhang al. 2023 Hosking al. 2023), and highlights the need for research on reference-free evaluation techniques.
Current models also very good producing fluent output, and the datasets used generate input prompts are unlikely to elicit harmful or out-of-scope responses, leading to very low error rates for these criteria (less than 1%). We therefore excluded them from our follow-up experiments, but we emphasize that they remain important considerations for models 'in the wild'.
Table Average error rates and overall quality scores from the experiments in Section by model. 'Unbiased" and 'Biased' refer to the overall quality scores.
Contradiction
Inconsistency
Factuality
Relevance
Formatting
Repetition
'Unbiased'
Refusal
Fluency
Harmful
'Biased'
Scope
Model
Command 52B
2.50
4.44
5.00
4.44
1.67
5.56
0.00
1.39
0.83
3.51
Command 6B
2.50
8.33
6.94
6.11
3.33
3.33
5.56
1.11
0.00
3.49
3.52
Falcon 40B
1.94
8.06
4.17
6.94
5.00
2.78
2.22
2.50
1.39
0.00
3.79
3.66
MPT 30B
0.28
3.06
5.56
2.22
1.11
0.28
0.28
0.28
0.83
0.00
3.76
3.77
References
0.56
4.72
8.33
6.11
1.67
0.00
0.56
0.28
0.83
0.56
3.62
3.50
F SUPPLEMENTARY RESULTS
18

---

Published as a conference paper at ICLR 2024
Complexity
Baseline
Complexity++
Frequency
10
20
Reading Difficulty
Figure 10: Frequency of Flesch- Kincaid reading grade for each preamble type. The shift distri- butions indicates that the preambles have successfully modified the complexity of the outputs.
Assertiveness--
Baseline
Assertiveness**
Frequency
Sentiment
Figure 11: Frequency of automatic sentiment for each preamble type. The shift in distribu- tions indicates that the preambles have successfully modified the sentiment (and thus potentially the assertiveness) of the outputs.
Contradiction
Error rate
0.2
Inconsistency
Factuality
Relevance
0.1
Formatting
Repetition
0.0
2
3
5
Complexity
Figure 12: Variation in enror rates with perceived complexity scores. More complex outputs are somewhat less likely to be considered containing enrors, although this effect is less strong than for assertiveness.
19

---

Published as a conference paper at ICLR 2024
Table 3: Full results of annotated error rates for each preamble type, according to crowdworkers ('Ann.") and experts (the authors, Exp.). The difference between the two annotation sources is also shown as 8. Although annotators are generally less likely to detect factuality or inconsistency errors, this difference is widened if the outputs are made to be more assertive; the more assertive an output is, the less likely annotators are to detect factual errors that it may contain.
Exp. Contradiction
Exp. Inconsistency
Exp. Formatting
Exp. Factuality
Exp. Relevance
Exp. Repetition
Ann.
Ann.
Ann.
Ann.
Ann.
Ann.
Preamble
Baseline
0.2 1.7
-15
15.2
100.3
4.3
20.3
-66.1
6.3
10.2
3.8
3.2
3.2
5.1
-1.8
Assertive--
2.7
0.6
8.5
14.6
-6.0
7.1
12.5
-5.4
14.6
14.6
0.0
9.022
4.8
2.3
4.2
-1.9
Assertive++
0.3 1.8
-1.4
2.6
19.3
16.77
24.6
22.3
3.9
533
-13
2.00000
2.0
1.2
10.5
-9.3
Complex--
0.5
0.5
4.1
16.7
12.6
5.1 25.0
19.9
4.1
0,0
4.1
2.9
0.4
0.3
1.7
-1.3
Complex+
0.8
0.8
2.4
10.2
-7.8
4.4
18.6
14.3
5.0
3.4
1.6
3.7
3.4
0.3
3.2
15.2
-12.1
Preamble Type
Baseline
1.2%
9.8%
15.5%
2.5%
3.2%
1.8%
-18%
20
Assertiveness
1.5%
0.0%
-5.0%
4.0%
5.2%
4.0%
-1.0%
Assertiveness 1.3%
15.2%
20.8
0.2%
-0.3%
-1.8%
88.7%
Complexity- 0.5%
11.77%
19.7%
5.0%
-0.3%
2.8%
-1.3%
-20
Complexity++ 0.8%
-7.3%
-1400%
1.8%
0.3%
-0,8%
-118%%
Contradiction
Inconsistencl
Factuality
Relevance
Formatting
Refusal
Repetition
Error type
Figure 13: The difference enror rates between crowdsourced annotations and 'expert' annotations from the authors, including samples that were marked as refusing respond.
Table 4: Average rates and overall quality scores from the experiments in Section by model.
'Unbiased' Quality
'Blased' Quality
Contradiction
Inconsistency
Formatting
Factuality
Relevance
Repetition
Refusal
Model
Command 52B
0.67
4.00
3.67
3.17
2.33
4.67
1.33
3.75
3.72
Command 68
1.17
4.17
5.17
4.00
4.17
7.00
4.17
3.57
3.51
Falcon 40B
1.00
8.17
5.50
9.67
8.83
9.50
3.00
3.53
3.51
MPT 30B
0.33
4.33
4.17
4.83
3.17
3.50
0.50
3.50
3.39
Llama 13B
2.17
7.00
4.83
19.83
6.17
9.67
1.33
3.63
3.52
20


Reviews:
[{"review_id": "6nSJ0Ct1kG", "reviewer": "Reviewer_knEY", "summary": "This paper studies an important topic that is very prevalent these days in the LLM field especially (but generally as well)  - use of human feedback to evaluate large language models as well as use the coarse grained data of human preference to train/tune the LLMs - the work studies, defines, disentangles and subjectivity of the human preferences and tries to ‘explain’ how various types of errors are correlated to this preference. It further suggests that human preferences do not correlate well to the important dimension of factuality of the LLM generated information.  \n\nIn particular, the paper tries to define a broad set of discrete dimensions of various properties/attributes of fine-grained evaluation criteria for LLM model for human judgment - this is trying to map the one preference score to the underlying error categories that the one score should ideally represent (somewhat equally in principle) - however these categories are meant to be defined to capture the epistemological meaning of the human preference and map that to a simple oenology of the error types described in the paper. \n\nThe paper then uses 3 datasets and collects the human preference annotations (with quality control and best practices) )on the various recent model outputs and the ontological error types. The  paper uses a Lasso regression model to fit the preference score to error categories for studying how various error categories contribute to the overall mapping to the single human preference. Some error types have a stronger contribution to the mapping function (eg. Refusal to answer which is very objective dimension unlike others) which some other do not contribute (eg. Harmfulness). Some other important dimensions like factuality and inconsistency have less weights compared to the others. \n\nThe paper then tries to study the ‘halo effect’ cognitive bias of the LLM generated output and how humans perceive and prefer more assertive generation and more complex generation over less assertiveness and less complexity. The paper notes that increased assertiveness and complexity both lead to slightly higher perceived quality, while low assertiveness\nleads to the worst rated responses. respond. Authors also note that non-expert annotators tend to underestimate the rate of inconsistency or factuality errors, and they are less likely to spot these errors in outputs that are assertive. The authors suggest how controlling the prompts for assertiveness (eg. having low assertiveness) could perhaps be used to promote safer refusals.\n\nSince perceived preference is correlated with assertiveness, the relationship suggests that using human feedback as a training objective could inadvertently increase the complexity and assertiveness of outputs (as a side-effect) and thus introduce these biases in the trained models.", "questions": "(please address the weaknesses above)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- RLHF and human preference alignment topic for LLM human-AI alignment is one of the highly discussed topics today for chatbot training. RLHF is used to teach the models to generate safer outputs (for example, refuse to respond when inappropriate, promote politeness and reduce toxicity and biases). The studies in the paper suggest that the human preferences can easily get confounded with assertive and complex text and prefer those therefore introduce this 'assertiveness bias' and 'complexity bias' in the modeling process. \n\n- The paper also discusses important findings around factuality (and the important topic of hallucinations studies in LLMs). \"Crowdworkers underestimate the rate of factuality and inconsistency errors. This difference is increased for high assertiveness responses, and decreased\nfor low assertiveness responses. In other words, annotators are more trusting of assertive responses, and are less likely to identify factuality or inconsistency errors within them\" -> in other works it could be possible that RLHF methods may promote hallucinations, and introduce more assertiveness and complexity in the generated output.", "weaknesses": "- it is not clear if this error type categorization is comprehensive enough; authors haven't provided any empirical/experimental support for these error categories.\n- There are other important biases from AI safety perspective that isn't explicitly studied in the paper (could possibly make the paper more relevant) - hallucinations, more fine-grained categorization of inconsistencies, toxicity, etc \n- the datasets studied may not be comprehensive enough to have captured some of the important dimensions of AI safety (toxicity, attribution, etc) \n- Besides the Lasso regression model, other interpretable/explainable ML methods might have shed more light on error categories and their correlation with both human preference and model output. \n- authors note that human preferences are mostly absolute or relative - there isn't any comparison between the two in this research. Since the paper provides insights into understanding these human preferences, more focus on various types of human preferences would have strengthened the finding of the paper\n- Authors very briefly touch upon the AI safety implications but do not give justice to this very important piece of information in the paper - what are the AI safety implications as a result of using human preferences for training and evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7qyvLR4uKz", "reviewer": "Reviewer_iLy7", "summary": "This paper analyzes potential confounding factors in human judgment of LLM-generated text, and hypothesizes that these confounders lead to optimizing the wrong metrics in RLHF (i.e., optimizing for assertiveness rather than factuality). The main result is that they find human annotators rely on features such as assertiveness and complexity when judging the overall quality of a piece of text, rather than relying on harder-to-evaluate features like factuality. The paper also evaluates quality judgments for existing models trained with RLHF on human preferences, again finding a correlation between assertiveness and quality judgments.", "questions": "* Why were the error types in Section 2 the ones chosen? There is a brief discussion (inspiration from Xu et al. 2023, Gricean maxims) but it would be nice to have more discussion on why these particular types were chosen.\n* Why do contradiction/scope/fluency/harmful have no contribution in Figure 1?\n* Did you analyze where disagreements came from in factuality judgments in Section 2.1? Where agreement was 0.64\n* What was the hypothesis of assertiveness and complexity being confounders founded upon? Are there other confounding factors that could influence this, e.g., length?\n* \"incorrectly perceived as being less repetitive\" -- is repetitiveness objective? It seems a lot more subjective than something like factuality.\n* What are the two lines in Figure 6?\n* Figure 5 shows annotated error rates -- are these expert annotations or general crowdsourced annotations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper studies a very important problem: that without understanding the underlying factors influencing human judgments of text, in light of ambiguous and underspecified annotation guidelines asking for nebulous quality ratings, when performing standard RLHF we are optimizing models in unintentional directions. The analysis of annotations is relatively thorough and easy to understand.", "weaknesses": "A couple of points:\n* I wish there were discussion on not just the problem of individual judgments being ambiguous, but that standard RLHF is optimizing towards a single user preference per example rather than considering factors resulting in a distribution over judgments for a number of different annotators (a point that is made e.g., as a motivation for jury learning, Gordon et al. 2022).\n* I would also like more discussion on what to do in light of these findings. There were hints at one possible perspective of a solution -- optimizing nebulous \"preference\" is far from what people do when using language, which is more based on utility and communicative success (which preference is not tied to at all).\n* I think the analysis in Section 4 is conflating a few different factors in comparing models -- the fact that Command is trained off-policy and Llama 2 is on-policy isn't discussed in enough depth for me to be convinced this is relevant to mention. I don't have a good intuition on why that might influence the model behavior, especially if the two models are trained on different data as well.\n* Following the above, I would have liked to see an experiment that directly shows how optimizing for human preference influences outputs across different dimensions explored in this paper (e.g, assertivness). The evidence from this paper is post-hoc analysis of existing models (in Section 4), and the earlier experiments showing annotators generally conflate quality with features like assertiveness/complexity. There are a couple of things this analysis conflates itself, e.g., the actual data that Command and Llama 2 are trained on, and their learning algorithms. But an even stronger experiment would perform RLHF-based finetuning (or some kind of finetuning with human preference judgments) on data where we have judgments of or could control complexity and assertiveness, and measure how the final models perform on factuality judgments after this process. E.g., perhaps this would show that optimizing for assertiveness++ responses hurts factuality. Or maybe it would show that with the exact same data, fine-tuning using a reward model exacerbates this problem more than just fine-tuning directly. But with the current analysis presented, it's hard to make any concrete conclusion beyond that all these models result in judgments conflating quality with style.\n* I would also like to see an experiment controlling for factuality, like in Section 3.2. E.g., if it's possible to even post-hoc separate the results in Figs 3 and 4 by whether an expert judged the output as factual or not. Basically, do you find that assertive false responses are rated as factual by annotators more often than inassertive true responses?\n* I think the main table showing expert judgments for Sec 3.2 is in the appendix, but in my opinion it should be in the main paper. \n\nIn general, I think the direction this paper investigates is super important, and it indeed highlights a number of issues with current methods. But I also think it could be made a lot stronger, especially in directly showing how these annotation artifacts influence model behavior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wyH8nMZIbk", "reviewer": "Reviewer_Rbzk", "summary": "This paper studies what attributes human preferences models over LLM output stend to capture, and which they don’t. Specifically, it considers ten error types: harmfulness, fluency, scope, repetition, refusal, formatting, relevance, factuality, inconsistency, and contradiction, and has human annotators label whether a failure arises in any of these categories, then uses lasso on top of these categories to to predict the annotated quality of an output (measured by different annotators). They find that refusal, formatting, relevance, repetition, and factuality, and inconsistency lead to reductions in overall score, while contradictions, scope, fluency, and harmfulness do not. In a second experiment, the annotators find that low assertiveness tends to lower annotator scores, and raising assertiveness tends to improve scores across subcategories (e.g., factuality, relevance). The paper finally finds that RLHF increases assertiveness, suggesting that the annotator preferences get baked into the objective.", "questions": "* How do the lasso weights change when you optimize on different subsets of the properties?\n* How much does the choice of ten properties you label for matter? What would happen if you included different properties?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper studies an important problem: what attributes do human labelers actually care about in model outputs, and does this match the attributes we'd like\n* The paper writing is clear throughout and easy to follow\n* Many of the empirical results would likely be interesting to the community at large; I especially liked Figure 5, which shows that simply increasing assertiveness reduces the error rates for many categories (e.g., factuality)", "weaknesses": "* The paper feels a bit ad-hoc; the properties tested seemed kind of arbitrarily chosen, but the specific set tested should have a significant impact on the learned lasso weights (e.g., if one feature is more predictive than the rest, the lasso weight would put all of it on that)\n* Some aspects of the paper do not engage with prior work. For example, it claims that “human feedback the de facto standard for evaluation” with no citation, and does not engage with the extensive work benchmarking LLMs \n* The RLHF increases assertiveness claim is tested indirectly; it compares Llama 2 13B and Command 52B, which have many confounders beyond RLHF.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lVkWSPKBfi", "reviewer": "Reviewer_mWEt", "summary": "The paper is a careful study of biases in human feedback, a critical component of influential approaches to aligning (e.g. RLHF) and evaluating LMs. The authors conduct extensive experiments having humans evaluate responses of several LMs (Cohere models, Mosaic models and LLaMA-2).\n\nFirst, the authors (i) ask crowdworkers to evaluate LMs’ outputs according to a set of separate criteria as well as give overall score, (ii) train a linear model to predict overall score from scores for separate criteria, (iii) use coefficients of the linear model as a measure of importance of each separate criterion. Model refusing to response is the most important criterion. When the crowdworkers are shown distractor samples (query and response are mismatched), factuality and contradiction (within the output) are both rated worse which implies that annotators fail to disentangle these criteria from the overall quality of a response.\n\nNext, the authors investigate how assertiveness and complexity of LM response affect human judgements. They found that annotators are more trusting of assertive responses, and are less likely to identify factuality or inconsistency errors within them. Furthermore, more complex or more assertive responses are incorrectly perceived as being less repetitive.\n\nBoth of these findings are cast a shadow of doubt on whether reward derived from human feedback is the right optimization target for aligning LMs or the right criterion for evaluating them.", "questions": "This is not really a flaw, but it would be much better if the experiments were done on responses from frontier models (such as GPT-4, Claude 2 and possibly LLaMA-2-70B). Was budget the limiting factor?\n\nA minor remark: The idea of predicting human preference judgements using a logistic regression model on top of human-interpretable features was recently explored in two concurrent papers that might be worth taking a look at: https://arxiv.org/abs/2310.13548 and https://arxiv.org/abs/2310.13011", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. I think the paper addresses a very important and understudied problem, at the very heart of AI safety: how to supervise and evaluated highly capable LMs. The fact that humans are easily misled and prefer convincing-and-assertive-but-subtly-wrong replies over truly correct replies is extremely worrying as it suggests we are training LMs to be deceptive.\n2. The paper is well-written and easy to follow. \n3. The analyses in the paper are carefully designed and comprehensive. I appreciate that the authors took care of estimating crowdworkers reliability, that their human feedback collection procedure is informed by a body of NLP and linguistics and that they conduct experiments with relatively large models (13B-52B).", "weaknesses": "1. One big question not addressed in the paper is how the biases in human judgement scale with human capabilities (e.g. question difficulty, annotator competence) and model capabilities (e.g. model size). Are humans more biased when evaluating responses they’re less competent to evaluate (e.g. medical advice, science explanations)? Are larger models more susceptible to exploit human biases? Overall, I think a good perspective for grounding this paper is scalable oversight [1]: the problem of evaluating agents more capable than the evaluators. LM capabilities will soon surpass human capabilities on many tasks and it’s critical to know whether human feedback will be increasingly misleading. That’s of course a big open problem and I don’t take not addressing it against the paper, it but it would be relatively easy to do experiments with models of the same family but different size or group questions by (perceived) difficulty.\n2. I’m not sure I’m convinced that “Human evaluation is necessary” (as stated in the conclusion). Techniques such as RL from AI feedback have been shown to be as effective as RLHF for increasing the harmlessness of LMs [2]. They ultimately rely on humans, but these are human experts involved in prompt engineering and designing rules and constitutions used for generating training data for SFT and training preference models. A big open questions is whether LMs-as-evaluators inherit human biases, even when carefully prompted. A simple experiment addressing this question would involve having an LM play the role of an evaluator and conducting a similar analysis for its judgements.\n3. The crowdworkes were asked to rate LM responses according to each criterion in series and then to give an overall score. I wonder such ordering of the task is not itself a confounder. Would overall scores be different if the crowsworkers were not asked to give scores for criteria before? Would a score for `Formatting` be different if the crowdworkers were not ask for `Factuality` before? Do authors control for this anyhow?\n4. It’s subtly incorrect to say that “Perez et al. (2023) identified similar ‘inverse-scaling’ behaviour, where training on a RLHF objective worsens sycophancy”. I think the paper showed that base LMs are already pretty sycophantic and that RLHF does not fix that. I think from that paper alone it’s still unclear that RLHF increases sycophancy; there is no robust trend. However, Sharma et al. [3] have recently provided much stronger evidence for the role of human feedback and RLHF objective in the emergence of sycophancy.\n\n[1] https://arxiv.org/abs/2211.03540 \n\n[2] https://arxiv.org/abs/2212.08073\n\n[3] https://arxiv.org/abs/2310.13548", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)