Using Natural Language Explanations to Rescale Human Judgments
Authors: Manya Wadhwa
Jifan Chen
Junyi Jessy Li
Greg Durrett
Venue: COLM (2024)
Keywords: human evaluation, natural language explanation, likert ratings, question answering, LLM
TL;DR: In the era of subjective annotations, eliciting natural language explanations from human judges can surface nuances in evaluation. We propose a method that operationalizes these explanations in a principled manner.

Abstract: The rise of large language models (LLMs) has brought a critical need for high-quality human-labeled data, particularly for processes like human feedback and evaluation. A common practice is to label data via consensus annotation over human judgments. However, annotators' judgments for subjective tasks can differ in many ways: they may reflect different qualitative judgments about an example, and they may be mapped to a labeling scheme in different ways. We show that these nuances can be captured by natural language explanations, and propose a method to rescale ordinal annotations and explanations using LLMs. Specifically, we feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce a numeric score anchored in a scoring rubric. These scores should reflect the annotators' underlying assessments of the example. The rubric can be designed or modified after annotation, and include distinctions that may not have been known when the original error taxonomy was devised. We explore our technique in the context of rating system outputs for a document-grounded question answering task, where LLMs achieve near-human performance. Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.

Published as a conference paper at COLM 2024
Using Natural Language Explanations to Rescale Human Judgments
Manya Wadhwa, Jifan Chen, Junyi Jessy Li, Greg Durrett Department of Computer Science
University of Texas, Austin manya. , wadhwa@utexas,.ed edu
Abstract
The rise of large language models (LLMs) has brought critical need for high-quality human-labeled data, particularly for processes like human feedback and evaluation. A common practice is to label data via consensus annotation over human judgments. However, annotators' judgments for subjective tasks can differ in many ways: they may reflect different qualita- tive judgments about an example, and they may be mapped to labeling scheme in different ways. We show that these nuances can be captured by natural language explanations, and propose a method to rescale ordinal annotations and explanations using LLMs. Specifically, we feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce numeric score anchored in scoring rubric. These scores should reflect the annotators' underlying assessments of the example. The rubric can be designed or modified after annotation, and include distinc- tions that may not have been known when the original error taxonomy was devised. We explore our technique in the context of rating system outputs for document--rounnder question answering task, where LLMs achieve near-human performance. Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.
1
Introduction
With the rise in model performance due to large language models (Brown et al., 2020; Ouyang et al., 2022, LLMs), there is a shift towards working on and evaluating more subjective tasks. On tasks like news summarization, LLMs can no longer reliably be judged by automatic metrics (Goyal et al., 2022a; Xu et al., 2023; Wang et al., 2023b) and achieve near-human performance (Zhang et al., 2023; Zhan et al., 2023). This high performance makes harder than ever to annotate model outputs for errors (Saunders et al., 2022; Dou et al., 2022; Chang et al., 2023; Goyal et al., 2022b; Liu et al., 2023; Chen et al., 2024), important ingredient as models are deployed in high-stake scenarios.
This shift towards more subjective tasks and difficulty in identifying er- rors has also led to a change in the role of human judgments in NLP. Ear- lier crowdsourcing work on simple la- beling tasks explored capturing anno- tator bias with item-response models (Dawid & Skene, 1979; Smyth et al., 1994), especially to learn annotator quality (Hovy et al., 2013). As the am- bit of NLP research has expanded to in- clude more sophisticated downstream tasks, taking into account subjectivity (and thus, inherent disagreement) in
Explanation-based rescaled scores
score: 50/100
scorer
Q: Why is this the case?
This should' denoii... missing fullinigg-
The article says...
Document
Answer (machine), to be judged
missinng all info
missing majorr info
missing minur info
complete
Figure 1: Overview of our method. By feeding explanations that annotators write into an LLM, we can rescale their coarse-grained judgment 100-point scale.

---

Published as a conference paper at COLM 2024
Input
Judgmect
Refereace Rescaled Scores ruberic
Rescaled
Resxaled
LLM
Scores
Question: Why was the entire clinbinng seison in doups! Anover: The Sherpas hat walked our peotest of the deatts of 16 of teir collesguzs
wi rubric
ovalanche, and their demands for better
Label: missing sajjo
besimeent and benells.
NLE: Imporant information was neglected, sentences and 15 missing
55
50
Articie: KATMANDU, Nepul
Sherpe quides camp
Washewaa,, ailer
collcagaas
Labdl complete
exposed an indercument resertment by Shorpas
NLE: The govemment urwilling to give the Sherpes decent sage or benofits for risking their lives.
100
100
100
pay, mecament
seasca incersinals fleeer doata, goversiees quillly ansouteed that top warsom ofteais would base camp
Thinsky aepatianc
Steppae and encoumgs them
rour woek [..]] (8) work the povemnet accrpe their demands. [-. (10) thr
unclear whather
setum
Label: complete
NLE: Senience contains the needed information
100
the Sharpos said they dewrrea mor- including mom imarance
sutmmoized the answer.
money, noce finnciall and for the victims funilles cew regeiations to ensure climbers rights. (1I) Without the help
pha who are key guides and also has lom of gear up the
Label: missing Senteners Missing: 15,
Everrst [- (15) is yod impossill for many of ss amiimm:: climbing while thene and thnez our fricral burted in the suw.
mountais. it would be acadly impossible for climbers to scale
machine respouse answered the question correctly but missed some relevint in informtiion that
75
70
said Dogje Sherps, on experienced tiverst guide from the any Hirrolgga communiy that has bocome fimoss for its high- attitue skills and eedarunce .... (44) The assrrance pasout for those killed in the ovalanche, which now stands S10,400, will also be incrceeal iu s15 x20 or million rxpoo. the Miniary of Tourien said for less than the Sherps' denand for $20,800, [...] (45) Hundreds of have ched trying.
clarifies the Sherpos' demands.
Label: missing minor
Sentences Missing: & also unkaown Sherpus would accept
70
and return.
Figure Example from INQUISITIVE- where annotators labeled a question-answer pair on a Likert scale and gave explanations for their judgments. Separate human labelers rescore these explanations according to an aspect specific scoring rubric. Our proposed explanation-based rescaling (EBR) method maps judgments to scores using that rubric, producing scores closer the reference scores by accounting for factors mentioned in the explanations.
annotated data has surfaced as key direction (Plank, 2022; Uma et al., 2021; Basile et al., 2021; Nguyen et al., 2016).
While existing work developed label aggregation methods and ways to incorporate different labels (Plank, 2022), information from the labels alone still limited. Instead, intricacies in human judgments can be captured by natural language explanations provided during data annotation, which capture more nuanced subjectivity. They enable us to go beyond direct labeling, where annotators can choose different coarse labels for the same reason or the same coarse label for different reasons. The question is, how do we transfer least some of that expressiveness into numeric expressions that are more friendly for model training and evaluation?
This paper proposes explanation-based rescaling (EBR) that enables us to transfer the expressiveness of natural language explanations into numeric forms. Our key idea is to make the ordinal label space (e.g., a coarse Likert scale) fine-grained, namely 0-100 scale (Kocmi& & Federmann, 2023), which then enables us to place the initial annotations in it by leveraging natural language explanations in principled manner. Our approach begins by gathering explanations for each judgment during the annotation process. Next, we leverage an LLM to convert both a Likert judgment and its associated explanation into a numerical rating between 0-100. Crucially, this rescaling is guided by a predefined aspect-based scoring rubric, which can be defined entirely post-annotation, and provides task-specific guidance to the LLM on the placement of labels on the 0-100 scale. We find that an LLM with rubric produces scores consistent with those that human produces when using the same rubric.
We consider the task of evaluating LLM-produced answers for document-grounded, non- factoid question answering. We use annotated high-level questions (targeting discourse understanding) from the INQUISITIVE dataset (Ko al., 2020), supplementing these with freshly collected questions from geographically diverse and recent text sources. We collect outputs from several LLMs, then annotate answers chiefly for answer completeness, with workers giving Likert judgment and explanation. Through annotator recruitment and filtering, we selected a set of crowdworkers who gave high-quality, reasoned decisions, yet still had differences in opinion. Figure gives example from our dataset where annotators chose different Likert judgments for information completeness, but the explanations show that some of them were looking at the same factor. Rescaling with scoring rubric is able to
2

---

Published as a conference paper at COLM 2024
impart fine-grained distinctions based on these explanations, and in particular, works better than rescaling without a rubric.
We evaluate approach on whether LLM rescaling (1) can discern subtleties in natural language feedback as well as humans do; (2) changes correlation between annotators. The proposed approach brings scores closer to how humans would do this rescaling without impacting agreement, while retaining subjectivity.
Our main contributions are: (1) A method for rescaling Likert judgments with explanations using LLMs anchored in a scoring rubric. (2) A dataset of 12.6k human judgments (with explanations) on the completeness and correctness of answers produced by strong systems (GPT-3 and GPT-4) for document-grounded high-level question answering.
2 Background and Task Desiderata
Motivating Example Figure shows motivating example for approach on evaluating LLMs when they answer document-groundee questions. Given an article and a question, we use an LLM to generate a freeform answer. The answer to the question is high quality, but we want to be able to evaluate the LLM output precisely and assign granular scores. Our approach targeted towards tasks like this kind of LLM evaluation, where humans have to work hard to articulate subtle distinctions in their annotation.
We have five crowd workers give judgments about the answer this question. They disagree about whether the question is missing major or minor information. For this task, it would be difficult to specify concrete enough annotation guidelines to make the label fully unambiguous. However, some of the explanations call out the same relevant feature of the answer, which is the answer was missing specific sentences talking about sherpas possibly not returning back. Our approach accounts for subjectivity in assessing quality, but assumes that there is an underlying ground truth answer which annotators can identify.
An LLM can judge the provided labels and explanation, simultaneously giving finer-grained information ratings on a scale of 0-100. These ratings are anchored in a scoring rubric, which is defined post-hoc after looking at collection of explanations for the aspect being evaluated (in our case is for completeness). As mentioned above, the 0-100 scale offers granularity to effectively map variety of explanations to numeric values, and the aspect-specific scoring rubric ensures consistency in this mapping, as opposed to mapping examples without on the numeric scale. We describe our method in detail in Section 5.
Need for new data We need datasets that contain human judgments accompanied by informative, well-formed natural language explanations, where the underlying task satisfies the two properties bolded above (strong model performance and the right amount of subjectivity). We found three datasets that are relevant but they are not compatible with these characteristics. SQuALITY (Wang et al., 2022) includes evaluation of both human and model summaries by other humans. However, this task is very demanding due to the long document length, and we found that the feedback provided in the existing dataset is short and underspecified. Saunders et al. (2022) also have dataset with human critiques of topic summaries. However, the evaluation criteria are defined loosely, leading to critiques that are too generic. There is no multiply-annotated data for us to reliably perform analysis on the effect of our rescaling methods. Finally, Filighera et al. (2022) releases dataset for short answer grading with explanations; however their data does not contain annotator information. Note that other existing datasets for explanations in NLP, such as those mentioned in Wiegreffe & Marasovic (2021), do not align with our task requirements, since classification tasks do not involve fine-grained critiquing of outputs.
3 Collecting QA pairs
We study human judgments and critiques for non-factoid, document-groundet question answering, focusing on judgments of information completeness, i.e, whether machine- generated answers contain all key pieces of information from document. We use two
3

---

Published as a conference paper at COLM 2024
sources of data for this task. First, we use the questions collected in the INQUISITIVE dataset (Ko et al., 2020). Second, following the annotation guidelines from INQUISITIVE, linguistics student wrote questions western geographic entities (Hong Kong, Singapore, and India). This reduces leakage about notable prior news events for the L.Mss in question. A summary of the two splits is given in Table 5. All text is in English. We call our dataset INQUISITIVE-RRAAI See Appendix A for examples of articles for the task.
articles focused recent news events and non-
The questions target high-level comprehension of text, making the answers complex with information distributed across multiple sentences. They contain a mixture of causal ques- tions (e.g., why are they now liberalizing bank laws?), procedural/ eelabooaaaooiionnn questions (e-g., how will the exceptions to the verification concept be taken care of?), background information questions (e.g., what is the main focus of this movement?), and instantiation questions (e.g., which groups were the human rights activists working on behalf of?) (Ko et al., 2020), all of which require discourse-level processes and reasoning.
This task meets our desiderata by striking a balance between subjec- tivity and objectivity. There are di- vergent opinions about how much information should be included, but because the correct answer is grounded in the article, it should be relatively easy to judge if key information is missing. This con- trasts with other long-form ques- tion answering tasks like EL.5 (?), where the lack of grounding makes it difficult to judge information completeness and requires subject matter experts for each question (Xu et al., 2023).
Missing Missing Missing Complete All
Major
Minor
Overall
18
4
11
67
text-davinci text-davinci-003
10
33
8
3
12
77
GPT-4
8
3
8
81
EXPERT-HUMAN
11 8
11
70
Table 1:
Unaggregated % of labels in each category across all QA systems and for each system.
We use three off-the-shelf LLMs (Davinci, GPT-3.5-turbo and GPT-4) to answer questions from INQUISITIVE-RRAA We also have two human experts answer a subset of the ques- tions. These systems are shown in Table 6 and the prompts used are given in Figure 8.
Human Evaluation of QA pairs
Using Mechanical Turk, we enlist 8 qualified crowdworkers to conduct human evaluation of the question-answer pairs in our dataset. Each instance is evaluated by 5 crowdworkers. The study leads to a dataset of 12.6k annotations consisting of discrete labels along with natural language explanations.
Each crowdworker was given the article, question, and answer, and was asked to evaluate the answer on completeness and correctness attributes. For completeness, they were asked to choose Likert scale the amount of information missing in the answer with respect to all relevant information present in the document. Four options were given: complete, missing minor information, missing major information, and missing all information. For correctness, they were asked to mark if the information in the answer is faithful to the information present in the document. Crowdworkers were also asked enumerate missing sentences and give rationale in the form of natural language explanation for their decision.
Appendix B.1 describes our process of recruiting and qualifying crowdworkers in more detail and shows screenshots of the interface (Figure 7). We ensure the crowdworker pay is ~$15/hour Despite disagreements annotation, we have substantial evidence, as well as third party validation (Appendix B.2), that these crowdworkers are attentive to the task and labeling high-quality data.
'hee label distributions in the two data splits is similar, suggesting that leakage has no noticeable impact. Thus all results are reported on the entire BROUAAITIVE-BROAD
4

---

Published as a conference paper at COLM 2024
4.1 Dataset Statistics
Table shows the un-aggregated % of labels in each category for the completeness attribute for INQUISITIE--RRADE The distribution is skewed towards the complete label. Table also shows the distribution of labels for different QA systems being evaluated. Together, these show the challenges of judging the outputs of these systems. They are performing at human level according to these annotators and make relatively infrequent mistakes.
Similarly, for the correctness attribute, 70% of QA pairs are marked as correct. Those marked as incorrect constitute of unanswerable questions where the model hallucinated information instead of accurately stating the information was missing from text. We thus focus our analysis only the completeness attribute.
4.2 Human Label Variation
We also measure inter-annotator agreement across the discrete ratings. If we collapse to two classes, complete vs. not, the Fleiss Kappa value is 0.328. Kendall's Tau-b (T₂) correlation across all labels is 0.325. This "fair" agreement, and given the perceived quality of the natural language explanations, we view this as evidence of genuine subjectivity.
Please refer to the Appendix (Table 10)
Rescaling Prompt
for the distribution of labels across 3 an-
The main goal of your task is to score a machine generated response to question. Scoring is on the "completeness"
notators. Table 14 gives examples of ex-
planations that differ in their label deci-
attribute. A complete answer will have all relevant information from the article required to answer the question and an incomplete answer won't 't. However, you not dircctly scoring the machine response, but instead using a given feedback and amount of missing information. The details given below:
sion but agree on details in their explana- tions.
Explanation-Based Rescaling (EBR)
Article which the question was asked: (article) Feedback given to machine response: (feedback)
Sentences marked as missing: (enumerated missing sentences)
In this section we propose a rescal-
Level of the missing information definition)
ing method that captures natural lan- guage explanations by using them to re- evaluate the Likert labels. We take inspi- ration from the scoring mechanism pre- sented in Kocmi & Federmann (2023) for machine translation evaluation, which validated the effectiveness of a 0-100 scale compared to other options. The key element here is that a chosen scale pro- vides higher granularity than the coarse grained scale on which the initial Likert labels are collected. Instead of using an LLM to directly evaluate the task, we use a more detailed scoring rubric.
On a scale of 0-100 how will you scure machine response using the feedback and level of missing information stated above? Use the rubric below for scoring:
1. if the answer is complete, give 100 points
2. if the answer is missing one more minor details then have deductions ranging from to 30 points based on the severity of missing details
3. if the answer is missing majur facet of information, it results in deduction of least 40 points and more than 50 points are deducted if less than half of the correct information was given.
if the answer contains correct information but only marginally relevant information from the article, 70 points are deducted
5. if the answer contains no correct information but the article clearly has information present, 100 points are deducted
What is the score? Give number.
Formalization Assume we have a col- lection of items Xi,..., Xn which are be- ing rated by annotators AI,...,dh in sparse manner, i.e., not every annotator rates every item. Each annotator assigns an item discrete rating Tk and writes an explanation Êzi, where refers to the item and refers to the annotator. We rescale annotator judgment as following: $/4 = f(rk (())
Figure 3: considers the aspect definition (eg: completeness, defined in the prompt above), a scoring scale (0- 100), the task input (the article), human judgment (Likert rating, natural language explanation and missing sentences) and a scoring rubric.
The rescaling prompt for GPT-4, which
Details about the rubric creation process given in Appendix C. We also explore rescaling variations without rubric. Variants of the rescaling prompt are given in
Appendix D.1 and report our results in Section 7.1. Note, we do not rescale instances that get the extreme
5

---

Published as a conference paper at COLM 2024
labels complete and missing all, since in these cases rarely observed nuanced explanations from which rescaling would be possible.
Prompt To compute f, we invoke gpt-4-0613, taking input both the explanation egi along with the discrete rating label rki- The proposed rescaling prompt is given in Figure The rescaling prompt is structured following:
1. Aspect definition:
defines the aspect being evaluated (in our case, completeness)
2. Scoring scale: defines the numeric scale to which the human judgment needs to be
mapped (0-100 for task; note that the LLM typically outputs multiples of 5)
3. Task input: helps contextualize the natural language feedback (the article)
4. Human judgment: Likert rating and the natural language explanation (and when available, missing sentences)
5. Scoring rubric with deductions: anchors the scoring to be consistent
6 Evaluation
With our initial human annotation and methodology established, now turn to evaluation. We first introduce notion of reference rescaling: given rubric, can expert annotators agree on how explanations from our dataset should be rescaled? We establish data and baselines for this evaluation, then turn to comparing human and automatic rescaling in Section 7.
6.1 Reference Rescaling
In order to evaluate the ability of LLMs to faithfully rescale natural language explanations, ask three experts to establish reference scores. We sample 145 instances, where each instance consists of a QA pair and corresponding human judgment in the form of the Likert label, natural language explanation and missing sentences. The distribution of labels in this subset is: complete: 20, missing minor: 52, missing major: 53, missing all: 20. We sample more from missing minor and missing major labels since these categories have nuanced explanations, compared to complete or missing all, which occur at extreme ends of the scoring scale. Each expert was given the same rubric information the rescaling prompt mentioned in Section 5. We refer to average of the expert rescaled scores as reference scores, represented by
Using this, we can answer two questions. (1) Can humans consistently rescale if given the rubric (i.e is our methodology sound)? (2) Can LLMs do this rescaling automatically in manner similar to humans when given the same rubric?
6.2 Metrics
We determine how well our rescaling method does by comparing the rescaled values with references scores and also by comparing inter-annotator agreement before and after rescaling. We use Mean Absolute Error (MAE) and Kendall's т correlation (Kendall, 1948).
Mean Absolute Error computes a difference between M ( exi) and R (TT where M is system (e.g., our proposed rescaling method), R refers to references scores (rescaling done by experts) and (Tki (k)) is the Likert rating and explanation for item Xg by annotator a;
Kendall's T is based on pairwise score comparisons, and thus reflects common ranking use case. We use Kendall's Th, which makes adjustments for ties. Using a method (e.g., our proposed rescaling method), we compute correlations between the proposed and reference rescaled scores T (M(iK,, ex:), R(Tki, ex))
We also compute pairwise correlations between two annotators' rankings, which we denote as T(M(re, ex), M(ripee;)) for annotators and j, where is a placeholder index. We then compute an aggregate correlation across all pairs of annotators: EmgafeAxAnita T(M(HHi ex), M(rk, 4,))), where A is the set of annotators.

---

Published as a conference paper at COLM 2024
6.3 Baselines
We compare the proposed rescaling with four baselines that map human judgments to numeric scale and calculate Kendall's T and MAE against reference scores for each method.
STATIC rescaling maps the Likert ratings to a static numeric scale by assigning complete to 100, missing minor to 70, missing major 30 and missing all to O. This mapping does not use any fine-grained information from the explanations.
AvG EBR rescaling leverages the explanations to come up with mapping for the four Likert labels to 0-100 scale. To find this mapping, we first use EBR to rescale human judgment the dataset. We then average all numeric scores under each of the labels. This method allows to incorporate explanations while keeping the original level of granularity. This also maps labels at a calibrated interval from each other instead of having them at equal intervals (as in the STATIC rescaling method). We get the following mapping for each label: complete: 100, missing minor: 78.6, missing major: 50.9, missing all: 0.0.
EBR W //O RUBRIC rescales using human judgment which includes Likert ratings and explanations to get a score without the aspect-specific scoring rubric. This method can theoretically map examples with any Likert rating anywhere in the 0-100 numeric scale.
MISSING SENTENCES HEURISTIC (MSH) uses the number of miss- ing sentences marked by the anno- tator a way rescale the orig- inal annotation. In this method, deduction of 16 points is made for each sentence that is marked as missing; this value was set to min- imize MAE. This scoring baseline solely dependent on the number of missing sentences and does not incorporate any natural language explanations for evaluation.
Missing Minor
Missing Major
Overall
(1,2)
0.62
0.25*
0.82
(2,3)
0.53t
0.19
0.81
(1,3)
0.51
0.16
0.78*
MAE
(1,2)
4.04
20.19
8.86
(2,3)
5.77
13.02
6.93
(1,3)
6.15
21.89
10.28
Table Kendall's т and MAE for expert rescaled
scores. t, Statistically significant 7 value with < 0.05.
Results
In this section we show the effectiveness of our approach by comparing LLM rescaling against reference scores. We also explore how the rescaling influences inter-annotator agreement.
7.1 Rescaling Results
Missing Minor
Missing Major
Overall
Are humans able to do the proposed rescaling consistently? To establish humans can do look
MAE 4| rt
MAE +
r MAE
Without Rubric
the proposed rescaling at Kendall's 's and MAE between pairs of experts as reported in Ta- ble 2. Correlation is high and MAE low when considering all la- bels. We also specifically look at missing minor and missing major la- bels; making distinctions within these requires the most nuance. Given the granularity of the pro- posed scoring scale (0-100) for our method, see experts showing high correlation and low MAE for missing minor. However, for miss-
Static MSH Avg EBR
15.1
12.8
0.85t
10.1
0.49
12.5
0.35
25.4
0.42
24.2
15.3
16.4
0.85t
18.6
EBR
0.05
15.6
0.06
19.8
0.69*
12.9
With Rubric
Avg EBR EBR (ours) 0.45*
8.7
14.1
0.85
8.4
7.9
0.20
14.4
0.83*
8.1
Table 3: Comparison of the proposed rescaling against reference scores using Kendall's sss 7 correlation and Mean Absolute Error. Reference scores are an average of three expert scores. EBR is the proposed method." ! Statisti- cally significant value with p 0.05
7

---

Published as a conference paper at COLM 2024
ing major, even though scores show low agreement we still see positive correlation granular scale. In Figure 4(a) we observe that one of the experts had higher spread of scores in missing major compared to the other two, highlighting subjectivity in explanations and underscoring the complexity of this task. Table 12 in the Appendix shows examples of explanations given for the missing major label category.
(a)
Does prompting GPT-4 with the rubric recover the rescaled values from humans? Table shows
100
expert 1
expert 2
80
Scores(o-100)
expert 3
how our proposed rescaling us-
60
ing GPTT4 compares to reference scores on 145 instances mentioned in Section 6.1. We look at Kendall's and MAE between reference scores (average of expert rescaled scores) and EBR scores. We also compare EBR with the baseline methods defined in Section 6.3. Again, we specifically look at miss- ing minor and missing major label categories, since they are the main source of subjectivity and have more nuanced explanations.
40
missing_all
missing_majp major
missing_mino mmmior
complete
category
(b)
80
Scores(o-100)
60
Our approach consistently outper- forms all other methods in terms of achieving the lowest MAE, indi- cating its ability to faithfully cap- ture subtle nuances in human ex- planations when compared to al- ternative rescaling techniques.
40
reference
20
EBR w/ rubric
EBR w/o rubric
missing_all
missing_mapp
missing_ minor
complete
category
Figure 4: (a) shows the distribution of scores by the three experts. (b) shows the distribution of scores us- ing EBR w RUBIC and EBR w / RUBRIC against average of the three expert scores (reference scores). EBR w/o /O RUBRIC is more scattered whereas the rubric causes the categories to track experts 2 and 3 more closely.
Overall correlation is consistently high across all methods. However, since this includes all labels, the complete and missing all categories can distort the overall metric. We instead choose to focus miss- ing minor and missing major specif- ically, looking at the correlation within these more subjective labels. Notably, with our proposed method both correla- tion values, i.e. 0.45 for missing minor and 0.20 for missing major are similar to the correlation values observed among experts, as shown in Table 2.
While MSH gets a higher correlation value than our baselines also leads high MAE, even though this is the criterion it was optimized for. Note that STATIC and AvG EBR do not produce rankings within these categories, and therefore are not evaluated this setting.
How important is the rubric? Table 3 shows the usefulness of the rubric. EBR with rubric consistently achieves lower MAE and higher correlation, across all labels and for missing minor and missing major.
In Figure 4(b), we also look at the distribution of EBR w/ RUBRIC scores and EBR W W/ RUBRIC scores within each label category against reference scores. EBR // RUBRIC scores fall into label order without any explicit constraint. EBR W / RUBRIC is more spread out and scores within category cross boundaries with those of other label categories. Note that this is not necessarily wrong, but we view it as an indicator that the scores are less calibrated when rubric is given.
8

---

Published as a conference paper at COLM 2024
Label distributien rescaling
Score distribution restalescaling
Annotator
complete
Label: missing major
Missing Sentences:
NhE:The answer correct, insuflicient
Anritatr
mincr
EBR 60
Areectator
60
Annotator
major
Label: missing major
Missing Sentences: 3, 5,7
20
NLE:The response is partially correct and missed the information stated sentences 3,4,577.
115
major
minor
complete
20
40
80
100
Annotator
Annolatio
Figure Class label distribution and post-rescaling model scores (EBR) for a pair of annotators. Note that horizontal jitter is added to differentiate points; all scores produced by our method are multiples of 5. The explanation-based rescaling imparts finer granularity to the judgments while retaining subjectivity. We show pair of explanations, where the two annotators agree on the label and but disagree on the finer details of what constitutes " "missing major" information. EBR reflects the disagreement.
7.2 Annotator Alignment
Our proposed method transforms human judgment to a 0-100 scale, offering a finer level of granularity compared to the original four-category Likert scale used for coarse labeling. In this section, we investigate how going more fine-grained affects inter-annotator agreement.
How does rescaling affect annotator alignment? To understand how mapping to a fine- grained scale changes annotator correlation, we look the average of pairwise Kendall's (as described in Section 6.2) for INQUISITIVE-RRADD
Table 4 shows that our method is able rescale human judgment a much more fine- grained scale without impacting agreement; that is, our scoring can be more granular without impacting the inter-annotator correlation.
We take closer look at pair of annotators in Figure 5. It shows the distribution of labels (pre-rescaling) and scores (post-rescaling). For this pair there no change in the agreement post-rescaling. Looking at an example of their annotations from the dataset, we see that they agree on the category label but identify different factors in their explanations. Our proposed method assigns different scores to each human judgment and is able to reflect this difference.
How does rescaling impact subjectivity? We emphasize that our goal is not to "smooth out" subjectivity in the an- notation. Rather, taking natural language explanations into account produces more calibrated and nuanced view that pre- serves inherent subjectivity while aligning differences between annotators when they actually agree.
Variation
Original Labels
0.33
EBR without rubric
0.32
EBR (ours)
0.32
Table 4: Average pairwise Kendall's T for INQUISI- TIVE-BROAD. We look at how rescaling influences pairwise correlation with and without the rubric.
Tables 13 and Tables 14 in Appendix G show examples of human judgments, which includes category label, explanation and missing sentences for two different questions. These show different styles of explanations, but with an overlap in missing information.
Related Work
LLMs for evaluation and annotation Kocmi & Federmann (2023) present a scoring mech- anism for machine translation that we take inspiration from in this work. Other work has investigated giving prompts directly to LLMs instead of annotators (Chiang & Lee, 2023; Wang et al., 2023a; Chen et al., 2023). Similarly, He et al. (2024); Gilardi al. (2023); Törnberg

---

Published as a conference paper at COLM 2024
(2023); Zhu et al. (2023) showed that GPT-4 can outperform average crowd workers in several annotation tasks including BoolQ (Clark et al., 2019), Twitter content moderation (Al- izadeh et al., 2022), sentiment analysis (Rosenthal et al., 2017), and more. Wang et al. (2023c) showed that GPT-3.5 can generate near expert-human instructions to align LLMs' behavior to instructions. Previous work has also explored the use of defined criteria for model-based fine-grained error localization and rationale generation (Kim et al., 2024; Jiang et al., 2024). Our work is assumes that some tasks on the frontiers of LLM capabilities will always need human judgments; our procedure aims to augment and improve the capabilities of the group of crowdworkers.
Building consensus among annotators Another method for rescaling annotations is the calibrated sigma method (Weijters et al., 2016). However, this method does not change the ranking of examples, only rescales their scores for each annotator while preserving the ranking. We therefore do not compare to here. Another line of work (Hovy et al., 2013; Gordon et al., 2021) focuses modeling annotators to build better consensus judgments, but does not address aligning pairs of annotators. Sakaguchi & Van Durme (2018) proposed method EASL, that adapts the original annotator labels to probabilistic ones based the labels of all annotators. Finally, Ethayarajh & Jurafsky (2022) present a new protocol for NLG evaluation in which annotators give judgments in the form of probabilities over sets. This work also demonstrates the flaws in Likert judgments and pursues an approach to improving them orthogonal to ours. Consensus building has also been extensively studied in the translation community (Bojar et al., 2016). Graham et al. (2013) showed that using continuous scale of 0-100 improved inter-annotator correlations on translation tasks.
Natural language explanations Natural language explanations (NLE) have been shown to be effective in improving model performance when used as additional features (Wu & Mooney, 2019; Murty et al., 2020; Liang et al., 2020), explaining model decisions (Camburu et al., 2018; Narang et al., 2020; Hase et al., 2020), boosting the performance of in-context learning of LLMs on variety of reasoning tasks (Nye et al., 2021; Wei et al., 2022; Lampinen et al., 2022). Prior work also demonstrated that natural language explanations help us holistically understand annotator perspectives in complex tasks (Ferracane et al., 2021; Goyal et al., 2022a), improve annotation quality (Alonso, 2013) and aid adjudication (Filighera et al., 2022). Our work uses a similar idea but focuses on rescaling human annotation specifically, which is not task addressed in this prior work.
9 Conclusion
In this work, we showed that LLMs can be used to rescale annotator judgments. We feed an annotator's 's label and explanation of that label into GPT-4 to produce rubric-grounded score from 0-100. On a new dataset we collect of doocuenn--grountee by I.MMs, we show that rescaled annotations align well with reference rescaled values produced by expert annotators. Overall annotator correlation does not change, but we show that our rescaling method is able to capture fine-grained nuances of the judgments, teasing apart subjectivity and scale use differences.
questions answered
Ethics Statement
Our work aims to broaden the role of human annotation in LLM development. By leveraging explanations of labels, we hope to use annotation in more nuanced way and enable more effective human feedback. While this work focused on a document-grounded question answering setting, we envision that this can be useful for RLHF settings. Stronger feedback mechanisms like the one our work provides can lead to better aligned LLM systems and enable wider range of (potentially less trained) annotators to steer this new technology.
10

---

Published as a conference paper at COLM 2024
Acknowledgments
This work was partially supported by a grant from Open Philanthropy and NSF CAREER Awards IIS-2145280 and IIS-2145479, Thanks to Keziah Kaylyn Reina for annotating the extension of INQUISITIVE and conducting the worker quality analysis. Thanks to Keziah Kaylyn Reina, Kathryn Kazanas and Karim Villaescusa F. for rescaling 145 instances of human judgment and providing feedback on the proposed task.
References
Meysam Alizadeh, Fabrizio Gilardi, Emma Hoes, K Jonathan Klüser, Mael Kubli, and Nahema Marchal. Content moderation political issue: The twitter discourse around trump's ban. Journal of Quantitative Description: Digital Media, 2, 2022.
Omar Alonso. Implementing crowdsourccigg-aasee relevance experimentation: an industrial perspective. Information retrieval, 16(2):101-120, 2013.
Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, Alexandra Uma, et al. We need to consider disagreement in evaluation. In Proceedings of the 1st workshop on benchmarking: past, present and future, pp. 15-21, 2021.
Ondłej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 2016 conference on machine translation. In Ondfej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Liane Guillou, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Aurélie Névéol, Mariana Neves, Pavel Pecina, Martin Popel, Philipp Koehn, Christof Monz, Matteo Negri, Matt Post, Lucia Specia, Karin Ver- spoor, Jörg Tiedemann, and Marco Turchi (eds.), Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pp. 131-198, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v 11.16663/v1/W16-2001 URL https://aclanthologg wwg/ 2301.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language mod- els are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https: /ttttt:::///oooe http://pooceeennncceeee/ www.innn..nnnnuine ipsrips.ooo cctp:cceppppppeeeeeeeeeeeeeeeeee /200///pppp////1llll 42f644--aaeerrrrraaeeeeaaa
4457c86bbcb449771116bbbacc
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-snli: Natural language inference with natural language explanations. Advances Neural Information Processing Systems, 31, 2018.
Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Tyyer. Booookscore: A systematic exploration of book-length summarization in the era of ilms. ArXiv, abs//310.00785, 2023. URL https://apl. semanticscholar. wwwp/oorps.D:266666.. 263605928.
Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, and Eunsol Choi. Complex claim verification with evidence retrieved in the wild. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3569-3587, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653 //11 /2w24.naaclllong..oo 196. URL https: http://aaaaaaollooooooooooo ogy. org/ 2024. naacl-long..... 196.
11

---

Published as a conference paper at COLM 2024
Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. Exploring the use of large language models for reference-free text quality evaluation: An empirical study. In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi (eds.), Findings of the Association for Computational Linguistics: IJCNI.P- AACL 2023 (Findings), pp- 361-374, Nusa Dua, Bali, November 2023. Association for Computational Linguistics. doi: 10.18653/ /t/////222../2/ htt.///...nniings-iiii....... findings-ijcnlp..22 URL https:// aclanthology. . org/2023. 2023. ponddings-jjnnl....
Cheng-Han Chiang and Hung-yi Lee. Can Large Language Models Be an Alternative to Human Evaluations? In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics, 2023.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no /no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Pp. 2924-2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1 /v/ N19-1300. URL https: / hatphoooollnnoooooooooooooooooo. org/N19- 1300.
A. P. Dawid and A. M. Skene. Maximum Likelihood Estimation of Observer Error-Rates Us- ing the EM Algorithm. Journal of the Royal Statistical Society, Series C (Applied Statistics), 28 (1):20-28, 1979. ISSN 00359254, 14679876. URL http: /twt:://www..e http://www..ttrrrr org/ htt///aabeee4668000
Yao Dou, Maxwell Forbes, Rik Kencel-Kedziorski, Noah A. Smith, and Yejin Choi. Is GPT-3 text indistinguishable from human text? scarecrow: framework for scrutinizing machine text. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7250-7274, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: URL https::/aclantthll.og
10.18653/ wwwp///22acllooon....
org acl-long. 501.
Kawin Ethayarajh and Dan Jurafsky. The authenticity gap in human evaluation. In Pro- ceedings of the 2022 Conference on Empirical Melhods in Natural Language Processing, pp- 6056-6070 Abu Dhabi, United Arab Emirates, December 2022. Association for Computa- tional Linguistics. URL https:tt/cclaatttiit http://aacannhoooggoooooo org/ 2022. mompp-main... maiin.46.. 406.
Elisa Ferracane, Greg Durrett, Junyi Jessy Li, and Katrin Erk. Did they answer? subjective acts and intents in conversational discourse. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1626-1644, 2021.
Anna Filighera, Siddharth Parihar, Tim Steuer, Tobias Meuser, and Sebastian Ochs. Your answer is incorrect... would you like to know why? introducing bilingual short an- swer feedback dataset. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8577-8591, 2022.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. ChatGPT outperforms croke-workers for text-annotation tasks. ar Xiv preprint ar arXio:2303.15055 2023.
Mitchell L. Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S. Bernstein. The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI 211 New York, NY, USA, 2021. Association for Computing URL https://don org/
Machinery. ISBN 4781450880966. doi: 10.1145/3411764.445555 10. No.1145/341178 3445423.
Tanya Goyal, Junyi Jessy I.I, and Greg Durrett. News Summarization and Evaluation in the Era of GPT-3. Xiv, 2022a. URL
12355.
Tanya G Jessy Li, and Greg Durrett. SNaC: Coherence error detection for narrative summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 444-463, Abu Dhabi, United Arab Emirates, December 2022b. Associ- ation for Computational Linguistics. URL https::/accaatthe http://aaaaaanaaaa.aaaaa .rr.looo..oog/// 29.
12

---

Published as a conference paper at COLM 2024
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. Continuous measure- ment scales in human evaluation of machine translation. In Antonio Pareja-Lora, Maria Liakata, and Stefanie Dipper (eds.), Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pp. 33-41, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL https: fttt::://aclaathhllee
Peter Hase, Shiyue Zhang, Harry Xie, and Mohit Bansal. Leakage-adjusted simulata- bility: Can models generate non-trivial explanations of their behavior in natural lan- guage? In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics; EMNLP 2020, pp. 4351-4367, Online, November 2020. Associ- ation for Computational Linguistics. doi: 10.18653/vV 2211/220..fnman wen.diiinnggeenll..... .900. URL www.:ll/oooooo///222/2
mondings-emnl.... 390.
Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. AnnoLLM: Making large lan- guage models to be better crowdsourced annotators. In Yi Yang, Aida Davani, Avi Sil, and Anoop Kumar (eds.), Proceedings of the 2024 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technolo- gies (Volume 6: Industry Track), Pp. 165-190, Mexico City, Mexico, June 2024. Associ- ation for Computational Linguistics. doi: 10.18653/v /1653vv/// / 2////2222222 maacll-nddsstry155 URL www.:/gg.....//// org/2024. naanl innel-iidustry.111
Dirk Hovy, Taylor Birg-Kirkppatrick Ashish Vaswani, and Eduard Hovy. Learning whom to trust with MACE. In Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff (eds.), Proceedings of the 2013 Conference of the North American Chapter of the Association for Computa- fional Linguistics: Human Language Technologies, pp. 1120-1130, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL https: / http:///aaaaaholoyyoooo org/N13-1132.
Dongfu liang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. TIGERScore: Towards building explainable metric for all text generation tasks. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL htttt::://operrveeeei www. id=EE1CBKCOSZ.
Maurice George Kendall. Rank correlation methods. 1948.
Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. Evallm: Interactive evaluation of large language model prompts on user-defined criteria. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703300. doi: 10.1145/3613904. 3642216. URL https: org/10.
10. 1145/361 3904 3642216.
Wei-Jen Ko, Te-yuan Chen, Yiyan Huang, Greg Durrett, and Junyi Jessy Li. Inquisitive question generation for high level text comprehension. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6544-6555, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1 1tt6/////1122200m///maaaai //acl antolooy . org/2020. emnlp-main. 530.
URL https:
Tom Koemi and Christian Federmann. Large language models are state-of-the-art evalua- tors of translation quality. In Mary Nurminen, Judith Brenner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Vanmassen- hove, Sergi Alvarez Vidal, Nora Aranberri, Mara Nunziatini, Carla Parra Escartín, Mikel Forcada, Maja Popovic, Carolina Scarton, and Helena Moniz (eds.), Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pp. 193-203, Tampere, Finland, June 2023. European Association for Machine Translation. URL 1.
http:://aclanthooogy.orr////
Andrew Lampinen, Ishita Dasgupta,Stephanic Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. Can language models learn from explanations in context? In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 537-563, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https: /tttls:///llaaiiea /ttpaaaaaaaanaaac.aaaaaa hwwphhllgg.orgg////// org/2022. findings-emni em..p.
13

---

Published as a conference paper at COLM 2024
Weixin Liang, James Zou, and Zhou Yu. ALICE: Active learning with contrastive natu- ral language explanations. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3380-4391 Online, November 2020. Association for Computational Lin- guistics. doi: 10.18653/v1 wwtp/200emmllmmaaai..... emnlp-main. 355.
URL https:: https://aclanthoooogy org/2020.
Nelson F. Liu, Tianyi Zhang, and Percy Liang. Evaluating Verifiability in Generative Search Engines. Xiv eprint Siti::2304008844 2023.
Shikhar Murty, Pang Wei Koh, and Percy Liang. ExpBERT:
Representation engineering with natural language explanations. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Amnual Meeting of the Association for Computationnl Linguistics, pp. 2113-2113, Online, July 2020. Association for Computational Linguistics. doi: 10.18653 /ttp//////2...mmmm.ama.... acl-main.190. URL https: aellanthooogy wrw.2020..... acl-main. 190.
Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. WT5?! Training Text-to-Text Models to Explain their Predictions. arv preprint ar X14:20004114466, 2020.
An Nguyen, Matthew Halpern, Byron Wallace, and Matthew Lease. Probabilistic modeling for crowdsourcing partially-subjective ratings. Proceedings the AAAI Conference Human Computation and Crowdsourcing, 4(1):149-158, Sep. 2016. doi: 10.1609/ 00.1609/ccomp.v4ij hcomp.v4i1.13274.
URL httpps:/ojss aaai wwtg/rdee....oooooooooooo..o..we hhtp://cooF//arrilleeeeeeeeeeeee
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint ar arXiv:2112.00114 2021.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv, 2022. URL
2203. 02155.
Barbara Plank. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 10671-10683 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://acclanthoo.g. org//022.. emnlp-main. 731.
Sara Rosenthal, Noura Farra, and Preslav Nakov. SemEval-2017 task 4: Sentiment analysis in Twitter. In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Moham- mad, Daniel Cer, and David Jurgens (eds.), Proceedings of the 11th International Work- shop on Semantic Evaluation (SemEval-2017), pp. 502-518, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/SS1-2288. URL https://aclantholo.g www/oorg/-////
/tlp://ntooooggooo//////// 2088.
Keisuke Sakaguchi and Benjamin Van Durme. Efficient online scalar annotation with bounded support. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 208-218, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: URL https: /attps://acleen http://////claattlogog...oo -
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiuingg models for assisting human evaluators. arxiv eprint (rriz:2006008800 2022.
14

---

Published as a conference paper at COLM 2024
Padhraic Smyth, Usama Fayyad, Michael Burl, Pietro Perona, and Pierre Baldi. Inferring Ground Truth from Subjective Labelling of Venus Images. In Proceedings of the 7th In- ternational Conference Neural Information Processing Systems, NIPS'94, pp. 1085-1092, Cambridge, MA, USA, 1994. MIT Press.
Petter Tömberg. ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning. ar arXiv preprint ar Xiio:2044.06588, 2023.
Alexandra Uma, Tommaso Fornaciari, Anca Dumitrache, Tristan Miller, Jon Chamberlain, Barbara Plank, Edwin Simpson, and Massimo Poesio. Semeval-2021 task 12: Learning with disagreements. In Proceedings of the 15th International Workshop Semantic Evaluation (SemEoul-2021), Pp. 338-347, 2021.
Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel R. Bowman. SQuALITY: Building a long-document summarization dataset the hard way. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1139-1156, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v /v1 2022.emnlp-main.755 URL
hwtp://hoooooooooo/22222 ewn.22...mmmmmmmmmmmmm emnlp-main main. 75.
Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is ChatGPT good NLG evaluator? a preliminary study. In Yue Dong, Wen Xiao, Lu Wang, Fei Liu, and Giuseppe Carenini (eds.), Proceedings of the 4th New Frontiers in Summarization Workshop, pp. 1-11, Singapore, December 2023a. URL
Association for Computational Linguistics. doi: 10.18653/ve / newsum-1.1.
https://aclanthollgg..orr///
Lucy Lu Wang, Yulia Otmakhova, Jay Ye Thinh Hung Truong, Bailey Kuehl, Erin Bransom, and Byron Wallace. Automated metrics for medical multi-document sunma- rization disagree with human evaluations. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp. 9871-9889, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10.18653 2t0/////1222233clloggg https://aclanthol..g org/2023. acl -logg. 549.
URL
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484-13508, Toronto, Canada, July 2023c. Association for Computational Linguistics. doi: 10.1 /v 201./2222clllooggggcc 754. URL httpttts: /atpsss//cclaaahooooooo 2023. acl-long. 754.
10.18653/w1/2223.aae
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
Bert Weijters, Hans Baumgartner, and Maggie Geuens. The calibrated sigma method: An efficient remedy for between-group differences in response category use on likert scales. International Journal of Research in Marketing, 33(4):944-96) 2016.
Sarah Wiegreffe and Ana Marasovic. Teach me to explain: A review of datasets for
explainable natural language processing. In Thirty-fifth Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https:
Jialin Wu and Raymond Mooney. Faithful multimodal explanation for visual question answering. In Tal Linzen, Grzegorz Chrupata, Yonatan Belinkov, and Dieuwke Hupkes (eds.), Proceedings of the 2019 ACL Workshop BlackboxNLP. Analyzing and Interpreting Neural Networks for NLP, pp. 103-112, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 11188633/////1
URL https: haw.aa/hooo.c...cooooooooooo org/ wrgg///--48111
15

---

Published as a conference paper at COLM 2024
Fangyuan Xu, Yixiao Song, Mohit lyyer, and Eunsol Choi. A critical evaluation of eval- uations for long-form question answering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Anmual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 2225-3245, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ 221/22222clloooo 181. URL https://aclanthoolggs org/2023 acl-long. 181.
Hongli Zhan, Desmond Ong, and Junyi Jessy Li. Evaluating subjective cognitive ap- praisals of emotions from large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14418-14446, Singapore, December 2023. Association for Computational Linguis- URL https://aalanthologg. org//023..
tics. doi: 10.18653
Sindings-emnJl 962.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tat- sunori B. Hashimoto. Benchmarking large language models for news summarization. Xrviv eprint arxio:2301.13848, 2023.
Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, and Gareth Tyson. Can ChatGPT Reproduce Human-Generated Labels? Study of Social Computing Tasks. arxiv preprint ar arXiv:2304.00144 2023.
A Data Collection
A.1 Question Dataset
Table 5 shows statistics about INQUISITIVE-BRAAD.
Property
INQUISITIVE EXTENDED SET Overall
num articles avg sents article
58
20
78
A.2 Dataset Examples
37.1
34.6
36.2
num questions avg toks toks/quese question num of annotations
358
231
589
Tables 7 and show examples of articles, questions, system re- sponses along with human notations (ratings and explana- tions) for two articles from the INQUISITIVE-BROAD dataset.
8.3
10.8
9.2
7058
5592
12650
Table 5: Dataset properties for two splits (questions taken from INQUISITIVE and our extended set) in INQUISITIE-BROAD.
A.3 Answer Collection
We prompt three LLMs and also ask two human experts (one of the co-authors and non-author lin- guistic student) to answer ques- tions from INQUISITIVE-RRAAT The systems along with their prop- erties are mentioned in Table 6. EX- PERT human in the table refers to linguistic undergraduate students who have worked on a series of computational linguistics annota- tion projects for 1-2 years. One stu- dent is now a lead data annotator at a small NLP company. These annotators are experienced at fol- lowing annotation guidelines and producing high-quality judgments. Figure 8.
Avg Answer Length (tokens)
Dataset Answered
QA System
Property
Text-Davinci (davinci)
not instruct tuned
INQUISITIVE BROAD
27.65
Text-Davinci-003 (gpt-3.5-turbo)
instruct tuned
INQUISITIVE
48.61
BROAD
GPT-4 (gpt-4)
instruct tuned
INQUISITIVE
48.54
+ RRHF
BROAD
INQUISITIVE
EXPERT human
50.40
EXTENDED
Table 6: Systems used to answer INQUISTIVE-BROAD Sample prompts for each of the systems is given in
16

---

Published as a conference paper at COLM 2024
Annotation Example-1
Article: CAIRO A night of largely peaceful protests ended early Monday in bloody clash between Muslim Brotherhood supporters and Egyptian soldiers, according to the Brotherhood and Egyptian media. Muslim Brotherhood officials, who are supporting ousted Islamist President Mohamed Morsi, said security forces raided their encampment outside the Republican Guard compound with tear gas and gunfire about a.m. Supporters of Morsi have camped there for days demanding the release of the former leader, who has been under arrest since military coup last week. Casualty figures were not immediately available, but Muslim Brotherhood officials said many people were killed and hundreds wounded. They called upon their supporters to donate blood and rush to the Nasr district of Cairo to assist the victims, 'Bloodbath!" tweeted Muslim Brotherhood spokesman Gehad Haddad. Egyptian television showed chaotic scenes of bloodied, unconscious protesters lying in makeshift triage facilities. They also showed images of more than dozen bodies lying under sheets and Egyptian flags. In an interview with Al-Jazeera television, Haddad said Egypt had returned to gell-fleeged police state in just five days'. Hours earlier, Egypt's new interim leadership had narrowed in on compromise candidate to serve as the next prime minister. The state-run Ahram website and other Egyptian media reported that the new front-runner is Ziad Bahaa El-Din, founding member of the Egyptian Social Democratic Party. El-Din is an attorney and former parliament member who previously served as an economic adviser, financial regulator and head of Egypt's General Authority for Investment under the government of deposed President Hosni Mubarak. El-Din is seen as a less divisive choice than secular opposition leader Mohamed BlBaradei, whose nomination was abruptly blocked day earlier. Under the latest scenario, ElBaradei would become vice president, Ahram reported. El-Din and interim government officials cautioned that no final decision has been made. The debate over the prime minister marked the first standoff within the unlikely coalition of divergent opposition parties that came together last week to support the military's 's toppling of Morsi The military and interim President Adly Mahmoud Mansour had hoped the swift appointment of prime minister would send a message of stability and confidence to both Egyptians and the international community. Instead, the fumbling raised fresh doubts about Egypt's political future. Though viewed as liberal, El-Din is seen as someone who will focus on improving Egypt's sagging economy and reforming corrupt government institutions. He has been outspoken police critic and blamed Morsi for failing adopt more inclusive approach during his year in office. Liberal youth activists, led by the Rebel movement that helped organize the mass protests that ousted Morsi, pushed hard for ElBaradei, a Nobel Peace Prize laureate and former head of the International Atomic Energy Agency. Mansour seemed set to appoint ElBaradei, and swearing-in was scheduled. But late Saturday the government abruptly backtracked after the ultra-religious Nour Party rejected him too liberal and divisive, threatening to quit the interim coalition if ElBaradei got the job. On Sunday, throngs of Egyptians from both sides had voiced their views in mass protests largely peaceful until the early morning violence. In the Nasr neighborhood, speakers denounced last week's coup as illegitimate and called on mid-level army officers to revolt against top generals who pushed Morsi out. 'We will stay here for as long as it takes and peacefully pressure the army reinstate Morsi, said Gamal Ragab, 41. And Tahrir Square, the number of anti-Morsi demonstrators surged into the tens of thousands early Sunday evening. 'We want to build the democracy that we dreamed of in 2011, not a religious state where religion is used to divide the whole population and exclude anyone who doesn't 't share the Brotherhood's ideology'' said Samira Kamel, 21, Cairo student. 'We want to show the world that this a popular revolution, not a coup'.
Question: Does he have lot of supporters?
Davinci-003: Yes, Muslim Brotherhood officials said security forces raided their encampment outside the Republican Guard compound with tear gas and gunfire about 4 a.m. and there were tens of thousands of anti-Morsi demonstrators in Tahrir Square.
Annotator 1: Label: Missing All, Explanation: The answer tells of anti-Morsi demonstrators not pro-Morsi. There is no information on the numbers of Morsi supporters
Annotator Label: Missing Major, Explanation: The answer focuses on the anti-Morsi side, though the text only mentions "throngs" of protestors "from both sides"
Table 7: Example article, questions, system responses and annotations from INQUISITIVE- BROAD
17

---

Published as a conference paper at COLM 2024
Annotation Example-2
Article: Touting her billionaire family' 's legacy of populism and massive election victories, Thailand Paetongtam Shinawatra is emerging as the candidate to beat in coming polls betting that nostalgia can win millions of working class votes. Paetongtarn, 36 is campaigning hard in the vote - rich rural strongholds of the Shinawatra family's Pheu Thai political juggernaut hoping to reignite the kind of fervour that swept father Thaksin and aunt Yingluck to power in unprecedented landslides. Political neophyte Paetongtarn promising Pheu Thai will complete unfinished business from three stints in office since 2001, all of which were cut short by court rulings and military coups that it says were orchestrated by Thailand 's conservative establishment. "We managed to fix everything in the first year but then four years later we were ousted by coup, so there are things that we have not achieved, " Paetongtarn said in her first formal interview with foreign media ahead of the election expected in May. " So we go on each stage to tell people how our policies can change their lives. And only through stable politics can people's 's lives change in sustainable manner, she said, while campaigning in the northeast. Thaksin and Yingluck were toppled by the army in 2006 and 2014, respectively , despite overseeing much economic growth. Both live in self- - imposed exile to avoid prison convictions their allies say were designed to prevent their political comebacks. The baton has passed to Paetongtarn, Thaksin youngest daughter, who is using the same playbook in offering minimum wage increases, utilities subsidies, and long promised high speed rail systems and infrastructure to manage floods and droughts Pheu Thai 's slogan is Think Big, Act Smart taking aim at incremental reforms by the military- backed governments of Prime Minister Prayuth Chanocha since he seized power in 2014. " The picture has be big and we must be able address long standing problems that festered. These must be completely dealt with, " Paetongtarn said. Though yet to be named as Pheu Thai's 's prime ministerial candidate, Paetongtarn is far ahead in the opinion polls, with twice the support of Prayuth. Pheu Thai is expected to win most votes, but could struggle to lead government given the military 's influence over an appointed Senate, which together with the elected lower house chooses the prime minister. Paetongtarn said she consults regularly and remains close with her father, , who lives mainly in Dubai. His chief worry was her campaigning while nearly seven months pregnant " But I'm OK, she said. " "This is my second pregnancy: I am aware of myself. I won't go too hard. ." Desppte their electoral popularity the Shinawatras are loathed in Thailand much as they are loved. They have long been accused by opponents of cronyism to enrich business friends and of buying off the poor with wasteful populist policies. The Shinawatras deny the charges. Thailand 's election is shaping up to be another grudge match between warring elites in Southeast Asia 's second biggest economy. Paetongtarn said she remains concerned about the impact of the country's 's intractable power struggle involving her family including coups, which she said makes Thailand go backwards " " It also makes the world see our country in a different light. They don't 't want to trade with us. It reduces the opportunities for everyone, " she said. Our country has been frozen for so long. So a coup should not take place again. The country must progress and people deserve to have better livelihoods." !"
,he said,
Question: How are foreign powers reacting to the election?
Davinci-003: Foreign powers have been cautious in their reactions to the upcoming election in Thailand. They are concerned about the country's s intractable power struggle and the potential for further coups, which could have a negative impact on trade and other opportunities. Annotator 1: Label: Complete, Explanation: The answer is correct and complete Annotator 2: Label: Missing Minor, Explanation: The answer fits with the overall theme of the article, but it attributes Paetongtarn Shinawatra's 's concerns to foreign investors
Table 8: Example article, questions, system responses and annotations from INQUISITIVE- BROAD
B Human Evaluation of QA
B.1 Annotation Task
We conducted four qualification rounds involving a group of 20 trusted and high-quality Turkers. These Turkers were required to meet specific criteria, including being located in the US, having a HIT approval rate of 98% or higher, and having completed more than 5000 approved HITs. They were assigned the task of annotating three documents, each containing five amustionnansswe pairs. The label distribution varied across the documents. To ensure the Turkers understood and performed the task correctly, we manually reviewed the annotations and explanations. This rigorous review process aimed to confirm the Turkers' comprehension and execution of the task.
18

---

Published as a conference paper at COLM 2024
Thank parkiquiing this dudy!
syste cridguing
soing, No personal idanifling information will be sollected from responses. You ard fro the study time closing your browser
meniored document ta tnowlodgg tron saowhore You nead dicido whether the maciin''s reapetaa cerrect and complete.
about
ylos
using information
Your
Read the docarend
Read the quertoen and aw genersted iespotie
ls machino respons conout madtene resporce conout the elommddor iiis from the dooument and relovort to answering the cuesion In this step allo sakl you to plape HDW much information the repporee mising Bio oo miiiiin?
You aloo nead to provile reasore for resporso Llowe quassiine.
Please role, iéfomationn neaded to answerr the quantton not the document the moctine right lovnlity this oirpu le ebatee Twe d011 snow". I.ll comncity icentilee this plaase mark incericaly statos
lo not document
incomnott
respoosoo
Figure 6: Screenshot of the task description visible to the annotators
Figure Screenshot of the annotation interface visible to the annotators
Following the qualification rounds, we selected 8 highly qualified Turkers. Given the substantial scale of our evaluation study, we actively monitored the feedback section of the HIT. If any ambiguities or issues arose, the Turkers had the option to contact us. Additionally, we promptly reached out the Turkers if they provided any general feedback on the task, aiming to address and clarify any concerns.
In order maintain quality of annotations, we assigned only one document per HIT. On average, each document had 4-5 questions. We compensated the Turkers $2 per HIT and gave them 20% bonus after each batch. Each Turker got 2 hours to finish a HIT they accepted. We released batches of only 20 HITs every 12-24 hours. This aimed to prevent any decline in annotation quality that could potentially rise from prolonged or excessive workload.
Figure shows screenshot the annotation interface visible to the annotators.
B.2 Quality of the Explanations
In order assess the quality of expla- nations we had human expert (an in- dependent judge not involved in this work) look at sizable sample of ex- planations for each annotator in our dataset and mark if they find the planations to be plausible. Specifically, the expert judge had access to the arti-
Worker ID
0
2
3
6
7
Quality agreement 87
94
81
93
94
74
Table 9: Human evaluation of explanations to gauge worker quality.
19

---

Published as a conference paper at COLM 2024
cle and the QA instance, well as the natural language explanation of the annotator; the judge studies whether the explanations reflect sound underlying reasoning of the answer.
The expert evaluated 630 expla- nations in total. For each annota- tor, we sampled roughly 80 expla- nations, equally split across each label. Overall, our expert judge agreed with 87% of the explana- tions finding the granularity of the missing information to be plausi- ble. Table presents percentage of explanations with which our judge agreed for each annotator.
missing all
missing major
missing minor
Worker ID
complete
0
5%
8%
18%
69%
1
22%
10%
25%
43%
32%
5%
15%
48%
Table 10: Distribution of labels for three annotators for the INQUISITIVE question set with davinci-003 answers. Worker 0 is more lenient whereas and 6 are stricter.
Some sources of disagreement in the remaining 13% were 1) instead of critiquing the answer, annotators wrote the "correct" answers instead; 2) politically contested topics that led to the reasoning in explanations not being supported by the document; 3) differences (between the judge and the annotator) in reading of the question and differences in inference from what was provided in the document.
B.3 Variation in Labels
Table 10 shows distribution of labels for three annotators. We see variations in annotator behavior, where annotator 0 is a little more lenient compared to annotator 1 and 6.
C Rubric Creation
An important component of our proposed rescaling prompt in Section 5 is the aspect specific scoring rubric which is designed post-annotation. To create the proposed rubric, authors of this paper examined 20 human annotations, which included explanations and Likert labels. They assigned point deductions by taking into account factors that surfaced only with annotator's 's explanations after their annotation.
For instance, explanations under the label missing minor, frequently highlighted the absence of certain names in the response or the omission of multiple small details. Likewise, within the label category labeled as missing major, explanations commonly emphasized the role of the absent information in completing the response. Some explanations explicitly stated that "the response is lacking half the information" along with references the specific sentence numbers that were missing. This information at granular level provided the authors with valuable insights into the variations in errors within each label category, which further helped refine point deductions for the final rubric.
D Prompting Details
D.1 Prompt Variants for rescaling human judgment
The following is a variation of the prompt without the scoring rubric:
The main goal of your task is to score a machine generated response to question. Scoring is on the "completeness" attribute. complete answer will have all relevant information from the article required to answer the question and an incomplete answer won't. However, you are not directly scoring the machine response, but instead using given feedback and amount of missing information. The details are given below.
Article on which the question was asked: 'article'
Based on the above article, the following question was asked: 'question'
2The judge did not annotators' Likert ratings on the QA instances, since we believed that would shift their focus the subjective question of whether the explanation justifies the rating.
20

---

Published as a conference paper at COLM 2024
GPT3-davinel
attille New YeeCNNN- Watching in persos will be cheaper ilan last yeur' gamne will stil cost you thassank of dollass StubHul
average price cuslemers
are paying fora ticket Thirsday was 56,500 The chrapest seats to Sunday's a match up betwoen the Philadeiphis Eagles and Kansss City Chiefs are sel ling at about $3,200 spicce Overall prices for the hig gime (llmdssk, Arizona are treninng down with pricee dropping each day. In fact, Stufiiie said that ticket prices luvee drepped more than 10% sincs teams were decided on Februury 12. Attnding this yourss 's game is costig fans subsuttialyy loss than last year's match up between the Cincinaat Dengals and Los Angeles Rams at LA's Saf' Stadium That's 's's in be expecial, since the some loan was playing in thei own city and the country' is securd-largee mutequoiian ceen. Tichets were averaging secally $10,000 each, Subliiis. SeatCleck another ticket selling websit,, shows ticket prices for Sunday's is game slowly declining and were avenging about 5,000 on Thundhy. The most expensive ticket its website costs $30,200 and is located near the feed; the cheapest is $4,200 At everage of 5445 per night, however, hotel prices the Phoenix area will be relatvelly exgensive, making the second highest level for a Super Bowl week, acconding to STR, a hespitality analytics firm that tracks prices The most expensive Super Dowl was 2016 in Sun Franciago when
averageal $451 sight In sddii ion to the Super Bowl, Bellly is also leoding PCA eveut tist Eattures goll's bigges playens, inclaling Rory around 160 per
Mellrny and lerdan Spieth. Those two events the weekend puis local hotel occupancy to 94%, the firm said The normal for hotel STR
After rouding semencer/ew YodKCNN
Watching Super Bowl LVU person will be cheaper than
game. will still you thousands of dollars'', a reader isked
following question.
Q:Why the Super Bowl cheaper this year?
GPT3-dawiini--033
vetille New YorkCNN Watching VVI in person will chesper than last year's 's game, but will stil osst thumandds of dollzes SubHlib said that the price cxstormer Philadeighia Liggls Kansas City Chiefs
paving ticket Thursday 56, 800. cheapest tending down with prizes dropping day. year's match up between the Cincinnati Bengals and Angeles
Sunday's
about $3.200
Oveual for game Glandal,
StuHHub uid that licket
lave droppsi
10% since
decided Febraary Attending year's game coetin fans subvaanially less countyys second-langeet metropuliian That's
expected,
team was playnng
own city
avorningg nerrly
were averzging about 56,500 on Thunday, The most locatal noir the field; the chappet is 54, $4,200 At an average of 5445 per night, howevet, lotel prices the Phoenis snca be relstively expensivee making it the second-hin tranhd-tighhtt level for a Super Bowl week, according to STR, hospitalit analytics firm that tracss prices. The most expensive Super Bowl was in 2016 in San Francisco when noms averaged 5451 per night. In addition to the Super Bool, the city is also hosting a PGA event thet Seatures gold's bigees players including Rocy Melizoy and Jordan Spieth. Those two events on the same weekend puts the local hotel occupancy rase to 94%, the firm said. The nornal rato for a hovel nom is around $160 per night, STR waid.
$10,000 each, Stabllub, SeatGeek another selling websitt, shows ticket proces for Sunday's game sowly decliting
expersive ticket on welsiic cass $30,000
After reading sentenoe: Overall prices for the big game in Glendale Arieon are trending down with prices drupping eoch day., reader asked the following question.
H.How do these trends compare other Super Bowl sales is previnus years?
A:
GPT-4
While reading the aniele below:
New YorKONN Watching LVI person will be chesper than last
but it will still cost thomands of dolloms Snahllub said that the price costomers Sunday's 's up the Philadelphia Eagles Karsas Chicß selling about splece, Overall prisee have duppal
paying for Thursday $5,000 The cheupest
for gume Gloudale
mecsllngg
with prius drppingg day,
SuisHuh
were decided febbuary Attending year's game costing substaniially Tha''s to expected,
last year's match up
Cincintori Bergils
Angeles Rams
home leam was playing
own city
country' second- wecond-largest metroolltann arca. Tickets were averaging nearly 10,000 were evecaging abou: $6,500 Thursda, most expensive
each, Stubilab. Scattiokk anccher diske selling websie, shows ticket prices Sunday's gast slowly deciining avengs $445 per night, however botel prices the Proenis srce will relstiveyy
tickst its webvite codss $10 and located near the ficki; chesposi $4,200, expensire, making it the second highess level for Super Dowl week, accorting STR, a hospitality analytics firm that tmiks prices. The most expensive Super Dowl was in 2016 in San Prancisco when rooms averaped $451 per night. addttion to the Super Bowl, the city also hosting a PGA event that festrres polf's bigges players inchiling Rory Mallrey and Jordan Spuech. These two evends on the same weekend pats the local hotel occupancy cate to 94% the fimm saad, The sommal miee for hotel room is around 5160 per night, STR sard.
had question at sentence:'New YorkCNN
Warching Super Bowl LVII in person will be cheoper than last year's game, but it will still cost you thousands of dollars..
Why the Super Bool cheaper this year?
Figure& 8: Examples of prompts for each QA system used to answer the INQUISITVE-BRRAAD Dataset
A machine responded with the following answer: 'answer'
Feedback given to the machine response: 'feedback' Level of the missing information: 'label definition' On a scale of 0-100 how will you score machine response using the feedback and level of missing information stated above? Give number.
It follows the same structure as the rescaling prompt in Figure 3 without the rubric.
D.2 Cost of prompting
All models used in this work are accessed through OpenAl 3PIs The total cost of prompt- ing LLMs in Table 6 for getting answers to INQUISITIVEBRRAAD is -330. All three models were prompted with the same set of articles and questions.
The total cost of prompting GPT-4 with EBR and the variations is $500.
E Average EBR scores per label
Table 11 shows the average score per label across all annotators using EBR for INQUISITIVE- BROAD, Although we do not impose any constraints regarding a score range per label when
Pricing information for the models used can be found here.
This includes multiple runs and prompt tuning on small set of examples
21

---

Published as a conference paper at COLM 2024
applying this method to each example, the average of all scores for label naturally aligns with the overall label order.
F Impact of Rescaling on Annotator Alignment
Label
Average Score
Figure 5 looks closely at a pair of annotators and their judgments. We see changes in label distribution pre rescaling and score distribution post rescaling.
missing
missing major
50.0
missing minor
79.9
complete
100.0
G Qualitative Impact of Rescaling
Table 11:
Average score of rescaled annota-
tions for each original label across all annota- tors under the INQUISITVE-BRRAAD dataset.
NLEs with same scores Table 12 shows ex- amples of explanations which got assigned the same score post-rescaling. We see some patterns in human judgment for these explana- tions. For example, missing minor explanations generally use phrases like 'somewhat correct' "answered correctly, but missed some relevant information' where as missing major expla- nations tend to jump right into the information that was missed, along with mentioning sentences that were missing.
NLEs for the same question Table 13 and Table 14 show human judgment well as EBR and reference scores for two different questions.
H Stability of rescale prompting
To check the stability of our proposed method, we re-run rescaling four times on the 145 instances that also have expert rescaled scores. Table H shows the overall Kendall's and MAE for the different runs.
I Dataset Release
As mentioned in Section 3, INQUISTIVE-
BROAD consists two subsets questions. The first set are questions collected in the IN- QUISITIVE dataset. Corresponding articles under this dataset are built prior work which sourced from Newsela and LDC. Newsela articles can be obtained from their website and LDC has standard purchasing guidelines. The second subset of questions is based on more recent events (March 2023). Due copyright constraints, we are unable to disclose the processed article text. How- ever, we will provide links to the original articles along with the code for processing them. Our question dataset will also be made publicly available.
Avg Score
MAE
Run
60.45
0.83*
8.18
Run 2
60.31
0,83t
8.11
Run
60.48
0.82
Run 60.10
0.83
8.16
Table 15: Results of running rescaling four times the 145 expert rescaled instances. We look at Kendall's T and MAE over four runs for the entire dataset. significant value with < 0.05
We will be releasing all the data collected (questions, machine responses and human judg- ment) under the CC BY-NC license.
J Limitations
The effectiveness our work hinges on the quality of the provided explanations and labels. Inconsistencies, such as the selection of incorrect labels by annotators (i.e., mistakes beyond
22

---

Published as a conference paper at COLM 2024
Label: Missing Minor, EBR: 85
The response is somewhat correct, but misses some relevant info Missing Sentences:
Explanation:
2 Explanation: The machine response answered the question correctly but missed some relevant information. would be useful include the fact that Argentinian passport holders enter 171 countries visa-free since machine response mentions Russians can enter only countries visa-free. Including that fact in the machine response without additional context does would not make much sense.
Missing Sentences: 10
Explanation: The response did not mention the street Missing Sentences:
per sentence 5.
Label: Missing Minor, EBR:
Explanation: The machine response answered the question correctly but had its last sentence in its response cut off and missed relevant information. The response should include that the fake documents were issued to allow the women to settle in Argentina.
Missing Sentences: 23,
2 Explanation: The article doesn't provide full answer the question but contains some relevant detail.
Missing Sentences: 12, 29
3 Explanation: Info from doc. sentence is used to appropriately make the first sentence of the answer. The second sentence draws from doc. sentence 12. The final answer sentence relevant as the question is not asking how they are able travel there. Info from sentence which discusses fleeing the war and getting access to better health care.
Missing Sentences: 17
Label: Missing Major, EBR: 50
Explanation: the machine response states what the helicopters were doing but fails to properly answer why they were operating (to combat rebel forces as indicated in sentences 15 and 35). Missing Sentences: 15, 35
2 Explanation: The machine response missed the question and provided an irrelevant response by restating the information the source sentence. The article does not state exactly when horticulture was a priority for Americans but does mention an ongoing decline from peak membership in horticultural association from the 1960s.
Missing Sentences: 24,
3 Explanation: The response is partially correct and failed to mention the domestic factors as per sentences 4,7,8,13,14.
Missing Sentences: 4, 4,7, 13, 14
Label: Missing Major, EBR: 40
Explanation: the machine response missed the question and provided mostly irrelevant response. line 27 explains that david hockney is 'one of the living masters of oil painting' and that was not present the response explaining why hockey britain's 's most celebrated living artist. additionally, the machine response conflates information presented the article. the variety of mediums was taken from line 6 but hockney did not create all of those works as explained on line 32.
2 Explanation: The text does not provide specific answer, but lot of detail could have been included in attempt address the question.
Missing Sentences: 17, 18, 30
3 Explanation: the answer makes use of sentence 1, but misses the specific people named in 3, 6, and 13:sergei grigoryants, vladimir oivin, marina shemakhanskaya.
Table 12: We give examples of explanations that get rescaled to the same score. These explanations are for different questions and by different annotators. All the explanations are fine-grained, pointing sentences that were missing from the machine response, along with the level of severity of this missing information.
subjectivity), may pose challenges that are difficult to overcome. Additionally, further investigation needed to determine the specific types of explanations that should be sought from annotators to facilitate more faithful rescaling.
Our proposed technique relies on a carefully curated scoring rubric. As outlined in Section 5, this process involves post-annotation analysis of the human judgment, which can be
23

---

Published as a conference paper at COLM 2024
Why is inflation expected to remain elevated?
Label: missing major Explanation: The response is partially correct missed other factors per sentences 8,12,13 and14.
Missing Sentences: 8, 12, 13, 14
EBR: 40
Reference 35
2 Label: missing minor
Explanation: This comprehensive answer, but cut off before it could finish supplying key information
Missing Sentences: 8, 10
EBR: 80
Reference score: 80
3
Label: missing major
Explanation: The first sentence in the response incorrect per sentences and 2. While the rest of the response correct, it missed the other factors mentioned in sentences 8,12,13 and Missing Sentences: 1, 2, 8, 12, 13
EBR: 30
Reference score: 26.67
4 Label: missing major
Explanation: The first two answer sentences not relevant as they stating things that are inflated in price rather than why they are that way. The rest of the answer lists accurate reasons but misses global energy prices being higher (sentence 4), strong housing demand and tight entitlement quotas (sentence 8), and persistent manpower shortages (14).
Missing Sentences: 4, 8, 14
EBR: 40
Reference score:
Label: missing major
Explanation: The response is partially correct and failed to mention the domestic factors as per sentences 4,7,8,13,14.
Missing Sentences: 4, 8, 13
EBR: 50
Reference score: 23.33
Table 13: Human judgment along with EBR and reference scores for the question Why is inflation expected to remain elevated?
time-consumingg, although this kind of "prompt engineering" is fixed cost regardless of dataset size. We consider the discovery of such rubrics from human judgments as potential avenue for future research.
Our analysis is also constrained by limited scale. Acquiring human annotations is expensive, which is why our study is restricted to one dataset and aspect. However, we believe that the proposed method can be extended to various aspects and tasks that necessitate nuanced evaluation. Our method provides template for future work, particularly if task designers are sensitive to the requirements of explanations for our method and infuse this understanding into the annotation process itself.
24

---

Published as a conference paper at COLM 2024
How long did take to end?
Label: missing major Explanation: The response isn't useful and misses important information that could better answer the question.
Missing Sentences: 16, 17, 22, EBR: 40
Reference score:
Label: missing minor
Explanation: The answer is pretty good. It is relevant and includes information from document, but fails to mention some of the details and other species that are still not recovered. Missing Sentences: 11, 17, 18, 19,34, 43
EBR: 70
Reference score: 68.33
3 Label: missing major
Explanation: The text does not provide specific answer, but of detail could have been included in an attempt to address the question
Missing Sentences: 17, 18, 30
EBR: 40
Reference score: 48.33
4 Label: missing minor
Explanation: Some more detail could have been included.
Missing Sentences: 46, 48, 50
EBR: 75
Reference scores: 63.33
5 Label: missing minor
Explanation: Some additional relevant information was included in the article Missing Sentences: 19, 21, 25, 46 EBR: 70
Reference score: 75
Table 14: Human judgment along with EBR and reference scores for the question How long did it take to end?
25


Reviews:
[{"review_id": "gQy6mwIXBP", "reviewer": "Reviewer_kZfw", "summary": "This paper proposes to use LMs to rescale human quality judgements by using explanations provided by the human annotators. The authors motivate the work by stressing the importance and value of human annotation in today's NLP landscape for model adaptation and evaluation, especially as models become more capable and automatic metrics are no longer reliable for increasingly subjective or complex tasks. The proposed automatic rescaling method can be used for a few purposes: to calibrate humans who have different interpretations of numeric rankings, to take a coarse-grained rating and make it more fine-grained (such as on a 100-point scale, as the authors propose to do), and/or to apply or specify an arbitrary rubric or criterion *after* the annotations are collected. The authors first collect a dataset of 12.6k high-quality human annotations of labels with explanations (5 per instance). They then test their rescaling method on a document-grounded non-factoid QA task where the goal is to rate \"completeness\" of a model-generated answer. They find that, despite the subjectivity of the task, their method can 1) rescale judgements while retaining the same level of human agreement as the original data annotations, and 2) closes some of the gap with expert rescaling judgements made using the new rubric, indicating that the method is at least partially successful.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed reviews and comments! Addressing some of your questions:\n\n\n> Will you release your code & dataset upon publication?\n\nYes, we will be releasing the dataset and code!\n\n> The motivation for using such a granular scale (1-100) could be made stronger [...] Could you explain this choice a bit more?\n\nThe 0-100 scale was a scale primarily chosen to be a natural prompting target for LLMs. We note that the models we used only produced scores that were multiples of 5, so in practice responses are mapped to 21 distinct values. We believe that a 1-10 or 1-20 scale could also work if our rubrics were defined in terms of these. The key element here is that the scale provides a bit more granularity than the original 4-point scale and enough granularity for the rubric to differentiate different responses.\n\n>  The main results (Table 3) support the inclusion of a rubric, but the avg. EBR baseline with a rubric is quite competitive\n\nWhile Avg EBR is comparable, it is still a four-point scale. It lacks the fine-grained aspect of evaluation which our approach enables us to achieve. EBR gives us the ability to tease out differences in human judgment and be more nuanced which otherwise is not possible with a four-point scale. \n\nWe will make sure to update the paper with the points you mentioned in your review!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is extremely clearly written and well-motivated. Related work coverage is comprehensive. \n- The research question is quite timely. The proposed rescaling is an interesting use-case of LMs that could resolve a number of issues in human annotation for subjective tasks. I think the paper will have a good audience at this conference.\n- The experiments are thorough and well-executed. Using a) LLMs to automate tasks and b) crowdsourcing can have a number of pitfalls if not assessed or evaluated rigorously, but the authors have done a good job in both of these areas, including elaborate quality control details in the Appendix.", "reasons_to_reject": "- A few methodological details could be expounded on or are limited in scope. The main one is a limited choice of models investigated. I elaborated on these below in \"questions/comments\". Overall, I don't think that these issues affect the contribution and validity of the paper, just constrain its scope a bit.\n- Ultimately, the proposed metric is not extremely strong. The main results (Table 3) support the inclusion of a rubric, but the avg. EBR baseline with a rubric is quite competitive, which may be support for the fact that the 1-100 Likert Scale is too fine-grained (as I've also touched on in \"questions/comments\" below). I'd appreciate if the authors could elaborate on their hypotheses for why this may be or their arguments against using the average approach.\n- I assume the authors will publicly release their dataset with publication, but if not, that will severely limit the impact of the paper.", "questions_to_authors": "Questions/comments:\n- Will you release your code & dataset upon publication?\n- The motivation for using such a granular scale (1-100) could be made stronger. It appears from the given example and reported human MAE that a 1-10 or 1-20 scale would be sufficient and that the super granular scale may invite noisy disagreement (for both humans and machines). Could you explain this choice a bit more?\n- Elaboration on \"limited choice of models\" mentioned above: Only one model (GPT4-0613) is studied for its ability to do the rescaling task, so its not clear how the correlation w/ human experts would change for other models and thus if this technique would be useful or valid for them, which is an important question for those doing non-API research. Similarly, only OpenAI API models (though 3 of them) are used for the initial question answering task.\n\nRelated work:\n- there are some interesting papers in psychology literature on the Likert Scale and humans' varying interpretations or calibrations; may be good to cite.\n\nTypos/misc:\n- many of your citations are not in parentheses but should be (i.e., by using `\\citep{}`).\n- please make Figs 2,4 larger given extra space in a camera-ready version.\n- abstract: what do you mean by \"propose a method to rescale ordinal annotations *and explanations* using LLMs\"? seems like it should say \"using explanations\"\n- Fig 1: would be good to also show the 3 expert scores here to get a sense of variance and that there is no 100% agreed ground-truth.\n- Seciton 6.2 last paragraph: here $M$ are the (unaveraged) individual reference scores, correct? It's a little confusing to double up on this $M$ notation.\n- Important to mention that being able to use an arbitrary scoring metric for rescaling *after* annotations are collected relies on the explanations containing enough detail for that particular aspect to be able to do so.", "extra_scores": {}}, {"review_id": "DS8Hw3egmA", "reviewer": "Reviewer_skN9", "summary": "This submission proposes a method to rescale coarse numerical ratings to fine-grained ones. The envisioned scenario is that for a task like document-based question answering, annotators asked to rate answers have given coarse-grained ratings on a four-point Likert scale (\"complete\", \"missing minor\", \"missing major\", \"missing all\"), along with short free text justifications of their rating. Noting the inherent subjectivity of these ratings and justifications, the authors assume that a finer-grained numerical scale from 0 to 100 points is more appropriate. To rescale from coarse to fine ratings, the authors prompt LLMs with the rating, justification, and an instruction that describes how to assign scores.\n\nThe evaluation of the rescaled scores focuses on the two middle labels (\"missing minor\" and \"missing major\") since the extreme points of the scale (\"complete\", \"missing all\") show little variation among raters. The evaluation is mainly performed by measuring agreement of LLM-rescaled scores with a rescaling performed by three experts on 145 instances. Agreement is measured via rank correlation (Kendall's Tau) and mean absolute error (MAE). In terms of rank correlation, the proposed method fails to outperform one of the simple baselines, namely the \"missing sentence heuristic\". In terms of MAE, the proposed method outpeforms all baselines, but unlike for rank correlation, there are no significance scores given, which is a concern considering the small sample size of 145 instances.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! Addressing your concerns/questions below:\n\n> In what manner the proposed approach is \"principled\" is not clear to me. [...] just because some instruction is included in the prompt does not mean that the LM adheres to it\n\nWe agree! This was the intention behind our evaluation in Table 3, where we measure how the proposed rescaling compares to human rescaling (also referred to as reference scores in the paper). We measure the correlation and MAE between the human and the model rescaled scores to show that the model rescaling is faithful to the instruction.\n\n>  The benefit of the fine-grained ratings is not shown\n\nAs we mentioned in response to reviewer kZfw as well, we don’t intend to claim that the full 100-point scale is needed. GPT-4 gives scores that are multiples of 5, so we aren’t using the full granularity of the scale. However, we *do* need more than a coarse 4-point Likert scale. We believe that our proposed method can work with roughly 10-20 score gradations. We will clarify this point in any future version.\n\n> \"Rescaling Prompt\" shown in Figure 3 does not specify what the initial score\n\nThe rescaling prompt in Figure 3 mentions the scale on which the scoring needs to be done. In our analysis and looking through the MAEs, we see the LLM operating as expected by deducting points from 100,  which is why we didn’t need to explicitly state the initial score.\n\n> It is also not clear how information is measured..[..]..subjective interpretation\n\nThe label description along with the specifics of missing information in the explanations enables the human annotator as well as GPT4 to measure how much information is marked as missing and apply the right deductions. This way of rescaling enables the scores to reflect the user’s measure of the information, so the same information can be weighed differently based on how the user expressed their judgment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-", "reasons_to_reject": "1. The proposed method is not motivated well, to the point that it is fundamentally unclear why and how it should address the research problem, for two reasons:\n\n1.1 It is left unclear why a finer rating scale is better suited to capture the inherent subjectivity of the task at hand. The only motivation given is a reference to Kocmi & Federmann (2023), \"which validated the effectiveness of a 0-100 scale compared to other options.\" What \"effectiveness\" means and if/how their results from machine translation apply to ratings of answers in QA tasks is unclear. If \"subjectivity\" in the task at hands means that different annotators have different views of what information is important in a document, then their ratings have no monotonic one-dimensional order and no amount of granularity can fix that.\n\n1.2. According to the introduction, \"subjectivity (and thus, inherent disagreement) in annotated data has surfaced as a key direction\" and the authors make the observation that \"intricacies in human judgments can be captured by natural language explanations provided during data annotation, which capture more nuanced subjectivity\". So far so good, but now the authors claim that their \"key idea is to make the ordinal label space (e.g., a coarse Likert scale) fine-grained, namely a 0-100 scale Kocmi & Federmann (2023), which then enables us to place the initial annotations in it by leveraging natural language explanations in a principled manner.\"\nIn what manner the proposed approach is \"principled\" is not clear to me. The authors are possibly referring to the scoring rubric that is included in the prompt, but just because some instruction is included in the prompt does not mean that the LM adheres to it, and it is even less clear if the LM faithfully can execute complex instructions that require it to repeatedly subtract a variable number of points, with the amount of points varying on the amount of information that is missing from the answer to be rated.\n\n2. The benefit of the finer-grained ratings is not shown. The evaluation stops at showing moderate agreement with expert-produced fine-grained ratings and stating that the rescaling does not impact inter-annotator agreement. The introduction claimed that the proposed method can \"capture more nuanced subjectivity\", but this is never directly shown.", "questions_to_authors": "- While the intention is clear, the \"Rescaling Prompt\" shown in Figure 3 does not specify what the initial score is from which points should be deducted\n\n- It is also not clear how information is measured, so a phrase like \"half of the correct information\" is left wide open to subjective interpretation", "extra_scores": {}}, {"review_id": "NxjEeBuLSY", "reviewer": "Reviewer_f43i", "summary": "The paper seeks a way to improve human evaluations of subjective tasks via large language models (LLMs). In particular, the authors exploit the natural language explanations to rescale ordinal annotations. The authors propose a method called Explanation-Based Rescaling (EBR) that utilizes natural language explanations provided by human annotators alongside their Likert scale ratings. EBR leverages an LLM to convert both the rating and explanation into a more fine-grained numeric score (0-100) guided by a post-hoc defined scoring rubric. The authors demonstrate the effectiveness of EBR on a document-grounded question answering task where LLMs achieve near-human performance. They show that EBR produces scores closer to reference scores generated by experts using the same rubric, without impacting inter-annotator agreement.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your reviews and detailed comments! Addressing your concerns below:\n\n> Achieving high agreement among [...] requires further exploration.\n\nOur main goal for using human annotators as reference was to evaluate whether or not LLMs can do our rescaling task according to the instruction (or task rubric). We definitely agree with you that the task is challenging and this is reflected in our experiments as well (in Table 2). We are exploring some of these challenges in our future work.\n\n\n> It was unclear whether fine-grained labels are necessary and how they could be helpful in improving LLMs or for NLP overall\n\nIn the era where LLMs make subtle mistakes and tasks are getting more subjective, we believe fine-grained scores are needed for evaluation. Fine-grained scales can give us a more expressive mechanism for representing differences in opinion where errors can be subtle: for instance, three distinct error types can be mapped to -5/-10/-15, all reflecting that a response is still quite good but at different levels. Note that our scores are all multiples of 5, so in reality our scale has 21 distinct values that are output from the model. A fine-grained scale can be coarsened if needed (eg: from 0-100 -> 1-4).\n\nThat said, collecting human annotations on a fine-grained scale might be harder and there can be scale usage differences, which is why we suggest a method that goes from explanations to fine-grained scores. \n\n\n> Regarding the evaluation, the authors mention that the work aims to demonstrate the usefulness of using NLEs to enhance rescaling performances.\n\nWe have comparisons to rescaling without explanations, reported in Table 3. In our proposed baselines, *Static* rescaling and *MSH* (Missing Sentences Heuristics) rescaling do not use explanations. We show that our proposed method that uses the explanation does substantially better compared to the non-explanation baselines.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1. Utilizing natural language explanations to refine and quantify human judgments is novel. EBR allows for a more fine-grained evaluation of LLM outputs by mapping them to a 0-100 scale. This is particularly useful for tasks with subtle distinctions where traditional Likert scales might be too coarse. Furthermore, its flexibility (the fact that the scoring rubric can be designed post-annotation) can give room for better optimization. \n\nS2. The authors conduct thorough experiments and analysis on a newly collected dataset, demonstrating the effectiveness of EBR compared to various baselines and highlighting its ability to capture subtle nuances in human explanations.", "reasons_to_reject": "W1. Achieving high agreement among human evaluators for subjective tasks is a significant challenge. As shown in Table 2, reaching a consensus among human annotators on missing majors seems to be particularly difficult. Given this, using the results of human annotators as references may not be the most effective method for evaluating the capabilities of LLMs on this task. While I don’t have a clear solution for this issue, it’s certainly a challenging area that requires further exploration.\n\nW2. It was unclear whether fine-grained labels are necessary and how they could be helpful in improving LLMs or for NLP overall. \n\nW3. Regarding the evaluation, the authors mention that the work aims to demonstrate the usefulness of using NLEs to enhance rescaling performances. To achieve this, the experiment should be designed to allow for a comparison between sets with and without explanations. The current setup does not directly evaluate this effect.\n\nW4. The evaluation is centered on a specific task of document-grounded question answering. Further investigation is needed to assess the generalizability of EBR to other tasks and domains. Specifically, in the presented task, the quality of the answers can be determined based on the missing sentences, which are relatively straightforward to quantify. However, in other scenarios, such clear criteria may not exist. \n\nW5. Lastly, the paper acknowledges that the creation of the rubric is currently a manual process. This could be time-consuming and subjective, potentially affecting the generalizability of the approach.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oX7EMPF9Sl", "reviewer": "Reviewer_8NAr", "summary": "This paper presents a method of using LLM to convert human Likert ratings to scalar values using the human explanations. \nThey first collected the Inquisitive-board dataset. The dataset consists of news articles and questions written by a linguistics student, and answers from LLMs. Then they ask annotators to evaluate the generated answers on how much information (major/minor/none) from the original articles the answers are missing, provide an explanation and how many sentences are missing. They found some disagreement in the Likert scale (major/minor/complete) ratings\n\nThey then provide some rubrics for converting the Likert scale ratings and explanations to a score in 0-100 for how much information is missing. They have 3 experts doing this rescaling for reference, and evaluate how well LLMs can do this rescaling. They found that the LLM rescaled values correlate with expert rescaled values and explanations provide fine-grained information beyond the Likert scale.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! Addressing your concerns below:\n\n> what does one do with this method?\n\nOne use case is LLM evaluation as you say. This can be with human or model generated feedback where directly scoring the generation is subjective and we need natural language feedback to tease out nuances. Recently, rubric based scoring is being adopted in NLP [1] [2], but it work assumes a static rubric and produces the feedback as an explanation or a rationale to the score. Our work focuses on using a rubric defined post-hoc, looking for patterns in the feedback so that the evaluation can be adapted to the underlying subjective judgment.\n\nAnother application could be fine-grained reward modeling. We see emerging frameworks like fine-grained RLHF [3] that provide ways to use fine-grained annotation scores. In line with that framework, we can evaluate the benefits of our proposed work to train better aspect-specific reward models. This can be further used to improve tasks like document-grounded QA, query focused summarization, or any task that involves human-LLM interaction. \n\nWe note that these are emerging research directions themselves and there are not “plug-and-play” applications for the evaluation methodology we’ve developed here. We believe it can be extended in current work and conceptually helps in many possible ways, but it is not easy to connect to one of these downstream applications within the scope of this current paper.\n\n> Can this method be extended to reconcile multiple [..] conflicting dimensions?\n\nOur method is tied to the scoring rubric and the quality of explanations, so more exploration would need to be done to see whether or not we can reconcile multiple dimensions in the rubric. This can definitely be an interesting future direction we explore. \n\n> I suspect the extent of information missing\n\nThis is an interesting hypothesis. In general, the information answering the questions can be located anywhere in the article, so we don’t strongly observe this. It would be interesting to connect more general notions of information relevance with these question answers, but that’s beyond the scope of the current work.\n\n\n[1] UltraFeedback: Boosting Language Models with High-quality Feedback https://arxiv.org/abs/2310.01377 \n\n[2] Prometheus: Inducing Fine-grained Evaluation Capability in Language Models https://arxiv.org/abs/2310.08491 \n\n[3] Fine-Grained Human Feedback Gives Better Rewards for Language Model Training https://arxiv.org/abs/2306.01693", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper provides an interesting method on rescaling Likert scale judgments and showed that the explanations provide insights into the discrete judgments.\n\nThe methodology is sound and the approach is interesting.", "reasons_to_reject": "The biggest question I’m left with after reading this paper is: what does one do with this method? The amount of missing information is a limited dimension of a generated answer. Can this method be extended to reconcile multiple and potentially conflicting dimensions? Since this paper is in the evaluation track, do the authors recommend using this method for future LLM evaluation? Does using the rescaled values as an evaluation metric leads to different models being preferred and are those models indeed better in some ways? Without clear takeaways/recommendations the impact of this paper is likely limited.", "questions_to_authors": "I suspect the extent of information missing probably also correlate with the location of the missing sentence, or more broadly the information structure. Generally the information in the beginning of the article is probably more prominent and major than the information at the end of the article. The question being asked will also impact which information is prominent. Have you looked at this in your analysis?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper uses language models (LMs) to rescale human quality judgments via annotator explanations. The method addresses the importance of human annotations for model adaptation and evaluation as models become more capable. It can calibrate differing human interpretations, refine coarse ratings, and apply rubrics post-annotation. Using a dataset of 12.6k annotations, the method was tested on a QA task, successfully rescaling judgments while maintaining agreement and aligning closer to expert evaluations.

Reviewers have mainly raised concerns about the validity of using a fine-grained 0-100 scale instead of a narrower range (e.g., 1-10).

In NLP, the 0-100 scale annotation has been primarily proposed and utilized in the machine translation community, supported by some advantages from a statistical point of view. It can capture subtle differences in the outputs, better aligning with human preference. 
Relevant works worth mentioning in the paper include:
- Graham et al. (2013): "Continuous Measurement Scales in Human Evaluation of Machine Translation"
- Bojar et al. (2016): "Findings of the 2016 Conference on Machine Translation"
- Sakaguchi et al. (2018): "Efficient Online Scalar Annotation with Bounded Support"

We expect the authors to address other concerns and questions in the final draft.
Decision: Accept