Using Natural Language Explanations to Rescale Human Judgments

Authors: manya  wadhwa
jifan  chen
jessy  li
greg  durrett
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 17:25:55

Abstract: The rise of large language models (LLMs) has brought a critical need for high-quality human-labeled data, particularly for processes like human feedback and evaluation. A common practice is to label data via consensus annotation over human judgments. However, annotators' judgments for subjective tasks can differ in many ways: they may reflect different qualitative judgments about an example, and they may be mapped to a labeling scheme in different ways. We show that these nuances can be captured by natural language explanations, and propose a method to rescale ordinal annotations and explanations using LLMs. Specifically, we feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce a numeric score anchored in a scoring rubric. These scores should reflect the annotators' underlying assessments of the example. The rubric can be designed or modified after annotation, and include distinctions that may not have been known when the original error taxonomy was devised. We explore our technique in the context of rating system outputs for a document-grounded question answering task, where LLMs achieve near-human performance. Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.

Main paper: 
 introduction: 
 With the rise in model performance due to large language models (Brown et al., 2020; Ouyang et al., 2022, LLMs) , there is a shift towards working on and evaluating more subjective tasks. On tasks like news summarization, LLMs can no longer reliably be judged by automatic metrics (Goyal et al., 2022a; Xu et al., 2023; Wang et al., 2023b) and achieve near-human performance (Zhang et al., 2023; Zhan et al., 2023) . This high performance makes it harder than ever to annotate model outputs for errors (Saunders et al., 2022; Dou et al., 2022; Chang et al., 2023; Goyal et al., 2022b; Liu et al., 2023; Chen et al., 2024) , an important ingredient as models are deployed in high-stake scenarios.This answer should've given more detail…it is missing the following..Answer (machine), to be judged This shift towards more subjective tasks and difficulty in identifying errors has also led to a change in the role of human judgments in NLP. Earlier crowdsourcing work on simple labeling tasks explored capturing annotator bias with item-response models (Dawid & Skene, 1979; Smyth et al., 1994) , especially to learn annotator quality (Hovy et al., 2013) . As the ambit of NLP research has expanded to include more sophisticated downstream tasks, taking into account subjectivity (and thus, inherent disagreement) in Separate human labelers rescore these explanations according to an aspect specific scoring rubric. Our proposed explanation-based rescaling (EBR) method maps judgments to scores using that rubric, producing scores closer to the reference scores by accounting for factors mentioned in the explanations.annotated data has surfaced as a key direction (Plank, 2022; Uma et al., 2021; Basile et al., 2021; Nguyen et al., 2016) .While existing work developed label aggregation methods and ways to incorporate different labels (Plank, 2022) , information from the labels alone is still limited. Instead, intricacies in human judgments can be captured by natural language explanations provided during data annotation, which capture more nuanced subjectivity. They enable us to go beyond direct labeling, where annotators can choose different coarse labels for the same reason or the same coarse label for different reasons. The question is, how do we transfer at least some of that expressiveness into numeric expressions that are more friendly for model training and evaluation?This paper proposes explanation-based rescaling (EBR) that enables us to transfer the expressiveness of natural language explanations into numeric forms. Our key idea is to make the ordinal label space (e.g., a coarse Likert scale) fine-grained, namely a 0-100 scale (Kocmi & Federmann, 2023) , which then enables us to place the initial annotations in it by leveraging natural language explanations in a principled manner. Our approach begins by gathering explanations for each judgment during the annotation process. Next, we leverage an LLM to convert both a Likert judgment and its associated explanation into a numerical rating between 0-100. Crucially, this rescaling is guided by a predefined aspect-based scoring rubric, which can be defined entirely post-annotation, and provides task-specific guidance to the LLM on the placement of labels on the 0-100 scale. We find that an LLM with a rubric produces scores consistent with those that a human produces when using the same rubric.We consider the task of evaluating LLM-produced answers for document-grounded, nonfactoid question answering. We use annotated high-level questions (targeting discourse understanding) from the INQUISITIVE dataset (Ko et al., 2020) , supplementing these with freshly collected questions from geographically diverse and recent text sources. We collect outputs from several LLMs, then annotate answers chiefly for answer completeness, with workers giving a Likert judgment and explanation. Through annotator recruitment and filtering, we selected a set of crowdworkers who gave high-quality, reasoned decisions, yet still had differences in opinion. Figure 2 gives an example from our dataset where annotators chose different Likert judgments for information completeness, but the explanations show that some of them were looking at the same factor. Rescaling with a scoring rubric is able to impart fine-grained distinctions based on these explanations, and in particular, works better than rescaling without a rubric.We evaluate our approach on whether LLM rescaling (1) can discern subtleties in natural language feedback as well as humans do; (2) changes correlation between annotators. The proposed approach brings scores closer to how humans would do this rescaling without impacting agreement, while retaining subjectivity.Our main contributions are: (1) A method for rescaling Likert judgments with explanations using LLMs anchored in a scoring rubric. (2) A dataset of 12.6k human judgments (with explanations) on the completeness and correctness of answers produced by strong systems (GPT-3 and GPT-4) for document-grounded high-level question answering. 
background and task desiderata: 
 Motivating Example Figure 2 shows a motivating example for our approach on evaluating LLMs when they answer document-grounded questions. Given an article and a question, we use an LLM to generate a freeform answer. The answer to the question is high quality, but we want to be able to evaluate the LLM output precisely and assign granular scores. Our approach is targeted towards tasks like this kind of LLM evaluation, where humans have to work hard to articulate subtle distinctions in their annotation.We have five crowd workers give judgments about the answer to this question. They disagree about whether the question is missing major or minor information. For this task, it would be difficult to specify concrete enough annotation guidelines to make the label fully unambiguous. However, some of the explanations call out the same relevant feature of the answer, which is the answer was missing specific sentences talking about sherpas possibly not returning back. Our approach accounts for subjectivity in assessing quality, but assumes that there is an underlying ground truth answer which annotators can identify.An LLM can judge the provided labels and explanation, simultaneously giving finer-grained information ratings on a scale of 0-100. These ratings are anchored in a scoring rubric, which is defined post-hoc after looking at a collection of explanations for the aspect being evaluated (in our case it is for completeness). As mentioned above, the 0-100 scale offers granularity to effectively map a variety of explanations to numeric values, and the aspect-specific scoring rubric ensures consistency in this mapping, as opposed to mapping examples without on the numeric scale. We describe our method in detail in Section 5.We need datasets that contain human judgments accompanied by informative, well-formed natural language explanations, where the underlying task satisfies the two properties bolded above (strong model performance and the right amount of subjectivity). We found three datasets that are relevant but they are not compatible with these characteristics. SQuALITY (Wang et al., 2022) includes evaluation of both human and model summaries by other humans. However, this task is very demanding due to the long document length, and we found that the feedback provided in the existing dataset is short and underspecified. Saunders et al. (2022) also have a dataset with human critiques of topic summaries. However, the evaluation criteria are defined loosely, leading to critiques that are too generic. There is no multiply-annotated data for us to reliably perform analysis on the effect of our rescaling methods. Finally, Filighera et al. (2022) releases a dataset for short answer grading with explanations; however their data does not contain annotator information. Note that other existing datasets for explanations in NLP, such as those mentioned in Wiegreffe & Marasovic (2021) , do not align with our task requirements, since classification tasks do not involve fine-grained critiquing of outputs. 
collecting qa pairs: 
 We study human judgments and critiques for non-factoid, document-grounded question answering, focusing on judgments of information completeness, i.e., whether machinegenerated answers contain all key pieces of information from a document. We use two sources of data for this task. First, we use the questions collected in the INQUISITIVE dataset (Ko et al., 2020) . Second, following the annotation guidelines from INQUISITIVE, a linguistics student wrote questions on articles focused on recent news events and nonwestern geographic entities (Hong Kong, Singapore, and India). This reduces leakage about notable prior news events for the LLMs in question. A summary of the two splits is given in Table 5 . All text is in English. We call our dataset INQUISITIVE-BROAD.1 See Appendix A for examples of articles for the task.The questions target high-level comprehension of text, making the answers complex with information distributed across multiple sentences. They contain a mixture of causal questions (e.g., why are they now liberalizing bank laws?), procedural/elaboration questions (e.g., how will the exceptions to the verification concept be taken care of?), background information questions (e.g., what is the main focus of this movement?), and instantiation questions (e.g., which groups were the human rights activists working on behalf of?) (Ko et al., 2020) , all of which require discourse-level processes and reasoning. This task meets our desiderata by striking a balance between subjectivity and objectivity. There are divergent opinions about how much information should be included, but because the correct answer is grounded in the article, it should be relatively easy to judge if key information is missing. This contrasts with other long-form question answering tasks like ELI5 (?), where the lack of grounding makes it difficult to judge information completeness and requires subject matter experts for each question (Xu et al., 2023) .We use three off-the-shelf LLMs (Davinci, GPT-3.5-turbo and GPT-4) to answer questions from INQUISITIVE-BROAD. We also have two human experts answer a subset of the questions. These systems are shown in Table 6 and the prompts used are given in Figure 8 . 
human evaluation of qa pairs: 
 Using Mechanical Turk, we enlist 8 qualified crowdworkers to conduct human evaluation of the question-answer pairs in our dataset. Each instance is evaluated by 5 crowdworkers.The study leads to a dataset of 12.6k annotations consisting of discrete labels along with natural language explanations.Each crowdworker was given the article, question, and answer, and was asked to evaluate the answer on completeness and correctness attributes. For completeness, they were asked to choose on a Likert scale the amount of information missing in the answer with respect to all relevant information present in the document. Four options were given: complete, missing minor information, missing major information, and missing all information. For correctness, they were asked to mark if the information in the answer is faithful to the information present in the document. Crowdworkers were also asked to enumerate missing sentences and give rationale in the form of natural language explanation for their decision.Appendix B.1 describes our process of recruiting and qualifying crowdworkers in more detail and shows screenshots of the interface (Figure 7 ). We ensure the crowdworker pay is ∼$15/hour. Despite disagreements in annotation, we have substantial evidence, as well as third party validation (Appendix B.2), that these crowdworkers are attentive to the task and labeling high-quality data.Table 1 shows the un-aggregated % of labels in each category for the completeness attribute for INQUISITIVE-BROAD. The distribution is skewed towards the complete label. Table 1 also shows the distribution of labels for different QA systems being evaluated. Together, these show the challenges of judging the outputs of these systems. They are performing at a human level according to these annotators and make relatively infrequent mistakes.Similarly, for the correctness attribute, 70% of QA pairs are marked as correct. Those marked as incorrect constitute of unanswerable questions where the model hallucinated information instead of accurately stating the information was missing from text. We thus focus our analysis on only the completeness attribute.We also measure inter-annotator agreement across the discrete ratings. If we collapse to two classes, complete vs. not, the Fleiss Kappa value is 0.328. Kendall's Tau-b (τ b ) correlation across all 4 labels is 0.325. This "fair" agreement, and given the perceived quality of the natural language explanations, we view this as evidence of genuine subjectivity.The main goal of your task is to score a machine generated response to a question. Scoring is on the "completeness" attribute. A complete answer will have all relevant information from the article required to answer the question and an incomplete answer won't. However, you are not directly scoring the machine response, but instead using a given feedback and amount of missing information. The details are given below:Article on which the question was asked: {article} Feedback given to the machine response: {feedback} Sentences marked as missing: {enumerated missing sentences} Level of the missing information:{label definition} What is the score? Give a number.On a scale of 0-100 how will you score machine response using the feedback and level of missing information stated above? Use the rubric below for scoring: 1. if the answer is complete, give 100 points 2. if the answer is missing one or more minor details then have deductions ranging from 5 to 30 points based on the severity of missing details 3. if the answer is missing a major facet of information, it results in a deduction of at least 40 points and more than 50 points are deducted if less than half of the correct information was given. 4. if the answer contains no correct information but only marginally relevant information from the article, 70 points are deducted 5. if the answer contains no correct information but the article clearly has information present, 100 points are deductedFigure 3 : The rescaling prompt for GPT-4, which considers the aspect definition (eg: completeness, defined in the prompt above), a scoring scale (0-100), the task input (the article), human judgment (Likert rating, natural language explanation and missing sentences) and a scoring rubric.Please refer to the Appendix (Table 10 ) for the distribution of labels across 3 annotators. Table 14 gives examples of explanations that differ in their label decision but agree on details in their explanations. 
explanation-based rescaling (ebr): 
 In this section we propose a rescaling method that captures natural language explanations by using them to reevaluate the Likert labels. We take inspiration from the scoring mechanism presented in Kocmi & Federmann (2023) for machine translation evaluation, which validated the effectiveness of a 0-100 scale compared to other options. The key element here is that a chosen scale provides higher granularity than the coarse grained scale on which the initial Likert labels are collected. Instead of using an LLM to directly evaluate the task, we use a more detailed scoring rubric.Formalization Assume we have a collection of items x 1 , . . . , x n which are being rated by annotators a 1 , . . . , a m in a sparse manner, i.e., not every annotator rates every item. Each annotator assigns an item a discrete rating r ki and writes an explanation e ki , where k refers to the item and i refers to the annotator. We rescale annotator judgment as following: s ′ ki = f (r ki , e ki ). Details about the rubric creation process are given in Appendix C. We also explore rescaling variations without rubric. Variants of the rescaling prompt are given in Appendix D.1 and we report our results in Section 7.1. Note, we do not rescale instances that get the extreme labels complete and missing all, since in these cases we rarely observed nuanced explanations from which rescaling would be possible.To compute f , we invoke gpt-4-0613, taking as input both the explanation e ki along with the discrete rating label r ki . The proposed rescaling prompt is given in Figure 3 . The rescaling prompt is structured as following:1. Aspect definition: defines the aspect being evaluated (in our case, completeness) 2. Scoring scale: defines the numeric scale to which the human judgment needs to be mapped (0-100 for our task; note that the LLM typically outputs multiples of 5) 3. Task input: helps contextualize the natural language feedback (the article) 4. Human judgment: Likert rating and the natural language explanation (and when available, missing sentences) 5. Scoring rubric with deductions: anchors the scoring to be consistent 
evaluation: 
 With our initial human annotation and methodology established, we now turn to evaluation. We first introduce a notion of reference rescaling: given a rubric, can expert annotators agree on how explanations from our dataset should be rescaled? We establish data and baselines for this evaluation, then turn to a comparing human and automatic rescaling in Section 7.In order to evaluate the ability of LLMs to faithfully rescale natural language explanations, we ask three experts to establish reference scores. We sample 145 instances, where each instance consists of a QA pair and a corresponding human judgment in the form of the Likert label, natural language explanation and missing sentences. The distribution of labels in this subset is: complete: 20, missing minor: 52, missing major: 53, missing all: 20. We sample more from missing minor and missing major labels since these categories have nuanced explanations, compared to complete or missing all, which occur at extreme ends of the scoring scale. Each expert was given the same rubric information as the rescaling prompt mentioned in Section 5. We refer to an average of the expert rescaled scores as reference scores, represented by R.Using this, we can answer two questions. (1) Can humans consistently rescale if given the rubric (i.e. is our methodology sound)? (2) Can LLMs do this rescaling automatically in a manner similar to humans when given the same rubric?We determine how well our rescaling method does by comparing the rescaled values with references scores and also by comparing inter-annotator agreement before and after rescaling. We use Mean Absolute Error (MAE) and Kendall's τ correlation (Kendall, 1948) .Mean Absolute Error computes a difference between M(r ki , e ki ) and R(r ki , e ki ), where M is a system (e.g., our proposed rescaling method), R refers to references scores (rescaling done by experts) and (r ki , e ki ) is the Likert rating and explanation for item x k by annotator a i .Kendall's τ is based on pairwise score comparisons, and thus reflects a common ranking use case. We use Kendall's τ b , which makes adjustments for ties. Using a method M (e.g., our proposed rescaling method), we compute correlations between the proposed and reference rescaled scores τ(M(r ki , e ki ), R(r ki , e ki )).We also compute pairwise correlations between two annotators' rankings, which we denote as τ(M(r ki , e ki ), M(r kj , e kj )) for annotators i and j, where k is a placeholder index. We then compute an aggregate correlation across all pairs of annotators:1 ( |A| 2 )∑ a i ,a j ∈A×A,a i ̸ =a j τ(M(r ki , e ki ), M(r kj , e kj )), where A is the set of annotators.We compare the proposed rescaling with four baselines that map human judgments to a numeric scale and calculate Kendall's τ and MAE against reference scores for each method.STATIC rescaling maps the Likert ratings to a static numeric scale by assigning complete to 100, missing minor to 70, missing major to 30 and missing all to 0. This mapping does not use any fine-grained information from the explanations.AVG EBR rescaling leverages the explanations to come up with a mapping for the four Likert labels to a 0-100 scale. To find this mapping, we first use EBR to rescale human judgment in the dataset. We then average all numeric scores under each of the labels. This method allows us to incorporate explanations while keeping the original level of granularity. This also maps labels at a calibrated interval from each other instead of having them at equal intervals (as in the STATIC rescaling method). We get the following mapping for each label: complete: 100, missing minor: 78.6, missing major: 50.9, missing all: 0.0.EBR W/O RUBRIC rescales using human judgment which includes Likert ratings and explanations to get a score without the aspect-specific scoring rubric. This method can theoretically map examples with any Likert rating anywhere in the 0-100 numeric scale.  
results: 
 In this section we show the effectiveness of our approach by comparing LLM rescaling against reference scores. We also explore how the rescaling influences inter-annotator agreement.Missing Major Overall τ ↑ MAE ↓ τ ↑ MAE ↓ τ ↑ MAE ↓ Without Rubric Static -To establish if humans can do the proposed rescaling we look at Kendall's τ and MAE between pairs of experts as reported in Table 2. Correlation is high and MAE is low when considering all labels. We also specifically look at missing minor and missing major labels; making distinctions within these requires the most nuance.Given the granularity of the proposed scoring scale (0-100) for our method, we see experts showing high correlation and low MAE for missing minor. However, for miss-ing major, even though scores show low agreement we still see positive correlation on a granular scale. In Figure 4 (a) we observe that one of the experts had higher spread of scores in missing major as compared to the other two, highlighting subjectivity in explanations and underscoring the complexity of this task. Does prompting GPT-4 with the rubric recover the rescaled values from humans? Table 3 shows how our proposed rescaling using GPT-4 compares to reference scores on 145 instances mentioned in Section 6.1. We look at Kendall's τ and MAE between reference scores (average of expert rescaled scores) and EBR scores. We also compare EBR with the baseline methods defined in Section 6.3. Again, we specifically look at missing minor and missing major label categories, since they are the main source of subjectivity and have more nuanced explanations.Our approach consistently outperforms all other methods in terms of achieving the lowest MAE, indicating its ability to faithfully capture subtle nuances in human explanations when compared to alternative rescaling techniques.Overall correlation is consistently high across all methods. However, since this includes all labels, the complete and missing all categories can distort the overall metric. We instead choose to focus on missing minor and missing major specifically, looking at the correlation within these more subjective labels. Notably, with our proposed method both correlation values, i.e., 0.45 for missing minor and 0.20 for missing major are similar to the correlation values observed among experts, as shown in Table 2 .While MSH gets a higher correlation value than our baselines it also leads to a high MAE, even though this is the criterion it was optimized for. Note that STATIC and AVG EBR do not produce rankings within these categories, and therefore are not evaluated in this setting.How important is the rubric? Table 3 shows the usefulness of the rubric. EBR with rubric consistently achieves lower MAE and higher correlation, across all labels and for missing minor and missing major.In Figure 4 (b), we also look at the distribution of EBR W/ RUBRIC scores and EBR W/O RUBRIC scores within each label category against reference scores. EBR W/ RUBRIC scores fall into label order without any explicit constraint. EBR W/O RUBRIC is more spread out and scores within a category cross boundaries with those of other label categories. Note that this is not necessarily wrong, but we view it as an indicator that the scores are less calibrated when no rubric is given. Note that horizontal jitter is added to differentiate points; all scores produced by our method are multiples of 5. The explanation-based rescaling imparts a finer granularity to the judgments while retaining subjectivity. We show a pair of explanations, where the two annotators agree on the label and but disagree on the finer details of what constitutes "missing major" information. EBR reflects the disagreement.Our proposed method transforms human judgment to a 0-100 scale, offering a finer level of granularity compared to the original four-category Likert scale used for coarse labeling. In this section, we investigate how going more fine-grained affects inter-annotator agreement.To understand how mapping to a finegrained scale changes annotator correlation, we look at the average of pairwise Kendall's τ (as described in Section 6.2) for INQUISITIVE-BROAD.Table 4 shows that our method is able to rescale human judgment to a much more finegrained scale without impacting agreement; that is, our scoring can be more granular without impacting the inter-annotator correlation.We take a closer look at a pair of annotators in Figure 5 . It shows the distribution of labels (pre-rescaling) and scores (post-rescaling). For this pair there is no change in the agreement post-rescaling. Looking at an example of their annotations from the dataset, we see that they agree on the category label but identify different factors in their explanations. Our proposed method assigns different scores to each human judgment and is able to reflect this difference. .We emphasize that our goal is not to "smooth out" subjectivity in the annotation. Rather, taking natural language explanations into account produces a more calibrated and nuanced view that preserves inherent subjectivity while aligning differences between annotators when they actually agree.Tables 13 and Tables 14 in Appendix G show examples of human judgments, which includes category label, explanation and missing sentences for two different questions. These show different styles of explanations, but with an overlap in missing information. 
related work: 
 LLMs for evaluation and annotation Kocmi & Federmann (2023) Our work is assumes that some tasks on the frontiers of LLM capabilities will always need human judgments; our procedure aims to augment and improve the capabilities of the group of crowdworkers.Another method for rescaling annotations is the calibrated sigma method (Weijters et al., 2016) . However, this method does not change the ranking of examples, only rescales their scores for each annotator while preserving the ranking. We therefore do not compare to it here. Another line of work (Hovy et al., 2013; Gordon et al., 2021) focuses on modeling annotators to build better consensus judgments, but does not address aligning pairs of annotators. Sakaguchi & Van Durme (2018) proposed a method EASL, that adapts the original annotator labels to probabilistic ones based on the labels of all annotators. Finally, Ethayarajh & Jurafsky (2022) present a new protocol for NLG evaluation in which annotators give judgments in the form of probabilities over sets. This work also demonstrates the flaws in Likert judgments and pursues an approach to improving them orthogonal to ours. Consensus building has also been extensively studied in the translation community (Bojar et al., 2016) . Graham et al. (2013) showed that using a continuous scale of 0-100 improved inter-annotator correlations on translation tasks.Natural language explanations Natural language explanations (NLE) have been shown to be effective in improving model performance when used as additional features (Wu & Mooney, 2019; Murty et al., 2020; Liang et al., 2020) , explaining model decisions (Camburu et al., 2018; Narang et al., 2020; Hase et al., 2020) , boosting the performance of in-context learning of LLMs on a variety of reasoning tasks (Nye et al., 2021; Wei et al., 2022; Lampinen et al., 2022) . Prior work also demonstrated that natural language explanations help us holistically understand annotator perspectives in complex tasks (Ferracane et al., 2021; Goyal et al., 2022a) , improve annotation quality (Alonso, 2013) and aid adjudication (Filighera et al., 2022) . Our work uses a similar idea but focuses on rescaling human annotation specifically, which is not a task addressed in this prior work. 
conclusion: 
 In this work, we showed that LLMs can be used to rescale annotator judgments. We feed an annotator's label and explanation of that label into GPT-4 to produce a rubric-grounded score from 0-100. On a new dataset we collect of document-grounded questions answered by LLMs, we show that rescaled annotations align well with reference rescaled values produced by expert annotators. Overall annotator correlation does not change, but we show that our rescaling method is able to capture fine-grained nuances of the judgments, teasing apart subjectivity and scale use differences. 
 
 Appendix: Our Tables 7 and 8 show examples of articles, questions, system responses along with human annotations (ratings and explanations) for two articles from the INQUISITIVE-BROAD dataset. We prompt three LLMs and also ask two human experts (one of the co-authors and a non-author linguistic student) to answer questions from INQUISITIVE-BROAD.The systems along with their properties are mentioned in Table 6 . EX-PERT human in the table refers to linguistic undergraduate students who have worked on a series of computational linguistics annotation projects for 1-2 years. One student is now a lead data annotator at a small NLP company. These annotators are experienced at following annotation guidelines and producing high-quality judgments. Sample prompts for each of the systems is given in Figure 8 . The debate over the prime minister marked the first standoff within the unlikely coalition of divergent opposition parties that came together last week to support the military's toppling of Morsi. The military and interim President Adly Mahmoud Mansour had hoped the swift appointment of a prime minister would send a message of stability and confidence to both Egyptians and the international community. Instead, the fumbling raised fresh doubts about Egypt's political future. Though viewed as a liberal, El-Din is seen as someone who will focus on improving Egypt's sagging economy and reforming corrupt government institutions. He has been an outspoken police critic and blamed Morsi for failing to adopt a more inclusive approach during his year in office. Liberal youth activists, led by the Rebel movement that helped organize the mass protests that ousted Morsi, pushed hard for ElBaradei, a Nobel Peace Prize laureate and former head of the International Atomic Energy Agency. Mansour seemed set to appoint ElBaradei, and a swearing-in was scheduled. But late Saturday the government abruptly backtracked after the ultra-religious Nour Party rejected him as too liberal and divisive, threatening to quit the interim coalition if ElBaradei got the job. On Sunday, throngs of Egyptians from both sides had voiced their views in mass protests -largely peaceful until the early morning violence. In the Nasr neighborhood, speakers denounced last week's coup as illegitimate and called on mid-level army officers to revolt against top generals who pushed Morsi out. 'We will stay here for as long as it takes and peacefully pressure the army to reinstate Morsi,' said Gamal Ragab, 41. And in Tahrir Square, the number of anti-Morsi demonstrators surged into the tens of thousands early Sunday evening. 'We want to build the democracy that we dreamed of in 2011, not a religious state where religion is used to divide the whole population and exclude anyone who doesn't share the Brotherhood's ideology,' said Samira Kamel, 21, a Cairo student. 'We want to show the world that this is a popular revolution, not a coup'.Question: Does he have a lot of supporters? Davinci-003: Yes, Muslim Brotherhood officials said security forces raided their encampment outside the Republican Guard compound with tear gas and gunfire about 4 a.m. and there were tens of thousands of anti-Morsi demonstrators in Tahrir Square.Annotator 1: Label: Missing All , Explanation: The answer tells of anti-Morsi demonstrators not pro-Morsi. There is no information on the numbers of Morsi supporters Annotator 2: Label: Missing Major , Explanation: The answer focuses on the anti-Morsi side, though the text only mentions "throngs" of protestors "from both sides" Following the qualification rounds, we selected 8 highly qualified Turkers. Given the substantial scale of our evaluation study, we actively monitored the feedback section of the HIT. If any ambiguities or issues arose, the Turkers had the option to contact us. Additionally, we promptly reached out to the Turkers if they provided any general feedback on the task, aiming to address and clarify any concerns.In order to maintain quality of annotations, we assigned only one document per HIT. On average, each document had 4-5 questions. We compensated the Turkers $2 per HIT and gave them a 20% bonus after each batch. Each Turker got 2 hours to finish a HIT they accepted. We released batches of only 20 HITs every 12-24 hours. This aimed to prevent any decline in annotation quality that could potentially rise from prolonged or excessive workload.Figure 7 shows a screenshot of the annotation interface visible to the annotators.Worker ID 0 1 2 3 4 5 6 7 % Quality agreement 87 90 94 81 93 86 94 74Table 9 : Human evaluation of explanations to gauge worker quality.In order to assess the quality of explanations we had a human expert (an independent judge not involved in this work) look at a sizable sample of explanations for each annotator in our dataset and mark if they find the explanations to be plausible. Specifically, the expert judge had access to the arti-cle and the QA instance, as well as the natural language explanation of the annotator;2 the judge studies whether the explanations reflect sound underlying reasoning of the answer. Worker 0 is more lenient whereas 1 and 6 are stricter.The expert evaluated 630 explanations in total. For each annotator, we sampled roughly 80 explanations, equally split across each label. Overall, our expert judge agreed with 87% of the explanations finding the granularity of the missing information to be plausible. Table 9 presents percentage of explanations with which our judge agreed for each annotator.Some sources of disagreement in the remaining 13% were 1) instead of critiquing the answer, annotators wrote the "correct" answers instead; 2) politically contested topics that led to the reasoning in explanations not being supported by the document; 3) differences (between the judge and the annotator) in reading of the question and differences in inference from what was provided in the document.Table 10 shows distribution of labels for three annotators. We see variations in annotator behavior, where annotator 0 is a little more lenient compared to annotator 1 and 6.An important component of our proposed rescaling prompt in Section 5 is the aspect specific scoring rubric which is designed post-annotation. To create the proposed rubric, authors of this paper examined 20 human annotations, which included explanations and Likert labels. They assigned point deductions by taking into account factors that surfaced only with annotator's explanations after their annotation.For instance, explanations under the label missing minor, frequently highlighted the absence of certain names in the response or the omission of multiple small details. Likewise, within the label category labeled as missing major, explanations commonly emphasized the role of the absent information in completing the response. Some explanations explicitly stated that "the response is lacking half the information" along with references to the specific sentence numbers that were missing. This information at a granular level provided the authors with valuable insights into the variations in errors within each label category, which further helped refine point deductions for the final rubric.The following is a variation of the prompt without the scoring rubric:The main goal of your task is to score a machine generated response to a question. Scoring is on the "completeness" attribute. A complete answer will have all relevant information from the article required to answer the question and an incomplete answer won't. However, you are not directly scoring the machine response, but instead using a given feedback and amount of missing information.The details are given below. Article on which the question was asked: 'article' Based on the above article, the following question was asked: 'question'article: New YorkCNN -Watching LVII in person will be cheaper than last year's game, but it will still cost you thousands of dollars. StubHub said that the average price customers are paying for a ticket Thursday was $6,800. The cheapest seats to Sunday's match up between the Philadelphia Eagles and Kansas City Chiefs are selling at about $3,200 apiece.Overall prices for the big game in Glendale, Arizona are trending down with prices dropping each day. In fact, StubHub said that ticket prices have dropped more than 10% since the teams were decided on February 12. Attending this year's game is costing fans substantially less than last year's match up between the Cincinnati Bengals and Los Angeles Rams at LA's SoFi Stadium. That's to be expected, since the home team was playing in their own city and the country's second-largest metropolitan area. Tickets were averaging nearly $10,000 each, StubHub. SeatGeek another ticket selling website, shows ticket prices for Sunday's game slowly declining and were averaging about $6,500 on Thursday. The most expensive ticket on its website costs $30,000 and is located near the field; the cheapest is $4,200. At an average of $445 per night, however, hotel prices in the Phoenix area will be relatively expensive, making it the second-highest level for a Super Bowl week, according to STR, a hospitality analytics firm that tracks prices. The most expensive Super Bowl was in 2016 in San Francisco when rooms averaged $451 per night. In addition to the Super Bowl, the city is also hosting a PGA event that features golf's biggest players, including Rory McIlroy and Jordan Spieth. Those two events on the same weekend puts the local hotel occupancy rate to 94%, the firm said. The normal rate for a hotel room is around $160 per night, STR said. After reading sentence:'New YorkCNN -Watching Super Bowl LVII in person will be cheaper than last year 's game , but it will still cost you thousands of dollars.', a reader asked the following question. Q:Why is the Super Bowl cheaper this year? A:article: New YorkCNN -Watching LVII in person will be cheaper than last year's game, but it will still cost you thousands of dollars. StubHub said that the average price customers are paying for a ticket Thursday was $6,800. The cheapest seats to Sunday's match up between the Philadelphia Eagles and Kansas City Chiefs are selling at about $3,200 apiece.Overall prices for the big game in Glendale, Arizona are trending down with prices dropping each day. In fact, StubHub said that ticket prices have dropped more than 10% since the teams were decided on February 12. Attending this year's game is costing fans substantially less than last year's match up between the Cincinnati Bengals and Los Angeles Rams at LA's SoFi Stadium. That's to be expected, since the home team was playing in their own city and the country's second-largest metropolitan area. Tickets were averaging nearly $10,000 each, StubHub. SeatGeek another ticket selling website, shows ticket prices for Sunday's game slowly declining and were averaging about $6,500 on Thursday. The most expensive ticket on its website costs $30,000 and is located near the field; the cheapest is $4,200. At an average of $445 per night, however, hotel prices in the Phoenix area will be relatively expensive, making it the second-highest level for a Super Bowl week, according to STR, a hospitality analytics firm that tracks prices. The most expensive Super Bowl was in 2016 in San Francisco when rooms averaged $451 per night. In addition to the Super Bowl, the city is also hosting a PGA event that features golf's biggest players, including Rory McIlroy and Jordan Spieth. Those two events on the same weekend puts the local hotel occupancy rate to 94%, the firm said. The normal rate for a hotel room is around $160 per night, STR said. After reading sentence:'Overall prices for the big game in Glendale , Arizona are trending down with prices dropping each day.', a reader asked the following question. Q:How do these trends compare to other Super Bowl sales in previous years? A:While reading the article below: New YorkCNN -Watching LVII in person will be cheaper than last year's game, but it will still cost you thousands of dollars. StubHub said that the average price customers are paying for a ticket Thursday was $6,800. The cheapest seats to Sunday's match up between the Philadelphia Eagles and Kansas City Chiefs are selling at about $3,200 apiece. Overall prices for the big game in Glendale, Arizona are trending down with prices dropping each day. In fact, StubHub said that ticket prices have dropped more than 10% since the teams were decided on February 12. Attending this year's game is costing fans substantially less than last year's match up between the Cincinnati Bengals and Los Angeles Rams at LA's SoFi Stadium. That's to be expected, since the home team was playing in their own city and the country's second-largest metropolitan area. Tickets were averaging nearly $10,000 each, StubHub. SeatGeek another ticket selling website, shows ticket prices for Sunday's game slowly declining and were averaging about $6,500 on Thursday. The most expensive ticket on its website costs $30,000 and is located near the field; the cheapest is $4,200. At an average of $445 per night, however, hotel prices in the Phoenix area will be relatively expensive, making it the second-highest level for a Super Bowl week, according to STR, a hospitality analytics firm that tracks prices. The most expensive Super Bowl was in 2016 in San Francisco when rooms averaged $451 per night. In addition to the Super Bowl, the city is also hosting a PGA event that features golf's biggest players, including Rory McIlroy and Jordan Spieth. Those two events on the same weekend puts the local hotel occupancy rate to 94%, the firm said. The normal rate for a hotel room is around $160 per night, STR said. I had a question at sentence:'New YorkCNN -Watching Super Bowl LVII in person will be cheaper than last year 's game , but it will still cost you thousands of dollars.', Why is the Super Bowl cheaper this year? A machine responded with the following answer: 'answer' Feedback given to the machine response: 'feedback' Level of the missing information: 'label definition' On a scale of 0-100 how will you score machine response using the feedback and level of missing information stated above? Give a number.It follows the same structure as the rescaling prompt in Figure 3 without the rubric.All models used in this work are accessed through OpenAI APIs3 . The total cost of prompting LLMs in Table 6 for getting answers to INQUISITIVE-BROAD is ∼$30. All three models were prompted with the same set of articles and questions.The total cost of prompting GPT-4 with EBR and the variations is ∼$500.4 Table 11 shows the average score per label across all annotators using EBR for INQUISITIVE-BROAD. Although we do not impose any constraints regarding a score range per label when applying this method to each example, the average of all scores for a label naturally aligns with the overall label order.Label: Missing Minor, EBR: 85 1 Explanation: The response is somewhat correct, but misses some relevant info Missing Sentences: 9 2 Explanation: The machine response answered the question correctly but missed some relevant information. It would be useful to include the fact that Argentinian passport holders can enter 171 countries visa-free since the machine response mentions Russians can enter only 87 countries visa-free. Including that fact in the machine response without additional context does would not make much sense. Missing Sentences: 10 3 Explanation: The response did not mention the street name as per sentence 5.Missing Sentences: 5Label: Missing Minor, EBR: 75 1 Explanation: The machine response answered the question correctly but had its last sentence in its response cut off and missed relevant information. The response should include that the fake documents were issued to allow the women to settle in Argentina.Missing Sentences: 23, 2 Explanation: The article doesn't provide a full answer to the question but contains some relevant detail. Missing Sentences: 12, 29 3 Explanation: Info from doc. sentence 3 is used to appropriately make the first sentence of the answer. The second sentence draws from doc. sentence 12. The final answer sentence is not relevant as the question is not asking how they are able to travel there. Info from sentence 17 which discusses fleeing the war and getting access to better health care. Missing Sentences: 17Label: Missing Major, EBR: 50 1 Explanation: the machine response states what the helicopters were doing but fails to properly answer why they were operating (to combat rebel forces as indicated in sentences 15 and 35). Missing Sentences: 15, 35 2 Explanation: The machine response missed the question and provided an irrelevant response by restating the information in the source sentence. The article does not state exactly when horticulture was a priority for Americans but does mention an ongoing decline from peak membership in a horticultural association from the 1960s. Missing Sentences: 24, 25 3 Explanation: The response is partially correct and failed to mention the domestic factors as per sentences 4,7,8,13,14. Missing Sentences: 4, 7, 8, 13, 14Label: Missing Major, EBR: 40 1 Explanation: the machine response missed the question and provided a mostly irrelevant response. line 27 explains that david hockney is 'one of the living masters of oil painting' and that was not present in the response explaining why hockey is britain's most celebrated living artist. additionally, the machine response conflates information presented in the article. the variety of mediums was taken from line 6 but hockney did not create all of those works as explained on line 32. 2 Explanation: The text does not provide a specific answer, but a lot of detail could have been included in an attempt to address the question. Missing Sentences: 17, 18, 30 3 Explanation: the answer makes use of sentence 1, but misses the specific people named in 3, 6, and 13:sergei grigoryants, vladimir oivin, marina shemakhanskaya.We give examples of explanations that get rescaled to the same score. These explanations are for different questions and by different annotators. All the explanations are fine-grained, pointing sentences that were missing from the machine response, along with the level of severity of this missing information.subjectivity), may pose challenges that are difficult to overcome. Additionally, further investigation is needed to determine the specific types of explanations that should be sought from annotators to facilitate more faithful rescaling.Our proposed technique relies on a carefully curated scoring rubric. As outlined in Section 5, this process involves post-annotation analysis of the human judgment, which can be Table 13 : Human judgment along with EBR and reference scores for the question Why is inflation expected to remain elevated? time-consuming, although this kind of "prompt engineering" is a fixed cost regardless of dataset size. We consider the discovery of such rubrics from human judgments as a potential avenue for future research.Our analysis is also constrained by limited scale. Acquiring human annotations is expensive, which is why our study is restricted to one dataset and one aspect. However, we believe that the proposed method can be extended to various aspects and tasks that necessitate nuanced evaluation. Our method provides a template for future work, particularly if task designers are sensitive to the requirements of explanations for our method and infuse this understanding into the annotation process itself.How long did it take to end? Table 14 : Human judgment along with EBR and reference scores for the question How long did it take to end?The label distributions in the two data splits is similar, suggesting that leakage has no noticeable impact. Thus all results are reported on the entire INQUISITIVE-BROAD.The judge did not see annotators' Likert ratings on the QA instances, since we believed that would shift their focus to the more subjective question of whether the explanation justifies the rating.Pricing information for the models used can be found here.This includes multiple runs and prompt tuning on a small set of examples 


Decision: This paper uses language models (LMs) to rescale human quality judgments via annotator explanations. The method addresses the importance of human annotations for model adaptation and evaluation as models become more capable. It can calibrate differing human interpretations, refine coarse ratings, and apply rubrics post-annotation. Using a dataset of 12.6k annotations, the method was tested on a QA task, successfully rescaling judgments while maintaining agreement and aligning closer to expert evaluations.

Reviewers have mainly raised concerns about the validity of using a fine-grained 0-100 scale instead of a narrower range (e.g., 1-10).

In NLP, the 0-100 scale annotation has been primarily proposed and utilized in the machine translation community, supported by some advantages from a statistical point of view. It can capture subtle differences in the outputs, better aligning with human preference. 
Relevant works worth mentioning in the paper include:
- Graham et al. (2013): "Continuous Measurement Scales in Human Evaluation of Machine Translation"
- Bojar et al. (2016): "Findings of the 2016 Conference on Machine Translation"
- Sakaguchi et al. (2018): "Efficient Online Scalar Annotation with Bounded Support"

We expect the authors to address other concerns and questions in the final draft.

Reviews: review: paper_summary: paper_summary: This paper proposes to use LMs to rescale human quality judgements by using explanations provided by the human annotators. The authors motivate the work by stressing the importance and value of human annotation in today's NLP landscape for model adaptation and evaluation, especially as models become more capable and automatic metrics are no longer reliable for increasingly subjective or complex tasks. The proposed automatic rescaling method can be used for a few purposes: to calibrate humans who have different interpretations of numeric rankings, to take a coarse-grained rating and make it more fine-grained (such as on a 100-point scale, as the authors propose to do), and/or to apply or specify an arbitrary rubric or criterion *after* the annotations are collected. The authors first collect a dataset of 12.6k high-quality human annotations of labels with explanations (5 per instance). They then test their rescaling method on a document-grounded non-factoid QA task where the goal is to rate "completeness" of a model-generated answer. They find that, despite the subjectivity of the task, their method can 1) rescale judgements while retaining the same level of human agreement as the original data annotations, and 2) closes some of the gap with expert rescaling judgements made using the new rubric, indicating that the method is at least partially successful.
strength_weakness: strength_weakness: - A few methodological details could be expounded on or are limited in scope. The main one is a limited choice of models investigated. I elaborated on these below in "questions/comments". Overall, I don't think that these issues affect the contribution and validity of the paper, just constrain its scope a bit.
- Ultimately, the proposed metric is not extremely strong. The main results (Table 3) support the inclusion of a rubric, but the avg. EBR baseline with a rubric is quite competitive, which may be support for the fact that the 1-100 Likert Scale is too fine-grained (as I've also touched on in "questions/comments" below). I'd appreciate if the authors could elaborate on their hypotheses for why this may be or their arguments against using the average approach.
- I assume the authors will publicly release their dataset with publication, but if not, that will severely limit the impact of the paper.
questions: questions: Questions/comments:
- Will you release your code & dataset upon publication?
- The motivation for using such a granular scale (1-100) could be made stronger. It appears from the given example and reported human MAE that a 1-10 or 1-20 scale would be sufficient and that the super granular scale may invite noisy disagreement (for both humans and machines). Could you explain this choice a bit more?
- Elaboration on "limited choice of models" mentioned above: Only one model (GPT4-0613) is studied for its ability to do the rescaling task, so its not clear how the correlation w/ human experts would change for other models and thus if this technique would be useful or valid for them, which is an important question for those doing non-API research. Similarly, only OpenAI API models (though 3 of them) are used for the initial question answering task.

Related work:
- there are some interesting papers in psychology literature on the Likert Scale and humans' varying interpretations or calibrations; may be good to cite.

Typos/misc:
- many of your citations are not in parentheses but should be (i.e., by using `\citep{}`).
- please make Figs 2,4 larger given extra space in a camera-ready version.
- abstract: what do you mean by "propose a method to rescale ordinal annotations *and explanations* using LLMs"? seems like it should say "using explanations"
- Fig 1: would be good to also show the 3 expert scores here to get a sense of variance and that there is no 100% agreed ground-truth.
- Seciton 6.2 last paragraph: here $M$ are the (unaveraged) individual reference scores, correct? It's a little confusing to double up on this $M$ notation.
- Important to mention that being able to use an arbitrary scoring metric for rescaling *after* annotations are collected relies on the explanations containing enough detail for that particular aspect to be able to do so.
paper_summary: paper_summary: This submission proposes a method to rescale coarse numerical ratings to fine-grained ones. The envisioned scenario is that for a task like document-based question answering, annotators asked to rate answers have given coarse-grained ratings on a four-point Likert scale ("complete", "missing minor", "missing major", "missing all"), along with short free text justifications of their rating. Noting the inherent subjectivity of these ratings and justifications, the authors assume that a finer-grained numerical scale from 0 to 100 points is more appropriate. To rescale from coarse to fine ratings, the authors prompt LLMs with the rating, justification, and an instruction that describes how to assign scores.

The evaluation of the rescaled scores focuses on the two middle labels ("missing minor" and "missing major") since the extreme points of the scale ("complete", "missing all") show little variation among raters. The evaluation is mainly performed by measuring agreement of LLM-rescaled scores with a rescaling performed by three experts on 145 instances. Agreement is measured via rank correlation (Kendall's Tau) and mean absolute error (MAE). In terms of rank correlation, the proposed method fails to outperform one of the simple baselines, namely the "missing sentence heuristic". In terms of MAE, the proposed method outpeforms all baselines, but unlike for rank correlation, there are no significance scores given, which is a concern considering the small sample size of 145 instances.
strength_weakness: strength_weakness: 1. The proposed method is not motivated well, to the point that it is fundamentally unclear why and how it should address the research problem, for two reasons:

1.1 It is left unclear why a finer rating scale is better suited to capture the inherent subjectivity of the task at hand. The only motivation given is a reference to Kocmi & Federmann (2023), "which validated the effectiveness of a 0-100 scale compared to other options." What "effectiveness" means and if/how their results from machine translation apply to ratings of answers in QA tasks is unclear. If "subjectivity" in the task at hands means that different annotators have different views of what information is important in a document, then their ratings have no monotonic one-dimensional order and no amount of granularity can fix that.

1.2. According to the introduction, "subjectivity (and thus, inherent disagreement) in annotated data has surfaced as a key direction" and the authors make the observation that "intricacies in human judgments can be captured by natural language explanations provided during data annotation, which capture more nuanced subjectivity". So far so good, but now the authors claim that their "key idea is to make the ordinal label space (e.g., a coarse Likert scale) fine-grained, namely a 0-100 scale Kocmi & Federmann (2023), which then enables us to place the initial annotations in it by leveraging natural language explanations in a principled manner."
In what manner the proposed approach is "principled" is not clear to me. The authors are possibly referring to the scoring rubric that is included in the prompt, but just because some instruction is included in the prompt does not mean that the LM adheres to it, and it is even less clear if the LM faithfully can execute complex instructions that require it to repeatedly subtract a variable number of points, with the amount of points varying on the amount of information that is missing from the answer to be rated.

2. The benefit of the finer-grained ratings is not shown. The evaluation stops at showing moderate agreement with expert-produced fine-grained ratings and stating that the rescaling does not impact inter-annotator agreement. The introduction claimed that the proposed method can "capture more nuanced subjectivity", but this is never directly shown.
questions: questions: - While the intention is clear, the "Rescaling Prompt" shown in Figure 3 does not specify what the initial score is from which points should be deducted

- It is also not clear how information is measured, so a phrase like "half of the correct information" is left wide open to subjective interpretation
paper_summary: paper_summary: The paper seeks a way to improve human evaluations of subjective tasks via large language models (LLMs). In particular, the authors exploit the natural language explanations to rescale ordinal annotations. The authors propose a method called Explanation-Based Rescaling (EBR) that utilizes natural language explanations provided by human annotators alongside their Likert scale ratings. EBR leverages an LLM to convert both the rating and explanation into a more fine-grained numeric score (0-100) guided by a post-hoc defined scoring rubric. The authors demonstrate the effectiveness of EBR on a document-grounded question answering task where LLMs achieve near-human performance. They show that EBR produces scores closer to reference scores generated by experts using the same rubric, without impacting inter-annotator agreement.
strength_weakness: strength_weakness: W1. Achieving high agreement among human evaluators for subjective tasks is a significant challenge. As shown in Table 2, reaching a consensus among human annotators on missing majors seems to be particularly difficult. Given this, using the results of human annotators as references may not be the most effective method for evaluating the capabilities of LLMs on this task. While I don’t have a clear solution for this issue, it’s certainly a challenging area that requires further exploration.

W2. It was unclear whether fine-grained labels are necessary and how they could be helpful in improving LLMs or for NLP overall. 

W3. Regarding the evaluation, the authors mention that the work aims to demonstrate the usefulness of using NLEs to enhance rescaling performances. To achieve this, the experiment should be designed to allow for a comparison between sets with and without explanations. The current setup does not directly evaluate this effect.

W4. The evaluation is centered on a specific task of document-grounded question answering. Further investigation is needed to assess the generalizability of EBR to other tasks and domains. Specifically, in the presented task, the quality of the answers can be determined based on the missing sentences, which are relatively straightforward to quantify. However, in other scenarios, such clear criteria may not exist. 

W5. Lastly, the paper acknowledges that the creation of the rubric is currently a manual process. This could be time-consuming and subjective, potentially affecting the generalizability of the approach.
paper_summary: paper_summary: This paper presents a method of using LLM to convert human Likert ratings to scalar values using the human explanations. 
They first collected the Inquisitive-board dataset. The dataset consists of news articles and questions written by a linguistics student, and answers from LLMs. Then they ask annotators to evaluate the generated answers on how much information (major/minor/none) from the original articles the answers are missing, provide an explanation and how many sentences are missing. They found some disagreement in the Likert scale (major/minor/complete) ratings

They then provide some rubrics for converting the Likert scale ratings and explanations to a score in 0-100 for how much information is missing. They have 3 experts doing this rescaling for reference, and evaluate how well LLMs can do this rescaling. They found that the LLM rescaled values correlate with expert rescaled values and explanations provide fine-grained information beyond the Likert scale.
strength_weakness: strength_weakness: The biggest question I’m left with after reading this paper is: what does one do with this method? The amount of missing information is a limited dimension of a generated answer. Can this method be extended to reconcile multiple and potentially conflicting dimensions? Since this paper is in the evaluation track, do the authors recommend using this method for future LLM evaluation? Does using the rescaled values as an evaluation metric leads to different models being preferred and are those models indeed better in some ways? Without clear takeaways/recommendations the impact of this paper is likely limited.
questions: questions: I suspect the extent of information missing probably also correlate with the location of the missing sentence, or more broadly the information structure. Generally the information in the beginning of the article is probably more prominent and major than the information at the end of the article. The question being asked will also impact which information is prominent. Have you looked at this in your analysis?

Comments: title: Response
comment: Thanks for the response, and glad to hear you will be releasing the dataset!

As for the motivation for the 0-100 scale: it is still not clear to me why such a large scale was selected, though I can see that this is not a crucial design choice. I'd encourage you to make this a little more clear in the paper (that a scale with so many intervals is not needed, and the same results could (likely?) be achieved with a scale with fewer intervals but no multiple-of-5 limitation).

>While Avg EBR is comparable, it is still a four-point scale. It lacks the fine-grained aspect of evaluation which our approach enables us to achieve. EBR gives us the ability to tease out differences in human judgment and be more nuanced which otherwise is not possible with a four-point scale.

I may be misunderstanding Table 3, but doesn't it follow from a comparable MAE between avg. EBR and EBR, meaning that each method's error in aligning with human judgement scores (0-100) is about the same, that either 1) a more fine-grained scale *does not* give the ability to tease out differences in human judgment, or 2) the human judgements do not have such nuances? I see nuanced human judgements in Figure 4a, so I assumed it was the former.

I will keep my score.