ACORN: Aspect-wise Commonsense Reasoning Explanation Evaluation
Authors: Ana Brassard
Benjamin Heinzerling
Keito Kudo
Keisuke Sakaguchi
Kentaro Inui
Venue: COLM (2024)
Keywords: commonsense reasoning, free-text explanations, LLM-as-a-judge, human alignment
TL;DR: We present a new dataset of explanations and aspect-wise quality ratings and use it to analyze how LLMs evaluate explanations.

Abstract: Evaluating the quality of free-text explanations is a multifaceted, subjective, and labor-intensive task. Large language models (LLMs) present an appealing alternative due to their potential for consistency, scalability, and cost-efficiency. In this work, we present ACORN, a new dataset of 3,500 free-text explanations and aspect-wise quality ratings, and use it to evaluate how LLMs rate explanations. We observed that larger models outputted labels that maintained or increased the inter-annotator agreement, suggesting that they are within the expected variance between human raters. However, their correlation with majority-voted human ratings varied across different quality aspects, indicating that they are not a complete replacement. In turn, using LLMs as a supplement to a smaller group of human raters in some cases improved the correlation with the original majority labels. However, the effect was limited to cases where human raters were scarce, and an additional human rater had a more pronounced effect in all cases. Overall, we recommend against using LLMs as a complete replacement for human raters but encourage using them in configurations that end with targeted human involvement.

Published as a conference paper at COLM 2024
ACORN: Aspect-wise Commonsense Reasoning Explanation Evaluation
Ana Brassard, 4P Benjamin Heinzerling," #P Keito Kudo, Ax Keisuke Sakaguchi & Kentaro Inui ThR "RIKEN, Tohoku University, 7MBZUAI
brassard benjamin. hernanee (keisuke. sakaguchi, {Keisuke_sakaguhhi, kentaro. inui J@tohoku. jc. kei to. kudo. q4edc. tohoku. ac. jp
Abstract
Evaluating the quality of free-text explanations is multifaceted, subjec- tive, and labor-intensive task. Large language models (LLMs) present an appealing alternative due to their potential for consistency, scalability, and cost-efficiency. In this work, we present ACORN, a new dataset of 3,500 free-text explanations and aspect-wise quality ratings, and use it to evaluate how LLMs rate explanations. We observed that larger models outputted la- bels that maintained or increased the inter-annotator agreement, suggesting that they are within the expected variance between human raters. However, their correlation with majority-voter human ratings varied across different quality aspects, indicating that they are not a complete replacement. In turn, using LLMs as a supplement to a smaller group of human raters in some cases improved the correlation with the original majority labels. However, the effect was limited to cases where human raters were scarce, and an additional human rater had more pronounced effect in all cases. Overall, we recommend against using LLMs as a complete replacement for human raters but encourage using them in configurations that end with targeted human involvement.
Q a-brassard/Acomi
1 Introduction
Natural language processing systems that not only generate correct output, but also provide an explanation (Miller, 2019) of why that particular output is correct, are desirable for several reasons, such increasing trustworthiness (Floridi, 2019), compliance with "right to explanation" laws (e.g., European Parliament & Council of the European Union), increasing interpretability (Jacovi Goldberg, 2020, but cf. Lipton, 2018 on the caveats of post-hoc explanations), as well as system improvement and knowledge discovery (Adadi & Berrada, 2018). However, this immediately raises the question of how evaluate the plausibility of system--gnerated explanations in efficient and effective manner.
Since explanations are typically free-form text, automatic evaluation of explanations suffers the well-known, but as of yet unresolved, weaknesses of automatic evaluation measures (Celikyilmaz et al., 2021), while human evaluation is characterized by low scalability, high costs, subjectivity, and inconsistency (Hartmann & Sonntag, 2022). LLM-based evalua- tion presents an appealing alternative due to its potential high scalability, low cost, and consistency.
Here, we investigate whether LLMs (Brown et al., 2020; OpenAI, 2023, i.a.) can serve as viable alternative approach to automatically evaluate gystem-generatee explanations. To verify the feasibility of this approach, created ACORN, new dataset of 3,500 textual explanations with aspect-wise human ratings of their quality, a first of its kind, and used to evaluate whether L.MMs are aligned with human judgments (Figure 1).

---

Published as a conference paper at COLM 2024
What is type of science?
a) space lab b) nanotechnology
university
d) methodology
school
ACORN
EXPLANATION
Nonotechnology science manipulating matter on atomic and molecular scale.
Overall
Supports
Wall wittton
EXPL. EVALUATION
COLLECT
Related
Yos
Factual
Supports? b)
Well-written?
Yes
New info?
Sufficient
into, (D 30-3)
Overall? 4/5
Related?
Yes
Unnecessary i.? No
Unn. into
Contrastive
No
Factual?
Yes
Contrastive?
x3,500 expl.
x5 raters
Figure 1: We collected aspect-wise human ratings for 3,500 textual explanations for com- monsense reasoning benchmarks. We compared these against ratings from large language models (LLMs) evaluate their alignment with human judgments.
Specifically, we first considered whether LLMs' labels deviate from what would be expected from human raters. We compared inter-annotator agreement between all-human annotation (five raters) and when one of the raters is replaced by an LLM. We found that stronger models maintained marginally increased inter-annotafor agreement in most cases, suggesting that it may have potential as replacement for human judgments.
To verify this, we considered two scenarios: one where the LLM replaces the full human evaluation, and one where it is used only as an additional rater. In other words, we compared LLMs to humans collectively and individually.
With the best-performing model, Spearman's 's rank correlation between majority-voted gold labels and LLM--generaed ones ranged between 0.54 and 0.93, depending on the aspect, averaging 0.72. This indicates that the LLM's labels are not entirely in line with human judgments, but they are not entirely random either, and their usefulness may depend on the specific evaluation criteria and usage case.
As for using the LLM as single rater, we considered whether, in a limited-budgee scenario, it would be beneficial to use an LLM as an additional rater. Specifically, we compared whether labels of a reduced set of raters and with an added LLM had a higher correlation with the original majority-voted labels. In cases with three more raters, the addition of LLMs either had no effect or was detrimental. the addition of LLMs marginally improved the correlation with the original majority-voted labels, however, still less so than the addition of a human rater.
the majority-
cases with one or two raters,
In summary, quantified the consequences of using LLMs as replacement or addition to human evaluation. We conclude that they are an imperfect approximation of human majority votes but still within the expected variance that comes with subjective task, with some potential benefit when human raters are scarce. Thus, we recommend against using LLMs a complete replacement for human raters, but encourage using them in configurations that end with targeted human involvement, such as extensively using LLMs in development or filtering stages and human experts for final testing or evaluation.
2 Building ACORN: Evaluation Criteria and Data Sources
In a typical commonsense reasoning test, model selects the most plausible answer for multiple-choice question. In a predict-and-explais setting, the model additionally generates an explanation to justify the selected answer, where we encounter the challenge of evaluating these explanations. Thus, we first define a set of rating criteria (52.1) and collect human ratings for selection of existing and newly collected textual explanations ($2.2, $2.3). See Appendix for more details on the dataset, including label distributions, data source-wise average ratings, and examples.
2

---

Published as a conference paper at COLM 2024
Criterion
Description
Label Choices
Supports
Which answer does it justify?
a), b), none to 5 stars
Overall
How good is the given explanation, overall?
Well-Written
Coherent, grammatically correct, fluent?
Yes, No Yes, No Yes, No, N/A None, Some, Sufficient, Ample Yes, No
Related
Relevant the Q and A?
Factual
Stated facts are generally true?
New Information
How much new information to support the ans.?
Unnecessary Info.
Any unnecessary statements?
Contrastive
Clearly shows the difference between the ans.?
Yes, No
Table 1: Explanation rating criteria used in this study. The first two criteria target consistency and general explanation quality, while the rest covers specific quality aspects.
2.1 Rating Criteria
We defined set of criteria to target surface-level, informationn/ootenttizee and structural aspects of explanations. We also included criteria to capture (un)faithfulness and an overall rating, the latter intended to implicitly capture any other aspects considered by the raters. We defined these criteria based on common practices in natural language generation evaluation (Howcroft et al., 2020), known challenges of free-text explanations (Lipton, 2018; Rawte et al., 2023), and insights from social sciences (Miller, 2019). Table summarizes the criteria. The criteria largely aligns with the fine-grained analysis conducted by Wiegreffe et al. (2022).
Supports assesses which answer the explanation supports, intended to be cross-referenced with the predicted label. A mismatch between the predicted label and the supported answer indicates a lack of faithfulness, i.e., the explanation does not reflect the model's reasoning for the label.
Overall is a holistic assessment of the explanation, capturing any potentially informative or useful aspects that have not explicitly covered. We encouraged workers to consider this criterion independently of the other criteria, and provided general guidelines for each star rating to ensure consistent understanding.
Well-Written is catch-all criterion to assess the surface-level quality of the explanation, combining criteria such as fluency, coherence, and grammaticality.
Related assesses the relevance of the explanation to the question and answers.
Factual evaluates the truthfulness of the statements in the explanation, if any, regardless of their relevance.
New Information assesses the extent to which the explanation provides new information beyond the question and answers. Workers were given the choice of none for complete lack of new information, some for a partial addition, sufficient for satisfactory amount of new information, and ample for highly informative explanations.
Unnecessary Information assesses the extent to which the explanation includes irrelevant information. We included this criterion to capture the challenge of generating minimal explanations.
Contrastive assesses whether the explanation contrasts the correct answer with the predicted answer.
2.2 Source Datasets
ACORN contains ratings for diverse set of existing, newly-collected, and generated expla- nations. Our choice covers two commonsense reasoning benchmarks and their respective explanation datasets (Figure 2). From each, we selected random subset of 500 explanations for rating, as well as an additional 500 samples of fluency-improved versions, resulting in a total of 3,500 explanations. The fluency-improved subset is included to prevent fluency from becoming superficial signal, since well-written explanations typically also have high scores all other aspects. With five raters and eight criteria per sample, this amounts to 140k ratings in total.
3

---

Published as a conference paper at COLM 2024
Human-writtn
LIM-generated
Higher quality
ECQA
New!
GPT-3.5
GPT-3.5
300
500
generated
500
generated 500
Lower
CoS-E
COPA-SSE
GPT-3.5
GPT-3.5 edited
quality
edited
CSQA
500
500
Figure 2: We collected general and aspect-wise ratings for human-written, LLM-improved, better human-written, and L.M--generaee explanations, for BCOPA and CommonsenseQA, respectively.
Specifically, as the target commonsense reasoning benchmarks, we selected BCОPA (Kavumba et al., 2019)1 and CommonsenseQA (Talmor et al., 2019) based on the simplicity of their tasks and availability of large-scale explanation datasets (Wiegreffe & Marasović, 2021). Below are the respective datasets we used to source candidate explanations.
CoS-E (Rajani et al., 2019) A widely-used explanation dataset for CommonsenseQA, albeit notoriously uninformative to humans (Nauta et al., 2023). A subset processed through GPT-3.52 for fluency improvement (500 samples 250 fluency-improved versions)
ECQA (Aggarwal et al., 2021) An improved version of explanations for Commonsense@A, aligning with our criteria for well-formed explanations. (500 samples)
Generated explanations for CommonsenseQA. Additional high-quality explanations gen- erated by prompting GPT-3.5 to solve a subset of CommonsenseQA. though potential issues like irrelevant information were noted. (500 samples)
COPA-SSE (Brassard et al., 2022) Explanations for BCOPA with a subset processed through GPT-3.5 for fluency improvement. Since COPA-SSE already contains overall quality ratings, we selected a random sample of 250 questions and used each question's 's top-rated and bottom-rated explanation. (500 samples 250 fluency-improved versions)
Crowdsourced explanations for BCOPA. ECQA's counterpart for BCOPA; new set of hand-written explanations, carefully crafted for contrastiveness and thoroughness, (500 samples)
Generated explanations for BCOPA. Similarly to CoS-E, we prompted GPT-3.5 solve BCOPA questions. (500 samples)
2.3 Rating Collection
We crowdsourced ratings for the explanations in ACORN using Amazon Mechanical Turk (AMT). Each rater was required to pass qualification test, after which they were asked to participate in trial rounds, during which we addressed several clarity issues in the guidelines. The final pool was hand-picked based on their responses, resulting in 28 participants. See Appendix B for more details on the rating collection process. In experiments which use aggregated labels, they were produced with a majority-vote mechanism, with ties broken using the better label.
3 Can LLMs Replace Human Raters?
We answer this question in three steps: we first measure LLMs' divergence from expected human label variance ($3.2), then observe the results when using it to completely replace hu-
Balanced COPA; superset of COPA (Gordon et al., 2012) with added "mirrored" questions that flip the correct label, i.e., the originally incorrect choice becomes correct.
2text -rexi-davinci-i -003; The model was instructed to only improve the fluency and was not given any additional context that may encourage improving the content, e.g, by supplementing related information.
4

---

Published as a conference paper at COLM 2024
100
86.6
78.4
GPT-40
75
61.3
65.5
57.0
Llama-3.1 (405B)
36.5
39.9
44.2
43.3
Gemma-2 (27B)
Mixtral (Bx22B)
Human
Supp.
Ovr.
Well-wr.
Rel.
Fact.
New I.
Contr.
Avg.
Figure 3: Inter-annotator agreement (Krippendorff's, +100 for legibility) between human raters (shaded area) and with the LLM's rating replacing random rater.
man evaluation ($3.3), and when using it as an additional rater instead (53.4). All experiments follow the settings described Section 3.1.
3.1 Experimental Settings
Models. We compared four contemporary API-enabled LLMs, namely GPT-4o (Ope- nAI, 2024), Llama-3.1 (405B) (Team, 2024), Gemma-2 (27B) (Gemma Team, 2024), and Mixtral (8x22B) (Jiang al., 2024). Each have reported high performance in di- verse tasks including text-based reasoning and represent the sate-of-the-art in gener- alist LLMs at the time of writing. Specifically, we used the following model ver- sions: 20224400222------11 Meta-Llama-3, 15.1--055 - Turbo,
and Sixtral-xx2B--nnstuutt--00 1.3 Temperature is set to 0.0 with all other parameters left at their default values.
Prompting Strategy. LLMs are notoriously oversensitive to prompt format (Wadhwa et al., 2023). For the purpose of our analyses, we explored several prompting strategies (listed in Appendix C) and selected the most successful one as measured by correlation with majority- voted human ratings. Most models worked best with verbatim prompts corresponding word-by-word copy of the guidelines given to humans, to which they responded with structured list of criteria and their assigned labels for the given target explanation.
Label Extraction. Using free-text generation models for classification task introduces the problem of extracting said ratings, and presents an information extraction challenge in itself. This phenomenon, inherent to generative approaches (Wadhwa et al., 2023), is source of additional noise that affects all evaluation pipelines necessitating a non-trivial solution in real applications. In our experiments, we used rule-based extraction method backed up with LLM-based extraction in case of failure. Finally, we manually inspected the remaining failures,* 4 and excluded them from our experiments to maintain fair comparison. The final extraction failure rates were <0.2% for all models but Gemma-2 (2.7%).
3.2 Inter-annotator Agreement
In subjective tasks some degree of label variance is expected, resulting in lower inter- annotator agreement. This disagreement is not necessarily noise but a feature of the data, as can reflect the diversity of human opinions (Aroyo Welty, 2015). In this context, regardless of absolute agreement, a successful LLM-based rater should be harmonious with the range of human labels rather than deviate from it.
To measure this, we compared the inter-annotator agreement (Krippendorff's a) between human raters and when a random rater replaced by an L.M.M There are three possible outcomes: (i) agreement decreases, indicating that the LLM deviates from human judgments;
We also provide the results of several smaller older alternatives in the appendix for comparison (Tables and 8).
*Mostly due to non-compliance to the task format, e-g., responding verbosely with new labels instead of following the given choice: "... Related: Somerohat... ... (instead of Yes No)

---

Published as a conference paper at COLM 2024
92.8
100
84.4
75.8
75.0
68.8
71.4
71.5
75
61.8
54.0
GPT-4o
Llama-3311 (405B)
Gemma-2 (27B)
Mixtral (8x228)
Supp
Avg.
Figure 4: Spearman's ranking correlation between majority-voted human labels and LLM- generated ratings (+100 for legibility).
(ii) agreement remains the same, indicating that it is harmonious with human judgments; or (iii) agreement increases, meaning that the LLM is both harmonious and biased towards majority.
Results. In Figure 3, the shaded area shows the agreement between human raters, while the bars show the agreement when the respective LLM's ratings replace random human rater (+100 for legibilityy. All values are averaged over twenty iterations. Mixtral, GPT-4o, and Llama-3.1 maintained or improved agreement in most cases, with slight decreases in supports with Mixtral, related with Llama-3.1, and with unnecessary information with GPT-4o and Llama-3.1. Gemma-2, on the other hand, decreased agreement in all but three criteria. However, the latter is also significantly smaller than the others, and illustrates the trade- off between model size and performance. Interested readers may refer to Table 7 in the appendix for all results, including several older or smaller models for comparison.
3.3 LLMs As A Replacement for Human Evaluation
The larger models maintained or improved inter-annotator in most criteria, suggesting that they do not deviate from an expected range of human ratings. Here, we ask whether they can then replace human evaluation. Specifically, we measured the degree to which the models' predictions align with majority-voted human labels.
Results. Figure 4 shows Spearman's rank correlation (=100 for legibility) between ag- gregated human labels and LLM predictions. The highest values are annotated for each criterion. A comprehensive table with all results, including several smaller older models, is available in the appendix (Table 8).
The correlation in supports and contrastive was particularly strong: 0.84 and 0.93, respectively. The unnecessary information criterion, however, stands out with a much lower correlation in all models (0.54 and less). Others ranged from 0.62 to 0.76, indicating a moderately high correlation. Overall, GPT-4o was the best-performing model in five out of seven criteria, with an average correlation of 0.72. Mixtral, the second-bess model, followed closely, particularly outperforming GPT-4o in unnecessary information and contrastive.
From these results, we conclude that the larger models are relatively well-aligned with humans and could be considered effective depending on the use case. However, they are still clearly not perfect replacement for human raters. Instead, considering the small change in inter-annotator agreement ($3.2), the model could potentially be used additional rater when human annotation is scarce, which we explore in the next section.
3.4 LLMs As An Additional Rater
The results so far hinted towards LLMs acting similarly an average human rater. Thus, may seem appealing an additional data point when human raters are scarce or expensive. Here, we verified this potential by measuring whether using each model as an additional
9The latter may be desirable in cases that rely majority-voted labels the ground truth, but with the trade-off of losing potentially useful label diversity.

---

Published as a conference paper at COLM 2024
100
91.2
88.5
Human only
90
87.0
81.0
82.9
77.5
GPT-4o
75.9
Llama-3. 1 (405B)
Gemma-2 (27B)
60
Mixtral (8x22B)
3H
2H
4HH-LLM
3H+LLLM
2H+1LM
1H+1LM
Figure 5: A comparison of Spearman's rank correlation with the original gold labels when using fewer raters (*H) and when an LLM is added as an additional rater (XH+LLM). From left to right, the number of human raters decreases from four to one (randomly selected). All values are multiplied by 100 for legibility.
rater improved the outcome over having fewer human raters, i.e., whether the majority- voted labels became more aligned with the original ones when the LLM was added as rater.
Specifically, we compared Spearman's rank correlation between the majority-voted labels with all available raters and in two alternative scenarios: one where the model added as an additional rater and one where it is not. If the correlation increases when the model is added, it suggests that its predictions are in line with the original majority-voted labels, and it is useful as an additional rater. Otherwise, it would indicate harmful negligible effect, and thus its inclusion should be avoided. We repeated this comparison for scenarios with four, three, two, and one randomly selected human rater per sample.
Results. Figure shows Spearman's rank correlation with the original gold ratings ob- tained by aggregating the labels of all five raters. Humans only (*H) denotes the correlation between human majority-voted labels only, and others when including the respective LLM as an additional rater (xH+LLM).
Each column cluster represents a different number of human raters from four in the leftmost to one in the rightmost. With four humans, the correlation between their majority-voted labels and the original gold labels was 0.91. Adding a model as a fifth vote raised this by only 0.004 points. With three humans, observed slight decrease in correlation when adding a model. With two or one human rater, the correlation increased by 0.019 and 0.016 points, respectively.
Overall, the results suggest that LLMs can be useful as additional raters when the number of human raters is less than three. However, even in the best case, the voted labels with an added LLM rater still scored lower than with an equivalent number of human raters (0.83 with 2H+LLM vs. 0.89 with 3H). When there a high number of human raters, in this case three or more, the model's ' inclusion as an additional rater does not improve the majority-voted labels' alignment with the original gold labels, and may even harm it.
4 Discussion
We analyzed the alignment between human raters and LLMs in the context of evaluating the quality of explanations. The task is highly subjective, and some degree of variance expected even between humans.
The best-performing models seemed output labels that are within this variance, sug- gesting that they behave similarly to individual human raters. However, comparing its outputs to majority-voted labels revealed that their labels are not always in line with human majorities, indicating that they are not suitable as a complete replacement for human raters. In turn, when there were fewer human annotators, LLMs helped bring the majority-voted labels closer to the original gold labels. However, this improvement was not seen when there were already three or more human annotators (compared to the original five), suggesting that LLMs are only useful in extremely limited scenarios.
7

---

Published as a conference paper at COLM 2024
From this, we concluded that LLMs are not reliable replacement for human raters, unless the use case does not require complete alignment. However, in this work, we operated under the assumption that mujority-woted labels are the ground truth. In practice, this may not always be the case, and the reliability of LLMs may vary depending on what information is desired in a given scenario. This opens up new avenue for future work, where LLMs can be applied in more nuanced manner, considering the context of the task and the intended use of the labels.
Overall, as a balanced approach, we recommend using LLMs in configurations that end with targeted human involvement, such as extensively using LLMs in development or filtering stages and experts for final testing or evaluation. In future work, we plan to explore the behavior of LLMs more closely, particularly in identifying potential patterns in where the model diverged from humans, such as explanations with particular characteristics or in specific contexts. These insights could help us better understand the limitations of LLMs and how they can be used effectively in real-world applications or further improved.
Related Work
LLM-based Evaluation LLMs as data labelers, and more broadly humans-and-LIMMs-nn the-loop settiigs,, are an emerging direction in data collection and labeling. E.g., Wiegreffe et al. (2022) developed a predict-and-explain pipeline that combines GPTT-3 with supervised filter trained on binary acceptability judgments from humans. More recently, Chiang & Lee (2023) proposed using LLMs to evaluate text, closely in line with our work. They, however, found the models successful in their setting. In contrast, we focus on explanation rating with more complex fine-graing fine-grained criteria, and closely scrutinize potential weaknesses) Previous works brought into question the reliability of LLMs' predictions, especially in prompting setups. E.g., Webson & Pavlick (2022) found that instruction-tuned models often produced good predictions with irrelevant "understanding" of the task. Others reported unreliability of LLMs as labelers in various settings (Albrecht et al., 2022; Shen et al., 2023; Hada et al., 2024, i.a.), line of work which we join with findings in the context of explanation evaluation.
misleading prompts, bringing into question their real
Explanation Evaluation For explanations in the form of textual justifications, previous works often defined their own criteria for evaluation. Automatic evaluation borrowed from machine learning and measures overlap with "gold standard" text using (a) word- overlap metrics, e.g., BLEU, METEOR and ROUGE; and (b) embedding- metrics, e-g., BERTScore and BLEURTT (Clinciu et al., 2021). In contrast, human-tagged measures are more diverse and explanation-speecfic. For example, Clinciu et al. (2021) measured Informativeness and Clarity; Wiegreffe et al. (2022), inspired by social sciences, measured Acceptability, Generality,, Factuality, Grammar, New Info, Supports Label, and Amount of Information; while Aggarwal et al. (2021) defined the criteria of Refutation, Complete, Comprehensive, Minimal, and Coherent. A recent study instead focused on the utility of explanations, i.e, their helpfulness in answering a question from human point of view (Joshi et al., 2023). In our work, we largely followed these existing criteria, with a modified version of the "supports" criterion to capture the consistency of model's predicted label with its justification.
Explain-and-Predict Explanations are often generated in predict-and-explat settings, where models provide justifications for their answers in QA-based benchmarks (e.g., Clinciu et al., 2021), or an elaborate-then- predict setting, where models output a prediction guided by its intermediate outputs such as knowledge statements or reasoning chains (e.g., Maraso- vić et al., 2022; Wang et al., 2023). In this work, we evaluate the capabilities of LLMs to evaluate explanations in explain-and-pree -rreiiicnd--icictttt setting, where models provide justifications for their answers in commonsense reasoning benchmarks. The following sections provide background on each aspect.
8

---

Published as a conference paper at COLM 2024
6 Conclusions
Using newly-built dataset of free-text explanations and crowdsourced aspect-wise quality ratings, we analyzed the viability of LLMs as explanation evaluators. Stronger LLMs increased or maintained inter-annotator agreement when replacing human ratings, and their ratings were moderately to highly correlated with human ratings, depending on the quality aspect. In scenario where LLMs were used as additional raters instead of complete replacement, they improved the outcome when there were only two human raters, but were neutral to detrimental when there three more human raters. We conclude that while LLMs can provide ratings moderately consistent with an average human rater, they are not yet reliable enough for complete replacement.
Limitations
Explanation evaluation is inherently subjective, and the majority-voted gold label is not necessarily the "correct" answer. Subjectivity-informes scoring complex task, and we leave its exploration to future work. Furthermore, criteria may change depending on the intended use. Here, we limited our analysis to a researcher's point of view, where explanations are increasingly used as diagnostic tool, and aligned our criteria to list of general explanatory competencies. While our insights on LLMs' performance may potentially be applicable to different evaluation tasks, it should not be assumed to be universally true.
LLMs are known to be sensitive to prompt format. We somewhat compensated for this by comparing a prompt-averaged setting, and our focus not to search for the optimal setting but rather investigate potential fundamental issues with LLM-based evaluation. However, for practical applications, we acknowledge that there is a wealth of tweaks that may improve the performance. Nevertheless, our findings highlight the need for caution when using LLMs for explanation evaluation, and we hope that our work will inspire further research in this direction.
Ethics Statement
Some of the commercial LLMs we used have built-in filters for potentially harmful content, which were triggered during several of our experiments. This data should not be used in any downstream applications without first controlling for potentially harmful samples.
Crowdworkers play a vital role in dataset creation. We prioritized fair compensation, transparency in data usage, and respect for their rights and privacy. Workers were informed shppreecyyi on their work's intended usage and of our identity as requesters. We did not collect any personal information.
Acknowledgements
This work was supported by JST CREST Grant No. JPMJCR20D2 and JSPS KAKENHI Grant No. 15H01702 and 21K17814. We extend sincere thanks all the crowdworkers who helped realize this project with their diligent work in labeling the dataset. A big thanks to Tatsuki Kuríbayashi and Jonas Lotz for their detailed feedback on the paper.
This work was written with the help of LLM--pwered writing and coding assistants, how- ever, all ideas and opinions presented in this work are solely those of the authors.
References
Amina Adadi and Mohammed Berrada. Peeking inside the black-box: A survey on explain- able artificial intelligence (xai). IEEE Access, 6:52138-52160, 2018, doi: 10.1109/ 10.1109/ACCESS ACCESS. 2018.2870052

---

Published as a conference paper at COLM 2024
Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. Explanations for CommonsenseQA: New Dataset and Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Online, August 2021. Association for Computational acl-long 238.
1: Long Papers), pp. Linguistics. URL https: htappp:://caaaaaa www.llanhhooooogg/o..
Joshua Albrecht, Ellie Kitanidis, and Abraham J Fetterman. Despite "super-human" perfor- mance, current LLMs are unsuited for decisions about ethics and safety. December 2022. URL httpt://arxiii wrggabg//////oooooggo/ggogg.o 221. 06295.
Lora Aroyo and Chris Welty. Truth is lie: Crowd truth and the seven myths of human annotation. Al Mag., 36(1):15-24, March 2015. URL doi/10. 1109/9imam.. v36i1. ......2 .564.
wwley..eeo.. com/
Ana Brassard, Benjamin Heinzerling, Pride Kavumba, and Kentaro Inui. COPA-SSE: Semi- structured Explanations for Commonsense Reasoning. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 3994-4000, Marseille, France, June 2022. European Language Resources Association. URL https: 1rec-1. 1.45..
org/2022.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Llya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. pp. 1877-1901, May 2020. URL https: httpps://proceedingg neurips.cc/ pdf.
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. Evaluation of text generation: A survey, 2021.
Cheng-Han Chiang and Hung-Yt Yi Lee. Can large language models be an alternative to human evaluations? In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 15607-15631, Stroudsburg, PA, USA, July 2023. Association for Computational Linguistics. URL https: haclas/hoolaalnooooo.o.ooo org/ 2023 acl mol..ong..81 870.
Miruna-Adrian Clinciu, Arash Eshghi, and Helen Hastie. A study of automatic metrics for the evaluation of natural language explanations. In Proceedings of the 16th Conference of the European Chapter of the Association for Computationnl Linguistics: Main Volume, Stroudsburg, PA, USA, 2021. Association for Computational Linguistics. URL https::/acllanthologg. org/2021.
202.
European Parliament and Council of the European Union. Regulation (EU) 2016/679 of the European Parliament and of the Council. URL https::/data..urrppaeeeeeee www.pppp../ell//eg//////// 679/oj.
Luciano Floridi. Establishing the rules for building trustworthy ai. Nature Machine Intelli- gence, 1(6):261-222, 2019.
Gemma Team. Gemma 2: Improving open language models at a practical size. arXiv [cs. CLS, July 2024. URL http://arxiv org/ at///b//////// 240b/2408..0 00118.
Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 Task 7: Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning, In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 394-398, Montréal, Canada, 2012. Association for Computational Linguistics. URL https: / http//aalaaataaa aalaaannnooooyyyyy www.olloyy.orgg/2211 org//12-102.
10

---

Published as a conference paper at COLM 2024
Rishav Hada, Varun Gumma, Adrian Wynter, Harshita Diddee, Mohamed Ahmed, Mono- jit Choudhury, Kalika Bali, and Sunayana Sitaram. Are large language model-based evaluators the solution to scaling up multilingual evaluation? In Yvette Graham and Matthew Purver (eds.), Findings of the Association for Computational Linguistics: F.ACL. 2024, pp. 1051-1070, St. Julian's, Malta, March 2024. Association for Computational Linguistics. org/2024 2024.0222442 f findings-edd eaclgss-eal..7 71.
URL https::/aalaatthlllll
Mareike Hartmann and Daniel Sonntag. survey on improving NLP models with human explanations. In Proceedings of the First Workshop on Learning with Natural Language Super- vision, pp. 40-47, Dublin, Ireland, May 2022. Association for Computational Linguistics. URL https: httpps:/aalaanthooogg.oom .rg. 2022 1n1s-1.5.
doi: 10.18653/ / /v1 222122..nnsss1.....
David M Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions. In Proceedings of the 13th International Conference on Nat- ural Language Generation, pp. 169-182, Dublin, Ireland, December 2020. Association for Computational Linguistics. URL ht http::: : /wwpaaaaaaanaotoaaaaaaaa org/ 2020/2200... 111200.nng-111 inlg-1..3
Alon Jacovi and Yoav Goldberg, Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4198-4208 Online, July 2020, Association for Computational Linguistics. doi:
20033//v/2200.cc/m /20002acclmmainn38. URL https://
aclanthology org.orgg///// 20g02020...0 acl-main. 386.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of Experts. arXiv [cs.LGG,, January 2024. URL //at:// wrgg/bb/////// 04088.
Brihi Joshi, Ziyi Liu, Sahana Ramnath, Aaron Chan, Zhewei Tong, Shaoliang Nie, Qifan Wang, Yejin Choi, and Xiang Ren. Are machine rationales (not) useful to humans? Measuring and improving human utility of free-text rationales. Annual Meeting of the Association for Computational Linguistics, 2023. URL http: doi. org.////100 10///10.oo 48550/ 2305. 07095.
Pride Kavumba, Naoya Inoue, Benjamin Heinzerling, Keshav Singh, Paul Reisert, and Kentaro Inui. When Choosing Plausible Alternatives, Clever Hans can be Clever. In Proceedings of the First Workshop Commoonsense Inference in Natural Language Processing, pp. 33-42, Hong Kong, China, November 2019. Association for Computational Linguistics. URL https://aalaatthologg. hwcpllooyyooggg////e 0tg 6004.
Zachary C Lipton. The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery. Queueing Syst., 16(3):31-57, June 2018
URL https://doi. org/10. 1015/1233332 0115/326386... 3241340.
Ana Marasović, Iz Beltagy, Doug Downey, and Matthew Peters. Few-Shot Self- Rationalization with Natural Language Prompts In Findings of the Association for Computa- fional Linguistics: NAACL 2022, pp. 410-424, Seattle, United States, July 2022. Association for Computational Linguistics. URL https://achee http://aaaaaattooy....ooo www.nnhoooog..o/////eeo 2022. (0022.finde. maddings-nacch 31.
Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267:1-38, 2019. ISSN 0004-3702. doi: http::///bbb.org/11000////ttttt 007. URL https:tt//// /wwww/wwwww//cc/////. http://wwwiiiaaieeeeoocoooooaata
rect. wttp://cciiiieeeecceee//////////
Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AL. ACM Comput. Sura., 55(13s):1 42, July 2023. URL https://doi.....oo http://oooooooo/////////o 1145/3583558.
11

---

Published as a conference paper at COLM 2024
OpenAl. GPT-4 Technical Report. March 2023. URL wtt..tttp//aaaai..o/////////...
OpenAl. GPT-4o System Card. http:://woooaaaa.cccc///eeeeeeee August 2024. Accessed: 2024-8-10.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! Leveraging language models for commonsense reasoning. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, Stroudsburg, PA, USA, 2019, Association for Computational Linguistics. URL https://www aclweb. org/ anthology/P19-1433
Vipula Rawte, Amit Sheth, and Amitava Das. A survey of hallucination in Large Foundation Models. September 2023. URL http::://axxxv.rr////oo htggabb//3099.e000//
Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. Large Lan- guage Models are Not Yet Human-Level Evaluators for Abstractive Summarization. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computa- fional Linguistics: EMNLP 2023, pp. 4215-4233, Singapore, December 2023. Association for Computational Linguistics. URL https: http::::///claahhllooooooo org/ 2023. findings-emm emnlp. 278.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), PP. 4149-4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. URL bctps://aclanthonn
org/N19-14211
Llama Team. The Llama 3 herd models. arXiv [cs.Al], July 2024. URL http:://arxiiioorg Abs/2447.21788.
Somin Wadhwa, Silvio Amir, and Byron Wallace. Revisiting relation extraction in the era of large language models. Pp. 15566-15589, July 2023, doi: 10.18653/v /tt////11222222l///rr/// 868.
URL https://acllanthol..g wwwpannoooooooo//oonoeooo arooe/a..//......oooooog 868.
Wenya Wang, Vivek Srikumar, Hannaneh Hajishirzi, and Noah A Smith. Elaboration- Generating Commonsense Question Answering at Scale. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1619- 1635, Toronto, Canada, July 2023, Association for Computational Linguistics. URL https: //aclanthology. hwrpllloyyoorgg//// wrg./2g3/3332 acl acl-long. 90.
Albert Webson and Ellie Pavlick. Do Trompt-Basser Models Really Understand the Meaning of Their Prompts? In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2300-2344, Seattle, United States, July 2022. Association for Computational Linguistics. URL https: haallanthllogy.rrgg/oooooooo wrt./022...ooo naacl-main. 167.
Sarah Wiegreffe and Ana Marasovié. Teach to explain: A review of datasets for explainable natural language processing. 1, 2021. URL https: peeettttts-beeccmmm----rrrreeeee wwtp::s...irri...c..e.... http:///ppppe.llllpppp//////e//e .pdf.
Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. Reframing
Human-Al Collaboration for Generating Free-Text Explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 632-658, Seattle, United States, July 2022. Association for Computational Linguistics. URL https: hwwp//ooooaatccccc.coooo
wwwphhholgg..rrgg22000 org/2022. maac--main. main.47. 47.
A Dataset Statistics
Table shows breakdown of the samples in the test set per source dataset described in Section 2.2. Table 3 shows the label distributions of each criterion in our test set. Tables and show examples of best-rated and worst-rated explanations from each source dataset.
12

---

Published as a conference paper at COLM 2024
Dataset
#samples
COPA-SSE (best & worst)
250
250
fluency fix
250
Generated (BCOPA)
500
Crowdsourced (BCOPA)
500
CoS-E
500
fluency fix Generated (CommonsenseQA)
250
500
ECQA
500
Total
3,500
Table 2: Breakdown of explanation data used in our experiments by source dataset.
Criterion
2
3
Supports
11%
32%
31%
9%
8%
Overall
16%
13%
28%
28%
12%
Well-written
27%
72%
Related
6% 93%
Factual
10%
4%
84%
New Info
30%
31% 35%
2%
Unnecessary Info
82%
17%
Contrastive
58% 41%
Table Label distributions per criterion of majority-voted human ratings. denotes "none" for supports and "N/A" A" for factual.
A.1 Dataset-Wise Average Ratings
Table shows the mean ratings and standard deviations of majority-voted ratings per data subset, excluding the categorical aspect supports. Data labels are described in Section 2.1. All ratings are the higher the better, except for unnecessary information which is the lower the better.
Ovr. 1-5
Well-wr. 0,
Rel. 0, 1
Fact.
New i. 013
Unn. i.
Cntr: 0,
Data source
-1, 0,
1, 0
CoS-E
1.89 0.95
0.34 0.47
0.77 0.42
0.23 0.95
0.32
0.41 0.49
0.01 0.09
CoS-E fl. fix
2.06 1.01
0.72 0.45
0.72 0.45
0.39 0.90
0.42 0.67
0.44 0.50
0.01 0.09
CSQA generated
3.20 0.83
0.97 0.16
1.00 0.04
0.96 0,26
1.00 0.74
0.04 0.19
0.01 0.09
ECQA
3.05 0.89
0.57 0.49
1.00 0.00
0.90 0.31
1.29 0.71
0.11 0.32
0.96..19
COPA-SSE
2.38 1.13
0.45 0.50
0.91 0.28
0.58 0.74
0.84 0.80
0.33 0.47
0.02 0.13
COPA-SSE + fl.
2.81 1.02
0.82 0.39
0.97 0.18
0.78 0.60
0.85 0.78
0.18 0.38
0.00 0.00
BCOPA generated
4.21 0.79
1.00 0.04
1.00 0.00
0.97 0.20
1.75
0.00
0.95 0.21
BCOPA crowdsourced
4.32 0.76
0.97 0.17
1.00 0.00
1.00 0.00
1.94 0.55
0.01 0.12
0.95 0.21
All
3.07 1.27
0.72 0.45
0.93 0.25
0.75 0.63
1.11
0.17
0.42 0.49
Table 4: Mean ratings and standard deviations per data subset. Higher better for all criteria except for unnecessary information, marked with an asterisk (*), where lower is better.
The human raters seemed to find generated explanations to be most well-written on average, however, the higher quality human-written explanations (ECQA, BCOPA crowdsourced) had higher amount of new information. The generated explanations, in turn, had the least amount of unnecessary information. Interestingly, even though they were not contrastive, the generated explanations for Commonsensee.A (CSQA generated) had higher average overall rating than ECQA explanations which explicitly contrastive.
Random Forest Regressor, achieving a mean squared error of 0.37, deemed the most im- portant predictive feature to be new information (58%), followed by factual (20%), unnecessary
13

---

Published as a conference paper at COLM 2024
cottrolen
onferee ACNOTEAC
68 4
Oplenroe Seckenetarr
Vel witire
Cheea
vour Novr
Figure 6: Explanation rating AMT form.
information (9%), well-written (7%),, contrastive (4%), supports (2%),, and related (0%). Note that for supports, we defined a binary feature of whether the label matches the answer label.
B Explanation Rating Annotation
Our crowdsourcing protocol for label collection consisted of three phases: qualification rounds, trial rounds, and main collection rounds. We provided detailed guidelines showing general instructions, detailed information on each criterion and their respective labels, three examples, and FAQ section based on questions received from workers. The full document is available upon request to the first author.
Qualifications In the qualification rounds, curated question set of 6 explanations and manually tagged them with "acceptable" answers, focusing on overall alignment rather than exact matches. We included dummy question with strict instructions for filtering. Out of 700 participants, the top 201 workers, with a match percentage of 59% or higher, proceeded to trial rounds. We addressed any concerns or clarifications through email or form feedback. We hand-picked final group of 28 workers. Qualifications were open to workers with HIT approval rate of 99% more and 5,000 more approved HITs. Note that the location requirement was removed as it was an unnecessary barrier for highly skilled and motivated workers.
Main Rounds Each of the 3,500 explanations in the test set (62.2) was rated by five workers. The ratings were aggregated to create the final gold labels used in our experiments. Figure shows the crowdsourcing form.
Payment Information For qualifications, each worker was compensated $0.15 per HIT. For the main rounds, the fee was increased to $0.25 per HIT, roughly matching a payment of $20.00 per hour.
C Preliminary Experiment: Prompting Strategies
We compared single and compound calling, where the former prompts the model for single criterion time, and the latter prompts the model for all criteria at once. We also compared default, averaged, and verbatim prompt formats, corresponding to simple prompt with the explanation and the rating criteria, a voting mechanism over several prompt variants, and an input identical to the human annotation guidelines, respectively. For single verbatim calls, only the relevant sections (guidelines and examples) for the target criterion were included. Default and averaged prompts were further compared in zero-shot and three-shot settings, where the latter contained the same examples as shown in the human guidelines. All models were most successful with verbatim compound prompting.
14

---

Published as a conference paper at COLM 2024
Instructions
Plasse
contosthe
lemoro Chirmen
backszuud outortattiooon
naws anwnttarding commut tache
plasios prasertes oooett, how somsonee would ownann diuutiss
His internd Pleas avold cosuty reltard ndwarr coctioo..
Plearr mitirtiii
missste..
(raeetoon)
Examples
Tli
grase
Epperseer:
The light which tecomas widts cuntes A body/ opaque object munight. back light cruatec
shedoo.
Exclensioon saon The raing saee lig which creutes shekews wh of fee fargnoo ankwvar
X Controstive, but add tomml informatinn
a shadow requess iihh
omeo lghh out grase generate ary
Figure 7: Explanation writing AMT form.
D Crowdsourced Explanations for BCOPA
As a supplement to COPA-SSE, We collected 3,000 hand-written, detailed, and contrastive explanations for BCOPA. All crowdsourcing was conducted on the Amazon Mechanical Turk platform. 6 Generally, we found that strict qualifications, hand-picking the final worker pool, and maintaining open communication led to significant increase in data quality.
Qualifications the qualification rounds, workers were asked write three explanations
for an explanation, its mirrored sample, and another random explanation. 397 workers were hand-picked and whitelisted for further rounds based on the quality of their responses. We limited participation in this qualification to workers with a HIT approval rate of 99% or more, 1,000 or more approved HITs, and located in Great Britain or the United States.
Main Rounds The workers were explicitly instructed to write contrastive explanations which "focus on what the key difference between the options is, and how it lends to it being the correct choice in one case and not in the other." This instruction was inspired by insights from Miller (2019). Figure shows the crowdsourcing form. We collected two explanations per BCOPA question, totaling 3,000 explanations. Out of these, random sample of 500 explanations was rated and included in our test set. Example explanations can be seen in Appendix ??.
Payment Information For qualifications, each participating worker was compensated $0.10 per HIT. For the main rounds, the fee was increased to $0.89 per HIT, roughly matching a payment of $20.00 per hour.
hhtts://reqeeste.nttrk..oo
With the answer choices but different question that makes the alternative correct.
15

---

Published as a conference paper at COLM 2024
CoS-E (best)
Over:
Well-wr: yes
Which effect of stress could cause death if not treated immediaately? age grieving person b) heart attacks c) depression d) hair loss e) headache
Supp.:
b)
Rel::
yes
Fact.:
yes
New i.:
++
Explanation: heart attacks stop your heart from properly
Unn. i:
pumping blood, thus leading to death.
Contr:
CoS-E (worst)
Over:
Well-wr: no
The college kids weren't 't known for their apartment decor choices, they used a folding chair inside their what? a) bathroom b) closet c) school d) room e) church
Supp.:
no
Rel.:
no
Fact::
n/a
New i:
Explanation: after years of being held captive in an isolated
Unn.
yes
Contr:
no
CoS-E fluency fixed (best)
Over:
Well-wr: yes
What kind of driving leads to accidents?
Supp.:
b)
Rel.:
yes
stressful b) dangerous c) fun illegal e) deadly
Fact::
yes
Explanation: Driving dangerously creates accidents.
New i.:
Unn. i::
no
Contr: no
CoS-E fluency fixed (worst)
Over::
Well-wr: no
What is grumpy person likely to have?
Supp.:
none
Rel.:
no
headache b) understand themseloes c) compliments own house happiness
Fact.:
n/a
New i.:
Explanation: peculiar person.
Unn. i.:
yes
Contr: no
Generated (best)
Over:
Well-wr: yes
The cloth's are not yet placed where they belong, where should put them?
Supp.:
a)
Rel.:
yes
Fact::
yes
linen closet b) kitchen c) on the space heater d) fabric store e) bedroom
New i.:
++
Unn. i.:
Explanation: linen closets are typically used to store items such as cloths.
Contr:
no
Generated (worst)
Over: Supp.:
Well-wr.: yes
Sarah opened his chest and found a second heart. Her patient might be what?
d)
Rel.:
yes
Fact.:
no
a) istanbul b) animal c) antique shop d) human being e) bedroom Explanation: a human being is the only creature known to have two hearts.
New i.:
Unn.
no
Contr: no
ECQA (best)
Over:
Well-wr: yes
Where would one find captive monkey?
Supp.:
d)
Rel.:
yes
rain forest b) tree tops c) toy store d) zoological gardens e) great outdoors
Fact.:
New i.:
++
Explanation: A monkey can be kept captive at zoological garden as many animals are kept there for people to see. All the other options incorrect as they do not have captive monkeys.
Unn.
no
Contr:
yes
ECQA (worst)
Over: Supp.:
2 Well-wr:: no
Where are there lots of seats placed rows surrounding a court? a) show b) auditorium c) movies d) soccer stadium e) hockey game Explanation: In auditorium, there are lots of seats placed in rows. Auditorium seats are surrounding court. In show, there are no surrounding court placement of seats. Movies is not place. In soccer stadium, seats are not placed in In hockey game, there are no proper seats.
b)
Rel.:
Fact::
New i.:
Unn. i.: no
Contr: yes
Table 5: Top- and bottom-rated example explanations for CommonsenseQA in ACORN. New info. labels are denoted follows: +++ (ample), ++ (sufficient), (some), (none).
16

---

Published as a conference paper at COLM 2024
COPA-SSE (best)
Over:
5
Well-wr:
yes
The velticles travelled slowly. Cause? a) It was foggy. b) It was cloudy.
Supp.:
a)
Rel.:
yes
Fact.:
yes
Explanation: Fog is a thick mist which prevents you from driving quickly as you cannot see far in front of your vehicle. Driving slow through is motivated by the desire not to crash.
New i.:
Unn. i:
no
Contr:
no
COPA-SSE (worst)
Over:
Well-wr: no
The boy found the television show interesting. Result?
Supp.:
none
Rel.:
yes
He shut off. b) He recorded
Fact::
n/a
Explanation: The result causes the record shut off.
New i:
Unn.
yes
Contr:
no
COPA-SSE Fluency fixed (best)
Over:
Well-wr: yes
The woman tolerated her friend's difficult behavior. Cause?
Supp.:
a)
Rel.:
yes
The woman knew her friend was going through hard time. b) The woman felt that her friend took advantage of her kindness.
Fact::
yes
New i.:
++
Explanation: Going through a hard time can cause people act in a difficult way. A friend can be motivated to support their friend when they're having tough time.
Unn. i:
no
Contr: no
COPA-SSE Fluency fixed (worst)
Over:
Well-wr::
The shirt did not have a hole anymore. Cause?
Supp.:
none
Rel.:
no
I patched the shirt. b) I bleached the shirt. Explanation: I desire to look good.
Fact.:
n/a
New i.:
Unn. i::
yes
Contr:
no
Generated (best)
Over:
5 Well-wr. yes
The boy was struggling to fall asleep. Cause? He turmed his nightlight. b) He watched horror movie. Explanation: Watching horror movie can create fear and anxiety that make it difficult for someone to fall asleep. The nightlight not likely be the cause since it meant help someone sleep.
Supp.:
b)
Rel.:
yes
Fact.:
yes
New i.:
++
Unn. i.:
no
Contr: yes
Generated (worst)
Over::
Well-wr.:
The woman awakened. Cause?
Supp.:
Rel.:
yes
a) She took sleeping pills. b) Her lusband was snoring. Explanation: Taking sleeping, pills a direct cause awakening, whereas husband snoring is an indirect cause.
Fact::
New i.:
Unn. i.:
no
Contr:
Crowdsourced (best)
Over::
5 Well-wr: yes
The man was bitten by mosquitoes. Cause?
Supp.:
b)
Rel.:
a) He fell asleep on his couch. b) He went camping in the woods. Explanation: Sleeping on a couch would mean you are indoors where there are rarely mosquitoes. Mosquitoes are prevalent in wooded areas, the man would be more likely to be camping in the woods, if he was bit by mosquitoes.
Fact.:
yes
New i.:
Unn. i.:
no
Contr:
yes
Crowdsourced (worst)
Over:
Well-wr: yes
I refilled my water bottle. Cause? drank the water in
Supp.:
a)
Rel.:
yes
b) I kept the fridge.
Fact::
Explanation: If you kept your water in the fridge, it would not need to be refilled.
New i::
Unn. i:
no
Contr: yes
Table 6: Top- and bottom-rated example explanations for BCOPA in ACORN. New info. labels are denoted as follows: +++ (ample), ++ (sufficient), + (some), (none).
17

---

Published as a conference paper at COLM 2024
Model version
Supp.
Ovr.
W-wr.
Rel.
Fact.
New i.
Un. i.
Cntr.
Avg.
5-turbo 0613
0.861
0.561
0.327
0.644
0.413
0.458
0.386
0.756
0.551
gpt-4-0613
0.873
0.624
0.346
0.673
0.427
0.476
0.397
0.812
0,578
gpt-40-2824--5-11
0.859
0.624
0.409
0.666
0.441
0.485
0.416
0.794
0.587
spt-40-mini-28224-777
0.852
0.619
0.411
0.648
0.423
0.474
0.393
0.791
0.576
text-bison-001
0.846
0.588
0.344
0.629
0.407
0.468
0.411
0.649
0.543
gemini-1.0-pra
0.834
0.597
0.359
0.619
0.404
0.474
0.408
0.714
0.551
gemma-2-9b-it
0.836
0.553
0.360
0.634
0.403
0.407
0.368
0.748
0.539
gemma-2-27b-it
0.823
0.582
0.387
0.633
0.414
0.411
0.336
0.788
0.547
Neta-Llama-3.11-2 1-88----------
0.802
0.593
0.396
0.630
0.366
0.439
0.415
0.717
0.545
Meta-Llama-3.1-708-
0.860
0.623
0.412
0.647
0.416
0.480
0.378
0.792
0.576
Meta-Llama-3.. 140088-x+
0.637
0.405
0.644
0.424
0.482
0.385
0.804
0.580
0.834
0.586
0.356
0.622
0.397
0,455
0.358
0.703
0.539
Mixtral-8x22B-xx
0.848
0.629
0.386
0.645
0.428
0.472
0.428
0.808
0.580
Human
0.866
0.613
0.365
0.655
0.399
0.442
0.433
0.784
0.570
Table 7: Full results of the experiments comparing the difference in inter-annotator agree- ment between humans and with a random rater replaced by an LLM ($3.2). All values represent Krippendorff's 's averaged over 20 iterations. Extraction failures are excluded from analysis. Replace with "Instruct" and with "Turbo" in the model names.
Model version
Supp.
Ovr.
W-wr.
Rel.
Fact.
New i.
Un. i.
Cntr.
Avg.
gpt-3.
0.846
0.627
0.409
0.696
0.621
0.611
0.383
0.786
0.622
gpt-4-0613
0.900
0.740
0.515
0.778
0.675
0.691
0.528
0.946
gpt-40-2824-05-12
0.844
0.758
0.611
0.750
0.688
0.714
0.457
0.896
0.715
3pt-40-mini-2224--7-11
0.853
0.708
0.620
0.700
0.628
0.678
0.463
0.888
0.692
text-bison-001
0.773
0.685
0.456
0.654
0.653
0.640
0.515
0.607
0.623
gemini-1.8-pro
0.787
0.681
0.497
0.613
0.549
0.671
0.451
0.648
0.612
gemma-2-9b-it
0.798
0.645
0.492
0.674
0.584
0.616
0.388
0.742
0.617
gemma-2-27b-it
0.769
0.626
0.574
0.668
0.603
0.537
0.191
0.867
0.604
Meta-Llama-3.1 1--8-x---118-4
0.632
0.638
0.562
0.639
0.535
0.555
0.469
0.662
0.586
Meta-Llamaa331-7788---
0.847
0.732
0.632
0.703
0.652
0.680
0.311
0.886
0.680
Meta-Llama-3.1-408B
0.833
0.745
0.618
0.699
0.631
0.686
0.351
0,916
0.685
Mixtral-6x78---v0.1 0.747
0.624
0.487
0.616
0.543
0.604
0.314
0.640
0.572
Mixtral-8x228-x-000
0.799
0.733
0.588
0.688
0.642
0.681
0.540
0.928
0.700
Table 8: Full results of the experiments measuring Spearman's rank correlation between majority-voted human labels and LLM-generated ones ($3.3). Extraction failures are ex- cluded from analysis. Replace with "Instruct" and # with "Turbo" in the model names.
18


Reviews:
[{"review_id": "agSXzXb85Z", "reviewer": "Reviewer_RR4D", "summary": "This work studies the problem that whether LLM can serve to evaluate free-text reasoning explanations. First of all this work constructs a dataset with 3.5k examples, and by conduction experiments on this dataset, this paper reveals that the inter-annotator agreement is decreased when LLM-generated ratings are used to replace human ratings. More fine-grained results conclude that while LLMs can provide ratings moderately consistent with an average crowdworker, but they are not reliable enough for complete replacement.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for the comments. We will update our paper to clarify the points you raised, and we hope our response will resolve your reservations.\n\n1. **Dataset size.**\n\nEach sample includes eight aspect-wise quality labels from five annotators, resulting in ***140,000 data points.*** This is much larger than many analytical works. For example, [Xu et al. (2023)](https://aclanthology.org/2023.acl-long.181/) collected 260 expert judgments for long-form QA, and [Wiegreffe et al. (2022)](https://aclanthology.org/2022.naacl-main.47/) collected ratings for 250 samples by three raters over seven quality aspects, i.e., 5,250 data points. Our dataset is large enough to be suitable for fine-tuning applications.\n \nAlso, note that the total construction cost was around 8,000 USD. If we count the additional high-quality explanations for BCOPA (3,000 in total), the full value surpasses 10,000 USD, a substantial investment for most.\n\n2. **How *can* LLMs be used?** \n\nWe evaluated the models under the strict condition of replacing humans. However, we still view them as relatively successful and useful where *complete* replacement is not required, such as massive-scale data filtering, flagging data for further analysis, or use in model development stages. Our work quantifies current gaps and shows a way to assess future evaluator setups.\n\n3. **Other settings, e.g., evaluating each criterion separately?** \n\nWe considered several setups, including the suggested one (see \"Prompting Strategy\" in Section 3.1). We chose the one that yielded the highest correlation with aggregated human ratings to create a controlled setting with reasonably strong performance. For example, in our comparisons, compound/separated average Spearman with *verbatim* prompts (best results) was 0.72/0.71 for GPT-4 and 0.62/0.51 for GPT-3.5. All other settings we considered are listed in Appendix C.\n\n4. **Open-source models?** \n\nThank you for the suggestion. We selected the strongest model available to us at the time of writing (GPT-4) as a best-case scenario to eliminate doubt that a better model could alleviate the issues we report. The weaker alternatives were included to contextualize the performance and show the outcome with a more affordable alternative.\n\nWe will add open-source models to the camera-ready version. However, our main contribution remains the ***dataset and methodology*** for evaluating ***any*** explanation evaluation system, regardless of the models we feature in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The conclusion in this work can inspire futher works on using LLM for automatic evaluations. \n\n2. A dataset with 3500 examples is constructed and can be used for futher research on this task.", "reasons_to_reject": "Although this work studies whether LLM can be used for evaluating the quality of free-text explanations, the conclusions in this work are a bit weak. First of all the scale of 3500 examples is not large, and meanwhile, with the conclusion that LLMs are not reliable for explanation evaluation, how LLMs can be used for this task are less discussed in this work. For example, whether evaluating on each criteria and later gathering can help the performance.", "questions_to_authors": "1. Why not also compare open-sourced LLMs like Mistral/Llama in this work? The numbers on the open-sourced models are also meaningful.", "extra_scores": {}}, {"review_id": "iuPy9BuN4q", "reviewer": "Reviewer_RfVm", "summary": "This paper evaluates the utility of LLMs for evaluating AI-generated free-text explanations of model predictions. In particular, the quality of the aggregate ratings is tested in various settings where a random human rater is replaced with a suitably prompted LLM. For the most part, LLMs reduce overall annotator agreement and drag the aggregate label distribution away from the majority vote, but one small exception is the case of GPT-4 breaking ties between two humans.\n\nThe contribution of the work consists of some commonsense QA explanation data, a pretty straightforward and sound evaluation paradigm for testing whether LMs add productively on top of human raters, and the results of experiments testing several LLMs.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful comments.\n\n**Re: Faithfulness.**\n\nOur intention behind mentioning use cases of explanations was to stress their increasing importance and prevalence, arguably increasing the need for automatic evaluation. We completely agree that faithfulness is an important aspect of explanations (We cite Jacovi & Goldberg (2020) in the introduction, the work that popularized the distinction between plausibility and faithfulness.) \n\nWe also see how this might give the impression that faithfulness should be a larger factor in our analysis since one could reasonably expect that a \"right-to-explanation\" grants the right to a faithful, not just plausible-sounding, explanation. We will clarify this point by explicitly mentioning the distinction in the introduction and the limitation that our analysis focuses on plausible explanations.\n\nOur reasoning behind the \"supports\" criterion, which we will also clarify, is the following: we argue for a minimal condition of faithfulness to be that *the predicted answer and the explanation match.* That is, if an explanation points to a different answer than the system's prediction, it proves that it is unfaithful. Thus, we can use the \"supports\" label to cross-reference with the prediction and potentially identify inappropriate mismatches. \n\n**Q: Quantitative differences between source datasets?** Due to their respective intended usage, CoS-E and COPA-SSE tolerated surface-level flaws by design. In contrast, ECQA and our collection used guidelines explicitly aligned with human preferences. Dataset-wise Overall ratings, to be added, quantitatively confirm this difference.\n\n**Q: Why fluency-improved versions?** Well-written explanations typically also have high scores in all other aspects. To prevent fluency from becoming a superficial signal, we created fluent but imperfect samples w.r.t. other aspects by improving the fluency of low-quality explanations. Even if new errors were introduced, it would still be advantageous for this purpose.\n\n**Q: Why only half?** We deprioritized labeling the augmented samples from unrealistic distributions to maximize the number of informative samples within a limited annotation budget. Their total number still matches the other subsets.\n\n**Q: Few-shot prompting, maximize performance?** Our experiments are equivalent to three-shot prompting. Other settings we tried, as mentioned in Section 3.1, are listed in Appendix C. Please also see our response to Reviewer RR4D, point 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The specific methodology for testing LMs as replacing human raters seems really strong to me, and the discussion of how to interpret these results (i.e., if agreement decreases, remains the same, or increases) was solid. I think this paper serves as a good example for future work evaluating similar kinds of questions.\n* Not critical, but I think it'd be interesting & informative to add another bar to Figure 4 corresponding to a single random human, to serve as a baseline. As the human labelers are noisy, they shouldn't have 1.0 correlation with the majority vote, and it'd be nice to see how each model compares with that case.", "reasons_to_reject": "The paper seriously conflates some important issues around explanation quality. The motivation provided in the introduction is about model trustworthiness, interpretability, compliance with \"right to explanation\" laws, but all of these use-cases of explanations depend on them being faithful to the model's decision making process, i.e., predictive of model behavior. The evaluations done in this paper have little bearing on faithfulness, as they only assess whether the explanation is coherent, correct, and supports the given answer. While explanation coherence is necessary for faithfulness (as incoherent explanations cannot give much information about how the model behaves), the other aspects being measured are about the quality of the explanation with respect to the _actual reason_ a label is correct. A model's ability to report a reason that supports an answer — even the correct reason — is neither necessary nor sufficient for faithfulness, which is about how the explanation relates to the model's decision making process, and cannot be inferred from a single example alone (see e.g. the discussion of \"coherent biased reasoning\" in https://arxiv.org/abs/2403.05518). Faithfulness, furthermore, is the aspect that is critical for all of the uses of explainability promoted in the intro. Given all of this, I think the paper is not really about explainability, but about evaluating model performance at a specific task, which is producing correct-with-respect-to-the-real-world explanations for classification decisions. I think an accurate presentation of the work would frame it in this way.\n\nAll of that said, this framing issue is not crucial to the contribution of the paper, and the experiments and evaluations all seem pretty reasonable to me. So I don't think this is a dealbreaker or anything, I just strongly urge the authors to rework the motivation to avoid any undue implications about the relevance of this work to model trustworthiness, reliability, interpretability. etc., and ask that the discussion of aspects in Table 1 avoid implying that the \"Supports\" or \"Overall\" aspects address faithfulness.\n\nThe main reason I can imagine this being rejected is that it's overall a pretty small contribution. I don't begrudge it this necessarily but it doesn't feel like it needed the full space it used. Thankfully, the authors did not unnecessarily pad it out to 9 pages, finishing within 8, but there's still quite a bit of fluff and repetition — I feel the paper could have probably fit just fine in 4 pages, and I think it would've made a great short paper. COLM doesn't have a short paper track, and of course this is the first COLM, so I don't really know how to handle this case. I'm recommending the paper for acceptance, as I think it's sound, but leave it to the meta-reviewer to judge.", "questions_to_authors": "General comment: I find the description of the data sources hard to follow. I don't know how to evaluate claims like \"improved version\" or \"aligning with our criteria for well-formed explanations\". These seem like subjective qualitative judgments. When describing the provenance of your data, please describe in concrete, non-subjective terms what was done to create it. Some things are also curious to me & unexplained — why did you have to post-process CoS-E and BCOPA for fluency improvement? Could this have introduced errors? Why were only half of them fluency-improved? Does this not also mean that some of your test examples are correlated, hurting your statistical power later on? A lot of things about the setup here seem messy. Can you please clarify in the paper how you processed the data and why?\n\nQuestion: did you try few-shot prompting? I found a brief mention of it in Appendix D, but I couldn't figure out how it worked into the experiments and whether you tried it. It seems to me that you should probably be doing the best you can to get good performance out of these models and see if they work. Needing a few human labels for few-shot prompting doesn't weaken the case IMO, as you're already assuming access to some human labels anyway (e.g., when adding an LM to a pool of human raters).\n\nIn the ethics statement, you say:\n> Some of the datasets used in our research have been found to contain biases that can be further perpetuated through explanations supporting them.\n\nWhich datasets? Please clarify this and provide citations if it's the case.", "extra_scores": {}}, {"review_id": "R38vFTReo9", "reviewer": "Reviewer_e4Ah", "summary": "This paper examines the potential of large language models (LLMs) to automatically evaluate the quality of explanations in commonsense reasoning tasks. The authors introduce ACORN, a novel dataset of 3,500 explanations with human ratings across eight different quality aspects. The authors explore the capabilities of various LLMs to replace or supplement human evaluation, analyzing inter-annotator agreement and correlation with gold-standard labels. They find that LLMs often deviate from human judgments, but GPT-4 shows some potential, particularly as an additional rater when human resources are limited.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for highlighting the value of our work.\n\nA deeper model performance analysis, such as a quality-wise breakdown, is a great addition. Thank you for the suggestion; we will update our paper to include it. A quick run with GPT-4 revealed that it improved inter-annotator agreement more in lower-rated explanations than in higher-rated ones:\n\n| Overall rating | $\\Delta \\alpha *100$  |\n|---|---|\n| 1 star | +2.24 |\n| 2 stars | +3.86 |\n| 3 stars | +2.13 |\n| 4 stars | +0.33  |\n| 5 stars | -0.79  |\n| All samples | +0.91 | \n\n(As a reminder, the values represent the difference in inter-annotator agreement when the LLM replaces one random rater out of the five, averaged over 20 iterations and *100 for legibility. The samples are bucketed by the aggregated overall rating given by human annotators. This is related to the first setting, i.e., Section 3.2.)\n\n\nAs for why we include fluency-improved versions, we hope our answer to Reviewer RfVm regarding the same question clarifies this point:\n\n> Well-written explanations typically also have high scores in all other aspects. To prevent fluency from becoming a superficial signal, we created fluent but imperfect samples w.r.t. other aspects by improving the fluency of low-quality explanations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1. The paper thoroughly investigates various LLMs and their impacts on inter-annotator agreement and correlation with human judgments. The experimental design, which examines LLMs as potential replacements for human annotators, was particularly interesting. The results offer insightful ideas on how to use LLMs to complement human evaluations.\n\nS2. ACORN provides a valuable resource for research into automatic explanation evaluation, offering a diverse set of explanations and comprehensive human ratings across varying qualities.\n\nS3. Overall, the paper is well-written and easy to follow. The authors clearly document their data collection and analysis procedures, enhancing the transparency and reproducibility of their findings.", "reasons_to_reject": "While the paper focuses on a topic that may interest a narrow audience (evaluating explanations), its contributions are clearly and concretely articulated. However, I have one minor concern. The datasets used are of varying quality, yet the analysis encompasses all datasets uniformly. It would have been advantageous if the authors had conducted analyses on subsets of the datasets to determine if LLMs can be more effective with texts of particular quality levels. Additionally, the purpose of including fluency-improved versions in the datasets was not clear. Clarification on this aspect would be helpful.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper investigates the problem of whether LLM can serve to evaluate free-text reasoning explanations. A dataset called ACORN, containing 3,500 examples with human ratings across eight different quality aspects, is introduced and used for experiments. They find that LLMs often deviate from human judgments, but GPT-4 shows some potential, particularly as an additional rater when human resources are limited.

Pros:
- The introduction of the dataset with 3,500 examples is valuable, and can be used in future research.
- The paper is well-written, clear, and easy to follow, with transparent and reproducible data collection and analysis procedures.
- The methodology is thorough and well-executed.

Cons:
- Some reviewers expressed concerns about the paper's narrow scope.
Decision: Accept