Ecologically Valid Explanations for Label Variation in NLI
Authors: Nan-Jiang Jiang
Chenhao Tan
Marie-Catherine de Marneffe
Venue: EMNLP 2023 Findings (2023)
Keywords: annotation disagreement
explanation
interpretability
natural language inference
human label variation
textual inferences

Abstract: Human label variation, or annotation disagreement, exists in many natural language processing (NLP) tasks, including natural language inference (NLI). To gain direct evidence of how NLI label variation arises, we build LiveNLI, an English dataset of 1,415 ecologically valid explanations (annotators explain the NLI labels they chose) for 122 MNLI items (at least 10 explanations per item). The LiveNLI explanations confirm that people can systematically vary on their interpretation and highlight within-label variation: annotators sometimes choose the same label for different reasons. This suggests that explanations are crucial for navigating label interpretations in general. We few-shot prompt large language models to generate explanations but the results are inconsistent: they sometimes produces valid and informative explanations, but it also generates implausible ones that do not support the label, highlighting directions for improvement.

Ecologically Valid Explanations for Label Variation in NLI
Nan-Jiang Jiang"
Chenhao Tan
Marie-Catherinn de Marneffe FNRS, UCLouvain
Department of Linguistics Department of Computer Science
The Ohio State University jiang. 1879@osu. edu
The University of Chicago marie-catherine demarneffe chenhao@uchicago... edu
@uclouvain. be
Abstract
We introduce the LIVENLI' dataset, contain- ing 122 re-annotated MNLI (Williams al., 201 items, each with at least 10 highlights and free-text explanations for the labels chosen by the annota- tors, totaling 1,415 explanations. The dataset is rel- atively small because we intended it to be suitable for few-shot evaluation, following recent trends in BIG-Bench authors (2023).
Human label variation exists in many natural language processing (NLP) tasks, including nat- ural language inference (NLI). To gain direct evidence of how NLI label variation arises, we build LIVENLI, an English dataset of 1,415 ecologically valid explanations (annotators plain the NLI labels they chose) for 122 MNLI (Williams et al., 2018) items (at least 10 expla- nations per item). The LIVENLI explanations confirm that people can systematically vary on their interpretation and highlight wäthinllbbel variation: annotators sometimes choose the same label for different reasons. This suggests that explanations crucial for navigating la- bel interpretations in general. We few-shot prompt language models (LMs) to generate explanations but the results are inconsistent: the models sometimes produce valid and in- formative explanations, but they also generate implausible ones that do not support the label, highlighting directions for improvement.
The contribution of LIVENLI's explanations is their ecological validity: annotators provide both the label and the explanations, in contrast to many explanation datasets (e.g.. e-SNLI (Camburu et al., 2018)) in which explanations were collected for a given "ground truth" label. LIVENLI hence pro- vides direct evidence for how label variation among annotators arises. In addition, LIVENLI highlights another kind of variation, within-label variation: in some cases, humans converge to the same label for different reasons. Finally, we evaluate whether GPT-3 (Brown et al., 2020) can generate expla- nations for label variation. We found that its ex- planations can be fluent and informative but also implausible or not supporting the label.
1 Introduction
Practices for operationalizing annotations in NLP datasets have largely assumed one single label per item. However, human label variation (Plank, 2022) has been found in a wide range of NLP tasks (Plank et al., 2014; Poesio et al., 2018; Pavlick and K Kwiatkowski, 2019; Nie et al., 2020; Uma et al., 2021: Jiang and de Marneffe, 2022, i.a.). Aroyo and Welty (2015), i.a., argued that variation that is systematic in annotations should be considered signal, not noise. Specifically, the NLI task iden- tifying whether a hypothesis is true (Entailment). false (Contradiction), or neither (Neutral) given a premise - has embraced label variation and set out to predict it (Zhang et al., 2021; Zhou et al., 2022). However, it remains an open question where label variation in NLI stems from.
Data Collection
We re-annotated 122 items from the MNLI dev set where each item originally had 5 annotations:
60 items where only 2 annotators agreed on the label in the original MNLI annotations;
50 items where 3 annotators agreed on the label
in the original MNLI annotations, reannotated in Nie al. (2020)'s ChaosNLI with 100 annota-
tions;
12 items (4 for each NLI label) where all 5 anno- tators agreed on the label in the original MNLI annotations.
The first 110 items analyzed by Jiang and de Marneffe (2022) who proposed taxonomy of reasons for label variation.
'Label Variation and Explanation in NIJI, available at https://stt //gitt////ttbbb... http://ghhuuuucoojjiiieeeeee/eee
NNww at Google.

---

Ex.1 P: Most pundits side with bushy- headed George Stephanopoulos (This Week), arguing that only air strikes would politically palatable. H: Mr. Stephanopoulos has a very large pundit following due his stance on strikes only being politically palatable.
LIVENLI [E,N.C]: (0.4, 0.3, 0.3]
QUD: Does Stephanopoulos have a very large pundit following?
R.sp.1.1; This statement most likely to be true because the context is stated that "Most pundits" would side with Mr. Stephanopoulos. Most pundits could also mean a very large pundit following.
Resp.1.2: 1.2: N You cannot infer that the overall numher of pundits following the individual is large just because the majority of pundits follow the individual. He could just have out of total pundits following him, for instance. Furthermore, they may be following him for reasons outside his stance on air strikes.
QUD: Do pundits follow Stephanopoulos due to his stance on air strikes?
Resp.1.3: N George Stephanopoulos may have follow from pundits, but it might not be due to his support of drones.
Resp.1.4 1.4: C- He might have a large pundit following. but that would have to be for something before the current issue of air strikes since one event wouldn't 't get people a large following overnight.
Ex.2 P: for a change i got get sick of winter just looking every dead hate that
I'm so sick of summer.
LIVENLI [E.N,C]: [0, 0.35, 0.65]
Resp.2.1: J.I::.C. C
The context stating how one sick winter, not summer, as the statement describes.
Resp.2.2: C- The speaker hates winter because the foliage is dead, therefore he likely loves summer when everything is alive.
Resp.2.3: N The context mentions being sick of winter while statement mentions being sick summer. These could hoth
be true because the same person may still complain of summer's heat.
Table 1: Examples in LIVENLI. P: Premise. H: Hypothesis. [E,N,C]: the probability distributions over the labels (E)ntailment, (N)eutral, and (C)ontradiction, aggregated from the multilabel annotations LIvENI.I
Procedure Annotators read the premise (con- text) and hypothesis (statement), and were asked whether the statement is most likely be true/false/cither true or false (corresponding to En- respectively). Mul- tiple labels were allowed. They were then asked to write a free-text explanation for all the label(s) they chose. Inspired by Camburu et al. (2020); Wiegr- effe et al. (2022); Tan (2022), we asked annotators to give explanations that provide new information and refer to specific parts of the sentences, and avoid simply repeating the sentences. Annota- tors were also asked to highlight words from the premise/hypothesis most relevant for their explana- tions. 48 native speakers of English were recruited from Surge Al (surgehq. ai, details in A1).
80
Label Probabiity
80
tallmen/CConradiccionnNeee
40
110 Rems
12 htmms MNU agreement
Figure 1: Average distributions from the normalized multilabel responses for each item.
Annotators' reasons for label variation Jiang and de Marneffe (2022) analyzed the reasons for NLI label variation from a linguists' perspective and built a taxonomy of linguistic phenomena that can lead to different interpretations. For instance, in Ex. I (Table 1), does one infer a large follow- ing in the hypothesis from most in the premise? They hypothesized that such lexical indeterminacy contributes to the variation reflected in the NLI label distribution. However, their taxonomy was built post-hoc, detached from the annotators. Here we leverage LIVENLI's ecological validity: we analyze the reasons for label variation from the perspective of linguistically-naive annotators.
3 Analysis of LIVENLI
Label deseriptive statisties We normalize each multilabel response into distribution, then average the individual distributions to obtain the item's 's la- bel distribution. Figure 1, where each set of stacked bars gives the distributions for one item, shows that label variation is widespread, even on the 12 items that received full agreement in MNLI (right panel). The annotators provided legitimate explanations for the labels that differ from the original one (sce (1) in A1), suggesting that the variation is genuine and not noise. This reiterates Baan et al. (2022)'s point that considering one distribution to govern all judgments from the population is a simplification, and distributions can shift with annotator pools.
For each pair of premise/hypothesis, one author assigned one or more taxonomy categories exhib- ited its explanations (Table 5 in A2 shows the fre- quency of the categories). Across the board, label

---

variation arises in similar reasons to what Jiang and de Marneffe (2022) hypothesized. We calculated agreement between our taxonomy reannotations and the annotations from Jiang and de Marneffe (2022). The Krippendorff's a with MASI-distance is 0.30, suggesting low agreement (contingeney matrix in Figure 3 in A2). Specifically,, the average number of categories used here is higher 1.33 per item vs. 1.22 in Jiang and de Marneffe (2022), suggesting nontrivial differences between the an- notators' and linguists' reasons for label variation. Furthermore, the explanations uncover more ways in which label variation can occur. In par- ticular, we found that label variation often stems from annotators judging the truth of different at- issue content, which answers different Questions Under Discussion (QUDs) (Roberts, 2012). For example, in Ex.1, Responses 1.1 and (the first half of) 1.2 take the main point of the hypothesis to be Stephanopoulos having a very large pundit fol- lowing, but have different judgments on whether this main point is true. On the other hand, 1.3 and 1.4 focus on the reason for which pundits follow Stephanopoulos, and agree with 1.1 that he has large following. We thus added "QUD" category the taxonomy. The QUD category occurred in 28 items (out of 122) and is the third most fre- quently used category, thus an important aspect of language understanding. Incorporating QUD into NLP tasks and modeling (De Kuthy et al., 2018; Narayan et al., 2022) is an interesting direction.
is false. 2.2, on the other hand, does not assume that the premisc and the hypothesis refer to the same season, and infers through probabilistic en- richment that the speaker likes summer, making the hypothesis false. Thus both with and without the coreference assumption, annotators label the item as Contradiction.
Within-label variation shows up in ~33 of LIVENLI, and highlights the diversity of human explanations. This suggests that model generated explanations should be similarly diverse to imitate human decision makiin...
4 Predict and Explain Label Variation
Models should capture inherent label variation to achieve robust understanding. Explanations are es- sential for navigating the uncertainty rising from such variation. Here we use LIVENII to few-shot prompt LMs to jointly predict and explain label variation. We used text-davinci-003, a variant of GPT-3 (Brown et al., 2020) with few-shot prompt- ing (example in Table 7). Label variation in NLI has been predicted either as a distribution over three labels (Zhang et al., 2021; Zhou et al., 2022) or a multilabel (Jiang and de Marneffe, 2022; Liu et al., 2023). Here we use the multilabel approach for its greater interpretabilityy compared to distributions. We select gold answers labels with probability (from LIVENLI ChaosNLI) above 0.2, similar to Jiang and de Marneffe (2022).
We use Seaeetttthe-Exxplai (P-E) and Explain- then-Predict (E-P), a.k.a. chain-of-thought prompt- ing (Wei et al., 2022), which has been shown to improve performance of reasoning tasks like NLI (Ye and Durrett, 2022). (P)redict-only prompts the model without explanations for the same items, giv- ing baseline measuring the effect of explanations.
Within-label variation Our analysis reveals that there is also within-labed variation: annotators may vary in their text understanding when giving the same label. There are 16 instances of within-label variation: annotators chose the same labels but pro- vided very different explanations.? As mentioned earlier, 1.2 and 1.3 both chose Neutral but assumed different QUDs (1.2 also encompasses 1.3).
Train/test items Since Jiang and de Marneffe (2022) found that models are sensitive to label im- balance, we used a label-balanced set of 11 training items from LIVENLI, with one item per single- label and two items per multiple-label. The test set includes the rest of LIVENLI, augmented with ChaosNLI for infrequent labels, totaling 259 items.
As another example, in Ex.2, the explanations indicate that the annotators differ in whether to take the premise and hypothesis to be referring to the same entity/event. Response 2.1 assumes that the premise and hypothesis refer to the same season: since the premise talks about winter and not summer, the hypothesis I' m so sick of summer
The metries include exact-match accu- racy and macro F1. To measure how well the model can separate the agreement items from ones with label variation, we calculate a 4way-F1 score by
Metrics
PIt replaces two Jiang and de Marneffe (2022)'s cate- gories. "accommodating minimally added content" and "high overlap". which involve annotators having different readings and ignoring certain parts of the items.
*We consider two explanations to be different if they focus on different sentence parts or make different assumptions.
"We did not receive access to GPT-4 and ChaiGPT per- formed worse in preliminary experiments.

---

Accuracy
F1
4way
# labels
Ex. Roman times temple to Jupiter stood here, fol- lowed in the fourth century by the first Christian church, Saint- Etienne. H: Saint-Etienin a Christian church, had a temple to Jupiter and performed rituals during Roman times ChaosNIII [F,N,C]: [0.25, 0.25, 0.50]
P
25.25
70.701.70
37.05o.92
3.920.35
P-E E-P
24.7s.25
69.00s.as
37.151.77
1.780.32
19.90p.28
65.85,.46
27.551.48
1.900.28
ENC
14.30
72.70
18.20
3.00
Generic, Plausible, Not Support. No new info
P-E: C- the context states that there was a temple to Jupiter before the church was established, while the statement implies the church had a temple and performed rituals. This may or may not have been true, but it not something that is stated in the context.
Table 2: Metrics for each model and mean number (#) of predicted labels. The average number of gold labels in the test split is 1.71. Each value is the average of two random splits (standard deviation in subscript).
Specifie, Plausible, Support, New info
E-P: N it is possible that a church in Roman times may have done rituals for Jupiter, as Christianity was still a nascent religion at that time, but it is unknown based on the context.
Validity
Informativeness New Info
Plausible
Support
Specific
Total
Ex. P: Most pundits side with bushy-headed George Stephanopoulos (This Week), arguing that only air strikes would be politically palatable. H: Mr. Stephanopoulos has a very large pundit following due to his stance on air strikes only being politically palatable.
E-P P-E
75,0
80.9
38.2
10.3
68
77.1
81.4
37.1
5.7
70
Table 3: Percentage of explanations (out of Total) for 35 items marked as Plausible, Support label, Specific, and introduce New Info.
LIVENLI (E,N,C) 0.34,
Generic. Plausible, Support, No new info P-E: N - the context implics a large following but does not specify an exact amount. Therefore it cannot be definitively stated if there is a large following or not.
calculating macro Fl scores a 4way categoriza- tion of the multilabels: three single labels vs. "complicated" label if there are multiple labels.
Generic, Implausible, Support, No new info E-P: C- is is unknown to what extent the pundit's following is and isn't large. It has only been alluded to that many pundits are agrecing with him.
Classification Results Table 2 shows the perfor- mance of each model, together with the uniform baseline of always predicting all three labels ENC. Both P and P-E significantly outperforms E-P. This adds to the series of results where having post-prediction explanations outperforms chain-of- thought style prompting (Lampinen et al.. 2022; Zhou et al., 2023). In particular, P-E predicts fewer labels than E-P, suggesting that the models con- sider more labels be possible when generating explanations first. predicts the most number of labels, which can potentially be due to the fact that it does not need to generate an explanation for each label predicted, and hence does not pay the long sequence penalty when generating more labels.
Table 4: Generated explanations from both models.
their informativeness (Leake, 1991).
For random sample of 35 items, one author provided binary judgments on the four criteria for both models' explanations, with the model identity hidden. As shown in Table 3, the models do not differ much in their explanations quality, except for New Info. ~80% of the explanations support the labels they predict. Those not supporting the label are often Contradiction explanations (across mod- els, 67 8% Support for Contradiction vs. 88.2% for other labels). In Ex. 1 in Table 4, the explanation only states that the statement is not definitely true (hence not Entailment) but fails to state why it is false, thus not supporting the Contradiction label. Furthermore, only ~60% of the explanations are both plausible and support the label (hence valid).
Explanations Results We performed a qualita- tive analysis to evaluate the models' explanations on four criteria inspired by Wiegreffe et al. (2022): do the explanations plausibly deseribe sentence meaning (Plausible), support the label (Support), go beyond restating the sentences (Specific), in- troduce new information about the subject matter (New Info) (details in A4). The first two criteria target the explanations' validity, and the last two
The explanations are also lacking informative- ness: over half of the explanations are merely re- stating the premise/hypothesis (not Specific) and rarely have new information. However, when the generated explanations are both valid and informa- tive (as in the Neutral one for Ex. in Table 4), they do contribute to further understanding of why the label applies by providing interpretation.
'The ENC baseline scores high on Fl because it has a perfect recall, and a high precision given that more than half the items are multi-labeled.
Moreover, the generated explanations do not ex- hibit the range of within-label variation found in
"p= = 0.02 comparing P vs. E-P, = 0.03 for P-E vs. E-P. = 0.84 for P vs. P-E, McNemar's test.

---

the human explanations in Table 1. For Ex. 2 in Table 4, both generated explanations focus on the aspect of whether "most" implies "large", while the human explanations in Table identify two different QUDs the sentences can address.
thor. An evaluation scheme with multiple annota- tors would provide a better assessment of the model quality. We only tested one model: text-davinci- 003, because it was the best model available to us at the time development. We have not received ap- proval for access to GPT-4 from OpenAL. Further- more, due to the closed-nature of many commercial language models, including text-davinci-0033 we do not know what their training data contains. It is possible that the training set for text-davinci-003 in- cludes the items from MNLI (where the LIVENLI items are sourced from) and therefore contributed to some of the advantages of the models shown here.
5 Conclusion and Discussion
We built LIVENLI providing ccologically valid ex- planations for NLI items with label variation. We showed that LIVENLI's is explanations help under- stand where label variation stems from, and identify Question Under Discussion as additional source of label variation. We further emphasize a deeper level of variation: within-label variation. Our work illustrates the utility of explanations and provides future avenues for improving model capability of leveraging and generating explanations.
Broader Impact
We showed that GPT-3 can generate explanations that are detailed and closely resembling the struc- ture of human explanations. However, the ex- planations may not be as grounded in the input premise/hypothesis and the labels they predict. This can lead to potential misuse the explanations can be used to manipulate humans into believing the conclusion/label being explained, especially when the explanations are specific and detailed, giving the impression of correctness and authority. Therefore, these models should not be deployed without further studying their social implications.
Limitations
The LIVENLI items comes from the MNLI dataset, which is known to contain annotation artifacts and biases (Gururangan et al., 2018; Geva et al., 2019). It is possible that those biases also contribute to the variation found here. Further investigations on other datasets or tasks are needed to better un- derstand the prevalence of the label variation and within-label variation.
As shown in A1, filtering responses by answers to control items the usual technique for qual- ity control and reducing noise no longer works here. It is possible that using different set of con- trol items, perhaps hand-written instead of sourced from an existing corpus, would yield different re- sults. In any case, the most effective way to reduce noise is to control quality of the workers, which is what Nie et al. (2020) and we did. Still, noise is in- evitable in any data collection process, and we did find some amount of noise in LIVENLI: there are 6 responses from 3 items where the explanations reveal that the annotators misread the sentences. Those 6 responses have labels that differ from the rest of the responses for those 3 items. Therefore, a small amount of label variation found here is contributed by noise.
A Acknowledgments
We thank Michael White, Micha Elsner, the mem- bers of the Clippers group at OSU, the C.Psyd group at Comell, and the CHAI group at UChicago for their insightful feedback. We thank the work- ers at Surge Al who provided valuable data that enabled this work. This material is based upon work supported by the National Science Founda- tion under grant no. IIS-1845122. Marie-Catherine de Marneffe is Research Associate of the Fonds de la Recherche Scientifique FNRS.
References
Lora Aroyo and Chris Welty. 2015. Truth Lie: Crowd Truth and the Seven Myths of Human Anno- tation. AI Magazine, 36(1):15.
Our dataset is English-only. Studying label vari- ation in other languages may reveal new linguistic phenomena contributing variation that are not seen in English, and further reshape the taxonomy of reasons for label variation.
BIG-Bench authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of lan- guage models. Transactions on Machine Learning Research.
Our evaluation of the explanations generated are on small sample and is performed by one au-
Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. 2022, Stop measuring calibration when

---

humans disagree. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 1892-1915, Abu Dhabi, United Arab Emirates, Association for Computational Lin- guistics.
Nan-Jiang Jiang and Marie-Catherine de Marneffe. 2022. Investigating Reasons for Disagreement in Natural Language Inference. Transactions of the Association for Computational Linguistics, 10:1357- 1374.
Andrew Lampinen, Ishita Dasgupta. Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell. James McClelland, Jane Wang, and Felix Hill. 2022. Can language models learn from explanations in con- text? In Findlings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 537-563, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel BerrerttVosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
David B. Leake. 1991. Goal-based explanation evalua- tion. Cognitive Science, 15(4):509-545.
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta. Noah A. Smith, and Yejin Choi. 2023. We 'e afraid language models aren't modeling ambiguity.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natu- ral language inference with natural language explana- tions. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Shashi Narayan, Joshua Maynez, Reinald Kim Am- playo. Kuzman Ganchev, Annie Louis, Fantine Huot, Dipanjan Das, and Mirella Lapata. 2022. Conditional generation with question- answering blueprint.
Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on nat- ural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131 11193 Online. Association for Computational Linguistics.
Oana-Maria Camburu, Brendan Shillingfod,, Pasquale Minervini, Thomas Lukasiewicz, and Phil Blunsom. 2020. Make up your mind! adversarial generation of inconsistent natural language explamations. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 4157- 4165, Online. Association for Computational Lin- guistics.
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transuc- tions of the Association for Computational Linguis- tics, 7:677-694.
Kordula De Kuthy, Nils Reiter, and Arndt Riester. 2018. QUD-based annotation of discourse structure and information structure: Tool and evaluation. In Pro- ceedings of the Eleventh International Conference on Language Rexources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources As- sociation (ELRA).
Barbara Plank. 2022. The "problem" human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 10671-10682, Abu Dhabi, United Arab Emirates. A Association for Computational Linguistics.
Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019, Are we modeling the task or the annotator? an inves- tigation of annotator bias in natural language under- standing datasets. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLLP pages 1 1161-1166, Hong Kong. China. Association for Computational Linguistics.
Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Learning pfft-of-speech taggers with inter-annotator agreement loss. In Proceedings of the 14th Confer- ence of the European Chapter of the Association for Computational Linguistics, pages 742-751.
Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina, Juntao Yu, and Heike Zinsmeister. 2018. Anaphora resolu- tion with the ARRAU corpus. In Proceedings of the First Workshop on Computational Models of Refer- ence, Anaphora and Coreference, pages 11-22, New Orleans, Louisiana. Association for Computational Linguistics.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language infer- ence data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Popers), pages 122 New Orleans, Louisiana, Association for Computa- tional Linguistics.
Craige Roberts. 2012. Information structure in dis- course: Towards an integrated formal theory of prag- matics. Semantics and Pragmatics, 5.

---

Chenhao Tan, 2022. On the diversity and limits of human explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2173-2188, Seattle, United States. Association for Computational Lin- guistics.
Appendix
A1 Details on Data Collection Procedure
Annotators were paid $0.6 per item (hourly wage: $12 to $18). Each item has at least 10 annotations. On average, each annotator labeled 29.3 items. Our data collection was performed with IRB approval. SurgeAI guarantees that the annotators are native English speakers and have passed certain reading and writing tests. Figure shows a sereenshot of the annotation interface.
Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Sil- viu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 12:1385-1470.
(1) shows an example where the original MNLI annotations are unanimously Neutral but exhibit- ing genuine variation in LIVENLI annotations, to- gether with the LIVENLI explanations.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le. and Denny Zhou. 2022. Chain thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
(1) P:
The original wax models of the river gods on display in the Civic Museum. H: Thousands of people come to the wax models.
Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2022. Reframing
human-Al collaboration for generating free-text ex- planations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, pages 632-655 658, Seattle, United States. Association for Computational Linguisties.
original MNLI: [0.1,0]
LIVENLI [E,N.C]: [.23, 163..7.,, .7,, 04]
N context refers to the wax model dis- plays in the museum. The context makes no mention of the number of visitors mentioned in the statement.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume I (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
E Museums are generally places where many people come, so if the original wax models are there is likely thousands of people will come to see them.
Entailment Contradiction It's unlikely a museum could stay open for very long without thousands of visitors, so it's likely true that thousands of people come to see these mdoels. Unless, of course, it's big museum with many attractions more interesting than the models, which case the statement is likely be false.
Xi Ye and Greg Durrett. 2022. The unreliability of ex- planations in few-shot prompting for textual reason- ing. Advances in Neural Information Processing Systems.
Yi Zhang, Sujay Kumar Jauhar, Julia Kiseleva, Ryen White, and Dan Roth. 2021. Learning to decom-
pose and organize complex tasks. In Proceedings of the 2021 Conference of the North American Chap-
A2 Details on Taxonomy Reannotations
ter of the Association for Computational Linguistics:
Human Language Technologies, pages 2726 2735, Online. Association for Computational Linguistics.
Table shows the frequency of taxonomy reanno- tations. Figure compares our reannotations with Jiang and Marneffe (2022)'s annotations.
Xiang Zhou, Yixin Nie, and Mohit Bansal. 2022. Dis-
tributed NLI: Learning to predict human opinion dis- tributions for language reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 972 987, Dublin, Ireland. Association for Computational Linguistics.
Yangqiaoyu Zhou, Yiming Zhang, and Chenhao Tan. 2023. Flame: Few-shot learning from natural lan- guage explanations.

---

Read the following context and statement:
Context: Could you pleese speak issue, with regerd the social ramifications of gum chewing in public?
don't opinion on gum chewing public,
Choose one or more from the following:
you feel uncertain you multiple options apply, choose instead, even though it might feel contradictory.
Assuming context true,
most likely to be true
be either false
most likely to false
Explain, in few sentences, why you chose your answer.
chose than option, elaborate which circumstances each option possible.
Explein options you
specific parts sentences. should NOT simply repeat the sentences Avold "The context and statement means the same/opposite thing". Specify which part of the context and statement means the samejopposite thing.
Your expi anation should Include new Information
Avoid "Just because doesn't
under what ciroumstances does say that Y possible meanings why ambiguous.
ambiguousfit's
Minimum word count: 10 Words:
Highlight the words the Context and Statement that relevant your explanations. Highlight phrases your meniloned.
should words/parts Only highllght the words that are most important for the explanations.
Figure 2: Screenshot the data collection interface.
Combination of Taxonomy Categories
Frequency
Prohabilistic
28
Lexical
QUD
15
Coreference
Lexical Probabilistic
Coreference Probabilistic
Imperfection Lexical
Implicature
Imperfection Probabilistic
Prohabilistic QUD
Coreference Lexical
Lexical I QUD
Temporal
Appear once:
Implicature Interrogative; Implicature Lexical: Interroga- tive; Implicature QUD: Imperfection QUD: Interrogative Lexical; Coreference Temporal; Coreference Imperfection; Coreference QUD; Implicature Interrogative; Implicature
Lexical; Interrogative; Implicature Lexical Prohabilistic QUD; Probabilistic Temporal; Presupposition QUD: QUD Temporal
Table 5: Frequency of the taxonomy categories in LIVENLL. Label variation in annotators arises for sim- ilar reasons what Jiang and de Marneffe (2022) hy- pothesized. QUD is new category.

---

Coreference
4
Imperfection
Jiang and Hamette (2022) annotations
Interrogative
Lexical
12
Presupposition
Probabilistic
Temparal
Coreference, Imperfection onneftenncelmpeefeciioonn
Coreference,
Probabilistic
Lexical Probsbilisticc
Lexical, Teeppal
Presusposition
Probabillstic.Tempaaa
Lexical, Probebilistic.QUD
Coreference
Implicature
Interrogative
Lexical
disegreent Desgreemenn
Probabilistic
QUD
Temparal-
Coreference.Inne wwa.nneeemmeriiii pereectttooon
Coreferencel Lexical Lexical
Coreference, Probaoiistii
Coreference. OUD
Coreference Temporal
Imper fection, Lexicall
Imperfectt Imperfection, Decciinn,Pooii Probabbiiiii
Imperfection QUD
implicature.iini amplictuureiineemoon www.reiineemmggge linteeogative
pmpllcatuue..c Lexical
Implicature QUD
Interroattive Lexical-
Levical Po Probabllsstt
Lexical QUD
iessuppsntiic .UD
osstiontiionnoo Probabil .00D
Probabilizs probabilettic Temparal
QUD Temporal
Lexical, Probebi isbic QUD
ours
Figure Contingency matrix of current taxonomy reannotations vs. Jiang and de Marneffe (2022)'s annotations.

---

A3 Details on Prompting Setups
train
test total
LIVENLI
LIVENLI
ChaosNLI
E
1
37
18
19
N
1
37
13
24
C
1
37
8
29
NC
2
37
26
11
EC
2
37
36
EN
2
37
2
ENC
2
37
10
27
Total
11
259
111
148
Table 6: Number of items of each label in the train/test split from each source.
Instruction Given context, a statement can be true, false or either true or false. Different people may interpret the sentences differently and have differentt judgments. You are an assistant helping users identify what possible judgments peuple can have and provide an explanation each judgment. need to identify as many possible judgments as possible.
User:
Context: While probably that democracies unlikely go unless they're attacked, sometimes they are the first take the offensive.
Statement:
Democracies probably won't go unless
attacks them their soil
Assistant:
Given the context, some people consider the statement to be true, some people consider the statement to be either true false.
Explanations It can be true because nation is being attacked, odds are it's occuring on its soil (there's 's far more of that than it's likely true democracies won't go war unless they're attacked, say, foreign military bases whatever). The statement must and since the context taken can be either truc false because the the context. This could be false since the context only mentions being attacked, where the attack would tuke place.
that
well.
adds
part with "on their soil" which explicitly mentioned in
Table Training example prompt for the Predict-then-Explaee setup. The boxed words indicate components of the prompt and not appear in the actual prompt. The Instruction only appears once before all the training items. The Explain-ther -hen-Pr prompt has the Prediction following Explanations The Predict-only prompt does not include Explanations

---

A4 Evaluating the Generated Explanations Table shows the definitions of the four binary criteria and examples of explanations judged ac- cording to these criteria.
Note that the sentence content partially affects the ratings for Specificity and New Information. If the label is straightforward and there is not much be explained, good explanation does not nec- essarily need be specific. Similarly, for New Information, the sentences may not discuss any real world events/entity, and there is not much new information that can or need to be added.
Table 9 shows the evaluation results breakdown by labels.
Hide model identity for evaluation The identity of the model is hidden by only presenting the por- tion the text that corresponds to the explanation and shuffling the order. The gold label distribution of the item is also hidden.

---

Specificity
Specific/ Generic
Is the explanation being detailed and specific in explaining the label, or is it more generically describing what the sentences state (or do not state) verbatim?
Plausible Implausible Is the interpretation inferences drawn from the context/statement plausible, that is, can someone reading the con- text/statement possibly read them in this way? If the explanation is generically describing what the sentences state (or do not state). is the description accurate?
Plausibility
Support / Not Support Does the explanation supports the lahel?? Does the conclusion reasonahly follow the description of the premise/typotheis?? If the explanation is implausible or inaccurately described the sentences, disregard the discrepancy and assume the explanation to be true.
Support the label
New Information
New info No new info
Does the explanation introduce facts information about the subject matter?
P: and to have children and just get day care or somcone to take care of it and not really have the honding process that takes place with babies and stuff you know
H: The day care is perfect for the children.
Specific Plausible
Support
New info
is not known from the context what the preference the parent is toward day care or having another person take care the child and the preference may vary from person to person. Furthermore, the statement does not clarify what the that individual.
definition of "perfect"
Specific Plausible
Not Support No new info
C- taking care of children and the bonding prucess are two different things one cannot infer that day care can provide same process as natural bonding that takes place after birth of child.
Specific Implausible Not Support No info
C- the bonding process with babies was mentioned in the context, suggesting that getting day would not idcal for the children's needs. The statement implies that the day care is perfect, which may be false.
Generic Implausible Support
No new info
-hehe context is speaking in general and does not specify that the day care is perfect. It just states that someone will take
care of the children.
Table 8: Definitions of the four proposed criteria: Specificity, Plausibility, Support the label, and Introduce New Information, and examples of explanations judged according to these criteria.
Plausible & Support
Setup
Label
Plausible
Support
Specific
New Info
Total
E
76.5
88.2
70.6
35.3
5.9
17
E-P
N
65.2
87.0
56.5
47.8
17.4
23
C
82.1
71.4
57.1
32.1
7.1
28
E 87.5
93.8
81.2
25.0
6.2
16
P-E
N
68.0
88.0
60.0
44.0
4.0
25
C
79.3
69.0
55.2
37.9
6.9
29
Table 9: Percentage of explanations of each model with each label, marked as being Specific, Plausible, Supporting the label, and Introduce New Information. An explanation is considered to be valid it is both plausible and supporting the label.


Reviews:
[{"review_id": "SJ6HAq68sq", "reviewer": "Reviewer_sZVS", "summary": "", "questions": "See reasons to reject", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper. We appreciate the recognition of human label variation as an important problem. We would like to point out that **our main contribution lies in the collection of ecologically valid explanations of label variation**. We were hoping to spur interest in the wider NLP community about variation in annotations, and we are glad that the reviewers recognized that aspect of our work.\n\n\n### Re: Size of the dataset and Lack of Broad Model Evaluation\nWe understand the limitations of this work being narrowly scoped, focusing on one task and collecting a small dataset. We intended our work to be a focused contribution, a proof of concept, and an initial exploration in this topic as a short paper. Even though the number of items (122) is small, we collected a large number of explanations for each item (10+, most explanation datasets contain only 3 or 5 explanations and sometimes even just one.). We performed manual examinations on all the explanations, as automatic methods that we experimented with did not reveal much insights. We only tested one model due to space constraints, and we did not have access to the more powerful GPT-4 at the time. We will address these limitations in the Limitations section in our revision. \n\n\n### Re: Unclear Generalizability\n\nEven though we only analyzed the NLI task, our findings can inform approaches to other tasks such as fact-checking. For example, our findings suggest that people’s judgments vary because they consider different Questions Under Discussion(QUD), which in turn suggests that variation can be reduced if we make QUDs explicit. This is important in the task of fact-checking, where all the QUDs need to be considered and a claim is only true if all of its possible implications, i.e. answers to all QUDs, are true. [Chen et al 2022](https://aclanthology.org/2022.emnlp-main.229/)) presents a step in this direction. They introduced a dataset/method for fact-checking complex claims where each claim is accompanied by a set of questions (that can be considered as QUDs), and a claim is fact-checked by checking answers to all these questions. By making questions explicit, it can reduce potential variation and facilitate checking the entire claim.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors' work presents new data resources, provides a new perspective on the interpretive process in NLI tasks, and conducts an NLP engineering experiment. The dataset presented can be a valuable resource for further research on NLI and the human label variation problem, providing both raw data and insights.\n- Exploration of Label Variation: The paper tackles the understudied issue of human label variation in NLI, shedding light on the factors that lead to such variation. This understanding can lead to more sophisticated approaches in NLI and similar tasks, by acknowledging and incorporating label variation rather than attempting to oversimplify the problem.\n- Identification of Within-Label Variation: The authors identified and highlighted a new type of variation, within-label variation, where annotators may choose the same label for different reasons. This finding adds a new layer of complexity to our understanding of label interpretation and could lead to more nuanced models and strategies in NLI tasks.\n", "reasons_to_reject": "- Size of the Dataset: The LiveNLI dataset, though novel and useful, is relatively small, with only 122 re-annotated MNLI items. Its small size may limit its effectiveness in providing comprehensive insights into label variation across a wide range of contexts and applications.\n- Lack of Broad Model Evaluation: The paper evaluates the ability of GPT-3 to generate explanations for label variation. However, the evaluation does not extend to other state-of-the-art models, e.g., GPT-4, which could offer a broader perspective on AI's capacity to handle such complex tasks.\n- Unclear Generalizability: The findings mainly pertain to label variation in NLI tasks. While these findings are valuable, it remains unclear how broadly they apply to other NLP tasks. More discussion or analysis on the generalizability of the findings to other areas of NLP would enhance the paper's impact.", "questions_to_authors": "See reasons to reject", "paper_topic_and_main_contributions": "This paper addresses the issue of human label variation in Natural Language Inference (NLI) tasks and explores how such variation arises. The authors observe that traditional practices for operationalizing annotations in NLP datasets have largely assumed a single label per item, a simplification that doesn't accurately capture the complexity of human label variation. This work explores where label variation in NLI comes from and how it can be better understood and potentially leveraged in future tasks.\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA"}, "extra_text": {}}, {"review_id": "gWPh7jLGLM", "reviewer": "Reviewer_9znK", "summary": "", "questions": "- In lines 40 - 42: \"each with at least 10 highlights and free-text explanations for the labels chosen by the annotators, totaling 1,415 explanations\", however, in lines 62 - 63: \"We re-annotated 122 items from the MNLI dev set where each item has 5 annotations\". How many annotations did you do exactly? I am confused here.\n- I don't quite get how you use text-davinci-003 to do multilabel classification with explanations. Do you instruct the model to output multiple labels at the same time and give explanations to each label? It would be helpful if you can put the prompts you used to generate the results in the appendix.\n- When using text-davinci-003, how many examples do you use as few-shot demonstrations and how stable those results are for different GPT calls? I am asking this because the test set is small and sometimes the instabilities of GPT calls can yield different conclusions.\n- Did you try zero-shot prompts and compare the results with few-shot prompts?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper.\n\n\n### Re: reasons to reject \n\n> I am not quite convinced by the conclusion stated in Line 221 \"This adds to the series of results where having post-prediction explanations outperforms chain-of-thought style prompting\"\n\nOn line 221, we were mainly comparing Predict-then-Explain vs. Explain-then-Predict. \n\n> Just like the authors mentioned in the Limitation section, it could be that the training data of text-davinci-003 already covers the test set, so that directly predicts a label achieves higher performance. I think to rule out this possibility, the authors could gather 10 or 20 new examples and test the same methods, which would make the results more convincing.\n\n\nYou are right that this could be a possibility. Collecting a few new examples takes more time than we have available during the author response period. Instead, we briefly tested the hypothesis of whether the model could be memorizing the MNLI labels. We zero-shot prompted 10 examples from the MNLI dev set to see if it memorizes the original labels. The prompt format is: “premise: ___ hypothesis: ___ entailment/neutral/contradiction? label:”. We specifically chose examples where only 3 out of 5 MNLI labels agreed, and they differ from LiveNLI labels. The goal is to have the original MNLI majority voted labels be more of an arbitrary choice, and if the model predicts the same label as the MNLI label, it is likely due to memorization. We found that the model predicts the same MNLI label on 5 (out of 10) examples, suggesting that even if the MNLI test set is in the training data, it does not consistently memorize the labels. \n\nOn the other hand, it is possible that training on the MNLI test set leads to other advantages such as being able to better extrapolate task-specific patterns and generate better explanations. There is indeed no definite conclusion to be drawn without more information about how the model text-davinci-003 was built, which is sadly information that is not available.\n\n### Answers to your questions \n\n- > In lines 40 - 42: \"each with at least 10 highlights and free-text explanations for the labels chosen by the annotators, totaling 1,415 explanations\", however, in lines 62 - 63: \"We re-annotated 122 items from the MNLI dev set where each item has 5 annotations\". How many annotations did you do exactly? I am confused here.\n\nWe collected at least 10 annotations for each item for LiveNLI. On line 62-63, we meant that the authors of MNLI collected 5 annotations for the dev set. We will rephrase to avoid the ambiguity.\n\n- > I don't quite get how you use text-davinci-003 to do multilabel classification with explanations. Do you instruct the model to output multiple labels at the same time and give explanations to each label? It would be helpful if you can put the prompts you used to generate the results in the appendix.\n\nHere’s the prompt format – Context: ____ Statement: ___ Given the context, some people consider the statement to be _lable1_, some people consider the statement to be _label2_. It can be label1 because ___ It can be label2 because ___. We will add the prompt in the appendix in the final version.\n\n- > When using text-davinci-003, how many examples do you use as few-shot demonstrations and how stable those results are for different GPT calls? I am asking this because the test set is small and sometimes the instabilities of GPT calls can yield different conclusions.\n\nAs we mentioned on line 204, we used 11 examples for few-shot demonstrations. In Table 2, we reported standard deviation of the metrics from two random splits. We can see that E-P underperforms the other two models in both runs, but the differences between P vs. P-E are small.  \n\n- > Did you try zero-shot prompts and compare the results with few-shot prompts?\n\nWe did not try zero-shot prompts for multilabel classification. We will consider this direction for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The introduction of the LIVENLI dataset, focusing on ecologically valid explanations, fills a gap in the field and provides a valuable resource for future research.\n- The authors conducted thorough human annotation and analysis to investigate the label variations of NLI, and they find that Questions Under Discussion (QUDs) is an important factor in label variation which adds a new dimension to the taxonomy of prior work.\n- The scope of this paper is well-suited for a short paper and the paper is well-presented.", "reasons_to_reject": "- I am not quite convinced by the conclusion stated in Line 221 \"This adds to the series of results where having post-prediction explanations outperforms chain-of-thought style prompting\". Just like the authors mentioned in the Limitation section, it could be that the training data of text-davinci-003 already covers the test set, so that directly predicts a label achieves higher performance. I think to rule out this possibility, the authors could gather 10 or 20 new examples and test the same methods, which would make the results more convincing.\n- I am a bit confused by several parts in the paper. See questions below for details.", "questions_to_authors": "- In lines 40 - 42: \"each with at least 10 highlights and free-text explanations for the labels chosen by the annotators, totaling 1,415 explanations\", however, in lines 62 - 63: \"We re-annotated 122 items from the MNLI dev set where each item has 5 annotations\". How many annotations did you do exactly? I am confused here.\n- I don't quite get how you use text-davinci-003 to do multilabel classification with explanations. Do you instruct the model to output multiple labels at the same time and give explanations to each label? It would be helpful if you can put the prompts you used to generate the results in the appendix.\n- When using text-davinci-003, how many examples do you use as few-shot demonstrations and how stable those results are for different GPT calls? I am asking this because the test set is small and sometimes the instabilities of GPT calls can yield different conclusions.\n- Did you try zero-shot prompts and compare the results with few-shot prompts?", "paper_topic_and_main_contributions": "This paper presented the LIVENLI dataset, containing 1,415 ecologically valid explanations for 122 MNLI items, to explore human label variation in natural language inference (NLI). The authors analyze label variation, including within-label variation, and find that annotators\n sometimes choose the same label for different reasons, which suggests that explanations are crucial for navigating label interpretations in\n general.  They also experiment with using LLMs to predict and explain label variation, emphasizing the need for diverse explanations to imitate human decision-making. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UT4E6JevoO", "reviewer": "Reviewer_SDvm", "summary": "", "questions": "- One could argue for an alternative perspective that the label variation problem is simply caused by some mistakes in the annotators (either those who annotated the original MNLI dataset or those recruited in this paper). Would you like to comment on this argument?  ", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper. We were hoping to spur interest in the wider NLP community about variation in annotations, and we are glad that the reviewers recognized that aspect of our work.\n\n### Re: lack of focus\n\nAs pointed out in our introduction, this paper builds on Jiang and de Marneffe (2022): that previous work offered an in-depth analysis of the NLI task, answering many of your questions. Here we build on Jiang and de Marneffe (2022) and investigate the problem of label variation through explanations. That is, *the focus of this paper is to understand the source of label variation*, through explanations provided by the annotators themselves (i.e., ecologically valid).\n\nWithin-label variation is something we discovered while analyzing label variation.  We did touch on why people can have different reasons for the same label (e.g. coreference, Question Under Discussion, line 151 - 172), and we can expand this analysis in the final version.\n\nAs you pointed out, within-label variation presents its own set of questions that warrants future research. Therefore, we hope that our paper can present these questions to the NLP community and a resource to start answering those questions, or encourage others to build more resources.\n\n### Re: the term “ecologically valid”\n\nWe meant that the explanations are ecologically valid, in that they are provided by the annotators who provide the labels. In comparison, other explanations datasets only collected the explanations while taking the ground truth answers for granted (e-SNLI, Camburu et al 2015, or ECQA Aggarwal et al 2021). \n\n\n### Re: annotator mistakes \n\nSome of the variation might indeed be due to genuine errors from the part of the annotators, as acknowledged in our Limitations section. In LiveNLI, we found 6 annotations (out of 1,415) where the explanations revealed that the annotators read the sentences incorrectly. The number of genuine mistakes is however small and label variation persists even if we discard those 6. The rest of the labels are all provided by the annotators with their best effort, as shown by the quality of their explanations. In ChaosNLI, the authors took great care into vetting the annotators, making sure that they paid attention. If label variation persists (and we show that it does), it is part of human behavior that is worth studying.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The label variation is an important issue.\n- This paper shows evidence that label variation occurs in both humans and LMs.  \n- This paper analyzes both the label variation and the within-label variation issues.  \n", "reasons_to_reject": "- Overall, this paper appears to lack a focus. The label variation and within-label variation appear like two distinct research problems, and I expect each question to be more thoroughly analyzed in a conference-level publication. These are two distinct problems, and I think analyzing one problem to a sufficient depth can constitute of a solid paper. If the central problem is to analyze the label variation, then I expect more discussions about the NLI task itself: are there naturally multiple possibilities towards the problems? Are there some mistakes in the dataset's labels? Why do the annotators write different reasons (for the within-label variation problem)? If the central problem is the within-label variation (of the models), then I expect to see some hypotheses about the models' mechanisms, some analysis to support these hypotheses, and some attempts towards mitigating the within-label variation problems.  \n- The problem scope is limited. While there are indeed a lot of efforts towards the NLI problem, it would be great that there is at least one other non-NLI datasets.  \n- A minor complaint: The term \"ecologically valid\" appears odd to me. Why do the annotated labels become ecologically valid when they come with explanations?", "questions_to_authors": "- One could argue for an alternative perspective that the label variation problem is simply caused by some mistakes in the annotators (either those who annotated the original MNLI dataset or those recruited in this paper). Would you like to comment on this argument?  ", "paper_topic_and_main_contributions": "This paper considers the problem of human label variation in natural language inference. \nFirst, this paper collects LiveNLI, a dataset containing 1415 annotation+explanation pairs for 122 MNLI items. This dataset confirms the label variations. Further, this dataset leads to another problem: within-label variation (annotators choose the same label for different reasons).  \nThen, this paper dives into the within-label variation problem. They few-shot prompt LMs to generations. The prompting experiments lead to findings including that the explanations sometimes do not support the labels.\n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "- Line 7: MNLI needs a citation (this is the first time it occurs in this paper).  ", "Justification_for_Ethical_Concerns": "I do not see ethical concerns with this paper. Note that the procedures in this paper passed an IRB approval."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses an interesting problem, one of the common objections is the limited scope and the small size of the provided resource. However, considering that it is a short paper, the scope and the contribution of the paper are adequate.
Decision: Accept-Findings