VARIERR NLI: Separating Annotation Error from Human Label Variation (2024)

Abstract: Human label variation arises when annotators assign different labels to the same item for valid reasons, while annotation errors occur when labels are assigned for invalid reasons. These two issues are prevalent in NLP benchmarks, yet existing research has studied them in isolation. To the best of our knowledge, there exists no prior work that focuses on teasing apart error from signal, especially in cases where signal is beyond black-and-white. To fill this gap, we introduce a systematic methodology and a new dataset, VARIERR (variation versus error), focusing on the NLI task in English. We propose a 2-round annotation procedure with annotators explaining each label and subsequently judging the validity of label-explanation pairs. VARIERR contains 7,732 validity judgments on 1,933 explanations for 500 re-annotated MNLI items. We assess the effectiveness of various automatic error detection (AED) methods and GPTs in uncovering errors versus human label variation. We find that state-of-the-art AED methods significantly underperform GPTs and humans. While GPT-4 is the best system, it still falls short of human performance. Our methodology is applicable beyond NLI, offering fertile ground for future research on error versus plausible variation, which in turn can yield better and more trustworthy NLP systems.

heading: 1 Introduction
text: Labeled data plays a crucial role in modern machine learning (ML) (e.g., Mazumder et al., 2023). Data quality impacts ML performance and, in turn, user trust. It is therefore of vital importance to aim at high-quality consistently-labeled benchmark data (e.g., Bowman and Dahl, 2021). However, recent research revealed a notable presence of annotation errors in widely-used NLP benchmarks (Klie et al., 2023; Rücker and Akbik, 2023). Similar observations were made recently in computer vi-
∗ Equal contribution.
sion (CV) (Northcutt et al., 2021; Vasudevan et al., 2022; Schmarje et al., 2024).
At the same time, there is increasing evidence that for many items in many tasks, more than a single label is valid. For some items, systematic variation exists for valid reasons, such as plausible disagreement or multiple interpretations. In other words, the world is not just black and white. Human label variation (HLV, as termed by Plank 2022) has been shown on a wide range of NLP tasks (de Marneffe et al., 2012; Plank et al., 2014; Aroyo and Welty, 2015), including in natural language inference (NLI; Pavlick and Kwiatkowski 2019; Zhang and de Marneffe 2021). NLI involves determining whether a hypothesis is true (Entailment), false (Contradiction), or neither (Neutral), assuming the truth of a given premise; see Figure 1 for an example with plausible labels.
Although high-quality, consistently labeled data may initially appear to conflict with the goal of accommodating HLV, it is important to emphasize that we do not perceive these as contradictory goals. While HLV exists, so do errors. We assert that annotators are inevitably prone to make errors, such as misunderstanding instructions or accidentally
2256
selecting a wrong label. Optimizing data quality is essential through providing clear instructions and effective training, and identifying annotation errors yields better datasets (Larson et al., 2019). However, still little is known about what constitutes an error versus plausible variation. We lack both a theory and operationalizable procedures to tease apart error from plausible HLV consistently and soundly. Some datasets with errors (and their corrections) exist, and there has been work on automatic error detection (AED). However, both have their limitations (§2). A crucial gap remains: a lack of examination in real-world scenarios where the signal is nuanced, not merely black-and-white.
To address this gap, this paper contributes: (i) VARIERR, a novel multi-annotator English NLI dataset with both plausible variation and detected errors. To the best of our knowledge, no such dataset exists yet. (ii) A new methodology to detect errors: we collect multiple annotations, where each label comes with an ecologically valid explanation inspired by Jiang et al. (2023), and propose to pair them with validity judgments to identify errors. (iii) Finally, we benchmark existing AED methods and GPTs in a challenging setup, where the task is to tease apart error from plausible human label variation. Our findings indicate that existing AED methods underperform humans and GPTs substantially on our self-validated VARIERR NLI dataset. We release our data and code to facilitate uptake.1

heading: 2 Related Work
text: Labeled data is the fuel of machine learning, as it drives both learning and evaluation. Following a data-centric view, we focus on improving data quality over data quantity (Motamedi et al., 2021; Swayamdipta et al., 2020; Zhang et al., 2021; Gordon et al., 2022). We aim to bring together work on data quality from two ends: annotation error vs. human label variation.
Annotation Errors and AED Several recent work found errors in widely used benchmarks, such as CoNLL 2003 for Named Entity Recognition (Wang et al., 2019; Reiss et al., 2020; Rücker and Akbik, 2023), TACRED for relation extraction (Alt et al., 2020), WSJ for syntax (Manning, 2011; Dickinson and Meurers, 2003), and ImageNet for object classification (Beyer et al., 2020; Northcutt et al., 2021; Vasudevan et al., 2022).
1https://github.com/mainlp/VariErr-NLI
AED has a long-standing tradition in NLP. Proposed methods range from early work that relies on variation-based methods positing that instances with similar surface forms tend to have the same label (Dickinson and Meurers, 2003; Plank et al., 2014) to more recent model-based approaches that either exploit predictions (Amiri et al., 2018; Arazo et al., 2019) or information derived from training dynamics (Swayamdipta et al., 2020); see Klie et al. (2023) for a survey on AED.
Flaggers and scorers for AED have been proposed (Klie et al., 2023). Flaggers detect errors by providing a hard decision on whether an instance is erroneous. Scorers, on the other hand, assign a score to each instance reflecting the likelihood of being an error, and the top-n scoring instances are then corrected. Here, we focus on scoring methods to rank instances. Most of the AED work mentioned has limitations as they either rely on posthoc mining of errors (and might therefore miss out on errors) in semi-automatic ways (e.g., Reiss et al., 2020), or they inject synthetic noise which has been shown to result in datasets where errors are easy to spot (Larson et al., 2019). Instead of using synthetic noise, we focus on realistic setups and re-annotate data in ecologically valid ways.
Human Label Variation (HLV) Recent studies have drawn attention to HLV in NLP (i.a., Uma et al., 2021; Plank, 2022). HLV has been described as annotator disagreement, which is not just noise but also signal since a sign of vagueness or ambiguity can benefit models (Aroyo and Welty, 2013). These include judgments that are not always categorical (de Marneffe et al., 2012), inherent disagreement (Pavlick and Kwiatkowski, 2019; Davani et al., 2022), or justified and informative disagreement (Sommerauer et al., 2020). For subjective NLP tasks, which by essence encourage annotator subjectivity (and hence variation), there is also a line of work referred to as perspectivism (Cabitza et al., 2023), connected to the descriptive data annotation framework proposed by Rottger et al. (2022).
HLV in NLI This paper focuses on NLI, known to contain HLV (Pavlick and Kwiatkowski, 2019; Nie et al., 2020; Jiang and de Marneffe, 2022; Jiang et al., 2023). Pavlick and Kwiatkowski (2019) reannotated nearly 500 NLI instances with 50 crowdworkers and showed that disagreements in NLI cannot be dismissed as annotation “noise.” ChaosNLI (Nie et al., 2020) pioneers large-scale NLI annota-
tion by collecting 100 annotations per instance for 3K items from SNLI (Bowman et al., 2015), αNLI (Bhagavatula et al., 2020), and MNLI (Williams et al., 2018) but for which the original annotations did not yield high agreement. They show that, for most of the items, HLV persists with more annotations. Further, their experiments show a large room for model improvement and a positive correlation between human agreement and label accuracy.
In another line of work, Jiang and de Marneffe (2022) identified reasons for observing variation in NLI, deriving a taxonomy based on linguistic properties of the items. Following up on that work, Jiang et al. (2023) proposed LIVENLI, to gain insights into the origins of label variation. They reannotated 122 NLI instances from ChaosNLI with ecologically valid explanations: annotators are instructed to not only provide NLI labels but also explanations for their label choices. This addresses a limitation of prior work that uses post-hoc explanations, which may not reflect the true reasons of the original annotators, thereby questioning the validity of the prior method. They show that ecologically valid explanations have an additional benefit: signaling within-label variation, i.e., annotators give the same label but for different reasons. While we do not focus on the latter here, we take inspiration from Jiang et al. (2023) to collect ecologically valid explanations (cf. §3.1).
To the best of our knowledge, there remains a gap for studies on both annotation errors and human label variation in a concentrated effort. It is thus an open challenge to define error in an ecologically valid way, and it is unknown to what extent existing AED methods help detect such errors and whether new methods are needed. To find answers to these challenging open questions, we believe it is important to move both directions forward.

heading: 3 VARIERR: Annotation Procedure
text: To tease apart human label variation from error, we create VARIERR (Variation versus Error), a NLI dataset with two rounds of annotations by four annotators:2 Round 1 for NLI labels and explanations (§3.1) and Round 2 for validity judgments (§3.2). Table 1 presents two VARIERR examples with tworound annotations, as well as their deduced label variations and errors to be discussed in §4.
2Annotators are Master’s students in Computational Linguistics and the first author of this paper, all paid according to national standards.

heading: 3.1 Round 1: NLI Labels & Explanations
text: We collect annotations from four annotators on 500 NLI items randomly sampled from the MNLI subset of ChaosNLI. Annotators were asked to provide not only one or more NLI labels (E: Entailment, N: Neutral, C: Contradiction) to each item but also a one-sentence explanation for each label they chose, as the same label could be chosen for different reasons (Jiang et al., 2023). Annotators could use a fourth “I don’t know” (IDK) label if none of the NLI labels seemed suitable. The Round 1 annotation sums up to 1,933 label-explanation pairs with the standard three NLI labels for the 500 items, and 331 “IDK” annotations (released in the data) which are discarded in Round 2.

heading: 3.2 Round 2: Validity Judgments
text: VARIERR’s key contribution lies in proposing a second round of validity judgment. Validity judgment mirrors conventional annotation adjudication in that annotators judge each other’s NLI labels and explanations. This information is delivered anonymously to annotators to reduce group dynamics. However, rather than agreeing on a single label or explanation altogether, annotators are free to make independent judgments on each label-explanation pair from Round 1, which enables inferring what is an error versus plausible variation (cf. §4.2).
So in Round 2, annotators become judges. For all 500 items, the 1,933 label-explanation pairs from Round 1 are distributed anonymously to the same four annotators. For each NLI item, each judge sees all label-explanation pairs annotated in Round 1, including their own, which they may or may not remember.3 For each label-explanation pair, the annotator judges whether the explanation makes sense for the NLI label, answering “yes” (✓), “no” (✕) or “IDK” (?, I don’t know) as shown in the four right columns of Table 1. Round 2 amounts to 7,732 validity judgments, including 158 “IDK”s.

heading: 4 VARIERR: Detecting Errors
text: Multiple validity judgments on label-explanations enable distinguishing annotation errors from HLV.
3Round 1 took place August-November 2023, and each annotator spent ∼26 hours to annotate 500 items. Round 2 took place from November 2023 to January 2024, and each spent∼15 hours. Annotators worked independently in different weeks. The interval between the end of Round 1 and the start of Round 2 was one month or longer for all annotators.
✓: ‘yes (makes sense)’; ✕: ‘no’; ?: ‘IDK (I don’t know)’; magenta : self-judgments, black: peer-judgments, Err : label error. Curly brackets in label-explanation pairs denote label counters, e.g., in 1a, Before:{E:1,C:4} means that there are one entailment and four contradiction label-explanation pairs before validation.

heading: 4.1 Self versus Peer
text: One consequential feature of our two-round multiannotator procedure is the post-hoc distinction between self- vs. peer-judgments. Self-judgments refer to Round 2 judgments on the judge’s own Round 1 label-explanation annotations ( ✓ , ✕ , ? in Table 1), whereas peer-judgments refer to judgments from other annotators (✓, ✕, ?). Since we have four annotators, each label-explanation pair receives one self-judgment and three peerjudgments. Note that the self vs. peer distinction only enters into effect after data collection.

heading: 4.2 Validating Labels
text: Let A = {a1, ..,a4} be the set of annotators. Self-validated Label-Explanation A labelexplanation pair given by annotator ak on an item in Round 1 is self-validated if ak marks the label-explanation pair as “yes” in Round 2.
Peer-validated Label-Explanation A labelexplanation pair given by annotator ak in Round 1 is peer-validated if the majority (≥2) of the other
three annotators A/{ak} marks the pair as “yes” in Round 2.
For example, the item in Table 1a received the Contradiction (C) label and accompanying explanations from all four annotators in Round 1. Among these four label-explanations, three are selfvalidated ( ✓ ) in Round 2. On the other hand, all four explanations for C are peer-validated since the majority (all in this case) of the peers voted “yes” (✓) for each label-explanation.

heading: 4.3 What counts as an error?
text: In the conventional setup of annotation adjudication, multiple annotators discuss the rationales for their labels and converge to an agreed label. The annotations that are originally different from and subsequently changed to the agreeing label are considered annotation errors. Similarly, in VARIERR, a label-explanation pair might be considered wrong in retrospect (i.e., in Round 2) by the annotator who wrote it after reading all label-explanation pairs given to that item by all annotators.
Thus, in this paper, we use Round 2 self-
judgments—approving or rejecting annotators’ own Round 1 label-explanation pairs—as the criteria for annotation errors.4 We define a NLI label as an error if all label-explanation pairs are not self-validated. In other words, a label is viewed as correctly attributed to an item if any of its explanations is self-validated. In Table 1a, the Contradiction (C) label has at least one self-validated ( ✓ ) explanation (it even has three), and is thus not deemed an error. In contrast, Entailment ( E ) is an error in Table 1a because none of its explanations is self-validated, similarly for E and C in Table 1b.

heading: 4.4 Data Statistics & IAA
text: Table 2 shows the frequencies of NLI labels across the four annotators on the 500 items and 1,933 explanations before and after validation. We include statistics on repeated frequency counts (e.g., E counts twice if it is given as a label by two annotators for the same item) and aggregated labels (repeated labels for a given item count once). Moreover, following Jiang et al. (2023), we compute inter-annotator agreement (IAA) on NLI labels using Krippendorff’s α (for multi-annotator) with MASI-distance (for multi-label).
Since all VARIERR items are sampled from ChaosNLI, which only includes MNLI items with two or all three of the NLI labels in the original annotations by design, we expect HLV and, thus, a medium-to-low IAA in our dataset. Indeed, VARIERR has an IAA of 0.35 (Krippendorff’s α with MASI-distance) before validation, which raises to 0.50 and 0.69 after self- and peer-validations, with the latter reaching substantial agreement (see A.1 for pairwise IAA). However, with the appreciation of HLV, as long as there are no errors in the data, we argue that perfect agreement is not reachable. As a matter of fact, though the repeated and aggregated
4We opted for a strict error definition here. Peer validation could be used in the future but requires additional decisions.
frequencies of NLI labels decrease adequately after validation, HLV still exists in self- and peervalidated annotations, averaging 1.50 (749/500) and 1.28 (642/500) labels/item.
We also observe in Table 2 that 88.57% (1,712/1,933) of Round 1 explanations in VARIERR were self-validated and 82.82% (1,601) were peer-validated. Figure 2a presents the number of label-explanation pairs rejected by both self- and peer-validations, by self-validation only, and by peer-validation only. Most Entailment and Contradiction annotations rejected by self are also rejected by peers (dark green). However, Neutral presents a challenging situation for self-validation where 60.13% (92/153) of Ns are only invalidated by the joint force of peers but not by one annotator alone.
Figure 2b gives the frequencies of aggregated label combinations per item before validation and after self- and peer-validations (see A.2 for labelexplanation pair frequencies). Frequencies of multilabeled items drop after self-validation and, more remarkably, after peer-validation. Inversely, the number of single-labeled items increases vastly, especially for Neutral. We also observe from VARIERR that a large portion of items, 37.6% (188/500), are self-identified as errors and 51.6% (258) are rejected by peer-validation.
In sum, though HLV remains in VARIERR, our validation process demonstrates that annotation errors are frequently concealed under label variations. We thus proceed with the challenging automatic error detection task in §5-6 to separate annotation errors from valid HLVs.

heading: 5 Automatic Error Detection (AED) on
text: VARIERR
We now describe our experiments to detect annotation errors using VARIERR automatically. We evaluate the capabilities of AED methods, LLMs, and human heuristics (all henceforth scorers) in capturing annotation errors.

heading: 5.1 Task Definition and Evaluation
text: Following Klie et al. (2023) and Weber and Plank (2023), we model AED as a ranking task. In this setting, the goal of the scorer is to provide a ranked list with the labels that are most likely errors at the top and the most likely correct ones at the bottom. This ranked list can then be used to guide reannotation efforts (Alt et al., 2020; Northcutt et al., 2021) or remove the most likely errors from the
entailment neutral contradiction 0
20
40
60
80
100 120 Co un t
52
26
52
35 35
21
56
92
54
self-and-peer self-only peer-only
(a) Number of label-explanations rejected by selfand-peer, self-only, and peer-only validations.
C+E+N C+E C+N E+N C E N 0
50
100
150
200
250
Co un
t
43 25
112
155
32 40
93
15 12
80
129
52 54
156
9 6
42
80 73 82
204
before validation self validated peer validated
(b) NLI label sets on non-, self- and peer-validated items.
Figure 2: Frequency statistics on VARIERR.
training data (Huang et al., 2019). Scorers produce such a list by assigning an error score to each assigned label in the dataset. They derive the ranked list by sorting it based on the assigned scores.
We evaluate scorers on VARIERR using the following protocol. A model receives the list of NLI items from VARIERR where each item is paired with the aggregated label(s) it received in Round 1. For the 500 items in VARIERR, the model is given a list of 878 item-label pairs (cf. Table 2). Based on that information, the model assigns an error score and ranks the labels by this score. We evaluate how well the model performs by comparing this ranked list with the self-flagged errors (§4.3). Following Klie et al. (2023), we use standard ranking metrics for evaluation: average precision (AP), i.e., the area under the precision/recall curve computed over all assigned labels, and precision/recall for the top 100 ranked labels, P@100 and R@100.

heading: 5.2 Models
text: We evaluate five different AED models: two variants of Datamaps (DM, Swayamdipta et al. 2020), Metadata Archaeology (MA, Siddiqui et al. 2023), and two GPTs. We report the mean and standard deviation over three random seeds for DM and MA.
Datamaps (DM) We use training dynamics (i.e., the collection of training statistics over epochs E) for each label. These statistics are obtained by training a DistilRoBERTa-base model5 (Sanh et al., 2019) following Klie et al. (2023) in a multi-label setting (Jiang and de Marneffe, 2022) on all labels
5https://huggingface.co/distilroberta-base
of VARIERR obtained in Round 1. We refer to the j’th label of the i’th example as labeli, j. The training dynamics are modeled by the probability pi, j,e that DistilRoBERTa predicts for labeli, j after the e’th epoch. Based on these probabilities, the two DM models we use are defined as follows:
DMmean = − 1E E∑ e=1 pi, j,e (1)
DMstd = ¿ÁÁÀ 1
E ( E∑ e=1 pi, j,e+DMmean)2 (2) Note that a low average probability for the label indicates a likely error. Because our evaluation setup requires the most likely errors to be ranked first, we negate the average probabilities. Metadata Archaeology (MA) MA models AED as a supervised task. It represents each instance (or label in our case) as the E-dimensional − log pi, j,e vector, where E is the number of epochs and pi, j,e is the probability the model assigns to the j’th label of the i’th NLI instance at epoch e. Then, it assumes that some instances are labeled with the property of interest (in our case, whether it is an erroneous label). It predicts whether an instance is an error by employing a k-nearest neighbors (kNN) classifier using the instance representations and error labels. We use the average number of annotated errors for the kNN to obtain a score for each instance. Following Siddiqui et al. (2023), we use k = 20. To obtain unbiased predictions, we require that the kNN training instances are distinct from those we want to obtain predictions for. We use a 2-fold cross-validation setup where we split VARIERR
into two folds, use one half as ground truth, obtain the predictions for the other, and vice versa.
GPT We also compare two large language models (LLMs): GPT-3.5 (Brown et al., 2020) and GPT-4 (OpenAI, 2023).6 We emulate the Round 2 annotation setting in §3.2 as closely as possible by prompting each model to provide a score reflecting how much each Round 1 explanation makes sense for a given label. We compute the score per label by averaging the GPT-assigned scores of all explanations for the label. We prompt GPT as follows, giving it the premise (context) and hypothesis (statement) of a NLI item as well as all label-explanation pairs, asking it then to provide a probability for each reason: System: You are an expert linguistic annotator.
User: We have collected annotations for a NLI
instance together with reasons for the labels. Your task is to judge whether the reasons make sense for the label. Provide the probability (0.0 - 1.0) that the reason makes sense for the label. Give ONLY the reason and the probability, no other words or explanation. For example:
Reason: <The verbatim copy of the reason> Probability: <the probability between 0.0 and
1.0 that the reason makes sense for the label, without any extra commentary whatsoever; just the probability!>.
Context: {CONTEXT} Statement: {STATEMENT}
Reason for label {LABEL}: {REASON_1} Reason for label {LABEL}: {REASON_2} [...] Reason for label {LABEL}: {REASON_n}
Reason {REASON_1} Probability:
We implement GPTs using sglang (Zheng et al., 2023) and its default sampling parameters. See Appendix B for a complete prompt example. Note that the GPTs have access to the explanations for the labels, whereas the other models described above only have access to the labels without explanations.

heading: 5.3 Human Heuristics
text: In addition to the above automatic means, we experiment with four human heuristics that use
6GPT models have previously seen the premise-hypothesis pairs from MNLI (Balloccu et al. 2024 and https:// hitz-zentroa.github.io/lm-contamination/), but not the ample new annotations, i.e., NLI labels, explanations, and self/peer-validation judgments from multiple annotators.
the human label distributions over NLI labels (E, N, C) from annotation efforts: label counts from ChaosNLI (100 annotators) and VARIERR (4 annotators). In addition, we compare to VARIERR’s total and average peer judgments over explanations.
Label Counts (LC): ChaosNLI & VARIERR We hypothesize that if multiple annotators choose the same label, there is a high likelihood that the label is a correct annotation. We implement two label count (LC) baselines: one using ChaosNLI (Nie et al., 2020) and one using VARIERR.7 Since VARIERR is a subset of ChaosNLI items, we use label counts from ChaosNLI (LCCHAOS) as a human heuristic to score Round 1 labels on each item, i.e., how many of the 100 crowd-workers annotated labeli, j on item i. For instance, the ChaosNLI human distribution is {N:25, E:72, C:3} for the example in Figure 1. Similarly, we include LCVARIERR that counts the number of annotators (4 in total) that assigns labeli, j to item i in VARIERR’s Round 1 NLI labels. We multiply both LCCHAOS and LCVARIERR by −1, proposing that if a label has a higher count, then it is less likely to be an error.
Peer-judgments (Peer) in VARIERR VARIERR’s 2-round annotations enable more finegrained human heuristics that engage judgments on label-explanation pairs. Since each labeli, j can be assigned by multiple annotators with different explanations, we count the number of “yes” judgments on explanations from peers, i.e., excluding self-judgments since those are used for gold error labels.
We implement two peer heuristics: Peersum and Peeravg. Peersum sums all “yes” judgments across multiple explanations on the same label, and Peeravg sums “yes” judgments within each explanation and then averages across explanations within the label. Given that one label can maximally receive four explanations, it can receive up to 12 peer-judgments, 3 per explanation. For example, C in Table 1a receives 11 peer-judged “yes” in total (Peersum= 3+2+3+3 = 11), and the average over four explanations is Peeravg= 11/4 = 2.75. Peeravg differentiates more from Peersum when there are multiple explanations, but each receives sparse “yes” judgments. For example, N in Table 5a (Appendix C) receives 3 “yes” judgments but across two explanations, resulting in Peersum=2+1=3 and
7We did not include a comparison with LiveNLI (Jiang et al., 2023) because among its re-annotated 122 MNLI items, only 15 are shared with VARIERR.
Peeravg= 3/2 = 1.5. Similarly to the label counts above, we multiply both Peersum and Peeravg by −1, hypothesizing that fewer “yes” judgments indicate a higher likelihood to be an annotation error.
Combining Label Counts and Models Ranking labels by the number of annotations they received in Round 1 is a very strong baseline; see LCVARIERR in Table 3. Inspired by Nogueira et al. (2019), we investigate an approach that re-ranks the predictions of LCVARIERR by breaking ties with the scores produced by another model (e.g., DM, MA or GPTs). Note that LCVARIERR produces many ties because its score is always one of {−1,−2,−3,−4}.

heading: 6 Results for AED on VARIERR
text: Table 3 presents human and model performances on VARIERR AED using the ranking setup in §5.

heading: 6.1 Human Performance
text: The best human heuristic is from peers (Peersum), reaching a performance of 46.5% AP, 47% precision@100, and 36.7% recall@100. These numbers support our hypothesis that human validation can be used as a strong means to detect annotation errors in a task with relatively high HLV because self- and peer-rejected label-explanation pairs overlap considerably (cf. Figure 2a). Interestingly, both peer-derived heuristics from VARIERR perform better than LCCHAOS (3 linguists versus 100 crowd-workers), which suggests that having few highly-trained expert annotators is sufficient for reliable error detection, outperforming a larger
number of crowd-workers. LCVARIERR outperforming LCCHAOS on all metrics strengthens this finding. Next we compare humans to automatic means.

heading: 6.2 Model Performance
text: Among the models, GPT-4 outperforms all other methods by a large margin with a 8.5/22.3/17.6 percentage points (pp.) improvement in terms of AP / P@100 / R@100 over the second best model DMmean. GPT-4 even outperforms LCCHAOS in P@100 and R@100 and is close to the best peer heuristic for these two metrics.
One might postulate that ChaosNLI could have been part of GPT-4’s training mixture, and GPT4 performed well by reproducing its probabilities. To check whether this is the case, we compute Pearson’s r between the predictions of all scorers (Figure 3). While GPT-4 has a slightly higher correlation (0.42) with LCCHAOS than with all other methods, it is still much lower than some correlations between other models, e.g., 0.61 between DMstd and MA. Thus, we conclude that GPT-4 does not solely rely on information from ChaosNLI but achieves its strong performance via some other mechanism. Another possible explanation is that it is the only model next to GPT-3.5 that has access to explanations. In the future, we would like to investigate the use of explanations further.
Moreover, Figure 3 allows for a more general interesting observation. There seems to be a clear cluster structure in which the training-dynamicsbased models (DM and MA) correlate highly with each other and GPT-4 clusters with human scorers. Notably, correlations across these two clusters are small to non-existent or sometimes even negative.

heading: 6.3 Influence of Human Label Variation
text: In which situations do AED methods make mistakes, such as detecting false positive errors? This is an open question. We hypothesize that many topranking labels would either be errors or come from instances displaying HLV, i.e., instances with multiple labels after self-validation. The rest should be instances with just one plausible label. To test this hypothesis, we compute the proportion of erroneous labels vs. valid labels from HLV instances vs. other (with a single plausible label and thus exhibiting neither errors nor HLV labels) for the top 100 ranking labels for each method.
The results for the GPTs and human heuristics in Figure 4 confirm our hypothesis: they place very few (0-11) labels that are neither errors nor HLV in the top 100. On the other hand, the trainingdynamics-based methods MA and DM assign between 17.6 and 29.8 of these items to the top 100. This suggests that increasing the separation between errors and HLV is only one part of improving training dynamics methods for AED. Another could be finding the characteristics of the top-ranking items that are neither errors nor HLV.

heading: 6.4 Reranking models using label counts
text: Column AP (rerank) in Table 3 presents our reranking results. We observe that re-ranking improves over vanilla LCVARIERR for all methods but GPT-3.5. Interestingly, the best performing methods—also compared to the non-re-ranking approaches—are DMmean and DMstd. They even perform better than Peersum, the best human approach. This suggests that combining statistics from multiple annotators with AED methods based on training dynamics is a promising future direction.

heading: 7 Conclusion
text: Errors exist in datasets, but so does plausible human label variation. This paper defines a general procedure to separate the two by leveraging ecologically valid explanations (where annotators provide their reasons for a label) and pairing these with annotators’ validations (to allow corrections). We provide a new VARIERR dataset for the task of NLI re-annotated from scratch. Our empirical investigation on VARIERR for NLI finds that traditional annotation error detection methods fare poorly and underperform humans and LLMs.
While this paper only applies our 2-round annotation procedure, VARIERR, to NLI data, our methodology is general, and we hope it inspires uptake. Future work includes adapting these approaches to other NLP tasks, probing differences between self- and peer-judgments, mapping such strategies to (large) language models, and linking VARIERR to experiments with LLMs’ explainability, self-correction, or multi-agent systems.
Limitations
We believe that our two-round annotation setup would work for eliciting ecologically valid error annotations in tasks or languages other than English NLI. However, we cannot be sure without trying it, which we did not do in this project. Further, we did not use all types of information that VARIERR contains for the training-dynamics-based AED methods. An interesting question would be whether exploiting the soft label distribution with methods from learning from disagreement (Uma et al., 2021) would improve AED results. Another potentially useful source of information is the explanations given by the annotators. Using this information for computing the training dynamics or directly modeling whether an explanation makes sense for a label in a supervised setting could potentially improve AED performance.

heading: Acknowledgements
text: We thank Huangyan Shan, Shijia Zhou, and Zihang Sun for their contributions and invaluable feedback on VARIERR. Thanks also to Verena Blaschke for giving feedback on earlier drafts of this paper, as well as to the reviewers for their feedback. MarieCatherine de Marneffe is a Research Associate of the Fonds de la Recherche Scientifique – FNRS. This work is funded by ERC Consolidator Grant DIALECT 101043235 and supported by project
KLIMA-MEMES funded by the Bavarian Research Institute for Digital Transformation (bidt), an institute of the Bavarian Academy of Sciences and Humanities. The authors are responsible for the content of this publication.

heading: A Data Statistics
text: A.1 Pair-wise inter-annotator agreements (Cohen’s kappa) with MASI-distance for non-validated, self-validated, and peer-validated versions
A.2 Frequency of NLI label on non-validation, self-validated, and peer-validated explanation-label pairs
B GPT Prompt
C More VARIERR Examples
Neutral, C: Contradiction; magenta : self-judgments, black: peer-judgments, Err : label error.

heading: Future Work
text: Future work includes adapting these approaches to other NLP tasks, probing differences between self- and peer-judgments, mapping such strategies to (large) language models, and linking VARIERR to experiments with LLMs’ explainability, self-correction, or multi-agent systems. Limitations
We believe that our two-round annotation setup would work for eliciting ecologically valid error annotations in tasks or languages other than English NLI. However, we cannot be sure without trying it, which we did not do in this project. Further, we did not use all types of information that VARIERR contains for the training-dynamics-based AED methods. An interesting question would be whether exploiting the soft label distribution with methods from learning from disagreement (Uma et al. , 2021) would improve AED results. Another potentially useful source of information is the explanations given by the annotators. Using this information for computing the training dynamics or directly modeling whether an explanation makes sense for a label in a supervised setting could potentially improve AED performance