ActiveAED: A Human in the Loop Improves Annotation Error Detection
Year: 2023

Abstract:
Manually annotated datasets are crucial for training and evaluating Natural Language Processing models. However, recent work has discovered that even widely-used benchmark datasets contain a substantial number of erroneous annotations. This problem has been addressed with Annotation Error Detection (AED) models, which can flag such errors for human re-annotation. However, even though many of these AED methods assume a final curation step in which a human annotator decides whether the annotation is erroneous, they have been developed as static models without any human-in-the-loop component. In this work, we propose ActiveAED, an AED method that can detect errors more accurately by repeatedly querying a human for error corrections in its prediction loop. We evaluate ActiveAED on eight datasets spanning five different tasks and find that it leads to improvements over the state of the art on seven of them, with gains of up to six percentage points in average precision.

heading: 1 Introduction
text: Correct labels are crucial for model training and evaluation. Wrongly labelled instances in the training data hamper model performance (Larson et al., 2020; Vlachos, 2006), whereas errors in the test data can lead to wrong estimates of model performance (Alt et al., 2020; Larson et al., 2020; Reiss et al., 2020). This is a problem in practice, as even widely used benchmark datasets can contain a nonnegligible number of erroneous annotations (Alt et al., 2020; Northcutt et al., 2021; Reiss et al., 2020). Researchers have developed a multitude of annotation error detection (AED) methods to detect such labelling errors as recently surveyed by Klie et al. (2022). After detection, there are multiple ways to deal with the found annotation errors. When it comes to training data, a reasonable strategy is to simply remove the instances flagged by an AED model (Huang et al., 2019). For evaluation
data, however, this is not viable, because in many cases this would remove a significant fraction of hard but correctly labelled instances in addition to the errors (Swayamdipta et al., 2020), which would lead to an overestimation of model performance. Instead, researchers resorted to manual correction of the labels flagged by the AED method (Alt et al., 2020; Reiss et al., 2020; Northcutt et al., 2021; Larson et al., 2020). Strikingly, even though this manual correction requires human input, the typical workflow is to first apply the AED method once and afterwards correct the flagged errors, without using the human feedback in the AED step.
We hypothesize that connecting the human input and the AED prediction in a human-in-theloop setup could increase the accuracy of the AED method without increasing the total amount of human intervention. To support this hypothesis, we propose ActiveAED, an AED method which includes human feedback in the annotation loop; see Figure 1 for an illustration. We base ActiveAED on the Area-under-the-Margin metric (AUM) (Pleiss et al., 2020), which was recently proposed to detect annotation errors in computer vision datasets. As an additional contribution, we
8834
propose a novel ensembling scheme to improve AUM’s performance. In experiments on eight datasets spanning five different tasks, we show that ActiveAED improves over three baselines that performed well in a recent evaluation (Klie et al., 2022). On seven datasets, we observe improvements, with gains of up to six percentage points (pp) in average precision. Our ablation study shows that both the human-in-the-loop component and the ensembling scheme contribute to the improvements. We make code and data available under https://github.com/mainlp/ActiveAED.

heading: 2 Related Work
text: AED for Natural Language Processing (NLP) datasets has a long tradition which has recently been comprehensively evaluated and surveyed by the seminal work of Klie et al. (2022). We base our evaluation setup on theirs. Existing AED methods can be divided into six different categories (Klie et al., 2022): variation-based (Dickinson and Meurers, 2003; Larson et al., 2020), model-based (Amiri et al., 2018; Yaghoub-ZadehFard et al., 2019; Chong et al., 2022), trainingdynamics-based (Swayamdipta et al., 2020; Pleiss et al., 2020; Siddiqui et al., 2022), vector-spaceproximity-based (Larson et al., 2019; Grivas et al., 2020), ensembling-based (Alt et al., 2020; Varshney et al., 2022) and rule-based (Kvĕtoň and Oliva, 2002). To the best of our knowledge, none of these AED methods has been developed or evaluated with a human in the loop, except for Vlachos (2006) who uses AED as part a larger framework for constructing a silver-standard dataset. Accordingly, they do not compare the performance of the AED component to competing approaches and they consider only a single dataset and task.
Additionally, one can distinguish between flaggers and scorers for AED (Klie et al., 2022). Flaggers output hard decisions of whether an instance contains an error, whereas scorers assign to each instance a score reflecting the likelihood of being an error. In this work, we focus on scoring methods, because ActiveAED requires error scores to rank the instances.

heading: 3 Active Annotation Error Detection
text: We propose ActiveAED, an AED method which uses the error corrections issued by an annotator in its prediction loop. The basic procedure of ActiveAED is this: In the first step, it uses a ranking-
based AED method to find the k most likely annotation errors across the dataset. In the second step, the presumed annotation errors are forwarded to an annotator who checks them and corrects the labels if necessary. After this, the dataset is updated with the corrections issued by the annotator and the procedure continues with the first step. This loop continues until a stopping condition is met, e.g. that the fraction of errors in the batch drops to a user-defined threshold. See Figure 1 for an illustration of the process.
We consider a scenario where an annotator wants to correct annotation errors in a dataset with a given annotation budget of n instances. There are two options of how to apply an annotation error detection (AED) method to support this. The first is the state-of-the-art and the second one is our proposed approach: (1) Run the AED method once on the dataset to retrieve a list of instances ranked by their probability of containing an annotation error. Then, spend the annotation budget by correcting the top-n instances. (2) Run the AED method and spend some of the annotation budget by correcting the top-k instances with k « n. Then, run the AED method again on the now partially corrected dataset and repeat until the annotation budget is exhausted. Note, both approaches involve ranking instances based on their probability of containing annotation errors, and selection of a subset of instances for annotation based on this ranking. As a result, the outputs of both approaches can be fairly compared, because they use the same annotation budget and the same ranking-based score.
More formally, we assume a dataset with inputs X , (potentially erroneous) labels y, and true labels y∗ which are initially unknown to us. After training the model for E epochs, we use (negative) AUM to assign error scores:
si = 1
E
E∑
e=1
max y′ ̸=yi
pθe(y ′|xi)− pθe(yi|xi), (1)
where pθe(yi|xi) is the probability of the label assigned to xi as estimated by θe and maxy′ ̸=yi pθe(y
′|xi) the probability of the highest scoring label that is not the assigned one. Intuitively, correctly labelled instances on average obtain smaller (negative) AUM scores (Eq. 1) than incorrect ones, because the model will confidently predict their correct label earlier in the training. We chose AUM, because it performed well in preliminary experiments on SI-Flights (Larson et al., 2020)
and ATIS (Hemphill et al., 1990). Note, that this formulation differs from the original one in Pleiss et al. (2020) that uses raw logits instead of probabilities. We chose to use probabilities because this performed better in our experiments (see Table 1).
We extend AUM with a novel ensembling scheme based on training dynamics. For this, we train a model for E epochs in a C-fold crossvalidation setup. For each fold c ∈ {1, ..., C} and epoch e ∈ {1, ..., E}, we obtain a model θc,e. We use the models of one fold c to assign an error score sc,i to each instance with AUM (Eq. 1). For each fold, we calculate the AUM score both on the train and on the test portion of the fold, which yields C − 1 training-based scores and one test-based score for each instance. For each instance, we first average the training-based scores and then compute the mean of this average and the test-based score, which results in the final score si:
straini = 1 E − 1 ∑
c∈traini sc,i (2)
si = 1
2 (straini + s test i ), (3)
where traini is the set of C − 1 folds in which instance i appears in the training portion. Then, we rank all uncorrected instances by si and route the k highest scoring ones to the annotator, who manually corrects their label by setting yi := y∗i . Finally, the procedure continues with the partially corrected dataset until a stopping condition is met. There are two kinds of motivation for the proposed ensembling scheme: strain should improve the calibration of the model (Ovadia et al., 2019), which Klie et al. (2022) show to be helpful for AED. stest derives from the observation that model-based AED methods benefit from computing statistics over unseen data (Klie et al., 2022).

heading: 4 Evaluation Protocol

heading: 4.1 Datasets & Evaluation Setting
text: We evaluate ActiveAED on eight datasets following the choice of datasets used by Klie et al. (2022):1
• The intent classification part of ATIS (Hemphill et al., 1990), for which we randomly perturb labels.
1From this list, we exclude Plank et al. (2014) because it contains only annotation ambiguities and not corrected errors which are required for our evaluation setting.
• The sentiment analysis dataset IMDb (Maas et al., 2011), for which Northcutt et al. (2021) provide semi-automatically detected annotation errors.
• The sentiment analysis dataset SST (Socher et al., 2013) with randomly perturbed labels.
• The UPOS annotations2 from the Georgetown University Multilayer Corpus (GUM; Zeldes (2017)) with randomly perturbed labels.
• The CoNLL-2003 Named Entity Recognition data (Tjong Kim Sang and De Meulder, 2003), for which Reiss et al. (2020) provide a version with corrected annotations.
• The slot three filling datasets SI Companies, SI Flights, and SI Forex (Larson et al., 2020) that contain manually corrected slot labels.
We provide Hugging Face datasets implementations and detailed statistics for all datasets; see Appendix A. Our evaluation setup for the sequence labelling datasets (GUM, CoNLL-2003, SI Companies, SI Flights, and SI Forex) differs from that proposed by Klie et al. (2022). We opt for a sequencelevel setting because it is closer to our envisioned application scenario, as it makes more sense for an annotator to correct the entire sequence of annotations instead of a single one at a time. Specifically, we define errors on the sequence level, i.e. if at least one token annotation differs from the gold annotation, the sequence is treated as an error both during ActiveAED prediction and for evaluation. During prediction, ActiveAED aggregates tokenlevel error scores by calculating the maximum over all tokens in the sequence. For the other parts of the evaluation setup we follow Klie et al. (2022).3
In all datasets in which we perturbed labels, we resample the label uniformly for 5% of all annotations. We use average precision (AP) as our evaluation metric, which we compute with scikit-learn v1.1.3 (Pedregosa et al., 2011). To be consistent with ActiveAED’s application scenario, we cannot
2https://github.com/UniversalDependencies/UD_ English-GUM
3Note, that our results are not comparable with the numbers for the state-of-the-art reported by Klie et al. (2022), because of the different treatment of sequence-labelling datasets. Additionally, for ATIS and SST the choice of randomly perturbed labels differs (but the fraction is the same) and for IMDb the dataset statistics reported by Klie et al. (2022) are different from those of the original dataset (Northcutt et al., 2021), which we use.
use the standard train/dev/test split practice from supervised learning, because we will not have access to any known errors which we could use for development when we apply ActiveAED to a new dataset. Thus, we select the two datasets ATIS and SI-Flights as development datasets on which we devise our method, and reserve the remaining datasets for the final evaluation. We report the average and standard deviation across three random seeds. We follow the standard practice in active learning research and simulate the annotator by using gold-standard corrections (Settles, 2012; Zhang et al., 2022). Note, that here, we simulate a single annotator without accounting for inter- and intraannotator variation (Jiang and de Marneffe, 2022; Plank, 2022). We set k = 50 (an ablation for k can be found in Section 5), because this is small enough so that an annotator can handle it in a single annotation session but large enough that gains can be observed after a single iteration on SI Flights. We stop the prediction loop after 40 iterations or when the whole dataset was annotated. We perform 10-fold cross validation in all experiments. We describe the remaining hyperparameters in Appendix B.

heading: 4.2 Baselines
text: As baselines, we choose the top-performing scorer methods recommended by Klie et al. (2022):
• (Negative) Area-under-the-margin (AUM) (Pleiss et al., 2020): sAUMi = 1 E ∑E e=1maxy′ ̸=yi pθe(y ′|xi)− pθce(yi|xi)
• (Negative) Data Map Confidence (DM) (Swayamdipta et al., 2020): sDMi = − 1E ∑E e=1 pθ(e)(yi|xi)
• Classification Uncertainty (CU) (Klie et al., 2022): sCUi = −pθ∗(yi|xi),
where AUM and DM are both computed over a single training run and CU is computed with crossvalidation over the test portions using the model θ∗ achieving the lowest test loss for the given fold.

heading: 5 Results
text: The results of our evaluation can be found in Table 1. ActiveAED outperforms the three baselines on seven of the eight datasets, with gains ranging from 0.6 to 6 pp AP. We observe a large variance of the AP scores across different datasets, which is in concordance with the findings of Klie et al. (2022). We suspect that the relatively low scores on IMDb and CoNLL-2003 are because the errors were manually annotated after automatic filtering and thus are limited by the recall of the filtering method. We disentangle the contribution of our proposed ensembling strategy from that of the human-in-the-loop component by ablating the human-in-the-loop (last row in Table 1). We find that on four of the eight datasets, the ensembling alone improves results, whereas on SI Companies, SI Flights, and SI Forex, the main driver for improvements is the humanin-the-loop component. Generally, the human-inthe-loop component improves over the non-active variant on seven out of eight datasets.
A natural question that arises is whether the human-in-the-loop procedure of ActiveAED can also improve AED methods other than our modified version of AUM. To investigate this, we evaluate unmodified versions of (negative) AUM and DM on SI Flights and ATIS with our human-in-the-loop setup. We find that, for SI Flights, AUM/DM improves by 7.4/6.9 pp AP, whereas for ATIS, DM improves by 0.8 pp and AUM’s result diminishes by 0.2 pp. This suggests that a human in the loop might not be helpful for all combinations of datasets and methods, but that it has the potential
to significantly improve results for other methods than ActiveAED.
It is instructive to compare the precision-recall curves of ActiveAED to that of its non-active variant. The graphs for datasets SI Flights and CoNLL2003 can be found in Figure 2. On both datasets, the precision gains are present in the mid-to-high recall regime (> 0.4), which intuitively makes sense, because ActiveAED requires a few rounds of human annotation to produce different outputs than its non-active variant. This suggests that one could increase the efficiency of ActiveAED by starting with a more lightweight AED method, e.g. one that does not require cross validation or ensembling and only later switch to the more compute-intensive ensembling of ActiveAED. We leave the investigation of this option for future work. We describe the ablation study of our proposed ensembling scheme and for different choices of k in Appendix C. Here, we find that test ensembling is crucial, that train
ensembling sometimes improves results and that increasing k for the small SI-Flights dataset harms results. We provide example outputs of ActiveAED in Appendix E.

heading: 6 Conclusion
text: We have proposed ActiveAED, an AED method that includes human feedback in its prediction loop. While the proposed approach could be used with every ranking-based AED method, we base ActiveAED on the recently proposed AUM score, which we augment with a novel ensembling scheme based on training dynamics. We evaluate ActiveAED on eight datasets spanning five different tasks and find that it improves results on seven of them, with gains of up to six pp AP. In future work, we plan on extending ActiveAED to generative models and structured prediction tasks. Additionally, we want to use ActiveAED to clean benchmark datasets. We also plan to investigate the reasons for the observed performance gains of ActiveAED, for instance by exploring the role of model capacity and dataset characteristics (Ethayarajh et al., 2022). Finally, we would like to study the interplay between ActiveAED and human label variation (Jiang and de Marneffe, 2022; Plank, 2022).
Limitations
A major limitation of ActiveAED is that it is significantly more compute-intensive than other scoringbased AED methods such as AUM or DM. This is inherent to the proposed method because the ensemble requires training of multiple models and, after receiving human feedback, the full ensemble has to be re-trained. Also, the ensembling of ActiveAED requires more training runs than trainingdynamics-based AED methods. However, most model-based methods require a cross-validation scheme (Klie et al., 2022). The ensembling component of ActiveAED is more data-efficient than these approaches, because it makes use of the training dynamics captured during cross-validation instead of discarding them. A second limitation of this work is that while we chose baselines that performed strongly in Klie et al. (2022), they represent only a fraction of the scoring-based AED methods described in the literature. Finally, our evaluation is limited to a single language model and it would be interesting to investigate how ActiveAED interacts with larger language models than DistilRoBERTa.
Ethics Statement
Datasets with fewer annotation errors can improve model training and evaluation. While this generally seems desirable, it is subject to the same dual-use concerns as the NLP models that are improved with AED methods. Additionally, using ActiveAED instead of AUM or DM can make the AED results more accurate, but that comes at the expense of a higher runtime. This, in turn, leads to increased energy consumption and, depending on the source of the energy, more CO2 released (Strubell et al., 2019), which is highly problematic in the face of the climate crisis.

heading: Acknowledgements
text: We thank the reviewers for their constructive feedback which helped to improve the paper. Many thanks to the members of MaiNLP and NLPNorth for their comments on the paper. This research is in parts supported by European Research Council (ERC) grant agreement No. 101043235.

heading: A Datasets
text: Table 2 lists statistics for all datasets that we used in this work, together with links to the HuggingFace datasets implementations we provide.

heading: B Hyperparameters
text: As base model, we choose the 82M parameter model DistilRoBERTa-base4 (Sanh et al., 2020), which is licensed under apache-2.0. In all experiments, we perform 10-fold cross validation. We manually optimize the hyperparameters of ActiveAED on ATIS and SI Flights, resulting in a learning rate of 5e-5 and a batch size of 64. We adapt the number of epochs to the size of the dataset: for the SI datasets, we set it to 40, for ATIS to 20, for GUM, CoNLL and SST to 10, and for IMDb to 5 and use Adam (Kingma and Ba, 2015). We set the number of instances that the annotator corrects in a single pass k to 50 for all datasets because this is small enough so that an annotator can handle it in a single annotation session but large enough that gains could be observed after a single iteration on SI Flights.

heading: C Further Ablation Studies
text: Table 3 gives results for our full ablation study. We find that for ATIS, where ensembling was the main driver of improved results, ablating both train and test ensembling leads to worse results. For SIFlights, the variant without test ensembling leads to worse results, whereas omitting train ensembling
4https://huggingface.co/distilroberta-base
improves results. We hypothesized that, for small datasets, increasing k would lead to worse results. Our results confirm this. Setting k = 100 leaves results almost unchanged, whereas k = 200 leads to a dramatic drop of 3.9 pp AP on SI-Flights, without affecting performance on the much larger ATIS.

heading: D Compute Resources for Experiments
text: We estimate the total computational cost of our experiments including development of the method to be around 1000 GPU hours on an 80GB A100. As per the ML CO2 Impact tool (Lacoste et al., 2019)5 and an average carbon intensity of electricity for Germany of 0.485 kgkWh CO2 (Moro and Lonza, 2018) this amounts to roughly 121 kg CO2 emitted.

heading: E Example Outputs
text: Example outputs for IMDb and CONLL-2003 can be found in Figure Appendix 3. We show the five instances with the highest error scores assigned by ActiveAED. All instances contain an annotation error.
5https://mlco2.github.io/impact/
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Left blank.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix A

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix A
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A
C 3 Did you run computational experiments? Sections 4 and 5, Appendices C, D, E, and F

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix E
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix B

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In future work, we plan on extending ActiveAED to generative models and structured prediction tasks. Additionally, we want to use ActiveAED to clean benchmark datasets. We also plan to investigate the reasons for the observed performance gains of ActiveAED, for instance by exploring the role of model capacity and dataset characteristics (Ethayarajh et al. , 2022). Finally, we would like to study the interplay between ActiveAED and human label variation (Jiang and de Marneffe, 2022; Plank, 2022). Limitations
A major limitation of ActiveAED is that it is significantly more compute-intensive than other scoringbased AED methods such as AUM or DM. This is inherent to the proposed method because the ensemble requires training of multiple models and, after receiving human feedback, the full ensemble has to be re-trained. Also, the ensembling of ActiveAED requires more training runs than trainingdynamics-based AED methods. However, most model-based methods require a cross-validation scheme (Klie et al. , 2022). The ensembling component of ActiveAED is more data-efficient than these approaches, because it makes use of the training dynamics captured during cross-validation instead of discarding them. A second limitation of this work is that while we chose baselines that performed strongly in Klie et al. (2022), they represent only a fraction of the scoring-based AED methods described in the literature. Finally, our evaluation is limited to a single language model and it would be interesting to investigate how ActiveAED interacts with larger language models than DistilRoBERTa