Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance

## Abstract

Abstract NLP benchmarks rely on standardized datasets for training and evaluating models and are crucial for advancing the field. Traditionally, expert annotations ensure high-quality labels; however, the cost of expert annotation does not scale well with the growing demand for larger datasets required by modern models.
While crowd-sourcing provides a more scalable solution, it often comes at the expense of annotation precision and consistency. Recent advancements in large language models (LLMs) offer new opportunities to enhance the annotation process, particularly for detecting label errors in existing datasets. In this work, we consider the recent approach of LLM-as-a-judge, leveraging an ensemble of LLMs to flag potentially mislabeled examples.
Through a case study of four datasets from the TRUE benchmark, covering different tasks and domains, we empirically analyze the labeling quality of existing datasets, and compare expert, crowd-sourced, and our LLM-based annotations in terms of agreement, label quality, and efficiency, demonstrating the strengths and limitations of each annotation method. Our findings reveal a substantial number of label errors, which, when corrected, induce a significant upward shift in reported model performance. This suggests that many of the LLMs so-called mistakes are due to label errors rather than genuine model failures. Additionally, we discuss the implications of mislabeled data and propose methods to mitigate them in training to improve model performance.

## 1 Introduction

Natural Language Processing (NLP) benchmarks have long served as a cornerstone for advancing the field, providing standardized datasets for training and evaluating methods and models (Wang et al., ; Hendrycks et al., ; Srivastava et al., ; Calderon et al., ).
These datasets have been developed over the years for various tasks and scales, annotated using different schemes. Initially, human domain expert annotation was preferred, as these experts possess the skills necessary to determine correct labels accurately. However, as models have increased in size (Devlin et al., ; Raffel et al., ; Brown et al., ), the demand for larger datasets has also grown (Kaplan et al., ). Since expert annotation is cost-prohibitive, it does not scale well to meet these new demands. The demand for large quantities of annotated data quickly and cost-effectively has led researchers to adopt crowd-sourcing, often sacrificing expertise for scale.

That way or another, constructing datasets heavily involves making compromises in annotation, trading off between scale, efficiency and expertise.
Even when annotated by experts, datasets can naturally contain labeling errors, arising from factors such as task subjectivity, annotator fatigue, inattention, insufficient guidelines, and more (Rogers et al., ; Reiss et al., ; Sylolypavan et al., ). Mislabeled data is even more pronounced when non-expert annotators are involved (Kennedy et al., ; Chong et al., ).
Widespread mislabeled data is particularly concerning because both the research community and the industry rely heavily on benchmarks. In training data, label errors harm model quality and hinder generalization, while in test sets, they lead to flawed comparisons, false conclusions, and prevent progress.

Recent advancements in large language models (LLMs) (Ouyang et al., ; Chiang & Lee, ; Li et al., ; Calderon & Reichart, ; Gat et al., ) present new opportunities to improve the annotation process, specifically in detecting label errors within existing datasets. Rather than re-annotating entire datasets (e.g., through experts or crowd-workers), we consider the recent approach of LLM-as-a-judge (Zheng et al., ), and propose a simple yet effective method by leveraging an ensemble of LLMs to flag a set of potentially mislabeled examples. These can then be sent to experts for re-annotation and correction, or even get filtered during training.

Specifically, we construct an ensemble model using multiple LLMs with diverse prompts, gathering both their predicted labels and corresponding confidence scores. These predictions are contrasted with the original labels, and instances where the LLMs *strongly disagree* with the original label (i.e., show high confidence in a different label) are flagged as potential mislabeling cases.
Additionally, we not only explore the role of LLMs in detecting errors but also evaluate their performance as annotators, comparing them with expert and crowd-sourced annotations. We assess these approaches in terms of agreement, label quality, and efficiency, highlighting their strengths and limitations.

We aim to answer the following questions through a comprehensive end-to-end study: (1) To which extent current benchmarks include mislabeled data? (2) Can LLMs detect label errors? (3) How do expert, crowd-sourced, and LLM-based annotations compare in quality and efficiency? and (4) What are the implications of mislabeled data on model performance and can we mitigate their impact?

To this end, we choose the TRUE benchmark (Honovich et al., ) – A collection consolidating 11 existing datasets annotated for factual consistency in
a unified format – as a case-study and empirically investigate its labeling quality. Specifically, we analyze four datasets from TRUE with binary factual consistency annotation originating from different tasks. This enables us to explore multiple tasks and domains while benefiting from a uniform labeling schema.

This paper presents both methodological and empirical contributions. We propose a straightforward approach for detecting potential mislabeled examples (as illustrated in Figure 1), revealing a substantial number of label errors in existing datasets, ranging from 6% to 21%. Additionally, we demonstrate that the precision of LLMs in identifying errors improves with their confidence in an incorrect label; when their confidence exceeds 95%, over two-thirds of those labels are, in fact, errors. Moreover, we show that LLM-based annotations not only excel in error detection but also perform similarly to, or better than, traditional annotation methods, offering better trade-offs between quality, scale, and efficiency. Finally, we empirically illustrate the negative impact of mislabeled data on model training and evaluation. We propose a simple, fully automated method for addressing label errors, improving the performance of fine-tuned models by up to 4%. In evaluation, we found that mislabeled data can significantly distort reported performance; LLMs may perform up to 15% better. This indicates that many so-called prediction errors are not genuine errors but are instead human annotation mistakes.

Figure: Figure 1: An illustration of our approach for detecting and addressing mislabeled data: (1) Re-label examples from existing datasets using an ensemble of LLMs. (2) Identify strong disagreements between the LLM’s predictions and the original labels (i.e., high confidence in a different label), flagging examples based on confidence levels. Our findings show that LLMs detect between 6% and 21% of label errors, and higher LLM confidence is strongly associated with improved precision in error detection. (3) In the training set, we either filter or flip flagged examples to improve model performance, leading to an increase of up to 4%. For the test set, flagged examples are re-annotated by experts to make sure the evaluation is accurate. We found that under accurate evaluation, the performance of LLMs is up to 15% higher than the original mislabeled data.
Refer to caption: extracted/5951932/graphics/Flow.png

## 2 Data Annotation Approaches

### 2.1 Traditional Annotation Approaches

Crowd-Sourcing.
Crowd-sourcing has been widely used to annotate large-scale NLP datasets (Rajpurkar et al., ; Williams et al., ; Wang et al., ) because it enables the rapid collection of labeled data at scale. However, the reliability of crowd-sourced annotations has been questioned, as quality control remains a challenge, with labeling inconsistencies becoming more frequent as dataset complexity increases (Lu et al., ; Allahbakhsh et al., ).
One of the key advantages of crowd-sourcing has traditionally been its ability to handle tasks requiring human creativity or subjective judgment — areas where models have historically struggled. However, this advantage is fading, as models now approach near-human performance on such tasks (Chiang & Lee, ; Chen & Ding, ), and crowd workers increasingly rely on models for assistance, diminishing the human element in the process (Veselovsky et al., ; ).

Human Experts.
Expert annotation is a reliable approach for NLP tasks that require domain-specific expertise (e.g., medical or legal domains) and for tasks that demand deep cognitive engagement, such as those requiring training to understand complex guidelines or intelligent, attentive annotators. However, this approach is slow and expensive compared to crowd-sourcing (Snow et al., ; Chau et al., ), limiting its scalability for the large datasets needed to train modern LLMs.
Managing the trade-off between cost-effectiveness and annotation quality is critical, especially in tasks that require domain-specific accuracy (Chau et al., ). Maintaining inter-annotator agreement among experts presents an additional challenge, further driving up costs (Baledent et al., ).
Hybrid approaches, combining expert and crowd-sourced annotations, can mitigate this trade-off, though expert involvement remains essential for high-quality labels (Nguyen et al., ).

### 2.2 LLM as an Annotator

As shown in recent studies (Gilardi et al., ; Li et al., ; Calderon & Reichart, ; Kholodna et al., ), LLMs can be integrated into the annotation process, as they are fast, relatively cheap, and obtain decent performance. Despite their promise, LLMs make mistakes, and therefore their annotations can not be considered as Gold labels (Chen et al., ; Bhat & Varma, ).
Still, incorporating LLMs in the labeling process offers cost and scalability benefits.

In this study we propose to utilize LLM annotations, alongside with their confidence, in order to automatically detect errors in existing labeled datasets. Specifically, we propose a general schema for re-classification via LLMs, described as follows. We re-label the dataset via LLM, and obtain a predicted probability for each class. When using a model with access to the parameters, these probabilities can be extracted directly at inference time. When using models available via public APIs, probabilities may be given as token log-probabilities (if available through the API) or approximated via sampling. After annotating via LLMs, examples for which there is a *strong disagreement* between the LLM annotation and the original label (i.e., high LLM probability for another label), are flagged as potentially mislabeled.
If in a test set, the flagged examples could then be re-examined by human experts to determine their true label. If in a training set, the same solution may apply, but such examples could also be filtered or have their label changed according to the LLMs prediction.

To overcome the variance in LLM-generated answers, we suggest averaging over various prompts, and over different models, obtaining one strong LLM ensemble. As shown in subsubsection 4.2.2, this approach not only reduces variance but also enhances model performance and detection capabilities.

## 3 Experimental Setup

### 3.1 Data

As a case-study, we choose to explore the extensive and widely used TRUE benchmark (Honovich et al., ), which is typically used as an evaluation set (Steen et al., ; Gekhman et al., ; Wang et al., ; Zha et al., ). It consists of 11 datasets from various NLP tasks such as summarization and knowledge-grounded dialogue. This benchmark is unique in its approach of bringing multiple datasets and tasks into a unified schema of binary factual consistency labels. Each dataset is transformed from its original structure (e.g., a source document and a summary) into two input texts, *Grounding* and *Generated Text*, and a binary label indicating whether the generated text is factually consistent w.r.t the grounding. This enables us to examine multiple tasks and domain under the same umbrella at once, while maintaining a unified binary-label schema.

Specifically, we focus on four TRUE datasets, one from each task (summarization, dialogue, fact verification, paraphrasing). Each of these datasets have been annotated with different guidelines, for a different purpose, and with a slightly different annotation procedure:

MNBM (Maynez et al., ): Summarization.
This dataset provides annotations for hallucinations in generated summaries from the XSum dataset (Narayan et al., ). *Grounding* refers to the source document that the summary is based on, while *Generated Text* consists of model-generated summaries, which may include hallucinated information not present in the source. Three human annotators, trained for the task through two pilot studies, annotated the dataset for the existence of hallucinations. In TRUE, the binary annotations were determined by majority vote.

BEGIN (Dziri et al., ): Dialogue.
This dataset evaluates groundedness in knowledge-grounded dialogue systems, where responses are expected to align with an external *Grounding* source, typically a span from Wikipedia. *Generated Text* refers to model-generated dialogue responses that were fine-tuned on datasets like Wizard of Wikipedia (Dinan et al., ).
Data was annotated into entailment/neutral/contradiction labels, by three human annotators, trained for the task through two pilot studies, aggregated by majority vote. In TRUE, binary annotations were then determined by the entailment/not-entailment partition.

VitaminC (Schuster et al., ): Fact Verification.
This dataset is based on factual revisions of Wikipedia. The evidence, or *Grounding*, consists of Wikipedia sentences, either before or after these revisions. Most human involvement came from creating *Generated Text* rather than the annotation process, with annotators writing claim/evidence pairs derived from Wikipedia revisions, inherently generating labeled data for fact verification. Synthetic examples from the FEVER dataset (Thorne et al., ) were also included. Additionally, three annotators reviewed 2,000 examples, presumably to ensure data quality.

PAWS (Zhang et al., ): Paraphrasing.
This dataset consists of paraphrase and non-paraphrase pairs. *Grounding* refers to source sentences drawn from Quora and Wikipedia, while *Generated Text* was automatically generated through controlled word swapping and back-translation.
Five human annotators annotated the dataset with binary labels w.r.t paraphrasing correctness. The dataset includes both high- and low-agreement annotations.

For each of the four datasets, we randomly sampled 1000 examples (or the whole dataset if the number of examples is smaller than 1000). These examples are annotated via LLMs as described in subsection 2.2. We set an evaluation (i.e., test set) based on 160 randomly sampled examples from each dataset (a total of 640), while the rest remain for training and validation (they will be relevant for subsection 6.1). In addition to the LLMs annotation, the evaluation set is also re-annotated by human experts, which are two of this paper’s authors, fully familiar with the task, and by three human annotators per example via crowd-sourcing.

### 3.2 Annotation Procedure

This subsection outlines the annotation procedures for the various approaches. For additional implementation and technical details not covered here, please refer to the Appendix A.

LLMs.
We follow the general schema described in subsection 2.2 by utilizing LLM labels with their confidence for each class to detect mislabeled data. To this end, we annotate the data with four different models: GPT-4 (OpenAI, ), PaLM2 (text-bison@002) (Anil et al., ), Mistral (7B)(^1^11[https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)) (Jiang et al., ) and Llama 3 (8B)(^2^22[https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)) (Dubey et al., ). We designed four different prompts, to control the variance caused by task description, and report more stable results. The prompts are designed as a zero-shot classification task: the requested output is a single token, either ’0’ for factual inconsistency or ’1’ for factual consistency. Instead of taking the binary output, we extract the corresponding probability from logits or log-probabilities, as estimation of the model’s confidence for the predicted class. Overall, for each example we have $4\times 4$ probabilities for binary labels $P_{model}^{prompt}(y=1|x)$.
As GPT-4 and PaLM2 showed better performance (i.e., higher ROC AUC) and consistency (i.e., higher IAA), in the following sections we denote their average probabilities as the single *LLM $p$* where $p=P(y=1|x)$ or *LLM annotation* in the binary case $\mathbb{I}\{p>0.5\}$.

Crowd-sourcing.
We utilize the platform of Amazon Mechanical Turk (MTurk) to recruit
crowd-workers for annotating 100 examples from each dataset (a total of 400), and to design the interface layout. Examples were randomly assigned to annotators.
Each annotated example was manually reviewed. Rejected examples were returned to the pool and re-annotated, until each example had been annotated by three different annotators. To prevent (as much as possible) LLM use in the crowd-sourcing annotation, we disabled the possibility of right-click and Ctrl+c in the platform (as suggested by Veselovsky et al., ).
To obtain a single label per example, we consider two different aggregations: (1) *Majority* - by majority vote, and (2) *Strict* - if any annotator marks it inconsistent, that becomes the label.

Experts.
All examples where the LLMs’ annotations differed from the original label, regardless of the LLMs’ confidence, were annotated by human experts. These experts were two of this paper’s authors, who are fully familiar with the guidelines and task characteristics.
Each example was annotated by both annotators independently on a scale of 0 (inconsistent) to 1 (consistent). Examples were shuffled and did not appear in any specific order, and neither the original nor LLM labels were presented, just the plain texts. Subsequently, on the examples where the annotators did not agree with each other, a reconciliation phase took place, where both annotators discussed, attempting to resolve the disagreement.
After re-annotating all the conflicted examples, we consider the *Gold label* to be either the original label, in case the LLM label agrees with it, or the experts resolution, in case of there was a disagreement between them.

## 4 Label Error Analysis and the Role of LLMs in Error Detection

### 4.1 Do current benchmarks include mislabeled data?

To address the first research question, we conducted a detailed analysis of current benchmarks, identifying the extent to which mislabeled data exists across various datasets. Following the procedure described in subsection 3.2, we annotate the test-set using LLMs. We then contrast these annotations with the original labels, to find disagreements. As shown in Table 2, the disagreement rate is significant and can be up to $\sim 40\%$ of the examples. For instance, for 34.4% of the examples in the BEGIN dataset, the LLM ensemble label differ from the original label. Example for such label error detection is presented in subsection 4.1.
While usually we would say that this means that the LLMs performed poorly, we choose to further investigate these examples and settle the disagreement. To this end, we asked human experts to re-annotate these examples (as described in subsection 3.2), without any knowledge on LLM’s $p$ or the original label. After experts re-annotation, we can conclude which is correct: the original label, or the LLMs?

Our findings show a considerable number of label errors for all examined datasets (see the %error column in Table 2).
Based on the experts Gold label and the sample sizes, we also estimate a lower bound for the total percentage of label errors in the full datasets. For this calculation, we employed the Clopper-Pearson exact method (Clopper & Pearson, ) to construct a 95% confidence interval for the binomial proportion, adjusted by a finite population correction (FPC) (see more details in Appendix, D.1). We provide the lower bound of these confidence intervals in parenthesis Table 2, in the %error column. For instance, we bounded the %error in MNBM to be at least 11.6%.

Table: Table 1: Example of an annotation error in the original datasets, discovered by LLMs and corrected by experts. In Appendix Table 8 we provide additional examples.

### 4.2 LLMs for Label Error Detection

In this subsection, we explore the role of LLMs in label error detection and emphasize the strengths of an ensemble approach for this purpose.

#### 4.2.1 Can LLMs Automatically Detect Label Errors?

As described in subsection 4.1, we utilize LLMs to flag candidates for mislabeling, and indeed find label errors. In this subsection, we focus on the LLM viewpoint, including the effect of LLM confidence, and the power of ensemble (i.e., aggregating multiple LLMs). LLM annotations are valuable for flagging mislabeled data, offering more than just binary labels. By considering LLM confidence scores alongside their predictions, we can improve the precision of automatic error detection.

To better assess the utility of LLMs in detecting label errors, we break down the flagged examples into confidence-based bins. The rationale behind this is that not all flagged examples should be treated equally. Some instances are flagged with lower confidence, indicating that the LLM recognizes a potential issue but is uncertain about the label it suggests. On the other hand, when the LLM is highly confident in assigning a label that opposes the original one, it serves as a stronger signal of a possible label error.

Figure: Figure 2: When LLMs disagrees with original labels - who is correct?. As the LLM’s confidence grows, so does the precision of identifying an error in the original labels.
Refer to caption: x1.png

Figure 2 shows the rate of the experts agreement with the LLMs compared to the agreement with original labels, divided into confidence-based bins. Each bin includes at least 35 examples, and is defined by a confidence interval of 95% based on bootstrap sampling (see Appendix D.2 for futher details). The bins reflect increasing levels of LLM confidence in its predicted label (i.e., a stronger disagreement between LLMs and the original labels).

From the figure, we observe a clear trend: as LLM confidence increases, so does its precision in detecting label errors in the original dataset. In the highest confidence bin, LLM annotations surpass the original labels in agreement with expert re-labeling, and this difference is statistically significant. This indicates that when the LLM is highly confident in its disagreement with the original label, the labeled example serves as a strong candidate for a labeling error. Note that even in cases where the expert agreement with LLMs were below 50%, mislabeled data was still discovered.

#### 4.2.2 The power of ensemble

Figure: Figure 3: LLM Ensemble of different sizes (random subsets). (Left) presents the performance of the ensemble in terms of ROC AUC compared to the gold labels. (Right) presents the increasing ability to detect label errors. F1 is computed over Error / Not Error predictions.
Refer to caption: x2.png

As mentioned in subsection 3.2, we treat the LLM annotations as an ensemble of 2 models combined with 4 different prompts, in order to ensure greater stability in the results. Where one LLM may succeed, the other may fail, and averaging all their probabilities enables us to have more confidence in the final answer.
In this subsection, we further analyzed the performance of LLMs by varying the size of the LLM ensemble, examining how this affects the model performance.
We evaluate two aspects of model performance. First, we assess how closely the ensemble’s annotations match the gold labels — essentially, how much we can trust the LLM annotations. We measure this aspect of label quality using the ROC AUC compared to the gold labels. The second aspect is the ensemble’s ability to detect label errors. For this, we compute the F1-score by averaging the recall of errors and the precision of correctly identifying a candidate as a true error.

Results are shown in Figure 3. For both aspects, we see a clear trend. As we increase the number of models in the ensemble, the performance increases. In terms of ROC AUC w.r.t the gold labels (left plot), this suggests better annotation quality, while the right plot, a higher F1 score indicates a stronger error detector, either by recalling more errors or improving precision, or through a balance of both. Additionally, for both measures the variance decreases as the ensemble size grows, which indicates more stable and consistent annotations and error detections. Although not yet discussed in the context of error detection with LLMs, these results align with previous work showing the power of ensemble (Dietterich, ). These observations justify our choice to use an ensemble of models rather than a single one.

## 5 Comparing Annotation Approaches

Figure: Figure 4: Annotation approaches comparison.
Refer to caption: extracted/5951932/graphics/Compare-dense__no-frame.png

As discussed in section 2, we focus on three main annotation approaches, each with its own benefits and drawbacks. These approaches differ in how they manage the trade-offs between label quality, scalability, and cost. In the following section, we discuss and compare their characteristics. A summary of this comparison is given in Figure 4.

### 5.1 Annotation Quality

When annotating or validating a dataset, one of our main concerns is the quality of the labels, or in other words, establishing a reliable gold standard. However, each annotation approach produces different labels. To estimate the quality of these approaches, we measure the agreement between different annotations using the weighted F1-score (which accounts for both classes). Note that this metric is not symmetric, meaning that treating one annotation as the *true* label and the other as the *prediction*, or vice versa, can result in different scores.

Figure 6 presents the F1-score between each pair of annotation approaches. As the figure shows, LLMs have disagreements with the *original* labels (0.72). Yet, as discussed in subsection 4.1, the original labels themselves contain mistakes, so this disagreement does not necessarily indicate poor performance of the LLMs. When considering the *Gold* as the true label, LLM performance increases to 0.83. This suggests that LLMs, despite their discrepancies with the original labels, perform closer to the truth than initially reported. The *Gold* label, obtained by experts, has high agreement with both the *Original* and *LLM* labels.
On the other hand, the *MTurk-Majority* approach performs poorly, with near-random F1-scores compared to both the original and gold labels, and even when compared to its stricter variant, *MTurk-Strict*.
The results indicate that basic crowd-sourcing, without additional training to enhance crowd-workers into specialized sub-experts, performs significantly worse compared to other approaches, including LLM-based methods.

Figure: Figure 5: Comparison between all annotation methods, measured by the weighted-F1-score. Rows represent the ”true” label and columns represent the ”prediction”. For instance, the score of *LLMs* compared to the *Original* label is 0.72.
Refer to caption: x3.png

Crowd-sourcing.  For crowd-sourcing, the reported F1-score does not provide the complete picture. When we focus on individual annotators, we see that those who annotate more examples generally deliver higher-quality annotations, achieving greater accuracy when compared to both the original and gold labels (see Figure 6). This phenomenon can be explained by two hypotheses: (1) a learning process— as the annotators see more examples, they improve at the task, or (2) users who dedicate time to annotating multiple examples are likely those who either read the guidelines carefully and strive to perform the task to the best of their ability, or are naturally proficient at the task and therefore continue annotating.
Even though annotators who label more instances tend to provide higher-quality annotations, they are less common—most annotators tend to stop after only a few examples. This distribution of annotators results in overall insufficient annotation quality. Pre-qualification tests are often used to shift this distribution from the ”average worker” towards more experienced or dedicated annotators; however, this requires a significantly larger budget and greater micro-management involvement from the researcher.

### 5.2 Consistency

Usually, when annotating a dataset, more than one annotator is involved. This applies to crowd-workers, experts, and even LLMs— in this study, we use an ensemble of different LLMs and prompts. The use of multiple annotators, similar to an ensemble, is meant to overcome the variance between individuals, which can arise from the subjective nature of NLP tasks, different interpretations of instructions, lack of experience, task difficulty, and cognitive bias (Uma et al., ).

As such, a common practice in the NLP community is to report Inter Annotator Agreement (IAA)— a set of statistical measures used to evaluate the agreement between individuals. Typically, IAA can be viewed as an adjustment of the proportion of pairwise agreements, where 0.0 indicates random agreement. We focus on Fleiss’s $\kappa$ (Fleiss, ), as it accounts for label imbalance and multiple ($>2$) annotators. High IAA, or low variance between independent annotators, is considered an indicator of high-quality annotation.
In Table Table 3, we report the agreement between annotators across different approaches. For LLMs, we report two variants: (1) same model, different prompts; and (2) different models, where each model’s result is the aggregation across prompts.
For reference, we also include the IAA from the original annotations, as reported in the original papers: MNBM reported an average Fleiss’s $\kappa$ of 0.696 for the hallucination annotation task; BEGIN reported Krippendorff’s $\alpha$ (a generalization of Fleiss’s $\kappa$) of 0.7; VitaminC reported Fleiss’s $\kappa$ of 0.7065 on a sample of 2,000 examples; and PAWS reported a 94.7% agreement between a single annotator’s label and the majority vote on the Wikipedia subset used in TRUE.

Table: Table 3: Inter-Annotator Agreement in different Annotator groups. %agreement is the proportion of pairwise annotator comparison. Fleiss’s $\kappa$ (disagree. subset) refers to the $\kappa$ over the subset of disagreement between LLM and the original label.

Experts.
While it’s true that reconciliation naturally leads to increased agreement, the significant improvement in IAA we observed highlights its importance.
Though this phase is less common in practice, it is crucial not only for increasing agreement but also for improving the overall quality of annotations and ensuring more reliable outcomes.
Interestingly, label changes in this phase were not symmetric, as most changes (69.3%) were in the direction of consistent $\rightarrow$ inconsistent, where one annotator found an inconsistency that the other did not (see all change details in Appendix 6). It is important to note that the $\kappa$ obtained by the experts (both before and after reconciliation) was calculated on a more challenging subset, where the original label differed from the LLM prediction, and should be interpreted with this context in mind. This is reflected in the decrease in $\kappa$ observed for all other annotator groups on this subset.

LLMs.
GPT-4 and PaLM2, the better-performing LLMs on this task, show high IAA, with $\kappa=0.706$ and $\kappa=0.75$, respectively, which is similar to the experts’ reported $\kappa$. This suggests a comparable level of variance and quality in annotation, providing further empirical evidence for considering LLMs as annotators. This property adds to previous studies showing LLMs’ quality as surrogates for human preferences (Zheng et al., ) or evaluations (Chiang & Lee, ).

Crowd-Sourcing.
Crowd workers showed near-random agreement ($\kappa=0.074$), indicating poor-quality annotations. Only 40.8% of the examples were labeled unanimously, while the rest included annotations from both classes. Even among the subset of examples unanimously labeled as consistent, 37.9% are labeled as inconsistent in both original and gold labels, pointing to a lack of attention and thoroughness. See more details on the crowd-sourced annotation distribution in Appendix A.1.2.

### 5.3 Cost and Scalability

LLM-based annotation is significantly cheaper and faster than crowd-sourcing platforms like MTurk, especially when considering the additional time required for human review cycles. It is estimated to be 100 to 1,000 times more cost-effective than using human annotators, including experts. This scalability and speed make LLMs a highly efficient alternative for large-scale annotation tasks.

## 6 Implications of Mislabeled Data

### 6.1 Training on Mislabeled Data

Training on mislabeled data can harm model performance and stability, as learning from errors makes it harder to identify consistent patterns. The impact depends on various factors , such as the fraction of mislabeled data and the training procedure. In this subsection, we show that addressing this issue, even heuristically, significantly improves model’s performance on a test set.

Handling Label Errors.
In order to handle label errors in the training set, and reduce its effect on model performance, we propose two manipulations. For both manipulations, we use similar methodology as discussed in this paper, and flag examples where the LLMs $p$ strongly disagree (i.e., above a certain confidence threshold) with the original label. The first manipulation is filtering flagged examples out, which maintains a ”cleaner” yet smaller training set. The second manipulation is label flipping for flagged examples, which maintains the same amount of data, but may also cause harm if flipping too many correct labels.

Experimental Setup.
We set the training set to be the additional data examples from the datasets (i.e., MNBM, BEGIN, VitaminC, PAWS), which are disjoint from the test set. Note that we posses gold labels for the test set alnoe, while for the training set we only extract $p$ via GPT4 and PaLM2.
The finetuning procedure includes splitting the training set into train and validation sets, and fine-tuning on the train set. We report results averaged over five seeds.

As an ablation study, we also apply these manipulations on a random subset of examples rather than the flagged examples.
TThe ablation study aims to maintain a consistent number of training examples, while the ablation for flipping aims to address the claim that in some cases, a relatively small fraction of label errors may be even considered as a noise that improves model robustness (e.g., as in label perturbation (Zhang et al., ) or label smoothing (Szegedy et al., )).

We conducted this experiment starting from two base models: DeBERTa-base-v3,(^3^33[https://huggingface.co/microsoft/deberta-v3-base](https://huggingface.co/microsoft/deberta-v3-base)) and a fine-tuned version of it on classic NLI datasets, which we will refer to as the NLI-base model(^4^44[https://huggingface.co/MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli](https://huggingface.co/MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli)). We chose the NLI-base model as NLI tasks closely resemble factual consistency evaluation (FCE), making it well-suited for this experiment. Given the similar trends, we present the results for the NLI model here. Additional experiments and implementation details can be found in Appendix C.1.

Figure: Figure 7: Fine-tuning a model on a transformed dataset. The gray bar is the original dataset - without any changes. The green bars present results for label flipping for a subset of examples, determined by LLMs-confidence (plain), or at random (dotted). The blue bars represent filtering of these examples.
Refer to caption: x5.png

Results.
Figure 7 shows the results of our experiments.
In our confidence-based approaches, we clearly see the trend that as the confidence threshold—according to which our manipulations are applied—grows, our manipulation results in improved ROC AUC for both models. This trend eventually (i.e., for high enough LLM confidence) brings these approaches to significantly outperform the baseline. In contrast, when we applied our manipulations on random subsets, we generally see a diminishing effect of manipulation, converging to the no-manipulation baseline.

Comparing between the handling approaches, it appears that flipping is better than filtering for high confidence. We hypothesize that this stems from the amount of data that remains after flipping (i.e., the same amount as before the flipping) compared to the filtering approach, combined with the high error rate in these datasets. Note that this is contrary to the random case where filtering is better than flipping, as flipping a subset with low error-rate brings more damage than value.

### 6.2 Evaluating on Mislabeled Data

In this subsection, we examine the impact of mislabeled data in evaluation sets and its potential to distort results. Labeling errors can mislead the evaluation process, resulting in inaccurate performance metrics and, in some cases, flawed model comparisons that lead to incorrect conclusions.

Experimental Setup.
To test this assumption, we evaluate the performance of nine models, mostly state-of-the-art LLMs, on the test datasets. We compare their performance between the *Original* labels, and the *Gold* labels. For LLMs, we used zero-shot prediction as described in subsection 2.2, and averaged over prompts. For DeBERTa-based models we used the fine-tuned models from subsection 6.1, and averaged over seeds.

Table: Table 4: Comparison of Model Performance on Original and Gold Labels. Ranking is defined over ROC AUC.

Results.
Prior to this work, an evaluation of these models would induce the values and ranking as in Table 4 under the Original sub-columns.
However, as shown before, these datasets include labeling errors, and therefore do not support fair evaluation. Considering the new gold labels, based on expert intervention (as described in subsection 3.2), we obtain different results, shown in the Gold sub-columns.
The first observed discrepancy is the ranking of models. For example, DeBERTa-v3 has shifted from being the second-best to the second-worst.
Beyond the change in ranking, all metrics’ (i.e., ROC AUC, F1-score and accuracy) range has shifted upward, indicating that LLMs perform better on this task than what was previously thought, likely due to label errors. If this phenomenon extends to other tasks and datasets beyond those examined in this study, it could suggest that LLMs are better than currently perceived.

## 7 Related Work

### 7.1 LLMs in the Annotation Loop

LLMs have been increasingly utilized as annotators in various NLP tasks, offering potential benefits in efficiency and scalability. Several studies have demonstrated that LLMs can effectively generate annotations from scratch, sometimes outperforming human annotators or crowd workers (He et al., ; Gilardi et al., ; Törnberg, ; Calderon & Reichart, ).
However, LLMs are not flawless and cannot be considered gold-standard annotators when used alone. They may produce incorrect annotations, especially in complex (Chen et al., ), social (Ventura et al., ; Felkner et al., ), emotional (Lissak et al., ), or low-resource (Bhat & Varma, ) contexts. These studies showed that LLMs can exhibit poor performance and biases, highlighting the necessity of human oversight to ensure quality or fairness. To address this issue, several approaches for collaborative (Kim et al., ; Li et al., ) or active learning (Zhang et al., ; Kholodna et al., ) were suggested, where LLMs and humans are both part of the annotation procedure.

### 7.2 Handling Label Errors

Label errors (also referred to as label noise) in training and evaluation datasets can significantly impair NLP model performance and reliability (Frénay & Verleysen, ). Fine-tuned models have been employed to detect mislabeled data by identifying examples with high loss or low confidence (Chong et al., ; Hao et al., ; Pleiss et al., ; Northcutt et al., ). For example, Chong et al. () demonstrated that fine-tuned pre-trained language models can effectively detect label errors by ranking data points based on the training loss.
Once these high-loss or low-confidence examples are flagged, they are typically filtered out (Nguyen et al., ; Northcutt et al., ), corrected automatically (Pleiss et al., ; Hao et al., ), or re-labeled by human annotators (Northcutt et al., ) to verify and improve dataset quality.

### 7.3 Factual Consistency Evaluation

Factual consistency evaluation (FCE) refers to the task of verifying that generated text remains true to the facts in the source content, addressing factual inaccuracies in models’ outputs. It has been applied to tasks like summarization (Kryscinski et al., ; Xie et al., ; Gekhman et al., ) and dialogue (Honovich et al., ; Xue et al., ), which are prone to suffer from hallucinated outputs. Benchmarks like the TRUE Honovich et al. () standardize evaluation across datasets. Common methods include entailment-based models (Laban et al., ) and QA-based approaches such as Q^2 (Honovich et al., ) and QAFactEval (Fabbri et al., ), with recent advancements like WeCheck (Wu et al., ) improving efficiency through weakly supervised learning.

## 8 Discussion

Labeling errors are a persistent issue in NLP datasets, negatively affecting model fine-tuning and evaluation. Our findings demonstrate that LLMs, particularly when highly confident, can effectively detect these errors, outperforming crowd workers in accuracy, consistency, and cost-efficiency. As LLM capabilities advance, their role in refining data quality will become central to improving NLP benchmarks. Future work could explore applying LLM-based error detection to a broader range of datasets and tasks, as well as refining methods for optimizing label correction strategies. We encourage researchers to adopt our methods and critically evaluate existing datasets to drive more robust, reliable results in the field.

#### Acknowledgements

This research is a collaboration between the Technion and Google Research, supported by the Google Cloud Research Credits program with the award GCP19980904.

## Ethics Statement

We address several ethical considerations related to human annotators and the research community.

First, we recognize the significant human effort and cost involved in creating the datasets used in this study. While we question certain labels in these datasets, this should not be seen as undermining their value or the hard work behind them. These datasets have been highly beneficial to the research community, and our aim is to help improve labeling quality, especially as powerful tools like LLMs become more capable in various tasks. Our goal is to highlight areas where improvements can be made, contributing to further advancements in the field.

Additionally, we used crowd-sourced human annotators for text labeling. All participants were paid fairly, in line with platform regulations and our institution’s policies. We ensured transparency in the process, treated participants with respect, and provided appropriate compensation for their efforts.

Lastly, we acknowledge the potential impact of LLMs on crowd-sourced workers who depend on these platforms for income. While we explore the use of LLMs to enhance or potentially replace certain aspects of annotation, we do not intend for this to harm human workers. Instead, we hope that crowd-sourced workers will adopt these tools, allowing them to become more efficient and skilled, which will improve both the scalability and quality of future datasets while maintaining a role for human oversight.

## Reproducibility Statement

In the study, we performed experiments involving human annotators, including both experts and crowd-workers. While the exact results of these experiments cannot be fully reproduced due to the inherent variability of human participants, the process can be replicated. To facilitate this, we provide detailed guidelines, platform setup, and technical specifications related to the annotation procedure of crowd-source (subsection 3.2, subsection A.1) and experts (subsection 3.2, subsection A.2).

For the experiments involving LLMs, we have included detailed procedural steps (subsection 2.2; subsection 3.2; and Appendix A.3), the prompts used (Figure 10), model versions (subsection 3.2; subsection 6.1; and Appendix C.2), hardware specifications and hyperparameter configurations (Appendix C.1), statistical measures (Appendix D), and evaluation metrics. These materials ensure that the LLM experiments are reproducible.

## Appendix

## Appendix A Annotation Approaches

### A.1 Crowd-source

#### A.1.1 Annotation Procedure.

Each example was annotated by three annotators, that in addition to the binary label were requested to provide their confidence in their answer, and also write a short explanation for why they chose this label.
Pre-qualifications included 50+ approved HITs, 97%+ approval rate, and locations from [USA, UK, Australia], which are all English-speaker countries.
We disabled the possibility of right-click and Ctrl+c in the platform (as suggested by (Veselovsky et al., )), to prevent (as much as possible) the case where generative-AI (e.g., ChatGPT) will be applied to solve the task instead of humans solving it themselves (as shown by (Veselovsky et al., )).
Maximum time allowed per HIT was 6 minutes, while the actual average execution time was 2:20 minutes for all assignments, and 3 minutes for approved assignments.
The guidelines provided to annotators and the annotation platform layout are presented in Figure 8.

Each annotation was manually reviewed and was rejected if the answers were not in line with the instructions, or if it was obvious that the task wasn’t done honestly. Overall, this task suffered from a high rejection rate of 49.2% (1163 rejected, 1200 approved). The main rejection reasons were:
lack of meaningful explanation, obvious copy-paste annotations across different examples, explanation contradict with the label annotation, and cases where the explanation was a copy-paste of either the grounding or the statement.

Figure: Figure 8: Platform for crowd-sourcing annotation in Amazon Mechanical Turk (MTurk). (Top) Guidelines for the task and definitions. (Bottom) Annotation layout for a single instance.
Refer to caption: x7.png

#### A.1.2 Consistency

Table: Table 5: Distribution of crowd-source annotators. Each example was annotated by 3 workers. Plain segments are unanimous annotation, while dotted segments indicate examples where some annotators labeled as inconsistent, and other as consistent. For example, 19.8% of the examples had two inconsistent annotation, and one consistent annotation.

Crowd workers showed near-random agreement, indicating relatively poor-quality annotations. Table 6 describes the distribution of annotations by MTurk workers. Only 40.8% from the examples were labeled unanimously, where the rest included annotations from both classes. In addition, if aggregating as majority vote, we get that 75.8% of the examples are labeled as consistent, which is far from the original distribution of classes. As mentioned before, even experts may miss a small inconsistency nuance, and finding it requires attention. Even from the subset of examples unanimously labeled as consistent, 37.9% have a label of inconsistent in both original and gold labels, which points at a lack of attention and thoroughness.

### A.2 Experts

Experts annotation was using the platform of Label Studio.(^5^55[https://labelstud.io/](https://labelstud.io/)) Layout design is presented in Figure 9. Examples were presented in random order, and neither the LLM prediction nor the original label were presented during the annotation. At the first stage, each example was annotated independently both experts. Afterwards, the human experts began in a second phase of a reconciliation, where a discussion was made over examples they disagree over. This reconciliation phase ended up with a much higher agreement, and higher-quality labels.

In the reconciliation phase, we observed that most changes (69.3%) were from label 1 to label 0, indicating that contradictions might be hard to find, and not all annotators catch them at first. For the full distribution of label change in the reconciliation phase, see Table 6.

Figure: Figure 9: Annotation platform on Label-Studio for experts
Refer to caption: x11.png

### A.3 LLMs

We used four LLMs for the task of annotating a total of $160\times 4=640$ examples from four different datasets.
Each model was run with four different prompts (see full prompts in Figure 10). We used a variety of terminology, as this task appear with different framing in different studies. For example, the premise-hypothesis terminology from classic NLI (MacCartney & Manning, ), or document-statement used in (Tam et al., ).

For API models (GPT-4, PaLM2), we set temperature=0.0 and extracted the logit of the generated token (functionality provided by both APIs), if the generated token was either ’0’ or ’1’ as expected. This logit was then transformed into a probability $p_{t}=P(y=t|x)$ via exponent corresponding the generated token $t$, and $1-p_{t}$ for the other label. To address the case where the first generated token was an unrelated token such as ’ ’, ’\n’, we set max_tokens=2 and took the first appearance of either ’0’ or ’1’. For all models, prompts and examples, ’0’ or ’1’ were in the first two generated tokens. Rest of parameters were set according to their default values.

For models available through the HuggingFace API (e.g., Mistral, Llama 3), we can load the model parameters and make inference locally. In that case, we get access to logits for all tokens, instead of just for the generated ones. Therefore, we applied a similar procedure, where we seek for the first appearance of either ’0’ or ’1’ to be the most probable token to be generated, and then directly extracted the logits of the ’0’ and ’1’ tokens. These logits were transformed into probabilities $(P(y=0|x),P(y=1|x))$ via a softmax function.

Figure: Figure 10: Four different prompt input templates to LLMs for obtaining binary labels
Refer to caption: x12.png

## Appendix B Cost and Scalability

In section 5 we compare the different annotation approaches on label quality, consistency and cost-effectiveness. Here are the full details regarding run-time and costs.

In MTurk platform, a total of $400\times 3=1200$ annotations cost $572\$$, including 2 small pilot experiments. All annotations were prepared within a few hours. However, it demanded an additional and significant time for review, after which rejected examples returned to the pool. This annotation-review cycle was conducted for $\sim 5$ iterations.
Inference via OpenAI’s API on GPT-4 cost $\sim 4.5\$$ per prompt.
Inference via VertexAI’s API on PaLM2 cost $\sim 0.15\$$ per prompt. Both took $\sim 8$ minutes per prompt.
Inference on Mistral and Llama3 was via the HuggingFace API, and its cost is estimated by the cost of using a suitable Virtual Machine (VM) on Google Cloud Platform (GCP) for the time of inference (1 minute per model)- $\sim 0.1\$$ per prompt.
In total, annotating with an LLM is $\sim 10^{2}$ to $10^{3}$ times cheaper than a human annotator in a crowd-sourcing platform, and of course than human experts. It is also faster if considering reviewing procedure (excluding code writing or layout designing).

## Appendix C Mislabeled Data Implications

### C.1 Fine-tuning

Hardware.  For the finetuning of DeBERTa models, both the base pre-trained model, and the NLI model which is in the same size, in subsection 6.1, we used 2 Quadro RTX6000 (24GB) GPUs.

Implementation.  We used HuggingFace trainer with early stopping of 4 epochs. The finetuning procedure includes splitting the training set into train and validation sets (where validation size is 25% and train 75%), fine-tuning on the train set, and choosing the best checkpoint based on the validation ROC AUC. We ran all experiments on five different seeds, affecting also the train-validation split and the random set chosen for ablation. We fine-tuned all variants with the same hyperparameters, determined by the best performing on the no-manipulation baseline. This includes 30 epochs at most, batch size of 16, learning rate of 5e-5 and weight-decay of 0.03. The rest were set as the trainer and model default.

Additional Experiments.
The left plot in Figure 11 presents the same experiment discussed in subsection 6.1, but starting from the pre-trained DeBERTa-v3-base. Same trends applies here, where our LLM-confidence-based manipulations of either flipping or filtering flagged examples outperforms the baselines.

The right plot in Figure 11 compares the performance of these methods (starting from the NLI model) when applied to both the training and validation sets (solid bars) or only the training set (dashed bars). The results are consistent, with no statistically significant differences between the two settings. Importantly, all variations outperform the baseline, underscoring the critical role of a well-curated training set in enhancing the model’s ability to generalize effectively.

Figure: Figure 11: Similar experiments to the one in Figure 7, with small alterations. (Left) Starting from a different base model - pre-trained DeBERTa-v3-base. (Right) Dashed columns present results for when flipping or filtering methods were applied only on the training set, but not the validation.
Refer to caption: x13.png

### C.2 Model Evaluation

In subsection 6.2 we evaluated the following models: GPT-4, PaLM2 (text-bison@002), Mistral-v0.2 (7B), and Llama3 (8B), which are covered in subsection 3.2; DeBERTa-v3 and NLI-model, which is a fine-tuned version of it on NLI datasets, as discussed in subsection 6.1; and GPT-4o, GPT-4o-mini, Mistral-v0.3,(^6^66[https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)) which share the same implementation as GPT-4 or Mistral-v0.2.

## Appendix D Statistical Analysis

### D.1 Clopper-Pearson

As mentioned in subsection 4.1, we we employed the Clopper-Pearson exact method (Clopper & Pearson, ) to construct a 95% confidence interval for the binomial proportion, adjusted by a finite population correction (FPC). As we only have a subset of examples we re-annotated by LLMs or experts, we can not precisely determine what is the error rate in the full dataset, but only construct a confidence interval based on the re-annotated subset.
The Clopper-Pearson method provides an exact confidence interval for a binomial proportion, which means it gives a reliable estimate even with small sample sizes. By applying the finite population correction (FPC), we adjust the interval because our sample is drawn from a limited population. This adjustment helps refine the estimate by taking into account the size of the overall dataset compared to the sample.

For your reference, the sizes of the complete datasets are provided in Table 7.

Table: Table 7: General datasets information.

### D.2 Bootstrap sampling

In subsection 4.1, we use bootstrap sampling to provide confidence intervals for each bin. Unlike the method in subsection D.1, we do not make claims about the entire dataset, but rather focus on the re-annotated subset we possess. To achieve this, we perform 100 bootstrap samples from the empirical distribution of each bin, sampling with replacement. We then measure the agreement between the experts’ resolutions and the LLM annotations, compared to its agreement with the original label.

## Appendix E Label Errors

Table 8 demonstrates one example per dataset, in which the original label is in fact an error, the LLM prediction marked it as a candidate, and the expert annotators determined the correct gold label.

Table: Table 8: Annotation errors in the original datasets, discovered by LLMs and corrected by experts.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: **Question:**
1. The paper claims that LLMs show considerable agreement with expert annotators in a zero-shot setting for mislabeled samples. How do the authors justify this claim, given that it is widely known that performance improves in a few-shot setting? Specifically, if LLMs outperform crowd-sourced annotators in matching expert annotations, a decrease in performance should be expected. This is because LLMs, by learning from in-context examples, would align more closely with expert annotations, resulting in greater disagreement with crowd-sourced annotations.

**Suggestions:**
1. In line 79, it would be clearer to provide a numerical value instead of a verbal description.
2. The specific checkpoint used for GPT-4 is not mentioned in the paper.
soundness: 1
strengths: 1. The paper is well-written.
2. The topic is interesting, and the paper addresses important questions.
3. The approach used in the paper is clear and aligns well with the research questions mentioned.
summary: The paper examines mislabeled instances in existing standard label-based datasets. The proposed method uses large language models (LLMs) to identify these instances, flagging them as potential errors when LLM predictions differ from the ground truth label. These flagged instances are then reviewed by expert human annotators for relabeling. Depending on the research question being answered, the paper compares LLM predictions and crowd-sourced annotations with expert annotations to identify mislabeled data and conduct further analysis. The findings reveal significant disagreement between expert and crowd-sourced annotations, resulting in a high rate of mislabeled data. Notably, LLM predictions align more closely with expert annotations.
weaknesses: 1. The experiments were conducted with a small sample size and a small number of datasets. While it is understandable to control the budget when using LLMs and crowd-sourced annotators, I believe the sample size and the number of datasets in this study are too small to support the conclusions drawn. This is because a change in the label of even a few samples could lead to a significant shift in performance, and the central argument of this paper relies on this performance change.

2. In line with the previous comment, I am uncertain about the reliability of the results reported in Figure 2 and Table 3. If crowd-sourcing performs this poorly in matching expert annotations (gold labels), how did LLMs learn to predict labels that align more closely with expert annotations, given that they were trained on standard datasets labeled through crowd-sourcing?

3. According to the paper, the authors used their own guidelines to relabel dataset instances for both crowd-sourced and expert annotators. However, these guidelines differ from the original guidelines used to label the datasets. This creates a major discrepancy, as it raises the question of whether the newly assigned labels truly reflect the same conditions outlined in the original guidelines for determining ground truth labels.

4. While there is demographic information about the crowd-sourcing participants, such as their status as native English speakers, there is no such information available for expert annotators.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Could the methodology be extended to detect errors in multi-class datasets or cases with label ambiguity (fuzzy label)?
- What plans do the authors have to validate this approach beyond the datasets examined?
soundness: 3
strengths: The paper provides a detailed empirical study that underscores the prevalence of labeling errors in commonly used NLP datasets. It is thorough in its methodological approach, with comprehensive experiments and multiple evaluation metrics. The use of LLM ensembles to assess annotation quality and error rates highlights the potential cost-effectiveness of this approach compared to traditional methods.
summary: This paper evaluates the potential of LLMs as tools to detect label errors in NLP datasets, comparing LLM-based annotations with crowd-sourced and expert annotations. Using four datasets from the TRUE benchmark, the authors aim to demonstrate that LLMs can effectively flag mislabeled instances and, by extension, improve the overall accuracy of model training and evaluation.
weaknesses: This paper lacks substantial innovation, as its primary contribution—using LLMs for label error detection—builds on existing methods without introducing a significant advancement. Similar approaches have already been explored in previous studies, and the work here does not establish a clear differentiation or improvement over these. Additionally, the paper could benefit from expanding its dataset scope and addressing challenges such as multi-class labels or ambiguous labeling scenarios. Without these considerations, the findings may not generalize well to other, more complex datasets.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. Did you look into why the Fleiss’s kappa of crowd-sourcing is that low? To my understanding, Fleiss’s kappa is usually higher than 0.2, even for some subjective tasks (e.g. preference of responses). It is weird to me that the kappa value is only 0.074. 

Typo:
line 422: alnoe -> alone
line 427: TThe -> The
soundness: 4
strengths: 1. The paper is well written and easy to follow.
2. This paper shows that LLMs can be used to annotate data or identify label errors with a high agreement with experts, which can be a good way to mitigate the scalability issue of dataset curation. 
3. The authors conducted extensive experiments, making their arguments convincing.
summary: This paper evaluate the label errors in existing datasets and examine the reliability of different annotating approaches: expert, crowd-sourcing, and LLM-as-a-judge. The authors prompt LLM to predict the label of four datasets and asked experts to verify the prediction if it disagrees with the original label. The result shows a 10~40% of label errors in four selected datasets. In addition, LLMs' prediction highly align to experts' label when the confidence score of LLMs is high. The authors also hire crowd-workers to annotating a subset of samples, and shows that the annotations of crowd-workers have a lower inter-annotation agreement and a higher disagreement with labels annotated by LLMs and experts, suggesting that LLM-as-a-judge is a reliable and efficient alternative for large-scale annotation tasks. In the end, the authors introduced two approaches to mitigate the label errors in a dataset: by filtering those data points and by flipping the wrong labels. Experimental result shows that models' fine-tuned on datasets cleaned with both approaches achieve higher performance.
weaknesses: 1. The approach and experiment are limited in binary classification tasks, specifically in factual consistency. The argument could be stronger if the authors conduct additional experiments in other types of task.
2. The experimental result of the crowd-sourcing part may be questionable. In Appendix B.1, the authors mentioned that the pre-qualifications of the MTurk task included 50+ approved HITs (line 1075). However, to get a higher quality result, people usually set the minimum number of approved HITs at 3000. This may explain why the authors got a high rejection rate of 49.2% and a very low Fleiss’s kappa value of the workers. As a result, the argument in Sec. 5 that the quality of crowd-sourcing annotation is insufficient may be too strong.

Meta-review: This paper investigates the use of LLMs to identify mislabeled instances in NLP datasets, comparing LLM predictions, expert annotations, and crowd-sourced labels. The study highlights the prevalence of labeling errors (10–40%) in four binary classification datasets and demonstrates that LLM predictions align more closely with expert annotations than crowd-sourced labels. While the findings are intriguing and the paper is well-written, several weaknesses limit its contribution. The study’s scope is narrow, focusing solely on binary classification and factual consistency tasks, which undermines the generalizability of the conclusions. Additionally, the small sample size and limited number of datasets reduce the robustness of the results, as even minor changes in annotations could significantly impact the findings. The experimental design also raises concerns, particularly regarding crowd-worker qualifications (low pre-qualification thresholds may have led to unreliable annotations) and the mismatch between relabeling guidelines and original dataset standards. Furthermore, the paper lacks innovation, as the approach of using LLMs for label error detection is incremental and builds on existing work without introducing significant methodological advancements. Given these limitations, the study falls short of providing sufficient novelty and generalizable insights to warrant acceptance.

Decision: Reject