Unveiling the Multi-Annotation Process: Examining the Influence of Annotation Quantity and Instance Difficulty on Model Performance
Authors: Pritam Kadasi
Mayank Singh
Venue: EMNLP 2023 Findings (2023)
Keywords: multi-annotation
label-distribution
PVI
annotator-set
$\mathcal{V}$-Information
entropy
annotation budget
datamaps
cartography
TL;DR: We showcase, more annotations per instance in the dataset doesn't necessarily mean better performance of the model when trained on these datasets

Abstract: The NLP community has long advocated for the construction of multi-annotator datasets to better capture the nuances of language interpretation, subjectivity, and ambiguity. This paper conducts a retrospective study to show how performance scores can vary when a dataset expands from a single annotation per instance to multiple annotations. We propose a novel multi-annotator simulation process to generate datasets with varying annotation budgets. We show that similar datasets with the same annotation budget can lead to varying performance gains. Our findings challenge the popular belief that models trained on multi-annotation examples always lead to better performance than models trained on single or few-annotation examples.

Unveiling the Multi-Annotation Process: Examining the Influence of Annotation Quantity and Instance Difficulty on Model Performance
Pritam Kadasi and Mayank Singh
Department of Computer Science and Engineering Indian Institute of Technology Gandhinagar Gujarat, India
{pritam.k, singh. Jii i wig.gagc.c.... ac.
Abstract
data with multiple annotators requires significant time, annotation budget, and annotator expertise to ensure the creation of high-quality datasets with diverse perspectives.
The NLP community has long advocated for the construction of multi-annotator datasets to better capture the nuances of language interpre- lation, suhjectivity, and ambiguity. This paper conducts a retrospective study to show how performance scores can vary when a dataset expands from single annotation per instance multiple annotations. We propose novel muuli--annotttor simulation process to generate datasets with varying annotation budgets. We show that similar datasets with the same anno- tation budget can lead to varying performance gains. Our findings challenge the popular belief that models trained on multi-annotation exam- ples always lead to better performance than models trained single or few-annotation ex- amples.
Morcover, with limited annotation budget, it becomes crucial to determine the optimal number of annotators within the given constraints. This not only helps save annotation time and budget but also ensures efficient utilization of available resources. While some research (Wan et al., 2023; Zhang et al., 2021) has provided insights and sug- gestions on finding the optimal number of annota- tors, definitive solution to this problem has yet be achieved.
Another challenge is the restricted number of annotations available per instance, typically not exceeding 6 10, even with a large number of cruited annotators (Plank, 2022). This limitation arises from the considerable annotation efforts re- quired for a large volume of instances. As a result, when models are trained on such datasets, they only capture the opinions and information of small subset of the annotator pool. Additionally, certain datasets have not released annotator unnotator-specifie labels or established mappings to individual annotators (Nie et al., 2020; Jigsaw, 2018: 8: Davidson et al., 2017). However, the trend is gradually shifting, and there is a growing recognition that annotator- level labels should be made available (Prabhakaran et al., 2021; Basile et al.. 2021; Denton et al., 2021). This study aims to tackle the challenge lack- ing annotator-specific labels by simulating a multi- annotation process. Through this study, we provide insights into how the inclusion of more annota- tors can introduce variations in model performance and identify the factors that influence this variation. Considering that previous research (Swayamdipta et al., 2020) has highlighted the influence of indi- vidual instance difficulty on model performance, we examine how the addition of more annotations alters the difficulty level of instances and conse-
1 Introduction
The process of creating datasets often involves prac- tical constraints such as time, resources, and budget that limit the number of annotators or experts avail- able for collecting annotations (Sheng et al., 2008). As a result, there is a prevalence of single or few la- bels per instance (depending on the limited number of annotators) in the collected data. However, train- ing models on these datasets pose challenges to their generalization abilities, primarily because the data lacks diversity. With scarcity of different per- spectives and variations in the training data (Basile et al., 2021; : Plank, 2022), models may struggle to learn robust representations and fail to generalize effectively (Nie et al., 2020; Meissner et al., 2021). To address these challenges, the NLP commu- nity has highlighted the advantages of utilizing multi-anne annotator datasets (Davani et al., 2022) and also emphasized the importance of releasing multi- annotator datasets and associated information (cul- tural and demographic, etc.) (Sap et al., 2022; Her- shcovich et al., 2022). However, this approach introduces its own set of challenges. Collecting

---

quently affects model performance.
V1.1
01.2
In summary, our main contributions are: simula-
22.1
92.2
M2.55
Y
We propose novel tion process to address the issue of missing annotator-specife labels.
MN,
MN.MM
We demonstrate, that increasing the number of annotations per instance does not necessarily result in significant performance gains.
A2
fan, 02)
As
(a d2.. (lke)
We also demonstrate, that altering the number of annotations per instance has a noticeable impact on the difficulty of instances as per- ceived by the model and consequently affects the model performance.
yh.1
81.2 :
MLL
1L.2
...
30.4
X,
X.Ab.
Ma.
50.2
205.0
BS.
Figure 1: A Toy Multi-Annotator Dataset
2 The Multi-annotated Dataset
do not include annotator-specific labels. Instead, the available information is limited the label distribution for each instance (Nie al., 2020; Jig- saw, 2018; Davidson et al., 2017). For instance, cases with M annotations per instance and three possible labels, the label distribution is commonly represented by a list [p, r), where p. q. and r are positive integers that sum up to M. To ad- dress this constraint, introduce simulation pro- for multi-annotator scenarios that leverages the instance-level label distribution. Our proposed approach (see Algorithm 1), encompasses the fol- lowing steps:
In practical scenarios, the annotation process be- gins by hiring one more annotators who annotate each instance in the dataset. To enhance the rep- resentation of the true label distribution, have the option extend this process by recruiting addi- tional annotators. We continue this iterative process until either the annotation budget is exceeded or we observe saturation in the model's performance in predicting the true label distribution. As a result, we obtain multiple annotations assigned to each instance in this multi-annotated dataset.
A annti-r dataset D is formally char- acterized triplet D AY) in this research paper. The set X represents N text instances, denoted as æ1, 21., corresponds to M annotators, represented as 01, .,,0a,..... AMM- The annotation matrix Y cap- tures the annotations, with rows indexed by X and columns indexed by A. Specifically, Y A[ ([11, #2, ZN; a1, ....... aM). In simpler terms, the entry Y[ ay] stores the label Vi, assigned instance x; by annotator aj. Fur- thermore, an annotator-ser Ag, which comprises annotators where 1 5 k < M,, is defined. Conse- quently, the subset D restricted to Aş is denoted as Dz (X, Ag, Y'). where Y' = Y[X; Ax). This paper refers to Dr as the dataset subset with k annotations per instance. Figure/ illustrates toy multi-annotaie dataset, showcasing M annotators, and N instances along with its subsets comprising 2 and k annotators.
Initially, we generate a list of annotations for each instance by considering the actual instance-level label distribution. [Line 1] Subsequently, we randomize these annotation lists using consistent random seed across instances. [Lines 5-6]
The set
Next, we select the first k annotations from each randomized list, creating the dataset sub- set Dt- [Lines 4-8]
By employing this algorithm, we can generate h annotations per instance, thereby addressing the limitation of annotator-specific labels in existing multi- multi-annotator datasets. By repeating the algo- rithm with different random seeds or parameters, we can create multiple datasets subsets DR. each containing k annotations per instance. This flex- ibility enables the generation of diverse subsets, expanding the range of multi-annotator scenarios that can be explored and analyzed in our research.
3 Simulating the Multi-annotation Process
4 Experiments
4.1 Datasets
Based on our current knowledge, worth not- ing that existing multi-annotator datasets typically
We selected the ChaosNLI dataset (Nic et al., 2020) for our study, as it contains the highest number of

---

Algorithm 1 Creation of Annotator Datasets
4.2 Prefrained Language Models (PLMs)
Input:
X: set of N instances
In our study, we utilize all the pretrained language models (PLMs) reported in the ChaosNLI work by Nie et al. (2020). Specifically, BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), XLNet (Yang al., 2020), ALBERT (Lan et al., 2020), and DistilBERT (Sanh al., 2020). is important to clarify that our objective not to showcase state-of-the-art (SOTA) performance using these models, but rather to demonstrate the variations in performance we incrementally add annotations to the dataset.
CL: list of C class labels
LC: label counts of shape N C
experiment with
M: number of annotators
Output: D' = (Di, D2, ................................ M
1: AL f GETANNOTATIONLIST(
2: Initialize an empty set
3: for k + to M do
4: Initialize an empty list y
5: for i 1 to N do
6: SL RANDOMSHUFRLEALLLS
7: Choose first k annotations from AL
and add it y
4.3 Training Strategies
8: end for
In this section, we describe two variants of training strategies.
9:
DR + (X
10: Add D2 to D
Majority Label (ML): The PLMs are finetuned using the majority label, which determined by aggregating annotations from the target list of an- notations. The training objective aims to minimize the cross-entropy between the output probability distribution and the one-hot encoded majority la- bel.
11: end for
12: Return D
annotations (=100) per instance among the pub- licly available datasets (Plank, 2022). ChaosNLI is Natural Language Inference (NLI) task dataset known for its high ambiguity. Additionally, the ChaosNLI dataset includes sub-datasets, namely ChaosNLI-S and ChaosNLI-M, which are subsets extracted from the development sets of SNLI (Bow- man et al., 2015) and MNLI-matched Williams et al., 2018 8), respectively. Another sub-dataset, ChaosNLI-o. is created from the entire develop- ment set of AbductiveNL. hereafter, referred to as a-NLI (Bhagavatula et al., 2019).
Label Distribution (LD): The PLMs are finetuned using the label distribution from the target list of annotations (Meissner et al., 2021). The training objective aims to minimize the cross-entropy be- tween the output probability distribution and the target label distribution.
4.4 Evaluation
To evaluate the performance of our models, we utilize the classification accuracy computed on the test dataset. In the MLL setting, the accuracy is computed by comparing the label associated with the highest softmax probability predicted by the model with the majority label derived from the target annotations. the LD setting, the aceuracy computed by comparing the label corresponding to the highest softmax probability predicted by the model with the label that has the highest relative frequency in the target label distribution.
The ChaosNLI dataset consists of 4,645 in- stances, cach annotated with 100 new annotations. Additionally, the dataset already includes 5 old an- notations for ChaosNLI-S and ChaosNLI-M, and old annotation for ChaosNLI-o. Subsequently, we create D (see $3) utilizing these datasets and then divide these Di into train, development, and test sets using an 80:10 10:10:10 ratio. Table provides detailed statistics of the datasets used our study.
Datasets
#Instances
#Annotations Per Instance
#Class Labels
4.5
Experimental Settings
Following the approaches described in the studies (Nie et al., 2020; Meissner al., 2021), struct base models by finetuning PLMs (described in $4.2) on the combined train sets of SNLI and
SNLI
150,
5
MNLI
392,702
S
G-NLI
169,654
ChaosNLI-S
1,524
100
ChaosNI.I-M
1,599
100
ChaosNLI-o
1,532
100
##Isttncces corresponding to SNLI. MNLI and e-NLI are of train set as only train set is used for training base models in our study.
Table Dataset Statistics'

---

Min. Accuracy
Max. Accuracy
ChooNNEES
ChanNNLLM
ChocsNLL-o
ChocsLEES
ChanNNII-M
ML
LD
ML
LD
ML
LD
ML
LD
ML
LDD
ML
L.D
(1)
(100)
0.741
(100)
(30)
XLNNet
0.647 (1)
0.643(())
0.564 (1)
0.561 (1)
0.647 (2)
0.648 (1)
(1745((100)
0.77 (100)
0.744 (100)
0.751 (80)
0,679 (30)
0.685 (30)
ALBERT
0.639 (1)
0.668
0.737 (100)
(100)
0.751
BERT
0.643 (1)
0.643(1)
0.579 (1) 0.579
0.598 (6)
0.585 (6)
0.753 (90)
0.757 (3000
0.751 (90)
0.709 (100)
B.613 (3)
0.616 (3)
DiSIBERT
0.32
0.632
0.533
0.53
0,582
0.584 (96)
0.724
0.73 (100)
0.692 (80)
0.682
0.608 (3) 0.61 (3)
Table 2: The performance of various models in both the ML and LD settings is presented in this table. Values
indicate accuracy, and values in braces indicate k. The values highlighted in bold indicate the optimal number of
annotators where the performance reaches its peak compared to the maximum annotation budget allocated (100), Conversely, the highlighted values in the minimum accuracy column indicate the lowest performance achieved compared to the minimum budget allocated (1). This information provides insights into the impact of the number of annotators on the model's performance.
MNLI for both ChaosNLI-S and ChaOsNLI-M. For the ChaosNLI-o dataset, we construct base models by finetuning on the train set of a-NLI. We further finetune these base models with increasing sizes of annotators. Specifically, we finetune models for each Da. where E [1, 100]. For each k, we re- port average performance scores over test sets of 10 D 's (see $33
in ChaosNLI-S, it is observed that the performance reaches saturation point between 20 to 80 anno- tations. This means that increasing the number of annotations beyond this range does not result in significant improvement in the accuracy scores. Table 2 provides a complementary perspective on the observed trends. It highlights that the min- imum performance not consistently associated with the dataset having the fewest annotations, and vice versa. In the case of ChaosNL.Ia with BERT and DistilBERT, it is interesting to note that the optimal performance is achieved with just three annotations. This represents extreme scenario where minimal number of annotations can lead to the best performance. In general, these findings shed light on the optimization of our annotation budget. Similarly, the performance gain (maximum minimum accuracy) across different datasets also significantly varies. The average performance gain for ChaosNLI-M, ChaosNLI-S and ChaosNLI-o is O. 101 0. 0.7 and 0.031, respectively. The no- table variability in performance gain across differ- datasets further emphasizes that the impact of increasing annotations on performance improve- ment not consistent. It underscores the need carefully analyze and understand the specific char- acteristics of each dataset and model combination to ascertain the relationship between annotation quantity and performance.
We choose hyperparameters from the experimen- tal settings of the following work (Nie et al., 2020;
Meissner et al., 2021; Bhagavatula al., 2019). Our optimization technique involves employing the A optimizer (Loshchilov and Hutter, 2019). More details on hyperparameters can be found in $A.2. To ensure reproducibility, we conduct our experiments using the open-source Hugging Face Transformers library (Wolf et al., 2020). Further- more, all experiments are performed using 2 x NVIDIA RTX 2080 Ti GPUs.
5 Results and Discussion
5.1 Is higher performance always guaranteed by increasing the number of annotations?
Figure presents the accuracy scores as the number of annotations increases. Notably, the trends observed in the performance of ChaosNLI-S, ChaosNLI-M, and ChaosNLI-o challenge the pre- vailing belief that increased annotations invariably lead to improved performance. Specifically, for ChaosNLI-S and ChaosNLI-M, the accuracy scores exhibit non- momooonoo increasing patter.. In con- trast, the trend observed for ChaosNLI-a,, particu- larly with BERT and DistilBERT models, deviates from this expected behavior. In these cases, the accuracy scores show a decreasing trend as the number of annotations increases. Upon examining the RoBERTa accuracy scores for the LD setting
To provide an explanation for the observed com- plex behavior, we utilize the V-Information (Etha- yarajh al., 2022). V-information is measure that quantifies the ease with which model can predict the output based on a given input. The higher the y- information, the easier is for the model to predict the output given input. Furthermore V-information cannot be negative unless model overfits, etc. (see $A.1).
hnttps://huggiingfaccccmm wtteeee..oooccseeeeeeeeeeoeeoese cw//ocs///aaeeeeeeeeee co/bbcsstrrnsformers/
Figure 3 provides a visual representation of

---

NLI-S
ChaosNLI-M
ChaosNLI-a
0.80
RoBERTa
Accuracy
0.70
0.55
XLNett
Accuracy
0.70
0.60
0.75
ALBERT
Accuracy
0.60
0.50
0.75
Acolracy
BERT
0.65
0.60
0.50
0.75
DistilBERT
Accurace
0.65
Label Distribution (LD)
Majority Label
0.50
280
Figure 2: The figure displays accuracy scores for various models across for datasets ChaosNLI-S, ChaosNLI-M and ChaosNLL-ax For every l X-axis, the mean and standard deviation of the accuracy scores of models trained on 10 D3's 's are displayed. The detailed plots for ChaosNLI-a BERT and ChaosNLI-a DistilBERT can be found in Figure in the Appendix.
the V-information scores for the three datasets across five different PLMs. As anticipated, the V-- information scores are higher for the ChaosNL.IS and ChaosNLI-M datasets. Models that exhibit higher V-information scores also tend to yield higher accuracy scores in the LD-based perfor- mance evaluation. For instance, RoBERTa out- performs other models (except XLNet, for which the performance similar) terms of accuracy for the ChaosNLI-S dataset. The saturation of V-information scores starting at k = 20 for the ChaosNLI-S dataset effectively explains the ob-
served saturation of LD-based accuracy after 20 annotations, as depicted in Figure 2. This phe- nomenon suggests that the model reaches point where additional annotations provide diminishing returns in terms of extracting valuable insights from the instances. Therefore, the model's performance ceases to improve significantly beyond this thresh- old. For the ChaosNLI-a dataset, except RoBERTa and XLNNe (V-Information E [0, 0.25). compara- tively low), all models yielded approximately zero V-information 3 This implies that adding
We used same hyperparameters for all k's due to which

---

ChaosNLI-S
ChaosNLI-M
ROBERTS
XUNt
SERT DIStIDERT
more annotations to the ChaosNLLIa dataset does not establish a clear relationship between the input and output label distribution. This observation sug- gests that, for this particular variant of the dataset, the model might rely on factors other than the pro- vided annotations make accurate predictions. Figure 3:
The figure displays the V-Information values for various models in the LD setting. A higher value indicates that the data is easier for the respective model with respect extracting information from it. These values can be compared across datasets and models.
them correctly over multiple epochs. Ambiguous (a) instances display high variability in predicted probabilities for the true label. We investigate the proportion of the transitions between these cate- gories with the incorporation of additional annota- represents proportion
tions. For example, of the transitions from easy-to-learn ambigu- ous category among all transitions. This provides valuable insights into the underlying factors that contribute to the observed improvements or lack thereof in the model's performance.
The aforementioned findings indicate that all datasets yield similar performance when trained un- der the same budget, underscoring the importance of selecting the appropriate dataset for a specific task. Furthermore, these findings emphasize the significance of determining the optimal number of annotators, as the model's performance varies with the increase in annotations.
Figure 4 illustrates an interesting pattern in ChaosNLI-S and ChaosNLI-M datasets: as the number of annotations increases, significant pro- portion of training instances transition from the a category. For instance, more than 60% of all transitions between I to 10 annotations involve in- stances moving from the a > category. However, beyond 10 annotations, the proportion of instances transitioning to the e from the a category does not show a substantial increase. On the other hand, the reverse transition from the a category is the second most common transition. with an average proportion of 20%. The difference in proportions between the transition from e and the tran- sition from + becomes more substantial (at least 29%) as more annotations are added. the ChaosNLLIM dataset, we observe a higher propor- tion of instances transitioning from category to category h compared to the ChaosNLI-S dataset. Specifically, over 15% of the ambiguous instances in ChaosNLI-M exhibit shift towards the hard region, which is more than 50% of similar transi- tions observed in ChaosNLI-S. We argue that this substantial difference in transition patterns has a direct impact on the performance of models on the ChaosNLI-S dataset compared to ChaosNLI-MM.
5.2 Does the number of annotations influence the difficulty of instances as perceived by the model?
To investigate this question, we employ the concept of dataset cartography proposed by Swayamdipta al. (2020), which leverages train- ing dynamics distinguish instances based on their (I) confidence, measured the mean probability of the correct label across epochs, and (2) variability, represented by the variance of the aforementioned confidence. This analysis generates a dataset map that identifies three distinct regions of difficulty: easy-to-learn, hard-to-learn, and instances that are ambiguous with respect to the trained model. Easy-to-learn (e) instances exhibit consistently high confidence and low variability, indicating that the model can classify them correctly with confi- dence. hard-to-learn (h) instances, on the other hand, have low confidence and low variability, indi- cating the model's struggle to consistently classify
models for k < 3 overfitted resulting in negative V- Information.

---

ChaosNLI-S
ChaosNLI-M
ChaosNLI-a
80
RoBERTa
Changa (%)
60
40
20
80
Change (%)
XLNet
60
40
20
80
Change (%)
ALBERT
60
40
20
80
Change 195
BERT
60
40
20
80
DistilBERT
Change
60
40
20
As Av
As- Avo
As Apa
As- Aso
Al Ase
A1 Agi
Ai As Ago
Ay
Ago Asoo
Aso
As Ava
Asa
Asi
App
Aua
Ave
Age
As Avi
A1 Asso
Ai Aia
As Alg
As Ag
As Ago
As Api
A Aya
Ago
A1 Ago
At Ave
As Arc
A
As
A1
Figure 4: The figure provides visual representation of the transition instances between different categories during training the number of annotators increase from A to Ano.... A1o0- indicates percentage of instances that transitoned from category e to a.
Despite the presence of higher proportions of a to transitions ChaosNLI-M compared ChaosNLI- S, the category consistently leads better performance on the ChaosNLI-S dataset across all models analyzed.
the instances and classify them with reduced con- fidence. For XLNet and ALBERT, the combined proportion of low confidence transitions, e a and - h either surpasses or remains equal to the proportion of high confidence transition a + e. In the case of RoBERTa, behaves the same as ChaosNLI-S and ChaosNLI-M.
ChaosNLI-o exhibits distinct trends across vari- models. Specifically, in the case of BERT and DistilIBERT, where accuracy scores decline as the annotation increases (see Figure 2), we witness sig- nificant proportions of a (~~ 80%) and + (~ 43%) transitions, respectively. These transitions suggest that the models struggle to comprehend
These results suggest adding more annotations has indeed its effects on the difficulty instance thereby affecting the performance of the model.

---

6 Related Works
combines signals from uneven training data. Fi- nally, study by (Chen al., 2022) proposed an approach that reserves a fraction of annotations to explicitly clean up highly probable error samples to optimize the annotation process. All these studies contribute to the understanding of how to maximize the performance of NLP models under restricted annotation budgets. Our study aimed to address specific question within this context: assuming fixed annotation budget, which dataset would yield the highest performance?
Human disagreements in annotations. Tradi- tional upproaches like majority voting averaging overlook important nuances in subjective NLP tasks, where human disagreements are prevalent. To address this issue, Multi-annotator models treat annotators' judgments as separate subtasks, cap- turing the distribution of human opinions, which challenges the validity of models relying on ma- jority label with the high agreement as ground truth (Davani et al., 2022; Nie et al., 2020). Human vari- ation in labeling, which is often considered noise (Pavlick and Kwiatkowski, 2019), should be ac- knowledged to optimize and maximize machine learning metrics, as impacts all stages of the MI. pipeline (Plank, 2022). Incorporating annotation instructions that consider instruction bias (Parmar et al., 2023), which leads to the over-representation of similar examples, is crucial. This bias can limit model generalizability and performance. Future data collection efforts should focus on evaluating model outputs against the distribution of collec- tive human opinions to address this issue. All of the above works study annotator disagreements and how they affect the performance of models on downstream tasks. However, in our work, consid- ering disagreements' effect on model performance, we try to find out how the model performance varies increase the number of annotations per instance, i.e., varying the annotator disagreement, Overall, we try to answer, does more annotation per instance leads better performance or is the other way around?
Previous studies have demonstrated that anno- tation disagreements affect model performance. However, our study aims to explore how perfor- mance varies as we change the level of disagree- ment. we consider ideas from (Zhang al., 2021) who proposed a learning algorithm that can learn from training examples with different amounts of annotation (5-way, 10-way, 20-way) a multilabel setting, but we expand the number of annotations from 1-way till 100-way and train our model in a label distribution setting rather than in multi-label setting. To investigate the reasons for performance variation as we increase the number of annotations, we incorporate (Swayamdipta al., 2020)'s 's ideas and (Ethayarajh et al., 2022)'s ' concepts dataset difficulty. While previous studies focused on build- ing datasets and models and their impact on per- formance when the annotation budget is restricted, our work answers whether increasing the annota- tion budget necessarily leads to improved model performance. Overall, our study aims to demon- strate that, even with less annotation budget than its upper bound, is possible to achieve optimal performance compared to the performance at the upper bound thereby saving annotation budget and time. Our findings provide insights into optimizing annotation budgets.
Annotation under restricted annotation bud- get. Also, prior studies have investigated how to achieve optimal performance in natural language processing (NLP) models under restricted anno- tation budgets. One such study by (Sheng et al., 2008) examined the impact of repeated labeling on the quality of data and model performance when labeling is imperfect and/or costly. Another study by (Bai et al., 2021) framed domain adaptation with a constrained budget as consumer choice problem and evaluated the utility of different com- binations of pretraining and data annotation der varying budget constraints. Another study by (Zhang et al., 2021) explored new annotation dis- tribution schemes, assigning multiple labels per example for a small subset of training examples, and proposed a learning algorithm that efficiently
Conclusion
In this paper, we introduced a novel approach to handle the absence of annotator-specific labels the dataset through a multi-annotator simulation process. Additionally, we investigated the impact of varying the number of annotations per instance on the difficulty of instances and its effeett model performance. Our results highlighted that increas- ing the number of annotations does not always lead to improved performance, emphasizing the need determine an optimal number of annotators. This has important implications for optimizing annota-

---

tion budgets and saving time. Our findings pro- vide valuable insights for optimizing annotation strategies and open up new possibilities for future research in this direction.
Yejin Choi. 2019, Abductive commonsense reason- ing. arXiv preprint arXiv:1908.05739
Samuel R. Bowman, Gabor Angeli. Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lishon, Portugal. Association for Compu- tational Linguistics.
Limitations
The current study acknowledges several limitations that deserve attention. Firstly, the experiments were conducted using small-size Language Models due resource constraints. It is important to recognize that employing larger language models, such as BLOOM, GPT, and others, could potentially yield different outcomes and should be explored in future research. Furthermore, the scope of the discussion is constrained by the availability of datasets with a large number of labels per instance, leading to the utilization of the ChaosNLI dataset (Nie et al., 2020). Consequently, the generalizability of the findings to other datasets, if they emerge in the future, might be restricted.
Derek Chen, Zhou Yu, and Samuel R. Bowman. 2022. Clean or annotate: How to spend limited data collec- tion budget. In Proceedings of the Third Workshop on Deep Learning for Low-Resoue Resoures Natural Language Processing, pages 152- 168, Hybrid. Association for Computational Linguistics.
Aida Mostafazadeh Davani, Mark Díaz, and Vinodku- mar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective an- notations. Transactions of the Association for Com- putational Linguistics, 10:92-110.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detec- tion and the problem of offensive language. Proceed- ings of the International AAAI Conference on Web and Social Media, 11(1).
Acknowledgements
We express our gratitude to the anonymous review- ers for their insightful feedback. Our research has received support through the UGC-JRF fellowship from the Ministry of Education, Government of India. Additionally, we would like to extend our thanks to our colleague, Mr. Shrutimoy Das, a Ph.D. student at IIT Gandhinagar, who provided the initial review of this paper and generously shared GPU resources to conduct essential side experi- ments during critical phases of our research. We are grateful for these contributions, which signifi- cantly contributed to the success of this study.
Emily Denton, Mark Diaz, Ian Kivlichan, Vinodku- Prabhakaran, and Rachel Rosen. 2021. Whose ground truth? accounting for individual and collec- tive identities underlying dataset annotation.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
References
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2022. Understanding dataset difficulty with V-usable information.
Fan Bai, Alan Ritter, and Wei Xu. 2021. Pre-train or an-
notate? domain adaptation with constrained budget. In Proceedings of the 2027 Conference on Empiri- cal Methods in Natural Language Processing, pages 5002-5015, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Daniel Hershcovich, Stella Frank, Heather Lent, Miryam Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Pi- queras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and strategies in cross- cultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 6997-7013, Dublin, Ireland. Association for Computational Lin- guistics.
Valerio Basile, Michael Fell, Tommaso Fomaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. We need to consider disagreement in evaluation. In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pages 15-21, Online. Association for Com- putational Linguisties.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya. Keisuke Sakaguchi, Ari Holtzman, Han- nah Rashkin, Doug Downey, Scott Wen-tau Yih, and
Jigsaw, 2018 Toxic comment classification challenge. Accessed: 2021-05-01.

---

Artur Kulmizev and Joakim Nivre. 2023, Investigat- ing UD treebanks via dataset difficulty measures. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 1076-1089, Dubrovnik, Croatia. As- sociation for Computational Linguistics.
Maarten Sap. Swabha Swayamdipta, Laura Vianna. Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5884-5906, Seattle, United States. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for sulf--spe learning of language representations.
Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? improving data quality and data mining using multiple, noisy label- ers. In Proceedings of the 14th ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining, KDD '08, page 614-622, New York, NY. USA. Association for Computing Machinery.
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Man- dar Joshi, Dangi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
Nya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization, In International Confer- ence on Learning Representations.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang. Hannanch Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275-9293, Online, Association for Computa- tional Linguisties.
Johannes Mario Meissner, Napat Thumwanit, Saku Sug- awara, and Akiko Aizawa. 2021. Embracing ambi- guity: Shifting the training target of NLI models. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 862-869, Online. Association for Computational Linguisties.
Ruyuan Wan, Jachyung Kim, and Dongyeop Kang. 2023. Everyone's voice matters: Quantifying anno- tation disagreement using demographic information. arXiv preprint urXivv2301.00036
Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What
can we learn from collective human opinions on nat- ural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131-9143, Online. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A croad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long Papers), pages 11 12-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Mihir Parmar, Swaroop Mishra, Mor Geva, and Chitta Baral, 2023. Don't I blame the annotator: Bias already starts in the annotation instructions.
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent Disagreements in Human Textual Inferences. Trans- actions of the Association for Computational Linguis- tics, 7:677-694
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jermite. Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online. Association for Computational Linguistics.
Barbara Plank. 2022. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 10671-10682. Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On releasing annotator-level labels and information in datasets. In Proceedings ngs of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 133-138, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car- bonell, Ruslan Salakhutdinov, and Quoe V. Le. 2020. Xlnet: Generalized autoregressive pretraining for lan- guage understanding.
Shujian Zhang, Chengyue Gong, and Eunsol Choi. 2021. Learning with different amounts of annotation: From zero to many labels. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 7620-7632, Online and
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter.

---

Punta Cana, Dominican Republic. Association for Computational Linguistics.
A.2 Hyperparameter Details
Referring to Table 4, we initially trained the mod- els using the hyperparameters provided by (Nie et al., 2020). However, during our experiments, we observed signs of overfitting to our datasets. Consequently,
Appendices
A More Details
adjusted the hyperparameters, leading to the set provided in the table. More hy- perparameter details can be found in Tables and
A.1 V-Information
V-Information (Kulmizev and Nivre, 2023; Etha- yarajh et al., 2022), where V represents specific model families such BERT, GPT, etc., measures the level of ease with which model V can predict the output variable Y given the input X, The higher the V- y-Infrrmaiion, the easier is for the model V to predict the output variable Y given X. To mea- sure V-Information, we use predictive V-entropy:
A.3 Detailed Plots for Figure 2
For more comprehensive view of the phenomenon where performance decreases with an increasing number of annotations, provide detailed plots for BERT and DistilBERT, shown in Figure 5. While Figure maintains consistent y-axis for datasets ChaosNLI-(S, M, and a), these plots fea- ture distinct axes.
HY = inf log2 (() fev
and conditional V-eetropy:
Hv(Y|X)= = in in[[- log2 [XX] JEV
034
In simple terms, our goal is to find the f E V that maximizes the log- lieliikd of the label data with and without input X. Using these two quantities, V-Information can be calculated using the formula:
0A1
Iv(X -) HY) - XX
It is important to note that V-Information is com- puted with respect to Hy(Y), so Ty(X - Y) 2 0. Additionally, if X is independent of Y, then (x(X + Y)= =0.
657
(a) ChaosNLI-a BERT
While V-Information functions as an aggregated measure calculated for the whole dataset, (Etha- yarajh et al., 2022) extended this measure to new measure called Pointwise Y-Information (PVT), which allows for the calculation of the difficulty of individual instances. The higher the PVI, the casier the instance is for V in the given distribution. It be depicted by the formula:
PvI(x + y) - log, Pr + log, Pr(y")))
05a-
where Se S' E V are models trained with and with- out input E X, respectively, and y" refers to the gold label. Unlike y-Information, PVT can be negative, indicating that the model predicts the jority class better without considering the input compared when considering the input.
(b) ChaosNI.I-a DistilBERT
Figure 5: The figure displays accuracy scores for BERT and DistilBERT across & for dataset ChaosNLI-o. For every & on X-axis, the mean and standard deviation of the accuracy scores of models trained on 10 Dj are displayed.
Refer to Table for sample of instances from the ChaosNLI-a dataset with very low PVI, which demonstrates the high ambiguity these instances.

---

ChaosNU--S
ChaosNLI-M
ChaosNLI-g
0.80
0.75
RoBERTa
0.70
0.65
0.60
0.55
0.75
XLNet
0.70
0.65
0.60
0.55
0.75
ALBERT
0,70
0.65
0.35
0.50
BERT
0.55
0.75
DistiBBERT
0.70
0.65
Label Distribution (LD)
Majority Label (ML)
100
Figure 6: The figure presents accuracy scores various models across different values & datasets ChaosNLI- S. ChaosNLI-M, and ChaosNI.Iox. Along the X-axis, each & value corresponds to the mean and standard deviation the accuracy scores, based on models evaluated on test instances with the absolute ground truth.
A.4 Data Maps
test set remains the same, retaining 100 annotations for the LD setting, where the label distribution of these 100 annotations is considered. In the ML setting, we use the majority label of the 100 anno- tations.
Refer the RoBERTa datamaps in the LD setting Figures 7, 8, and For ChaosNLI-a, you can find datamaps for BERT and DistilBERT in the LD setting in Figures 10 and 11, respectively.
In Figure 6, on the whole, we observe little to no change in performance as we incrementally in- crease the number of annotations except few cases. Additionally, it's important to note that the hyperpa- rameters for these models are consistent with those listed in Tables and 5.
B Results on Absolute Ground Truth
We have extended our evaluation by testing our models the absolute ground truth, which repre- sents the majority label derived from all 100 anno- tations. In Figure 6, we provide plots for models trained on datasets with identical training and vali- dation instances as the Dj datasets. However, the

---

Parameters
Models
Learning Rate
Batch Size
Weight Decay
Max. Epochs
Learning Rate Decay
Warmup Ratio
SNLIMNLI G-NLI
3e-5
0.0
Linear Lincar
0.1
Ic-5
8
0.0
4
0.2
Table 3: Hyperparameters for base models RoBERTa, XLNNtt, ALBERT, BERT and DistilBERT
Parameter
RoBERTa
XLNet
ALBERT
BERT
DistilBERT
Learning Rate
5c-6
5c-6
5c-6
5c-5
5c-6
Batch Size
8
8
8
8
8
Weight Decay
0.0
0.0
0.0
0.0
0.0
Max. Epochs
3
5
5
3
Learning Rate Decay
Linear
Linear
Linear
Linear
Linear
Warmup Ratio
0.1
0.1
0.1
0.1
0.1
Table 4: Hyperparameters for finetuned models for dataset ChaosNLI-o
Parameter
RoBERTa
XLNet
ALBERT
BERT
DistilBERT
Learning Rate
5e-5
Batch Size
32
Weight Decay
0.0
Max. Epochs
Learning Rate Decay
Linear
Warmup Ratio
0.0
Table 5: Hyperparameters for finetuned models for dataset ChaosNL.IS and ChaosNLI-M
Hypothesis
Hypothesis
Jimmy grow very pace.
family offered Jinuny has tuttioa.
He takk out lunn for schoul.
So he reporid them for cullege.
-5,552562
neaded pick son up from soccer practice.
forgot son foe hours at football practice.
-3004293
got cougitt wattee
for is for loog time.
the Sizr Wars
reading science fiction stortes.
was gifted science test book.
material
Lost's toacher condase
Shee picked knife started dissecting frog.
Lon's disecttng foogs.
Lurt's clas didn't diesection secious.
desk instruction
-4.11444
Lay eajoyed his job evea though good
Larry came across pile of coal.
Lary was happy and excited.
Lary was poor coal miner.
4,209170
2
Table High ambiguous instances ChaosNLI-oe dataset
RoBERTa D100 L.D

---

1.0
0.5
contorecce
confidunce
0.6
variaclliyy
variability
variabilty
Varfability
hard
andiguosa
embiguou
anoigusa
0.8-
contidencce
controlence
confidunnce
0.6-
0.4
variability
variability
varisbilty
Variability
hard
herd
88-
confidence
confidence
86
0.4-
variability
variability
variability
varfabiltyy
1ard artiiuus
fart arbiy
haid
lor paru
contocncc
0.6-
0.4
0.2
variabllity
verisbility
variability
variebility
Auce
hard
hard
herd
confiflonce
0.6-
0.4
0.2-
0.2
0.1
0.2 varisbility
Varlability
variability
Figure 7: Datamaps across different annotator sets for RoBERTa model trained on ChaosNLI-S dataset in LD setting

---

A
artipooua easy
onti
0.5
eaay
easy
0.6
contidiacce
confideecc
confiisunce
variabilit
variability Ae
variabilty
variailitty Ag
Ag
arde eay
entayyac
0.8-
0.6-
contidence
confide deriec
varlabiliyy A,
varfability
variability Avo
varfabilityy Aa
ambig JAu
antrgguuu
eaty
easy
Can confidenice
Car confiiuucce
0.6-
0.4
Variabillty
veriability
variability
Variability
Aso
entiuuaa
confibern
0.6
confiderce
contidunce
0.4
variablliyy
variability
var variability Aspe
variabiliy
herd aenoguee
confidecee
0.6-
0.4-
0.0
0.1
0.2
0.3
0.0
0.1
0,2
0.3
0.4
0.0
0.1
0.2
0.3
Varisbblity
variabilit
variability
Figure 8: Datamaps across different annotator sets for RoBERTa model trained on ChaosNLI-M dataset in LD setting

---

A:
fv
urtiiuoua easy
eaay
sary
0.6
conntiionce
contidencc
confiidunce
0.4
variability
veriabilit Ae
viriabilty
variabllity A₂
orde
eary
ease
newy
0.6-
contidence
confidence
confiidaac nce
0.4
0.2
Vartability
vartability
variability Ave
varlability
1.0
antiiuuuu
amigg atut
exty
0.6
confidenie
confideecce
canf corfiduncc
0.4
0.2-
variability
variability
variabiiity
vartability
1.0-
urdé-uuua
emodguoo eaay
enoiuuaa
contideece
confidence
0.4
0.2
variability
varisbility
variabiliy
variabiliy
Aspe
1.0-
herd you
herd aenopuee
0.8-
confideecce
contiennce
0.4-
4644
0.2-
0.0
0.0
0.1
0.2 Varisbility
0.3
0.4
0.0
0.1
0.2
0.3
0.0
0.2
0.2
0.3
0.4
variability
variabillyy
Figure 9: Datamaps across different annotator sets for RoBERTa model trained on ChaosNLI-o dataset in LD setting

---

1.0
VOS
8.9
0.8
confidence
confidence
0.6
0.4
0.2
Varlabiity
variabiity
variabilty
variablity
1.0
anbiguos
uno g40.3
0.8
contiduncc
0.6
0.5-
0.4-
0.3
0.2
variabiity
varisbiiity
variabilty
varisbilty
1.0
0.8
confidance
0.6-
0.5
0.4
0.3
variablliyy
variablityy
variabiliyy
variability
1.0
16/S artigous
1ard artRipu
0.8
CoP sbancce
Confiounce
0.6
0.5
0.3
variabiliy
variabiity
varisbilty Auge
1.0
hard
hore p20.a
arbipuusa
0.9
0.7
CON Sanioo
confissance
0.5
0.3
varisbility
varabblityy
varisbiliy
Figure 10: Datamaps across different annotator sets for BERT model trained on ChaosNLI-a dataset in LD setting

---

A
onti eary
caay
0.6
confidiance
contidencc
confistunce
0.4
are
hara
eary
variability
variabiliy Ae
variabilit A
variability Aa
A
onde eayy
cewy
0.6-
contidence
confidence
confiide Bar
0.4
0.2
0.0
vartability A,
variabliity
variability Avo
varfability
1.0
andiguout
amig atut
aasy
eao
easy
0.6
cenfioenice
confiolence
Can councce
0.4
0.2-
variabiltty
variability
variability
vartability
1.0-
urdiguoua
emagu.o eaa
enogusa
contidercce
contideecc
contiiiio contidunce Bunce
0.4
0.2
variability
varisbility
varisbiliyy Aspe
variabiliyy
1.0-
herd pod
herd
0.8-
conffieerce
conmilence
0.4-
0.2
0.0
0.0
0.1
0.2 varisbility
0.0
0.1
0,2
0.3
0.0
0.1
0.2 varisbility
varisbilit
Figure 11: Datamaps across different annotator sets for DistilBERT model trained on ChaosNLI-a dataset in LD setting


Reviews:
[{"review_id": "NiPnugEOd8", "reviewer": "Reviewer_hZk1", "summary": "", "questions": "A. Line 055: If you're saying, for example, that 5 isn't enough, how many annotations do you think are necessary?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback and would like to address the concerns regarding the novelty of our contributions. Regarding the claim that our main contribution about the impact of increasing annotators has been previously addressed, as quoted\n\n>  One of their main contributions of showing how increasing the number of annotators doesn't always lead to better performance has already been published by another paper 10 years ago.\n\nand\n\n> Line 087: Hovy et al. 2013 (https://aclanthology.org/N13-1132/) already show that increasing annotators stops being beneficial after a certain number, so this second contribution can't be considered novel\n\nwe would like to address this first:\n\n\n1. In their 2013 study (MACE), Hovy et al. employed majority label (ML) and MACE as aggregation techniques to establish a singular ground truth. In contrast, our research integrates both the ML and label distribution (LD) approach. The LD method preserves each annotator's individual perspective, unlike ML or MACE. ML disregards minority perspectives, while MACE employs additional annotations to detect and exclude spurious labels. Therefore, our inclusion of LD in experiments aims to encompass all annotator viewpoints. **It is crucial to highlight that the MACE paper addresses a fundamentally distinct problem by assuming a single true label. The context and focus of our experiments markedly deviate from the scope of the MACE paper.** Thus, we refrain from making direct comparisons due to these dissimilarities.\n2. Furthermore, as we delve into the graphical depictions in the MACE paper (Fig. 6), a clear upward trend in the accuracy curve, becomes apparent. **It is essential to highlight that their analysis is limited to a range of up to 10 annotations. In contrast, our investigation extended to encompass annotation count sufficiently larger than 10, consistently revealing an increasing trend in the accuracy curve.**\n    > Hovy et al. 2013 (https://aclanthology.org/N13-1132/) already show that increasing annotators stops being beneficial after a certain number\n\n    **Nowhere in the MACE paper is there a statement or demonstration that increasing the number of annotators ceases to provide benefits after a certain threshold**. We kindly ask you to identify the specific section or passage in the MACE paper that supports this claim.\n    \n4. Referring to the quote below: \n    > A. Line 055: If you're saying, for example, that 5 isn't enough, how many annotations do you think are necessary?\n\n    The paper offers insights that address this question. Notably, Figure 3 plays a crucial role by illustrating the curve of $\\mathcal{V}$-Information values. A significant observation emerges from this depiction: higher $\\mathcal{V}$-Information values correspond to increased accuracy. As a result, the identification of the optimal number of annotators can be rooted in pinpointing the peak $\\mathcal{V}$-Information value. However, it's important to recognize that arriving at a clear conclusion requires a more thorough investigation.\n    \n5. For the quote below\n    > Line 092: What do you mean the number of annotations has a noticeable impact on the difficulty of instances? The difficulty of the instance should be inherent to the data itself and shouldn't be affected by who is annotating it.\n\n    > **\"The difficulty of the instance should be inherent to the data\"**\n    \n    Certainly, We will rephrase that part as: \"We also demonstrate, altering the number of annotations per instance has a noticeable impact on the difficulty of instances **as perceived by the model** and consequently affects the model performance.\" Thanks for highlighting this point\n    \n6. Typos Grammar Style And Presentation Improvements (A & C): Thanks for the suggestion, we will add these changes to the updated version of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper addresses an important issue in annotated datasets, where label breakdowns are not released or certain information relevant to the dataset is not made available.", "reasons_to_reject": "One of their main contributions of showing how increasing the number of annotators doesn't always lead to better performance can be inferred from another paper 10 years ago (further explanation in Missing References). Additionally, their explanation of the simulation process lacks details that emphasize the novelty of their method.", "questions_to_authors": "A. Line 055: If you're saying, for example, that 5 isn't enough, how many annotations do you think are necessary?", "paper_topic_and_main_contributions": "The authors propose a multi-annotator simulation process which aims to fill in missing gaps in annotated datasets and improve model performance.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "A. Line 087: Hovy et al. 2013 (https://aclanthology.org/N13-1132/) already provide enough information to infer that increasing annotators stops being beneficial after a certain number. This paper discusses how to detect spammers (although they say \"MACE does not discard any annotators, but weighs their contributions differently. We are thus not losing information.\") and that \"if most annotators\nanswer correctly, majority voting is trivially correct\". Therefore, if their model is able to identify the high-quality annotations and having a majority of correct labels makes majority voting trivial, then one can also claim that increasing annotators stops being beneficial after a certain number. In other words, the second contribution statement is too general to be considered novel enough.", "Typos_Grammar_Style_and_Presentation_Improvements": "A. The abstract should be a little more specific. Can you say a little more about the simulation process you're proposing? What range of annotation budgets are you looking at? What tasks are you training your models for? You want to make sure you give the reader enough information to decide whether they want to read your paper or not (otherwise those who would've been interested may choose not to).\n\nB. Line 092: What do you mean the number of annotations has a noticeable impact on the difficulty of instances? The difficulty of the instance should be inherent to the data itself and shouldn't be affected by who is annotating it.\n\nC. Figure 1 seems to add more confusion than to add clarity. Can you think of a better way to visualize the concept? There are also a lot of new symbols between 106 and 125, which makes it hard to read. Can you show the equations on a different line maybe?\n\nD. I'm not sure the approach makes sense. it just sounds like you're creating a dataset of random labels?\n\nE. Lines 192-193: I would rephrase or remove this. When you clarify that you're not aiming for SOTA performance, it sounds like your paper isn't going to share anything novel."}}, {"review_id": "qE6SvfFDuo", "reviewer": "Reviewer_KTZY", "summary": "", "questions": "\nWhile it is clear from the results presented in Table 2 that the impact of the increasing number of annotations on the model performance is not consistent across the datasets and models, I was wondering why you did not explicitly mention that at least for ChaosNLI-S and ChaosNLI-M datasets the general trend is in fact that model's performance benefits from more annotations up to a certain number (eg. having around 20 annotations is always better than 2-3)?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wholeheartedly thank you for the feedback and we would like to address the concerns raised.\n\n1. Referring to the quote below:\n    > I missed the discussion and/or potential references why the proposed simulation of multi-annotator scenario should be correct, it is neither well motivated nor explained.\n\n    The rationale behind this approach is to handle situations where annotator-specific labels are absent. Several datasets, such as (Nie et al., 2020; Jigsaw, 2018; Davidson et al., 2017), lack annotator-specific labels. By introducing the multi-annotator simulation process, we aimed to bridge this gap and enable researchers to understand individual annotator patterns and biases. We encourage the release of annotator-specific labels as this facilitates the model's ability to learn from these individual patterns during training. We will provide a more comprehensive discussion of this motivation and its implications in the updated version of the paper. Thank you for raising this point.\n\n2. \n    > the reported experiments focus on one specific NLP task and one dataset, which makes the title and the claims made in the paper somewhat misleading.\n\n    We agree, we will address this in the updated version of this paper.\n    \n3.  > While it is clear from the results presented in Table 2 that the impact of the increasing number of annotations on the model performance is not consistent across the datasets and models, **I was wondering why you did not explicitly mention that at least for ChaosNLI-S and ChaosNLI-M datasets the general trend is in fact that model's performance benefits from more annotations up to a certain number (eg. having around 20 annotations is always better than 2-3)?**\n\n    Your insight sheds light on an important aspect, and we appreciate your input. In our initial focus on comparing against the maximum annotation budget of 100, we might have inadvertently overlooked these specific trends. Your suggestion is highly valuable, and we will make sure to integrate this viewpoint in the updated version of the paper. Thank you for bringing this to our attention.\n    \n4. Typos Grammar Style And Presentation Improvements:\n    >it is not straightforward to understand what is the task behind the ChaosNLI dataset. Clearly, one can read the reference, but for the readability it would be better to adde1-2 sentences about this dataset \n\n    Thanks for the suggestion, we will address this in the updated version of the paper.\n    \n    >Figure 5 should be mentioned in the main text I believe, \n\n    Yes, we agree, due to page limit, we had to add this figure in the Appendix. We will address this in the revised version of the paper.\n    \n    > I would suggest to change the title of the paper in order to explicitly reflect the name of the NLP task you dealt with since otherwise the reader might be mislead and infer that the presented findings are of more generic nature (which is not the case) \n\n    Thanks for the suggestion, our approach and findings are applicable for classification tasks, we will add these changes in the updated version of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n- the paper is very clearly written and provides sufficient level of detail to understand the carried-out research\n\n- the work provides interesting new insights into the topic of creation of multi-annotator datasets highly advocated by the community\n\n- the outcome of the study have some practical implications in the context of decisions  related to the creation of corpora vis-a-vis budget envisaged for that purpose\n\n- the authors are well aware of the limitations of the presented results \n", "reasons_to_reject": "\n- I missed the discussion and/or potential references why the proposed simulation of multi-annotator scenario should be correct, it is neither well motivated nor explained\n\n- the reported experiments focus on one specific NLP task and one dataset, which makes the title and the claims made in the paper somewhat misleading", "questions_to_authors": "\nWhile it is clear from the results presented in Table 2 that the impact of the increasing number of annotations on the model performance is not consistent across the datasets and models, I was wondering why you did not explicitly mention that at least for ChaosNLI-S and ChaosNLI-M datasets the general trend is in fact that model's performance benefits from more annotations up to a certain number (eg. having around 20 annotations is always better than 2-3)?", "paper_topic_and_main_contributions": "\nThe reported research studies how does an increased number of per-instance annotations impacts the model performance for a specific (although not explicitly mentioned) NLP task and demonstrates that \"more\" does not always convert into better model performance being the common belief. The research also explores how the number of per-label annotations impacts the \"difficulty\" of instances and how it affects the model performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- it is not straightforward to understand what is the task behind the ChaosNLI dataset. Clearly, one can read the reference, but for the readability it would be better to adde1-2 sentences about this dataset\n\n- Figure 5 should be mentioned in the main text I believe\n\n- I would suggest to change the title of the paper in order to explicitly reflect the name of the NLP task you dealt with since otherwise\nthe reader might be mislead and infer that the presented findings are of more generic nature (which is not the case)"}}, {"review_id": "fKdSqBQ8Qg", "reviewer": "Reviewer_qogc", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wholeheartedly thank you for the invaluable feedback that you have given to this manuscript. We would like to address the concerns raised here.\n\n\n1. > I believe that this work did not answer perhaps the most pertinent question that the authors had posed. Given a fixed budget, a researcher can vary between K - the number of annotators per instance, or N - the number of instances annotated. While the authors propose increasing K up to some level for better performance, this work doesn't explore what happens to model performance when varying N under the same budget. The tradeoff between these two parameters is at the heart of many annotation design choices, and in my view, the current contribution of this work is of little use until this question is addressed. I also suspect that the results in this work are strongly influenced by the small training set in ChaosNLI (~1200 training instances) and that some of these would be invalidated once larger datasets are used (i.e. larger N with the same K). The authors could have easily accomplished that analysis using the original NLI datasets that set K to 5.\n\n    \n   We did not pursue this specific experiment as it has already been addressed by Zhang et al. (2021) (https://aclanthology.org/2021.emnlp-main.601/). whose work has significantly inspired our own research presented in this paper. In their study, they employ a learning objective that enables models to learn from unlabeled, single and multi-label instances. They systematically explore the tradeoff between N and K while maintaining a fixed annotation budget. One of their findings reveal that introducing multi-label examples at the cost of annotating fewer instances yields notable improvements in the NLI task performance. Notably, Zhang et al. (2021) also conducted their analysis on the same ChaosNLI dataset.\n    \n   Furthermore, to address concerns related to the dataset size, In our experiments, we establish a base model by finetuning Pretrained Language Models (PLMs) on substantial original training sets, such as SNLI, MNLI, and AbductiveNLI, encompassing over 100K instances (>>1.2K). We further finetune these base models on the ChaosNLI dataset, as elaborated in our paper. Therefore, we believe the concern expressed about the influence of the small training set in ChaosNLI has been effectively mitigated. \n    \n   In summary, while we acknowledge the reviewer's important point about varying N under a fixed budget, we believe that this question has been investigated by Zhang et al. (2021), and our work builds upon their insights. Additionally, our experimentation with a comprehensive base model address concerns about the limited training set in ChaosNLI affecting the validity of our findings.\n    \n2. \n    > Another aspect that I think was overlooked in this study is the overall quality of the dataset attained at different Ks. The authors should measure the accuracy between the K-dataset and the \"absolute\" ground truth collected from 100 workers. When designing a dataset annotation protocol, the correctness of the resulting data is possibly more important than the performance of the model trained on the data. While a model could achieve good performance at K=3 for example, maybe the K-dataset does not show high inter-annotator agreement. Therefore, maybe the attained quality of the dataset is low and not desirable. Perhaps the analysis in this study should also display model performance against the \"absolute\" ground truth (K=100) as well.\n\n    We have indeed conducted the suggested experiment, but the findings were not reported in the paper due to constraints on page limit. However, we are fully prepared to incorporate these results into the revised version of the paper.\n\n\n3.  > Another problem in this work is that the title and introductory sections mislead the reader to believe this is a general-purpose study of annotation protocols in NLP, while this work is demonstrated only on textual entailment. Moreover, this work perhaps is more suited for classification tasks, and reproducing similar results on other tasks that require span selection (QA, NER, etc.) or text generation would be much more challenging.\n\n    We acknowledge your observation. The title and introduction may indeed have conveyed a broader scope than our work's specific focus on textual entailment. We recognize that the applicability of our approach and findings primarily pertains to classification tasks. Your feedback is valuable, and we will ensure to address this aspect in the revised version of the paper. Thank you for highlighting this point.\n\n4.  > I would also suggest that the authors add a std. dev. to the accuracy results in some of the charts (e.g. chart 2). I think that the variability in results attained at lower K's would be much more significant compared to higher K's, and that should definitely be also a factor when a researcher designs the annotation protocol.\n\n    We appreciate your suggestion. We will incorporate the standard deviation (STDDEV) in the accuracy plots in the revised version of the paper. Thank you for bringing this to our attention.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work tackles an important problem in the collection of new annotated datasets - how to assess the necessary number of answers per prompt to get a useful dataset under a limited budget. \nThe authors use non-trivial and illuminating analysis methods, such as dataset cartography techniques and other different measures.\nThe experiments span multiple NLI datasets, several models, and other experimental design choices (e.g. single ground-truth label vs. a gold distribution).\nThe manuscript is well-organized and easy to follow, even though it deals with many complicated details, it appears that a lot of effort has been put into this work.", "reasons_to_reject": "I believe that this work did not answer perhaps the most pertinent question that the authors had posed.\nGiven a fixed budget, a researcher can vary between K - the number of annotators per instance, or N - the number of instances annotated.\nWhile the authors propose increasing K up to some level for better performance, this work doesn't explore what happens to model performance when varying N under the same budget. \nThe tradeoff between these two parameters is at the heart of many annotation design choices, and in my view, the current contribution of this work is of little use until this question is addressed.\nI also suspect that the results in this work are strongly influenced by the small training set in ChaosNLI (~1200 training instances) and that some of these would be invalidated once larger datasets are used (i.e. larger N with the same K). The authors could have easily accomplished that analysis using the original NLI datasets that set K to 5.\n\nAnother aspect that I think was overlooked in this study is the overall quality of the dataset attained at different Ks. \nThe authors should measure the accuracy between the K-dataset and the \"absolute\" ground truth collected from 100 workers.\nWhen designing a dataset annotation protocol, the correctness of the resulting data is possibly more important than the performance of the model trained on the data. \nWhile a model could achieve good performance at K=3 for example, maybe the K-dataset does not show high inter-annotator agreement. Therefore, maybe the attained quality of the dataset is low and not desirable. Perhaps the analysis in this study should also display model performance against the \"absolute\" ground truth (K=100) as well.\n\nAnother problem in this work is that the title and introductory sections mislead the reader to believe this is a general-purpose study of annotation protocols in NLP, while this work is demonstrated only on textual entailment. Moreover, this work perhaps is more suited for classification tasks, and reproducing similar results on other tasks that require span selection (QA, NER, etc.) or text generation would be much more challenging.\n\nI would also suggest that the authors add a std. dev. to the accuracy results in some of the charts (e.g. chart 2). \nI think that the variability in results attained at lower K's would be much more significant compared to higher K's, and that should definitely be also a factor when a researcher designs the annotation protocol.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work studies annotation design choices when collecting new datasets. Specifically, it explores the effect of using a varying number of annotations per instance on the resulting performance of models trained on the collected data. The study is conducted on the textual entailment task using previously collected datasets (ChaosNLI) that contain 100 annotations from different workers per instance.\nThe authors simulate datasets with K-annotations per instance by sampling K labels from each original instance without replacement. \nIn the experiments, K varies from 1 to 100, and the annotator set is assumed to be very large. \nThe results mostly indicate that model performance saturates quickly with the number of annotations per example.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper examines the impact of number of annotators per instance on model performance for text classification. R2 and R3 see the topic of the paper as important and that the results have practical implications. Additionally, R3 sees the experimental setup as thorough for the particular tasks, datasets, and models used. However R1 indicates that one of their primary results may have already been demonstrated before, limiting the degree to which the paper makes a contribution to the literature on this topic. The authors address this in their rebuttal – there are some similarities in the finding but the scale of experimentation and the nature of the datasets used in this work differ sufficiently. The scope may potentially be limited, as the paper only looks at NLI, and R3 additionally describes some potential issues with the utility and generality of the findings. The discussion helped to clarify some of the concerns of the reviewers; while the experiments may be specific to the one dataset tested (ChaosNLI) the paper does give insight into the impact of annotator volume for ambiguous labeling tasks.
Decision: Accept-Findings