Annotation Sensitivity: Training Data Collection Methods Affect Model Performance
Authors: Christoph Kern
Stephanie Eckman
Jacob Beck
Rob Chew
Bolei Ma
Frauke Kreuter
Venue: EMNLP 2023 Findings (2023)
Keywords: Annotation sensitivity
human annotation
annotation instrument
task structure effects
TL;DR: The design of the instrument used to collect training data impacts the performance and predictions of models.

Abstract: When training data are collected from human annotators, the design of the annotation instrument, the instructions given to annotators, the characteristics of the annotators, and their interactions can impact training data. This study demonstrates that design choices made when creating an annotation instrument also impact the models trained on the resulting annotations. 

We introduce the term annotation sensitivity to refer to the impact of annotation data collection methods on the annotations themselves and on downstream model performance and predictions.

We collect annotations of hate speech and offensive language in five experimental conditions of an annotation instrument, randomly assigning annotators to conditions. We then fine-tune BERT models on each of the five resulting datasets and evaluate model performance on a holdout portion of each condition. We find considerable differences between the conditions for 1) the share of hate speech/offensive language annotations, 2) model performance, 3) model predictions, and 4) model learning curves.

Our results emphasize the crucial role played by the annotation instrument which has received little attention in the machine learning literature. We call for additional research into how and why the instrument impacts the annotations to inform the development of best practices in instrument design.

Annotation Sensitivity: Training Data Collection Methods Affect Model Performance
Christoph Kern* Stephanie Eckman Jacob Beck
Rob Chew
Bolei Ma Frauke Kreuter
*LM Munich
University Maryland, College Park
RTI International
(christoph.kern,
jacob.beck, bolei.ma, steph@umd. edu
frauke. Traakk..kreuerr@lle kreuter)@lmu.de de rchewerti.org
Abstract
Swayamdipta et al.. 2020), the annotation scheme (Northeutt al., 2021), and annotator characteris- tics (Geva et al., 2019; Al Kuwatly et al., 2020) can contribute to bias and variability in the an- notations. How the prompts are worded, the ar- rangement of buttons, and other seemingly minor changes to the instrument can also impact the an- notations collected (Beck al., 2022).
When training data are collected from human annotators, the design of the annotation instru- ment, the instructions given to annotators, the characteristics of the annotators, and their in- teractions can impact training data. This study demonstrates that design choices made when creating annotation instrument also impact the models trained on the resulting annotations.
In this work, we investigate the impact the anno- tation collection instrument on downstream model performance. We introduce the term annotation sensitivity to refer to the impact of data collec- tion methods on the annotations themselves and on model performance and predictions. We conduct experiments in the context of annotation of hate speech and offensive language in tweet corpus.
We introduce the term annotation sensitivity refer to the impact annotation data col- lection methods the annotations themselves and on downstream model performance and predictions.
We collect annotations hate speech and of- fensive language in five experimental condi- tions of an annotation instrument, randomly as- signing annotators conditions. We then fine- tune BERT models on each of the five resulting datasets and evaluate model performance on holdout portion of each condition. We find con- siderable differences between the conditions for the share of hate speech/offensiv language annotations, 2) model performance, 3) model predictions, and 4) model learning curves.
Our results contribute the growing literature on Data Centric AI (Zha et al., 2023), which finds that larger improvements model performance often come from improving the quality the train- ing data rather than model tuning. The results will inform the development of best practices for anno- tation collection.
Our results emphasize the crucial role played by the annotation instrument which has re- ceived little attention in the machine learning literature. We call for additional research into how and why the instrument impacts the an- notations to inform the development of best practices in instrument design.
Background and Related Work
Annotation sensitivity effects are predicted by find- ings in several fields, such survey methodol- ogy and social psychology. Decades of research in the survey methods literature find that the wording and order of questions and response options can change the answers collected (Tourangeau et al., 2000; Schuman and Presser, 1996). For example, small changes in survey questions ("Do you think the government should forbid cigarette advertise- ments on television?" versus "Do you think the government should allow cigarette advertisements television?") impact the answers respondents give. Questions about opinions are more sensitive to these effects than questions about facts (Schnell and Kreuter, 2005).
Keywords: Annotation sensitivity, human anno- tation, annotation instrument, task structure effects
1 Introduction
Supervised NLP models typically trained on human annotated data and assume that these an- notations represent an objective "ground truth." Mislabeled data can greatly reduce a model's abil- ity to learn and generalize effectively (Frénay and Verleysen, 2013). Item difficulty (Lalor et al., 2018;
Psychologists and survey methodologists have

---

also documented the cognitive shortcuts that re- spondents take reduce the length and burden of survey and how these impact data quality. Respon- dents often satisfice, choosing an answer category that is good enough, rather than expending cog- nitive effort to evaluate all response options thor- oughly (Krosnick et al., 1996). When the questions allow, respondents will go fürther, giving incorrect answers to reduce the burden of a survey (Kreuter al., 2011; Tourangeau al., 2012; Eckman et al., 2014).
neither). The distribution of the sample across the strata is shown in Table A.I in Appendix $A.1. We do not treat the previous annotations as ground truth and do not use them in our analysis. There is no reason to believe they are more correct than the annotations we collected, and ground truth is difficult to define in hate speech annotation (Arhin et al., 2021).'
We developed five experimental conditions which varied the annotation task structure (Figure 1). All tweets were annotated in each condition. Condition A presented the tweet and three options on a single screen: hate speech (HS), offensive lan- guage (OL), or neither. Annotators could select one or both of HS and OL, or indicate that neither applied. Conditions B and C split the annotation of single tweet across two screens (screens are shown in Figure black boxes). For Condition B, the first screen prompted the annotator to indicate whether the tweet contained HS. On the following screen, they were shown the tweet again and asked whether it contained OL. Condition C was similar to Condition B, but flipped the order of HS and OL for each tweet. Annotators assigned to Condition D first annotated HS for all assigned tweets and then annotated OL for the same set of tweets. Con- dition E worked the same way but started with the OL annotation task followed by the HS annotation task.
Prior research finds strong evidence of task struc- ture effects in annotations of hate speech and of- fensive language. When annotators coded both annotations one screen, they found 41.3% of tweets to contain hate speech and 40.0% con- tain offensive language. When the task was split over two screens, results differed significantly. The order of the annotations matters as well. When hate speech was asked first, 46.6% of tweets were coded as hate speech and 35.4% as offensive lan- guage. The converse was true when the order was switched (39.3% hate speech, 33.8% offensive lan- guage) (Beck et al., 2022). Our work expands upon these findings test the hypothesis that task struc- ture affects not just the distribution of the annota- tions but also downstream model performance and predictions.
3 Methods
In each condition, annotators first read through a tutorial (Appendix $A.3) which provided defini- tions of HS and OL (adapted from Davidson et al. 2017) and showed examples of tweets containing HS, OL, both, and neither. Annotators could ac- cess these definitions at any time. Annotators were randomly assigned one of the experimental con- ditions, which remained fixed throughout the task. The annotation task concluded with the collection of demographic variables and some task-specific questions (e.g., perception of the annotation task, social media use). We assessed demographic bal- ance across experimental conditions and found no evidence of meaningful imbalance (see Figure A.4 in Appendix $A.4).
We tested our ideas about the downstream effects of annotation task structure in the context of hate speech and offensive language in tweets. We used a corpus of tweets previously annotated for these tasks (Davidson et al., 2017), but replaced the orig- inal annotations with new ones collected under five experimental conditions. We then trained separate models on the annotations from each condition to understand how the annotation task structure im- pacts model performance and predictions.
3.1 Data Collection
We sampled tweets from the Davidson et al. corpus, which contains 25,000
language tweets. We selected 3,000 tweets, each annotated three times in the prior study. Selection was stratified by the distribution of the previous annotations to ensure that the sample contained varying levels of annotator disagreement. We created nine strata reflecting the number of annotations (0, 1, 2, or 3) of each type (hate speech, offensive language,
Each annotator annotated up to tweets. Those in Condition saw up to 50 screens, and those other conditions saw up to 100. We collected three annotations per tweet and condition, resulting in 15 annotations per tweet and 44,900 (weet-annotation
'The original annotation task defined hate speech and of- fensive language as mutually exclusive categories, which we felt was too restrictive and difficult for annotators.

---

Training and Test Setup We used the collected annotations to build prediction models for two bi- nary outcomes: Predicting whether tweet con- tains offensive language (1 = yes, no), and predicting whether tweet contains hate speech (I = yes, 0 == no). We split our data on the tweet level into a training set (2,250 tweets) and a test set (750 tweets). Because we collected three an- notations per tweet in each experimental condi- tion, we had five condition-specife training data sets which each contain 6,750 (2, 250 3) tweet- annotations. We similarly created five condition- specific test sets with 2,250 (750 x 3) tweet- annotations each and a combined test set that con- tains tweet-annotations from all five conditions. Note that the five condition-specinee training sets contain the same tweets, annotated under different experimental conditions, as do the five condition- specific test sets.
#2
A
HS
HS
OL
#1
#2
#2
B
OL
We used the five training sets build condition- specific prediction models for both OL and HS. That is, we trained a model on annotations col- lected only in Condition and another model on annotations collected in Condition B, and so on. Models were evaluated against the five condition- specific test sets as well the combined test set. For both model training and testing, we used the three annotations collected for each tweet in each condition; that is, we did not aggregate annotations to the tweet level (Aroyo and Welty, 2015).
#1
#2
#2
C
HS
#2
#1
#2
D
HS
HS
OL
#1
#
#2
E
OL
OL
HS
HS
Model Types We used Bidirectional Encoder Representations from Transformers (BERT, Devlin et 2019), which widely used for text classi- fication and OL/HS detection tasks (e.g. Vidgen et al., 2020; Al Kuwatly et al., 2020). BERT is a pre-trained model which we fine- fine-tuned on the anno- tations from each condition. We also trained Long Short-Term Memory models (LSTM, Hochreiter and Schmidhuber 1997) the condition-specific training data sets. The LSTM results are reported in Appendix $A.4.
Figure Illustration of Experimental Conditions
combinations."
We recruited 917 annotators via the crowdsoure- ing platform Prolific during November and Decem- ber 2022. Only platform members in the US were cligible. Annotators received fixed hourly wage in excess of the US federal minimum wage after completing the task.
The full dataset including the annotations and the demographic information of the annotators is available at the following HuggingFace repos- h co/ datasets/ wtwpoittatitinsnsnstitiiteteeeet
Training and Validation The condition-specifie training datasets were further split into train (80%) and development (20%) set for model valida- tion. During training, after each training epoch, we conducted a model validation on the development set based on accuracy. If the validation showed a higher accuracy score than in the previous epoch, we saved the model checkpoint within this epoch, After full training epochs, we saved the respec- tive best model and used it for final deployment and evaluation on the five condition-specifi test sets. We repeated this training process 10 times with different random seeds and report average performance results to make our findings more ro- bust. We report further model training details Appendix $A.2.
itory:
3.2 Model Training
The code for data processing, model training and evaluation, and list of software packages and libraries used available at the fol-
Reproducibility
'Annotations by two annotators of 50 tweets each were corrupted and omitted from analysis.

---

lowing GitHub repository: com/ chkern/
across conditions. The left table gives the agree- ment rates for the modal OL annotations and the right table for the modal HS annotations. For OL, Condition disagrees most often with the other conditions. For HS, the agreement rates are smaller, and Condition E has the most disagreement with the other conditions.
4 Results
To understand the influence of the annotation task structure on the annotations themselves, we first compare the percent of tweets annotated as OL and HS and the agreement rates across conditions. We then evaluate several measures of model per- formance to study the impact of the task structure the models, including balanced accuracy. ROC- AUC, learning curves, and model predictions. Cor- responding results for the LSTM models given in Appendix $A.4. All statistical test account for the clustering of tweets within annotators.
Cond.
OL
IIS
B
0.653
0.596
C
0.646
0.731
0.545
0.536
D
0.629
0.695
0.707
0.559
0,579
0.539
0.655 0.740 0.740
0.724
0.477 0.505 0.484 0.510
C D
D
Table 2: Agreement between modal labels across anno- tation conditions (Krippendorf"'s alpha)
Collected Annotations Table 1 contains the fre- quency of OL and HS annotations by experimental condition. The number of annotations and anno- tators was nearly equivalent across the five con- ditions. More tweets were annotated as OL than HS across all conditions. Condition A, which dis- played both HS and OL on one screen and asked annotators to select all that apply, resulted in the lowest percentage of OL and HS annotations. Con- ditions B and C resulted in equal rates of OL an- notations. However, Condition C, which collected HS on the second screen (see Figure 1), yielded a lower share of HS annotations than Condition B (t = 5.93, < 0.01). The highest share of HS annotations was observed in Condition D, which asked annotators to first annotate all tweets as HS and then repeat the process for OL. Condition E, which flipped the task order (first requesting 50 OL annotations followed by 50 HS annotations), resulted in significantly more OL annotations than Condition 100.2,, < 0.01).
Prediction Performance Table 3 shows the bal- anced accuracy and ROC-AUC metrics for the fine- tuned BERT models when evaluated against the combined test (with annotations from all con- ditions). The models achieve higher performance when predicting OL compared to HS, highlighting that HS detection more difficult (Table 3).
Bal. Accuracy
ROC-AUC
Cond.
OL
HS
OL
HS
A
0.772
0.690
0.846
0.806
B
0.792
0.704
0.866
0.803
C
0.802
0.681
0.862
0.800
D
0.797
0.701
0.857
0.801
E
0.794
0.696
0.863
0.794
Table 3: Balanced Accuracy and ROC-AUC by anno- tation condition combined test set, averaged over BERT models
Figure shows the performance results for all combinations of training and testing conditions. The cells on the main diagonal show the balanced accuracy of BERT models fine-tuned on training data from one experimental condition and evalu- ated on test data from the same condition, averaged across the 10 model runs. The off-diagonal cells show the performance of BERT models fine-tuned on one condition and evaluated on test data from different condition. The left panel, in blue, tains results from the OL models; the right panel, in orange, from the HS models.
Number of
Percent OL HS
Cond.
Annotations
Annotators
A 9,000
184
51.6
26.8
B 9,000
183
58.8 29.6
C 8,950
182
58.5 28.2
D 8,950
179
54.4
33.5
E 9,000
189
59.0
31.8
Table 1: Annotation results by condition
Table 2 highlights disagreement across condi- tions. We calculated the modal annotation for each tweet in each condition from the three annotations, separately for HS and OL, and compared these
Performance differs across training and testing conditions, particularly for OL, as evident the row- and column- patterns in both panels

---

0.75
0.78
0.78
0.71
0.80
0.75
0.82
Train
0.76
0.8
0.81
0.79
0.82
0.72
0.89
0.68
0.68
0.89
(a) Predicting Offensive Language
(b) Predicting Hate Speech
Figure 2: Performance (balanced accuracy) of BERT models across annotation conditions
0.84
0.86
0.86
0.82
0.88
A-
0.83
0.81
0.8
0.8
0.79
0.84
0.88
0.88
0.85
0.99
B-
0.83
0.8
0.8
0.8
0.79
Tran
0.83
0.87
0.88
0.85
0.80
0.83
0.8
0.79
0.8
0.78
0.89
0.82
4
Test
Test
(a) Predicting Offensive Language
(b) Predicting Hate Speech
Figure 3: Performance (ROC-AUC) BERT models across annotation conditions
Figure 2. We do not observe higher performance scores in the main diagonal: models trained and tested data from the experimental condi- tion do not perform better than those trained and tested in different conditions.
Conditions B and C collected the HS and OL annotations separate screens, differing only in the order of the HS and OL screens for each tweet (Figure 1). Models trained on annotations collected in Condition C, where HS was annotated second for each tweet, have lower balanced accuracy across test sets when predicting HS (Figure 2b). However, there no corresponding effect for Condition B when training models to predict OL.
Models trained with data from Condition A re- sulted in the lowest performance across all test sets for the OL outcome (Table 3). Similarly, evaluating OL models on test data from Condition A leads to lower than average performance compared to other test conditions (Figure 2a). These results echo the high disagreement rates in OL annotations between Condition A and the other conditions (Table 2). In contrast, models tested on data from Condition show the highest performance when predicting HS (Figure 2b).
The OL models perform worse when evaluated test data collected in Condition D than data collected in Condition E (see the strong column effects in the last columns of Figure 2a). Again, no similar column effect is visible the HS panel (Figure 2b). potential explanation is that the second batch of 50 annotations was of lower quality

---

63 Aocuras
Bal Accuracy
2.70
2,90
0:35
0.00
0.90
20t Reledive Size af Training Data
50%
125%
0%
20h
50% Reletive Sie al Traiiing Date
79%
100%
Condition
Corditon
(a) Predicting Offensive Language
(b) Predicting Hate Speech
Figure 4: Learning curves of BERT models compared by annotation conditions
due to annotator fatigue.
whether differences between conditions impact training efficiency. Figure shows learning curves for the BERT models by training condition (line plots and corresponding scatterplot smoothers). These curves show how test set model performance changes as the training data size increases. For each training condition, 1 % batches of data were suc- cessively added for model training and the model was evaluated the (combined) test set. To limit computational burden, one model was trained for each training set size and training condition.
Evaluating models across conditions with re- spect to ROC-AUC largely confirms these patterns (Figure 3). The performance differences across testing conditions (column effects) are particularly striking in the ROC-AUC figures, while differences between training conditions (row effects) are less pronounced.
Predicted Scores Comparing model predictions across conditions contextualizes our findings. Ta- ble 4 shows agreement (Krippendorf's alpha) be- tween the predictions produced by models fine- tuned on different conditions. All predictions in Table 4 are for tweets in the test sets. OL predic- tions from the model trained on Condition A show more disagreement with the predictions from the other models (similar the agreement rates in Ta- ble 2 between modal labels). Agreement rates in predictions between models trained on Condition B to are lower for HS than for OL.
While the differences in the learning curves across conditions for OL are small (Figure 4a), Conditions A, C. and E stand out among the HS learning curves (Figure 4b). Building models with annotations collected in Condition E less effi- cient: initially, more training data are needed to achieve acceptable performance.
Discussion
Design choices made when creating annotation in- struments impact the models trained on the result- ing annotations. In the context of annotating hate speech and offensive language in tweets, ated five experimental conditions of the annotation instrument. These conditions affected the percent- age tweets annotated as hate speech or offensive language as well as model performance, learning curves, and predictions.
Cond.
OL
HS
B
0.679
0.778
0.754
0.869
0.822 0.777
D
0.727
0.869
0.901
0.839 0.811
0.751
cre-
E
0.682 0.878 0.861 0.872
0.788 0.789 0.760 0.797
B
B
Table 4: Agreement between BERT predictions across annotation conditions (Krippendorff's alpha)
Our results underscore the critical role of the annotation instrument the model development process. Models are sensitive to how training data are collected in ways not previously appreciated
Learning Curves We further study how the an- notation conditions impact model performance un- der a range of training set sizes, investigating

---

the literature.
one annotation rather than all that apply (Krosnick et al.. 1996; Pew Research Center, 2019). Another explanation distinction bias: asking the annotator to consider both classes simultaneously may lead the annotators to over-examine the distinctions be- tween offensive language and hate speech (Hsee and Zhang, 2004).
We support the calls for improved transparency and documentation in the collection of annotations (Paullada et al., 2021; Denton et al., 2021). If we assume a universal "ground truth" annotation ex- ists, task structure effects are cause of annotation noise (Frénay and Verleysen, 2013). Alternatively, if the annotations generated under different task structures are equally valid, task structure effects are a mechanism of researcher-inuuced concept drift (Gama et al., 2014). Determining a ground truth annotation is difficult and in some instances impossible. In such situations, the documentation of instrument design choices is particularly impor- tant. Lack of thorough documentation may explain why attempts to replicate the CIFAR-10 10 and Im- ageNet creation process encountered difficulties (Recht et al., 2019).
Annotators in Conditions and E annotated 50 tweets and then saw the same tweets again. We hypothesize that the relatively poor performance of these conditions is the result of a fatigue effect. Annotators may give lower-quality annotations in the second half of the task due to boredom and fa- tigue, which could partially explain the patterns we see in model performance (Figures 2, 3). However, we have not yet tested for order or fatigue effects.
Our results do not clearly demonstrate the su- periority of any of the five conditions we tested. More research is needed to identify the best ap- proach. In the meantime, we suggest incorporating variation in the task structure when annotation in- struments are created for human annotators, a point also raised by (Recht et al., 2019). Such variation can also help in creating diverse test sets that pro- tect against the test (column) effects we observe (Figures 2, 3). Relying on only one annotation col- lection condition leads to performance assessments that are subject to annotation sensitivity, that is, dis- torted by the design of the annotation instrument.
Large language models (LLMs) can assist with annotation collection, but we expect that human involvement in annotation will continue to be nec- essary for critical tasks, especially those related to discrimination fairness. However, such opinion questions are exactly the kinds of questions that are most susceptible to wording and order effects in sur- veys (Schnell and Kreuter, 2005). Whether LLMs are also vulnerable to task structure effects is not yet known. In addition, we expect that researchers will increasingly use LLMs for pre-annotation and ask annotators review the suggested annotations. This review is vulnerable to anchoring or confirma- tion bias (Eckman and Kreuter, 201 1), where the annotators rely too much on the pre-annotations rather than independently evaluating the task at hand. Task structure effects may impact anchoring bias as well.
This research has explored annota- tion sensitivity only in English-language tweets annotated by Prolific panel members living in the United States. The variations in annotations, model performance, and model predictions that see across conditions could differ in other countries and cultures, especially in the context of hate speech and offensive language. The similar effects in sur- veys, which motivated our work, do differ across cultures (Tellis and Chandrasekaran, 2010; Lee et al., 2020). This work has also not demonstrated that task structure effects appear in other tasks, such as image and video annotation. In retrospect, should have included sixth condition that versed the display of the label options in Condition A, which would have let us estimate order effects.
Limitations
We want to give special attention to Condition A, which presents all annotation tasks for given item on the same screen. We suspect this condition is the most commonly used in practice because it requires fewer clicks and thus less annotator effort. The rate of HS and OL annotations was lowest in Condition A. Models trained on data collected in Condition A had the lowest performance for offensive language and the highest performance for hate speech. There are several possible explanations for the differences in the annotations and in model performance which we cannot disentangle with our data. One explana- tion is an order effect: the hate speech button was shown first in a list of classes (response options). Respondents may have satisficed by choosing just
We encourage future work in this area to address these limitations.
Future Work Our results suggest that there are order and fatigue effects in annotation collection. Future work should conduct experiments to esti-

---

mate these effects. In surveys, measurement error is more common in later questions (Egleston et al., 2011). Future work could also assess fatigue effects by incorporating paradata (Kreuter, 2013), such as mouse movements (Horwitz et al., 2017) and re- sponse times (Galesic and Bosnjak, 2009). These behavioral indicators could contextualize how fa- tigue impacts task performance and data quality. In subsequent work, we will explore how the an- notators' characteristics influence their judgments and interact with the task structure effects. We also plan to diversify the tasks to include image evaluations and assignments that allow for a more accurate determination of ground truth. These hancements will provide a more comprehensive understanding of annotation sensitivity. Working with tasks that have (near) ground truth annotations would help the field make progress towards the development of best practices in annotation collec- tion.
Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. Al Magazine, 36(1): 15-224
Jacob Beck. Stephanie Eckman, Rob Chew, and Frauke Kreuter. 2022. Improving labeling through social sci- ence insights: Results and research agenda. In HCI International 2022 Late Breaking Pupers: Interact- ing with eXtended Reality and Artificial Intelligence, pages 245-261, Cham. Springer Nature Switzerland.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detec- tion and the problem of offensive language. Proceed- ings of the International AAAT Conference on Web and Social Media, 1 11())5122-55.
Emily Denton, Alex Hamna, Razvan Amironesei, An- drew Smart, and Hilary Nicole. 2021. On the genealogy of machine learning datasets: crit- ical history of ImageNet. Big Data & Society, 8(2):205395 7221 10359.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ethics Statement
In our work, we deal with hate speech and offen- sive language, which could potentially cause harm (directly indirectly) vulnerable social groups. We do not support the views expressed in these hateful posts, we merely venture to analyze this online phenomenon and to study the annotation sensitivity.
Stephanie Fckman and Frauke Kreuter. 2011. Confir- mation bias in housing unit listing. Public Opinion Quarterly, 75(1):339-15)
The collection of annotations and annotator char- acteristics was reviewed by the IRB of RTI Interna- tional. Annotators were paid wage in excess of the US federal minimum wage.
Stephanie Eckman, Frauke Kreuter, Antje Kirchner, An- nette Jäckle, Roger Tourangeau, and Stanley Presser. 2014. Assessing the mechanisms of misreporting to filter questions in surveys. Public Opinion Quarterly. 78(3)772
A Acknowledgments
Brian L. Egleston, Suzanne Miller, and Neal Meropol. 2011. The impact of misclassification due to survey response fatigue estimation and identi- fiability of treatment effects. Statistics in Medicine. 30(30):3560-33 -3572.
This research received funding support from RTI International and BERD@NFF NFDI.
References
Benoft Frénay and Michel Verleysen. 2013. Classifica- tion in the presence of label noise: survey. IEEE transactions on neural networks and learning sys- tems, 25(5):845-86)
Hala Al Kuwatly, Maximilian Wich, and Georg Groh.
2020. Identifying and measuring annotator bias based annotators' demographic characteristics. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 184-190, Online. Association for Computational Linguistics.
Mirta Galesic and Michael Bosnjak. 2009. Effects of Questionnaire Length on Participation and Indica- tors of Response Quality in Web Survey. Public Opinion Quarterly, 7392):349-300.
Kofi Arhin, Ioana Baldini, Dennis Wei. Karthikeyan Natesan Ramamurthy, and Moninder Singh. 2021. Ground-truth, whose truth? examin- ing the challenges with annotating toxic text datasets. CoRR, abs/21 12.03529.
João Gama, Indré Žliohaite, Alhert Bifet, Mykola Pech- enizkiy, and Abdelhamid Bouchachia. 2014. A sur- vey on concept drift adaptation. ACM computing surveys (CSUR), 46(4):1-37.

---

Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019, Are we modeling the task or the annotator? an inves- tigation of annotator bias in natural language under- standing datasets. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNL.P-JCCNLP) pages 1161-1166, Hong Kong. China. Association for Computational Linguistics.
Amandalynne Paullada. Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. 2021. Data and its (dis)contents: A survey of dataset development and use in machine learning research. Patterns, 2: 1001033..
Pew Research Center. 2019. When Online Survey Re- spondents Only Select Some That Apply.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019. Do imagenet classifiers generalize imagenet?
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 978):1735-
1780.
Rainer Schnell and Frauke Kreuter, 2005, Separating interviewer and sampling point effects. Journal of Official Statistics, 21(3):389-411
Rachel Horwitz Frauke Kreuter, and Frederick Conrad. 2017. Using mouse movements predict web survey response difficulty. Social Science Computer Review, 35(3):388-405.
Howard Schuman and Stanley Presser. 1996. Questions and answers in auiitde surveys: Experiments on question form, wording, and context. Sage.
Christopher K. Hsee and Jiao Zhang. 2004. Distinction bias: Misprediction and mischoice due to joint evalu- ation. Journal of Personality and Social Psychology, 86(5):680-695.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang. Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)) pages 9275-9293, Online, Association for Computa- tional Linguistics.
Frauke Kreuter, editor. 2013. Improving Surveys with Paradata, edition. John Wiley& & Sons, Ltd.
Frauke Kreuter, Susan McCulloch, Stanley Presser, and Roger Tourangeau. 2011. The Effects of Asking Fil- ter Questions in Interleafed versus Grouped Format. Sociological Methods and Research, 40(88):88- -804.
Gerard J. Tellis and Deepa Chandrasekaran, 2010. Ex- tent and impact of response biases in cross-national survey research. International Journal of Research in Marketing, 27(4):329-341
Jon A. Krosnick, Sowmya Narayan, and Wendy R. Smith. 1996. Satisficing in surveys: Initial evidence. New Directions for Evaluation, 2996(70):29-44.
John P. Lalor, Hao Wu, Tsendsuren Munkhdalai, and Hong Yu. 2018 Understanding deep learning perfor- mance through an examination test set difficulty: A psychometric case study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4711-4716, Brussels, Belgium. Association for Computational Linguistics.
Roger Tourangeau, Frauke Kreuter, and Stephanic Eck- man. 2012. Motivated underreporting in sereening interviews. Public Opinion Quarterly. 76(3):453- 469. Roger Tourangeau, Lance Rips, and Kenneth A. Rasin- ski. 2000. The psychology of survey response, 10. print edition. Cambridge University Press and Cam- bridge Univ. Press.
Sunghee Lee, Mengyao Hu, Mingnan Liu, and Jennifer Kelley. 2020. Quantitative evaluation of response scale translation through randomized experiment of interview language with bilingual english- and latino respondents. In The Essen- tial Role of Language in Survey Research. RTI Press.
Bertie Vidgen, Scott Hale. Ella Guest, Helen Mar- getts, David Broniatowski, Zeerak Waseem, Austin Botelho, Mathhw Hall, and Rebekah Tromble. 2020. Detecting East Asian prejudice on social media. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 172-172, Online. Association for Computational Linguistics.
Curtis Northcutt, Lu Jiang, and Isaac Chuang. 2021. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 10:1373-1411.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shieifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of the-ahrr art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang, Zachary Vetito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-peeformance deep learning library. In Advances in Neural Information Process- ing Systems 32, pages 8024-8033 Curran Associates, Inc.

---

Daochen Zha, Zaid Pervaiz Bhat, Kwei-Hemg Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu. 2023. Data-centric artificial intelligence: A survey, arXiv preprint arXiv:2303.100 10758.
Hyperparameter
Value
encoder
bert-base-cased
spochs_trained trained learning rale
20
1c
A Appendix
batch size optimizer
AdaniM
A.1 Selection of Tweets
We selected 3,000 tweets from 25,000, stratified by the previous annotations (Davidson et al., 2017). The nine strata and the sample size each is shown in Table A.I.
Table A.2: Hyperparameter settings of BERT models
Hyperpurameter
Value
epochs_traiee
learning_rate
5e
# of Annotations
batch_size
Sample Size
Percent
HS
OL
Neither
embedding dim
512
istm_hidden_ste size
512
3
417
14
lincar_laye__st la__er_yerszee
256
2
417
14
dropout_ree rate
0.3
vocab_size
000
417
14
bidirectional
True
0
0
417
14
loss
BCELosS (binary entropy)
0
2
160
5
optimizer
Adam
417
14
97
3
Table A.3: Hyperparameter settings of LSTM models
2
0
417
14
3
0
241
8
approximately 40 seconds per training data condi- tion, and the BERT model takes approximately 15 minutes.
Table A.1: Distribution of sampled tweets by stratum
We do not weight for the probability of selection in our analysis, because our goal is not to make inferences to the Davidson et al. corpus, which is itself not random sample of tweets.
A.3 Annotator Instructions
You will see a series of tweets. For each tweet, we want you to code if it contains hate speech or contains offensive language. The next few sereens explain what hate speech and offensive language are, help with the coding.
A.2 Model Training Details
Our implementation of BERT and LSTM models was based on the libraries pytorch (Paszke et al., 2019) and transformers (Wolf et al., 2020). During training, we used the same hyperparameter settings of the respective LSTM and BERT models for our different training conditions to keep these variables consistent for comparison purposes. We report the hyperparameter settings of the models in Table A.2 and A.3. The number parameters for pre-trained BERT (base) is ~ 110M, and the number of parameters for LSTM is 7 025 153. To avoid random effects on training, we trained each model variation with 10 different random seeds {10, 42, 84, 420, 567, 888, 1100, 1234, 5566, 7890} and took the average across the models.
We define hate speech as:
Language that is used to expresses hatred to- wards a targeted group or language intended to be derogatory, to humiliate, to insult the members of the group.
Here are examples of tweets that contain hate speech, according our definition:
"You jiggaboo...!"
"they 're working bill to prevent retards from voting, who knew retards COULD vote? things are starting make sense now."
"Every slant #LA should be deported. Those seum have no right be here. Chinatown should be bulldozed,"
All the experiments were conducted on an NVIDIA R A100 80 GB RAM GPU. Within this computation infrastructure, the LSTM model takes
Remember, hate speech is: language that is used to express hatred towards targeted group or lan-

---

guage intended to be derogatory, humiliate, to insult the members of the group.
However, context matters. If a tweet discusses someone else's hate speech, it may use hateful terms, but the tweet is not hate speech. For ex- ample,
"Why no boycott of the racist "redskins"? #Red- skins @ @ChangeTTheNamee
This tweet contains a hateful term, but the tweeter is making an anti-racist statement. We would label tweet as containing offensive language.
We define offensive language as:
Language that is highly offensive to certain in- dividuals groups but does not meet the require- ments of hate speech.
"Guess who just got apartment in downtown Columbus? That's right bitch you guessed it, this guy."
This tweet contains offensive language, but is not hate speech. We would label as offensive language.
Some tweets contains both hate speech AND offensive language.
"Subtweet me one more time, you dirty chink whore"
Some tweets do not meet our definition of hate speech or offensive language.
"Great lead battle and then Ricky hits Danica for yellow. Oh boy. #NASCAR"
A.4 Additional Results
We provide additional results of L.TM models in Table A.4 and Figure A.I, A.2, A.3, as well comparisons of demographic covariates across conditions in Figure A.4.
Bal. Accuracy
ROC-AUC
Cond.
OL
HS
OL
HS
A
0.846
0.806
0.747
0.637
B
0.866
0.803
0.755
0.654
C
0.862
0.800
0.759
0.625
D
0.857
0.801
0.734
0.655
0.863
0.794
0.754
0.638
Table A.4: Balanced Accuracy and ROC-AUC by anno- tation condition in combined test set, averaged over 10
LSTM models

---

Train
0.74
0.84
0.71
0.74
0.78
0.73
0.75
0.87
0.96
0.86
0.65
0.65
0.73
0.78
0.77
0.74
0.77
0.86
0.84
0.64
0.63
0.63
(a) Predicting Offensive Language
(b) Predicting Hate Speech
Figure A.I: Performance (balanced accuracy) of L.TTM models across conditions
A-
0.82
0.83
0.88
0.78
0.83
A-
0.74
0.73
0.73
0.73
0.72
0.52
0.83
0.89
0.04
0.75
0.73
0.79
0.73
Train
0.82
Test
Test
(a) Predicting Offensive Language
(b) Predicting Hate Speech
Figure A.2: Performance (ROC-AUC) of L.SM models across annotation conditions
0.70
Bal Acaurayy
Bal. Acauney
3.00
0.55
0.50
0.5
Training
Training Conditon
Training Cor dtton
(a) Predicting Offensive Language
(b) Predicting Hate Speech
Figure A.3: Learning curves of L.TMM models compared by annotation conditions

---

Can
Use
You
Condllan
Party
Colig grotake
Highuboo
Dore
Motc
Efricay
Addin
Figure A.4: Comparison of demographic covariates across conditions


Reviews:
[{"review_id": "85ysLwIfUL", "reviewer": "Reviewer_gqvR", "summary": "", "questions": "A. What was the reasoning for using LSTMs instead of something newer (since it can be predicted that BERT would perform better)?\n\nB. Will you be uploading your data along with the instructions you gave your crowdsourcing workers? That would definitely be another contribution.\n\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Review:**\n\nThank you for taking the time and effort to review our manuscript. We appreciate the feedback provided and will respond to the specific points below. \n\nThe focus of our paper is to put forward a clean experimental setup demonstrating the downstream effects of variation in the design of the annotation instrument. In this context, our focus is primarily on *relative* performance differences across both training and evaluation conditions, given common model types and performance metrics that have been used for the prediction task at hand (e.g. Vidgen et al., 2020; Al Kuwatly et al., 2020). While we will update the disagreement metrics (see below), we also welcome further extensions to our work, and thus make the data from our annotation experiment available for the community.\n\nWe agree that Krippendorff's alpha is a superior metric of agreement. We will replace the calculations in Table 2 with the Krippendorff's alphas. Our updated analysis shows that the alpha scores largely align with our initial assessment of disagreement patterns across annotation conditions (see Table 1 and 2 below). We will also replace Figure 3 with a table that more concisely summarizes disagreement between the model predictions using Krippendorff's alpha (see Table 3 and 4).\n\n**Response to Questions For The Authors:** \n\n- A. Thank you for pointing this out. We agree that BERT models are more commonly used. We included the LSTM models for robustness and in reference to prior work. In the revised paper, we will move all discussion of the LSTM model results to the appendix.\n\n- B. An important point. As noted above, we will publish the data and a concise documentation file with the manuscript. As a first step, we already updated the [OSF repository](https://osf.io/mn9ux/?view_only=75c84803b70947cb9831bd897cf8f01e) and included the main data files that we used for our analysis. We will also include the instructions given to the annotator in an appendix.\n\nTable 1: Agreement between modal labels for offensive language across annotation conditions (Krippendorff's alpha)\n \n|   | A | B  | C  | D  |\n|---|---|---|---|---|\n| B | 0.653 |   |   |   |\n| C | 0.646 | 0.731 |   |   |\n| D | 0.629 | 0.695 | 0.707 |   |\n| E | 0.655 | 0.74 | 0.74 | 0.724 |\n\nTable 2: Agreement between modal labels for hate speech across annotation conditions (Krippendorff's alpha)\n\n|   | A | B  | C  | D  |\n|---|---|---|---|---|\n| B | 0.596 |   |   |   |\n| C | 0.545 | 0.536 |   |   |\n| D | 0.559 | 0.579 | 0.539 |   |\n| E | 0.477 | 0.505 | 0.484 | 0.51 |\n\nTable 3: Agreement between BERT predictions for offensive language across annotation conditions (Krippendorff's alpha)\n \n|   | A | B  | C  | D  |\n|---|---|---|---|---|\n| B | 0.679 |   |   |   |\n| C | 0.754 | 0.869 |   |   |\n| D | 0.727 | 0.869 | 0.901 |   |\n| E | 0.682 | 0.878 | 0.861 | 0.872 |\n\nTable 4: Agreement between BERT predictions for hate speech across annotation conditions (Krippendorff's alpha)\n\n|   | A | B  | C  | D  |\n|---|---|---|---|---|\n| B | 0.778 |   |   |   |\n| C | 0.822 | 0.777 |   |   |\n| D | 0.839 | 0.811 | 0.751 |   |\n| E | 0.788 | 0.789 | 0.76 |  0.797 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The annotations variations caused by differences in MTurk HIT designs are of interest to every and any field that utilizes crowdsourced workers and there is a strong need for understanding how these variations occur.", "reasons_to_reject": "I'm not sure about the novelty and thoroughness of the project, given that the authors use BERT and LSTMs as their primary models and do not offer metrics more complicated than percentage or accuracy. For any annotation-related study, I'd like to see disagreement metrics, such as Krippendorff's alpha, but that wasn't provided in this paper.", "questions_to_authors": "A. What was the reasoning for using LSTMs instead of something newer (since it can be predicted that BERT would perform better)?\n\nB. Will you be uploading your data along with the instructions you gave your crowdsourcing workers? That would definitely be another contribution.\n\n", "paper_topic_and_main_contributions": "This paper dials in on how experiment design decisions, such as the layout of the questions and order of the options, can affect how crowdsourcing workers label instances. They collect 5 sets of annotations using different problem layouts and observe how they influence the performance of their BERT and LSTM models.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 3 doesn't offer much insight despite the amount of space it takes. I would put it in the appendix.\n\nChange the contents of Table 2 from percentage to some disagreement metric. Percentages aren't great for showing how two sets of annotations differ since they also include differences/similarities by chance (For example, two people annotating a binary dataset would most likely have more agreements than two people annotating a dataset with 10 labels)"}}, {"review_id": "RtTL0viWIz", "reviewer": "Reviewer_JnQu", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Review:** \n\nWe would like to sincerely thank the reviewer for the thoughtful review. Please see below our responses to your comments -- thank you again for the opportunity to improve the scholarship of this work.  \n\nWe agree that additional explanation of the experimental conditions will help readers  better interpret the results. We've drafted additional explanations of the experimental conditions for the Appendix. Additionally, we will publish more information of the annotation instrument, as well as the underlying data set, on OSF to help other researchers more easily reproduce our results or extend our work.\n\nWe agree that there are several potential mechanisms that are causing the differences we observe. However, in order to attribute the differences to any single (or combination) of these factors, we believe that a follow-up experiment would need to be designed and conducted to test for these specific hypotheses. For example, in the current experiment for Condition A, hate speech was presented as first option in a list of classes (alongside options for \"offensive language\" and \"neither\" on the same screen). To be able to disentangle the contributions of order effects vs. distinction bias, we would ideally also need a version of Condition A that presents \"offensive language\" as the first option to quantify the order effect. We will highlight the need for follow-up attribution studies in Future Work and note this in the Limitations section. \n\nWe recognize that our study does not identify a best approach for collection of hate speech and offensive language labels, which is a limitation of the current study. Our long-term goal is to develop actionable best practices for annotation collection. We hope that this early work motivates the broader research community to study this phenomenon by bringing attention to it. However, the revised paper will dedicate more space to discussing potential mechanisms and experimental designs to motivate future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper should be accepted, but as a short paper, because it addresses the crucial issue of annotation sensitivity in machine learning training datasets, which has received little attention in the literature.  The strengths of this paper are:\n1- The paper addresses an underexplored topic, annotation sensitivity, which has significant implications for the quality and reliability of machine learning training datasets. By investigating the impact of annotation instrument design on the resulting annotations and model performance, the paper contributes valuable insights to an area that has not received adequate attention in the literature.\n2- The paper demonstrates a well-designed experiment with five different annotation conditions, allowing for a comprehensive analysis of the effects of task structure on the collected annotations and the performance of the resulting models. This rigorous approach provides a strong foundation for the findings and their implications.\n3- The paper highlights the need for improved transparency, documentation, and best practices in the collection of annotations. These practical implications are of great importance to the NLP community, as they emphasize the need for more robust and reliable training datasets to improve the performance and generalizability of machine learning models.\n4- Although the paper focuses on the annotation of hate speech and offensive language in Twitter data, the findings have broader implications for other NLP tasks and domains. The insights gained from this study can inform the design of annotation instruments and data collection strategies in various areas of NLP research.\n5- The paper serves as a call to action for the NLP community to further investigate the impact of annotation sensitivity and develop best practices in instrument design. By highlighting the importance of this issue and its potential consequences for machine learning model performance, the paper encourages researchers to explore this topic further and contribute to the development of more reliable and effective NLP techniques.", "reasons_to_reject": "The paper under review presents an interesting topic regarding the impact of annotation task structure on collected annotations and the performance of machine learning models. However, it might be a good candidate for a short paper publication.\n1- The paper does not provide a clear and detailed explanation of the five experimental conditions used in the study. It is important for the authors to elaborate on the reasons behind their choice of conditions and how they differ from each other. This would help readers understand the implications of the findings and their relevance to the research question.\n2- The paper presents various findings regarding the impact of task structure on annotations and model performance, but it does not delve deep enough into the reasons behind these differences such as order effects, distinction bias, or fatigue effects. The authors should conduct a more thorough analysis of the results, exploring possible explanations for the observed trends. \n3- The paper fails to demonstrate the superiority of any of the five tested conditions in collecting annotations. Without clear evidence supporting one approach over others, it is difficult for readers to draw actionable conclusions from the study. The authors should consider refining their experimental design and analysis to provide more conclusive results.\nDespite these limitations, the paper could be a good short paper as it highlights the importance of annotation sensitivity and the crucial role of the annotation instrument in machine learning. It calls for additional research into how and why the collection instrument impacts the annotations collected, which could inform the development of best practices in instrument design. The paper also emphasizes the need for improved transparency and documentation in the collection of annotations and raises interesting questions for future research.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates annotation sensitivity in machine learning models by examining the influence of factors such as the design of annotation instruments, instructions given to annotators, characteristics of annotators, and their interactions on the quality and variability of human-annotated data. The authors focus on annotating hate speech and offensive language in Twitter data, using five experimental conditions of the annotation instrument, each varying in task structure, order, and presentation of annotation options. The study demonstrates that these experimental conditions affect not only the percentage of tweets annotated as hate speech or offensive language, but also model performance in terms of balanced accuracy, ROC-AUC metrics, learning curves, and predictions. The results emphasize the crucial role played by the annotation instrument, which has received little attention in machine learning literature. The authors recommend increased transparency and documentation in the collection of annotations, as well as further research into how and why the collection instrument impacts the annotations collected. This will help support the development of best practices in instrument design. Additionally, the paper suggests incorporating variation in the task structure when creating annotation instruments for human annotators to improve performance assessments and reduce test effects caused by annotation sensitivity.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vAn2IGz0RK", "reviewer": "Reviewer_W7XF", "summary": "", "questions": "Minor questions (addressing them in the paper will help fill these gaps):\n1) Did annotators have the option to abstain from annotating because it was a difficult decision?\n2) ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Review:** \n\nThank you for your careful reading of our paper. \n\nWe apologize for not mentioning the language of the data set. It is indeed English, which we will clarify in the revised paper. As mentioned in the response to Reviewer 1, we will upload the data set and the instructions when we revise the paper. \n\nYou are right that we manipulated only one aspect of the annotation instrument. There are of course many design choices that go into an annotation instrument, such as font, color, vertical or horizontal display of the annotation options, button size and type, etc. We chose to focus on the order of the screens and the annotation options to isolate their influence. In future work we will experiment with other aspects of the instrument.\n\nWe coined the term annotation sensitivity to draw attention to the fact that the annotations we collect are influenced by (sensitive to) many factors. The term is similar to ``gluten sensitivity'' and other terms. Because this is the first paper where we (or anyone) are using this term, we are open to alternatives.\n\nWe agree that our results do not address the mechanisms behind the differences that we observed. Because the influence of minor changes to the annotation instrument design on annotations and models are not well known in the literature, we chose to first demonstrate that these effects occur in this paper. The paper does suggest that a fatigue effect may be at work, but additional research is needed to test for that effect, which we are planning to do.\n\n**Response to Questions For The Authors:** \n\nEarlier research (Beck et al 2022) experimented with including and excluding an option to allow annotators to respond ``Don't Know.'' They found that the option was seldom used (maximum of 2.5\\% of responses) and did not change the distribution of the annotations. Because inclusion of this option is not standard in annotation instruments we have seen, we did not include it in this experiment. We will include this information in the revision of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The study demonstrating the five conditions is examined in the context of the resulting dataset and classifiers trained on the dataset. It makes an interesting read.\n\n2) The observations in the paper open up interesting ideas for future exploration:\n(a) Why is hate speech influenced by the annotation condition more than offensive language?\n(b) Obtaining multiple labels in one screen is indeed a common annotation strategy, as mentioned in the paper. What does this mean for annotation effort and costs associated with the creation of large annotated datasets?\n(c) Are the observations in the paper an artifact of the research problem (OL/HS detection) or does it extend to other tasks?\n\n3) I skimmed through the code included in the link in the paper. The code looks reasonably reproducible.", "reasons_to_reject": "These are minor reasons, in my opinion. However,\n(a) The claims in the paper seem to be exaggerated. The paper mentions 'annotation instrument'. However, the paper describes only one aspect of the annotation instrument.\n(b) Annotation sensitivity gives the impression that it is an empirical metric. However, it is a qualitative notion presented in the paper.\n(c) While the paper reports many interesting results (as described in the 'contribution' section above), it is a report of an empirical study - without sufficient investigation into the reason for the observation.", "questions_to_authors": "Minor questions (addressing them in the paper will help fill these gaps):\n1) Did annotators have the option to abstain from annotating because it was a difficult decision?\n2) ", "paper_topic_and_main_contributions": "The paper shows how order of annotation questions may affect hate speech (HS)/offensive language (OL) detection. The authors compare five conditions (the conditions are as follows: both labels together, OS followed by HL one tweet at a time, HL followed by OS one tweet at a time, OS of all tweets followed by HL of all tweets, HL of all tweets followed by OS of all tweets).\n\nThe analysis/evaluation shows that: (a) Proportions of OS and HL may differ, (b) there may be discrepancies in the HS labels based on the conditions, (c) performance of BERT/LSTM-based classifiers on the same annotation strategy is shown, (d) cross-condition evaluation results may be different, and (e) additional evaluation on prediction scores and training set sizes.\n\nThe paper is an interesting study of annotation order, in the context of hate speech detection.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please include the language of the dataset in the paper. It is neither obvious nor default that English is the language of the dataset."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates into "annotator sensitivity", i.e., how the wording of instructions affect the annotations collected.

All reviewers agreed this is an interesting and novel topic to look into, but also all raised some concerns on study design and evaluation. In specific:
1. The use of models are not up-to-date
2. The evaluation metric used seem a bit shallow (focusing on model accuracy but not other asspects like annotator-agreements)
3. The investigation focuses on only one type of intervention which makes the claim of "annotator sensitivity" a bit too broad.
4. Multiple artifacts may affect the result, noise of the annotations. 

Personally, I find the topic refreshing, and while the study is not perfect, it is sound enough and will spark interest and discussion among NLP researchers. As such, I recommend accepting to Findings track, but would recommend the author to significantly revise the paper as they suggest in the rebuttal.

I would also like to point out a relevant work on instruction bias:
Parmar, Mihir, et al. "Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions." EACL 2023
Decision: Accept-Findings