You Are What You Annotate: Towards Better Models through Annotator Representations
Authors: Naihao Deng
Xinliang Frederick Zhang
Siyang Liu
Winston Wu
Lu Wang
Rada Mihalcea
Venue: EMNLP 2023 Findings (2023)
Keywords: annotator disagreement
annotator representation
TL;DR: enhance model learning from disagreement using annotator representations.

Abstract: Annotator disagreement is ubiquitous in natural language processing (NLP) tasks. There are multiple reasons for such disagreements, including the subjectivity of the task, difficult cases, unclear guidelines, and so on. Rather than simply aggregating labels to obtain data annotations, we instead try to directly model the diverse perspectives of the annotators, and explicitly account for annotators' idiosyncrasies in the modeling process by creating representations for each annotator (*annotator embeddings*) and also their annotations (*annotation embeddings*). 
In addition, we propose **TID-8**, **T**he **I**nherent **D**isagreement - **8** dataset, a benchmark that consists of eight existing language understanding datasets that have inherent annotator disagreement.
We test our approach on TID-8 and show that our approach helps models learn significantly better from disagreements on six different datasets in TID-8 while increasing model size by fewer than 1\% parameters. 
By capturing the unique tendencies and subjectivity of individual annotators through embeddings, our representations prime AI models to be inclusive of diverse viewpoints.

You Are What You Annotate:
Towards Better Models through Annotator Representations
Naihao Deng"
Xinliang Frederick Zhang
Siyang Liu
Winston Wu
Lu Wang
Rada Mihalcea
University of Michigan - Ann Arbor, USA {dnaihao, mihalcea hemacccee@umich..cmco edu
University of Hawai'i at Hilo, USA
Abstract
Humor
Text A: Being crushed by large objects can be very de- pressing.
Annotator disagreement ubiquitous nat- ural language processing (NLP) tasks. There are multiple reasons for such disagreements. including the subjectivity of the task, dif- ficult cases, unclear guidelines, and so on. Rather than simply aggregating labels to ob- tain data annotations, we instead try to directly model the diverse perspectives of the annota- tors, and explicitly account for annotators' id- iosyncrasies in the modeling process by creat- ing representations each annotator (anno- tutor embeddings) and also their annotations (annotation embeddings). In addition, we pro- pose TID-8, The Inherent Disagreement dataset, benchmark that consists of eight exist- ing language understanding datasets that have inherent annotator disagreement. We test our approach on TID-8 and show that our approach helps models learn significantly better from dis- agreements on six different datasets TID-8 while increasing model size by fewer than 1% parameters. By capturing the unique tenden- cies and subjectivity of individual annotators through embeddings, our representations prime AI models to be inclusive of diverse viewpoints.
Text B: As you make your bed, you will sleep it. ANN WHICH FUNNTER. X KEANS a mE! A, A. B, X, X
Table I: An example where annotators disagree. Table 7 in Appendix B.I shows more examples.
words or phrases might be harmful to specific eth- nic groups (Kirk et al., 2022). For instance, terms that white annotators regard as innocuous might be offensive to black or Asian annotators due to cul- tural nuances and experiences that shape the subjec- tive perceptions of hate speech. Adjudication over the annotation of hate speech assumes that there is standard "correct" way people should feel to- wards these texts, which ignores under-represented groups whose opinions may agree with the majority. Similarly, in humor detection, different people can have varying levels of amusement to- wards the same text (Ford et al., 2016; Jiang et al., 2019), making it difficult reach consensus on such a subjective task. Another example is natural language inference (NLI), where it has been shown that there are inherent disagreements in people's s judgments that cannot be smoothed out by hiring more workers (Pavlick and Kwiatkowski. 2019), 9), Aggregating labels in NLI tasks can disregard the reasoning and perspective of certain individuals, undermining their intellectual contributions.
1 Introduction
Annotator disagreement common challenge in NLP (Leonardelli et al., 2021; Fornaciari et al., 2021). The conventional approach to reconciling such disagreements is to assume there a sin- gle ground-truth label, and aggregate annotator labels on the same data instance (Paun and Simp- son, 2021). However, disagreement among anno- tators can arise from various factors, including dif- ferences in interpretation, certain preferences (c.g. due annotators' upbringing or ideology), diffi- cult cases (e.g.. g.. due to uncertainty or ambiguity), or multiple plausible answers (Plank, 2022). It is prob- lematic to simply treat disagreements as noise and reconcile the disagreements by aggregating differ- ent labels into a single one. To illustrate, consider the case of hate speech detection, where certain
To account for these disagreements, one ap- proach is to directly learn from the data that has annotation disagreements (Uma et al., 2021), but representing this information inside the models is often not trivial. Instead, leverage the diverse viewpoints brought by different annotators, we cre- ate representations for the annotators (annotator embeddings) and for their annotations (amnotation embeddings), with learnable matrices associated with both of these embeddings (see Section 4). On downstream tasks, we forward the weighted embed-

---

dings together with the text embeddings to the clas- sification model, which adjusts its prediction for each annotator. Intuitively, by modeling each anno- tator with a unique embedding, we accommodate their idiosyncrasies. By modeling the annotations themselves, we capture annotators' tendencies and views for individual annotation items.
methods can be potentially grounded real- world demographic features.
TID-8 is publically available on Hugging- face at https: httss//hhggingface.comm cttefaacecoo/ao co/datasets///////////// dnaihao/TID- M 8. Our code and implementation are available at https: //gi thub. com/MicchianNLP Emotaato-EEmeddiings
To test our methods, we propose TID-8, The Inherent Disagreement 8 dataset, benchmark that consists of eight existing language understand- ing datasets that have inherent annotator disagree- ment. TID-8 covers the tasks of NLI, sentiment analysis, hate speech detection, and humorousness comparison. Empirical results on TID-8 show that annotator embeddings improve performance on tasks where individual differences such as the sense humor matter, while annotation embeddings give rise to clusters, suggesting that annotation embed- dings aggregate annotators with similar annotation behaviors.
2 Related Work
Inherent Annotator Disagreement. Annotator disagreement well-known issue NLP. A com- mon approach to deal with annotator disagreement to aggregate labels by taking the average (Pavlick and Callison-Burchh 2016) or the majority vote (Sabou et al., 2014), or select a subset of the data with high annotator agreement rate (Jiang and de Marneffe, 2019a.b).
Researchers have criticized the conventional ap- proach of assuming single ground truth and ig- noring the inherent annotator disagreement (Plank, 2022). Various studies reveal that there exists gen- uine human variation in labeling because of the subjectivity of the task or multiple plausible an- swers (Passonneau al., 2012; Nie et al., 2020; Min et al., 2020; Ferracane et al., 2021; Jiang and Marneffe, 2022). For instance, the task of toxic language detection, not all text is equally toxic for everyone (Waseem, 2016; Al Kuwatly et al., 2020). The identities and beliefs of the annotator influence their view toward the toxic text (Sap et al., 2022). Therefore, such annotator disagreement should not be simply dismissed as annotation "noise" (Pavlick and Kwiatkowski, 2019). Recently, researchers have started to leverage the different labels from annotators better personalize the model for vari- ous users (Plepi al., 2022).
Our approach helps models learn significantly better from disagreements six datasets in TID-8 and yields performance gain between 4%,~ 17% on four datasets that contain more than 50 anno- tators, while adding fewer than 1% of model pa- rameters. We also conduct an ablation study and a comparison of the two embeddings over different datasets. By building and analyzing embeddings specific to the viewpoints of different annotators, we highlight the importance of considering annota- tor and annotation preferences when constructing models on data with disagreement. We hope to con- tribute towards democratizing Al by allowing for the representation of a diverse range of perspectives and experiences.
In summary, our contributions include:
Rather than aggregating labels, we propose a setting of training models to directly learn from data that contains inherent disagreements.
Modeling Annotator Disagreement. Re- searchers have proposed various approaches for studying datasets with annotator disagreement. Zhang and de Marneffe (2021) propose Artificial Annotators to simulate the uncertainty in the annotation process. Zhou et al. (2022) apply additional distribution estimation methods such as Monte Carlo (MC) Dropout, Deep Ensemble, Re-Calibration, and Distribution Distillation capture human judgment distribution. Meissner al. (2021) train models directly on the estimated label distribution of the annotators in the NLI task. Zhang et al. (2021) consider annotator disagreement a more general setting with a mixture of single-label, multi-label, and unlabeled
We propose TID-8, The Iherent Disagreement 8 dataset, a benchmark that consists of eight ex- isting language understanding datasets that have inherent annotator disagreements.
We propose weighted annotator and annotation embeddings, which are model-agnostic and im- prove model performances on six out of the eight datasets in TID-8.
We conduct detailed analysis on the perfor- mance variations of our methods and how our
2

---

Tranformer
4 Methods
To explicitly account for annotation idiosyncrasies, we propose to create representations for both anno- tators and annotations. We create two embeddings, annotator embeddings (En), and annotation embed- dings (En). associated with two learnable matrices (Ca. On)) respectively as shown in Figure 1. For the annotator embeddings, we assign each annotator a unique embedding that represents their individual annotating preferences. For the annotation embed- dings, first assign embeddings to each label in the dataset. We then take the average embedding of the labels annotated by an annotator on other examples as their annotation embedding. The intu- ition that an annotator's labels on other examples can be viewed as proxy of their annotation ten- dencies when they annotate the current example. We describe the two embedding methods in detail below.
x
Jycusp
Paod
Prizd
Piod
Concept diagram how we apply our methods. We add the weighted annotator and amno- tation embeddings to the embedding of the "[CLS]" to- ken, and feed the updated embedding sequence the transformer-bas based models.
Figure I:
examples. Gordon et al. (2022) introduce jury learning to model every annotator in a dataset with Deep and Cross Network (DCN) (Wang et al., 2021). They combine the text and annotator ID together with the predicted annotator's reaction from DCN for classification. In contrast, we propose to explicitly embed annotator and their labels, and we perform detailed analysis of these embeddings. Davani al. (2022) employ common shared learned representation while having different layers for each annotator. Similar to our work, Kocoń et al. (2021) also develop trainable embeddings for annotators. In contrast, we propose embedding annotators as well as their labels with learnalle matrices associated with each. We test our methods on eight datasets sourced from various domains, while Kocoń et al. (2021) conduct their experiments on four datasets all sourced from Wikipedia.
4.1 Embeddings
Annotator Embedding (Fa) We define a learn- able matrix EA C RN to represent embeddings for all the annotators, where N is the total num- ber of annotators, and Il is the hidden size of the model. The annotator embedding for an individual annotator is E,E RI*H
Annotation Embedding (E,,) We define learn- able matrix E, RM*H to represent embeddings for all the labels, where M the number of pos- sible labels within the benchmark, and H the hidden size of the model. The embedding for an individual label / is Eg E R During training, for the example λ. annotated by annotator í, we calculate the annotation embedding E, by taking the average of the label embeddings Eg for all other examples annotated by the same annotator i:
3
Task Setup
En
Ex(3)
K,
For the dataset D, where 9 = (Iz, Vis a;) 2; represents the input text, Yi represents the corre- sponding label assigned by the annotator a, for text 2;. The dataset D consists of E examples anno- tated by N unique annotators. We aim to optimize the model parameters 0 to maximize the likelihood of the correct labels given the annotator a; and all
where K; is the set of examples in the training set annotated by the annotator i, the cardinality symbol |- yields the number of elements within that set, and Ei()) indicates the embedding for label I assigned to example k.
During testing, we average all the annotation embeddings of the training examples annotated by the same annotator:
of their input text æ;;
E
0- = arg max
log (( (1s a;; 0)
En
Ex(e)
K K
kEKis
3

---

4.2 Embedding Weights
5 TID-8 Overview
The annotator and annotation embeddings in- tegrated into transformer- based classification model. First, we calculate the sentence embed- ding of the input text Eg RlxH by averaging the text embedding, Et R x over the number of tokens, by Equation (1), where Et the sum of the word embedding, type embedding, and position embedding from the original BERT embeddings.
We propose TID-8: The Inherent Disagreement 8 dataset. TID-8 consists of eight publicly avail- able classification datasets with inherent annotator disagreement. In addition, information the asso- ciation between annotators and labels is available for all the datasets in TID-8. TID-8 covers the tasks of natural language inference (NLI), sentiment and emotion classification, hate speech detection, and humorousness comparison.
5 Eg 9
E
(1)
5.1 Desiderata Dataset
t=1
When selecting datasets for TID-8, a major concern is the quality of the annotations. Although there is significant number of annotator disagreements arising from differences in interpretation, certain preferences, difficult cases, or multiple plausible answers, annotation errors could still be the rea- son for disagreements (Plank, 2022). Furthermore, there no easy way to determine whether a label assigned by mistake or because of subjective reasons.
To incorporate embedings, given the sentence embedding E RIxH and the annotator embed- ding Ea E RI*H we calculate the weight for the annotator embedding Qa Rix1 using Equa- tion (2), where W, H and Wa R are learnable matrices.
Ca - (W.ET)
(2)
Similarly, for the sentence embedding Eg E RIxH and the annotation embedding E,, E RIXH we cal- culate the weight for the annotation embedding On E R using Equation (3), where Wn E R is another learnable matrix.
Fortunately, each dataset has its own quality con- trol mechanisms, such as including control exam- ples (De Marneffe et al., 2019), various data analy- ses (Demszky et al., 2020), etc. For instance, dur- ing the collection process of the CommitmentBank dataset, De Marneffe et al. (2019) constructed trol examples to assess annotators' attention, where the control examples clearly indicated certain la- bels. De Marneffe al. (2019) filtered data from annotators who gave other responses for the con- trol examples. Appendix B.4 contains details of the quality control for each dataset.
On "VWE)"?(WWEE)
We experiment with the following three methods for defining E, the combined embedding used by the classification model:
E,; Text embedding and weighted annotator em- bedding. (E(CLs) En F., Fe,y where E|CLs] is the embedding of the first token, [CLS], the encoded representation of which used for classification.
5.2 TID-8 Overview
TID-8 consists of eight datasets described in Ta- ble 2.
Annotation Distribution. Figure 2 shows the annotation distributions for the datasets in TID- 8. In Sentiment (SNT) dataset, each annotator la- bels similar number of examples. In Go Emo- tions (GOE), CommitmentBank (COM), Humor (HUM), and MultiDomain Agreement (MDA), small group creates most of the dataset examples. though more than two-thirds of the annotators an- notate more than 2,000 examples in Go Emotions (GOE). In Friends QIA (FTA), HS-Brexit (HSB), and Pejorative (PEJ) datasets, there are only a few annotators who cach annotates the entire dataset, except for one annotator in the Pejorative (PEJ)
Ea: Text embedding and weighted annotation em- bedding. E [E[CLs] + EaEa, Er.1, ... E,,y }.
E,+ E,: Text, weighted annotator, and weighted annotation embedding. E (Bicis) + an + agEa, E..1, }.
The embedding E then propagates through the layer norm and the dropout function in the same way as the standard embedding calculation in the transformer based model. The output embedding then propagates to the encoder.
4

---

100
100
(TA(
Pejentive (7E7)
MDA
H5 (158) (ESB)
GOE
20
F1A
20-
COM
Commimen bank (C3M))
HSB
SNT
0
250
1250
1500
Number Armotatoon
Number Annocaatoss
ofAsswas
Few Annotators.
(b) Many Annotators.
(c) all.
Figures 2a and 2b show the proportion of examples covered by the number of annotators (sorted by number of annotations). Specifically, Figure 2a shows the pattern for Few Annotators Datasets in TID-8 which contain 10 annotators, while Figure 2b shows the pattern for Many Annotators Dalasets in TID-8 which contain > 50 annotators. Figure shows the proportion examples with different numbers of labels on the eight datasets. The y-axis for all three plots is example coverage (%)
Figure 2:
FIA Friends QIA (Damgaard et al., 2021) which classifies indirect answers to polar questions.
Data
Train
Test
#A
#E/#A
#L
FIA
4.4k
0.6k
1,873
1.5k
0.7k
3
724
PEJ
Pejorative (Dinu et al., 2021) which clas- sifies whether Tweets contain pejorative words.
HSB
4.7k
1k
952
MDA
33k
11k
819
60
GOE
136k
58k
82
2,361
HSB
HS-Breeit (Akhtar et al., 2021), an abu- sive language detection dataset.
HUM
99k
42k
10599
133
COM
7.8k
3.7k
496
24
SNT
59k
1.4k
1481
41
MDA MultiDomain Agreement (Lconardelli et al., 2021), a hate speech detection dataset.
Table Statistics of datasets in TID-8. "#A" is the number of annotators. "#E/#A" is the average number of examples annotated per annotator, "#L" is the number of possible labels in the dataset. Datasets above the line are Few Annotators Datasets while datasets below are Many Annotators Datasets.
GOE Go Emotions (Demszky al., 2020), emotion classification dataset.
HUM Humor (Simpson et al., 2019) which compares humorousness between pair of texts.
COM CommitmentBank (De Marneffe et al., 2019), NI.I corpus. Sentiment Analysis (Díaz et al., 2018), sentiment classification dataset.
or more labels. This aligns with the findings by Pavlick and Kwiatkowski (2019) that there are in- herent disagreements in people's judgments in natu- ral language inference tasks, especially considering the meticulous data collection process described in Section 5.1 that ensures high-quality and reliable datasets. Appendix B.6 provides more details of the number of examples corresponding different numbers of answers.
SNT
Table 2: Overview of datasets in TID-8. Appendix B.2 provides additional details for these datasets.
dataset who only annotates six examples. There- fore, we refer to FIA, HSB, and PEJ as Few An- notators Datasets as they contain fewer than 10 annotators. For comparison, we refer to the other datasets as Many Anmotators Datasets, as all of them are annotated by more than 50 annotators. Appendix B.5 provides more details of the number of examples annotated for each annotator.
Experiment Setup
In this paper, we investigate the setting of training models to directly learn from data that has inherent annotator disagreements. Therefore, instead of ag- gregating the labels, we consider each annotation as a separate example. other words, different labels may exist for the same text annotated by different annotators.
Label Disagreement Figure 2c shows the label distributions among the datasets in TID-8. For most datasets, the majority of the examples have < 3 possible labels. For CommitmentBank (COM), a significant proportion the examples have four
5

---

Models. As our methods are model-agnostic, we our methods with various language understand- ing models, including base and large versions of BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and DeBERTa Version 3 (He et al., 2021). Adding annotator or annotation embeddings only increases fewer than 1% of the original parameters for each model. Appendix C.3 provides the details of the calculation.
En
E.
E., E.
75,06
64.81
76.70
75.72
75.76
75,67
65.88
76.13
74.67
74.97
75.65
68.36
76.02
75.14
75.28
MDA
76.24
69.73
77.18
75.99
75.68
76.45
70.43
77.78
76.93
77.26
76.38
73.02
77.22
74.75
77.19
63,04
60.88
68 ,449
69.98
69,90
62.90
62.12
68.39
69.92
66.32
63.22
60.49
67 /44
69.22
68.54
GOE
63.19
58.86
64,41
65.71
68,46
63.59
62,28
68. 68.58
69.70
69.60
Baseline Models. We include four baseline mod- els to compare against:
62.94
58.60
65.18
66.52
69.74
54.26
52.05
56.72
58.15
$3.89
54.11
51.07
56.67
58.19
54,35
Random: Randomly select label.
54.43
47.16
$5.07
56.31
53.31
HUM
54.40
52.55
54.26
51.97
50.02
Majority Vote iMVina): Always choose the most frequent label assigned by annotator.
54.71
53.63
56.33
57.70
53.31
54.67
54.81
57.18
58.76
51.86
Majority Vote (MV Always choose the overall most frequent label across annotators,
40.83
40.78
44.00
44.22
44.41
40.47
40.08
43.11
44.09
43.86
41.44
41.61
40.81
42.62
43.00
COM
40.66
40.32
40.42
40.34
39.75
T: Only feed the example text to the model.
40.54
38.14
42.37
42.82
42.59
Naive Concat (NC): Naively concatenate the annotator ID with the example text and feed the concatenated string to the transformer-based models.
40.57
33.82
44.00
44.33
44.15
47.09
39.20
62.88
60.23
64.61
47.32
36.91
61.8%
56.20
63.65
46.40
43.32
60.30
45.57
59.65
SNT
47,88
43.82
58.19
46.50
55.16
45.75
43.62
61.21
52.57
60.83
Evaluation Metrics. We report exact match ac- curacy (EM accuracy) and macro Fl scores on annotator-specifie labels.
48.76
43.78
67.37
68.39
69.77
Table 4: EM accuracy scores for annotation split Many Annotator Datasets, averaged across 10 runs. The best results (statistically significant from baselines, test, 0.05) are shown in bold. For each dataset, the six rows (from top to bottom) correspond to the from BERT base, BERT large, RoBERTa base, RoBERTa large. DeBERTa V3 base, and DeBERTa V3 large. Table 9 in Appendix C.4 shows complete table including the remaining three baseline models and performances on Few Annotators Datasets.
Dataset Split. Table shows the statistics for each dataset in TID-8. We split the data annotated by each annotator into a train and test set (and dev set if the original dataset contains one), where the train and test set have the same set of annotators ("annotation split"). For Friends QIA, HS-Brexit, MultiDomain Agreement, and Sentiment Analy- sis datasets, we follow the split from the original dataset. For the rest, we split the data into 70% train set and 30% test set. Appendix B.3 provides conduct.
improve performance across different models and model sizes, which shows the effectiveness of anno- tator and annotation embeddings in helping models learn from erowd data that has disagreements. Ap- pendix C.4 provides macro Fl scores.
pre-processing Appendix C.1 provides more details of the perimental set-ups.
7 Results and Discussion
Despite increased performance, annotator and annotation embeddings have different effects. In Table 4, we that MultiDomain Agree- ment (MDA), Go Emotions (GOE), and Humor (HUM), adding either annotation annotator em- bedding yields the best performance across models. In contrast, on CommitmentBank (COM) and Sen- timent Analysis (SNT), adding both embeddings yields the best or second-to-ben performance. Al- though on these two datasets, annotator or annota-
Both proposed embeddings help them better learn from disagreement. Table 4 shows that an- notation and annotator embeddings improve accu- racy scores on Many Annotators Datasets in TID-8 up 17% compared to the question-only baselines. In addition, naively concatenating the annotator ID with the question text harms the model perfor- mance, suggesting the need for sophisticated meth- ods. Furthermore, we see our methods consistently
6

---

tion embedding might perform the best for some models, adding both embeddings yields similar score. Furthermore, on Humor (HUM), adding both embeddings yields a worse performance than the baseline, while on Sentiment Analysis (SNT), adding both embeddings for BERT models yields the best performance. This suggests that annotator and annotation embedding have different effects. Intuitively, annotator embeddings capture individ- ual differences. From Figure 5, we observe clusters emerge from annotation embedding. Therefore, we hypothesize that annotation embeddings align with group tendencies. We further discuss the effects of these embeddings in Section 8.
Text: We know it ancedotally from readers we' ve heard from who'v 've heen blatantly discriminated against he- cause they re older.
POSITIVE (2) <>> NEGATIVE (-2)
Annotator ID
1
2
3
4
Gold
-1
0
-2
-2
T F, E.g
-1
-1
-1
-1 0 -1 -2
Table 5: An example predicted by the BERT base model from Sentiment Analysis, where adding both annotator and annotation embeddings better accommodates anno- tators' preference.
Text Oag
70-
Scaling models up does not help models learn from disagreement. On the Many Annotators Datasets in TID-8, observe little to no perfor- mance difference between the base and large ver- sions of each model. In some cases, the large mod- els even underperform the base model, as seen with BERT on the Go Emotions (GOE) dataset. This result suggests that the increased capacity of the larger models does not necessarily translate into improved performance when dealing with datasets that exhibit significant annotation disagreement. However, when there is minimal disagreement, as observed in the Friends QIA (FIA) dataset where only four examples have multiple labels while the remaining 5.6k examples have single label (as shown in Table 8), the larger models consistently outperform the base versions for the text-only base- line (as shown in Table 9 in Appendix C.4). This trend could be attributed to the larger models' higher capacity and increased number of param- eters.
50-
30
10-
MDA
GOE
HUM
CoM
SNT
Figure Ablation of performance on annotation split in the test when using annotator and annotation embed- dings without text (Embedding Only), text embeddings (Text Only), or the combination (Combination). We use the BERT base model on Many Annotators Datasets in TID-88
ability to learn from data that exhibits disagree- ments. However, the performance of these models varies when incorporate annotation or annota- tor embeddings. This indicates that different pre- training strategies might have an impact on the effectiveness of incorporating annotator or annota- tion embeddings into a given model.
The superior performance of the larger models in -ow-disagreement scenarios suggests that they excel at capturing and leveraging subtle patterns present in the data. However, when faced with datasets that contain significant disagreement, the larger models may become more prone to overfit- ting. Their increased capacity and specificity the training data might hinder their ability to general- ize well to new or unseen examples, resulting in diminished performance.
Apart from these analyses, we discuss perfor- mance patterns on each dataset in Appendix C.5.
Further Analyses
Since observe similar pattern across different transformer-basee models, we use the BERT basc model for ablation and discussion in this section.
Our methods give annotator- predictions. Often, the baseline text-only model cannot accom- modate different annotators, as shown in Table 5. However, after we incorporate the annotator or annotation embedding, the model can adjust its prediction to better align with the annotation for different annotators.
Models' ability to learn from disagreements is similar for text-only baselines but varies with different embeddings. For Many Annotators Datasets in TID-8, we observe similar perfor- mances across models for text-only baselines. This suggests that these models possess a comparable
7

---

(a) Annotation Embedding
(b) Annotator Embedding
Annotation Embedding
(b) Annotator Embedding
Figure 4: TSNE plots for MultiDomain Agreement. The embeddings are learned with BERT base model. We try with various hyperparameters and all of the plots demonstrate similar patterns.
Figure 5: Annotation and annotator embedding Sen- timent Analysis. The embeddings learned with the BERT base model. Different colors in Figure 5a indi- cate different "groups" in Table 17 in Appendix
Text and annotator annotation embeddings used jointly yield the best performance. To re- veal the performance contribution of different com- ponents, we train a BERT base model with text and both embeddings (En+ Ea) and test with text, embeddings, or their combination separately. Fig- ure shows the test-time performance using both embeddings (Embedding Only), just the text embeddings (Text Only), and using combination of both (Combination). We can see that the +,+ E, and text embeddings need to work coopera- tively to yield the best performance. In addition, we investigate the effects of the weight associated with the annotation annotation embeddings in Appendix C.6.
CA: current living area
GU: grew up areu
AG: age
GE gender
Pl: political identification
BD: education
AH: annual houschold income
Figure 6: The prevalent demographic features for each cluster/group in Figure 5a in Sentiment Analysis (SNT), Appendix D provides details of these demographic fea- tures. Figure 10 provides the spread-out plots for each group.
the domain of the dataset. MultiDomain Agree- ment contains English tweets about Black Lives Matter, Elections, and Covid-19 (Appendix B.2). Regarding these topics, each annotator has their own political beliefs and attitudes towards these topics, and their annotation is good reflection of their beliefs and attitudes. Therefore, the annota- tion embeddings may reflect the collective tenden- cies of individuals sharing similar political beliefs and attitudes.
Annotator embeddings capture individual dif- ferences. On Go Emotions (GOE) and Humor (HUM), adding annotator embeddings yields the best performance (Table 4). We hypothesize this is because both emotion and humor are subjective feelings, and annotator embeddings capture individ- ual differences. As revealed by psychological stud- ies, emotion and humor are entangled with one's cognition, motivation, adaptation, and physiologi- cal activity (Lazarus, 1991; Rowe, 1997; Tellegen et al., 1988; Cherkas et al., 2000; Martin and Ford, 2018). Having a dedicated embedding for each annotator (annotator embedding) might better cap- ture the individual annotator differences in tasks dealing with emotion and humor.
Figure visualizes both annotator and annota- tion embeddings on Sentiment Analysis. We notice that clusters emerge in annotation embeddings, in- dicating there are collective labeling preferences among the annotators on this dataset.
Prevalent demographic features vary among clusters in annotation embeddings. To gain in- sights into the relationship between annotation em- beddings and demographic features, we perform clustering analysis the annotation embed- dings of Sentiment Analysis shown in Figure 5a. employ the K-means algorithm (Lloyd, 1982) and empirically choose K= clus- ters based on visual observations from Figure 5a. Next, we map the identified clusters back to the corresponding demographic features provided by the dataset, as illustrated in Figure 6. The mapping
Annotation embeddings align with group ten- dencies. The visualization of the embeddings on MultiDomain Agreement in Figure 4a reveals a spectrum in terms of annotation embeddings, where each point within the spectrum represents an individual. The spectrum encompasses individ- uals positioned at opposite ends, as well as others dispersed throughout. This could be explained by
Specifically,
8

---

T
NC
E,
E,
E,, + Ea
consisting of eight language understanding tasks that have inherent annotator disagreement. We in- troduced method for explicitly accounting for an- notator idiosyncrasies through the incorporation of annotation and annotator embeddings. Our results on TID-8 show that integrating these embeddings helps the models learn significantly better from data with disagreements, and better accommodates individual differences. Furthermore, our approach provides insights into differences in annotator per- spectives and has implications for promoting more inclusive and diverse perspectives in NLP models. We hope that TID-8 and our approach will inspire further research in this area and contribute the development of more effective and inclusive NLP methods.
MDA
74.91
67.26
73.55
73.90
74.24
GOE
62.86
61.01
61.33
61.98
61.96
HUM
54.33
52.24
53.15
53.53
53.51
COM
40.78
40.49
40.80
40.30
40.28
SNT
43.93
38.56
36.99
40.82
37.90
Table 6: EM accuracy the BERT base model on annotator split for Many Annotators Datasets, averaged across 10 runs. The best results are in bold if they yield statisticallyy significant difference from the baselines (t-test, - 0.05).
is performed using the methods described in Ap- pendix D. We find the prevalent demographic fea- tures associated with each cluster vary significantly. This analysis demonstrates that the learned anno- tation embeddings can be grounded in actual de- mographic features, allowing us to identify distinct patterns and tendencies within different clusters. By examining the relationships between annotation embeddings and demographic characteristics, we can gain a deeper understanding of the underlying dynamics at play in the dataset.
A Acknowledgement
We thank the anonymous reviewers for their feed- back.
thank Zhenjie Sun, Yinghui He, and Yufan Wu for their help on the data processing part of this project. We also thank members of the Lan- guage and Information Technologies (LIT) Lab at the University of Michigan for their constructive feedback. This project was partially funded by an award from the Templeton Foundation (#62256).
Performance decrease is minimal unknown annotators. The annotation and annotator em- beddings do not substantially degrade performance when testing on new annotators. We test the em- beddings on the setting where the annotators in the train and test set are distinct (" ("annotator split"). We include 70% of the annotators the train and 30% for the test in TID-8, and test with the BERT base model. Table 6 shows the EM accuracy scores for this annotator split. On most of the Many An- notators Datasetsin TID-8, such as Go Emotions (GOE), MultiDomain Agreement (MDA), Humor (HUM), and CommitmentBank (COM), the per- formance loss is minimal none. For Sentiment Analysis (SNT), the annotation embedding suffers lot, which shows the difficulty of learning the group tendencies for unknown annotators. In ad- dition, because sentiment and emotion are highly personalized feelings, annotator embeddings this case suffer less than annotation embedding, as the annotator embeddings better handle individual dif- ferences. We further discuss performance on un- known annotators in Appendix C.4.
Limitations
One limitation of our work is the limited explo- ration of the demographic effects annotations. This is because Sentiment Analysis is the only Many Annotator Dataset in TID-8 that provides publicly available demographie features for the notators. We encourage researchers collect the demographic features of the annotators when build- ing datasets in the future while ensuring robust privacy protection.
Our methods suffer performance loss on unseen annotators, although on four of the five Many An- notators Datasets in TID-8, the performance loss is minimal to none. We stress that this is not the main focus of this paper, and we conduct this study to provide more comprehensive understanding of the embeddings. Future studies might enhance methods to deal with annotator disagreement for "unseen" annotators.
Due to the scope of this project, we only stud- ied annotator disugreement on classification tasks. However, annotator disagreement also exists in other NLP tasks such as summarization, or tasks beyond NLP such as image classification. We leave these topics to future research.
9 Conclusion
Instead of aggregating labels, we introduce set- ting where we train models to directly learn from datasets that have inherent disagreements. We pro- pose TID-8, The Inherent Disagreement- - dataset,
9

---

Ethics Statement
proaches to Discousse, pages 1-11, Punta Cana, Do- minican Republic and Online. Association for Com- putational Linguistics.
In this paper, we do not collect any new datasets. Rather, we propose TID-8 based on eight publicly available datasets. All of the eight datasets in TID- 8 provide the information of which annotator an- notates the corresponding examples. In addition, HS-Brexit provides whether the annotator is Mus- lim immigrant or not. Sentiment Analysis pro- vides more comprehensive demographic features for each annotator. All annotator is anonymous and their personal information is not revealed.
Aida Mostafazadeh Davani, Mark Díaz, and Vinodku- mar Prabhakaran. 2022. Dealing with disagreements: Looking heyond the majority vote in subjective an- notations. Transactions of the Association for Com- putational Linguistics, 10.92-110.
Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error- rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistios), 28(1):20-25
Our methods require the information on which annotator annotates the corresponding examples, which we believe will accommodate annotators' preferences while protecting their personal infor- mation.
Carhe-CCteeriee De Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmentbank: Inves- tigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107-124.
Furthermore, though can observe that differ- ent prevalent demographic features vary for clusters in annotation embeddings, our methods do not rely on demographic features. This protects the privacy of the annotators.
Dorottya Demszky, Dana Movshovitz-Attia, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine- fine-grained emo- tions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040 4054, Online. Association for Computational Linguisties.
Our methods help various models accommodate predictions based on annotators' preferences. We make steps towards leveraging different perspec- tives from annotators to enhance models.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186. Minncapolis, Minnesota. Association for Computational Linguistics.
References
Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021. Whose opinions matter? perspective-aware mod- els identify opinions of hate speech victims in abusive language detection. arXiv preprint arXiv:2106. 15896.
Mark Díaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle. 2018. Addressing age- related bias sentiment analysis. In Proceedings of the 2018 chi conference on human factors in comput- ing systems, pages 1-14.
Hala Al Kuwatly, Maximilian Wich, and Georg Groh. 2020. Identifying and measuring annotator bias based on annotators' demographic characteristics. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 184-190, Online. Association for Computational Linguistics.
Liviu P. Dinu, loan-Bogdan Tordache, Ana Sabina Uban, and Marcos Zampieri. 2021. A computational ex- ploration of pejorative language in social media. In Findings of the Association for Computational Lin- guisties: EMNLP 2021, pages 3493-3496 Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Laura Biester. Vanita Sharma, Ashkan Kazemi, Nai- hao Deng. Steven Wilson, and Rada Mihalcea. 2022. Analyzing the effects of annotator gender across nlp tasks. In Proceedings of the 1st Workshop on Per- spectivist Approaches to NL.@@ 1REC2022, pages 10-19.
Elisa Ferracane, Greg Durrett, Junyi Jessy Li, and Ka- trin Erk. 2021. Did they answer? subjective acts and intents in conversational discourse. Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1626-1644, Online. Association for Computational Linguistics.
Lynn Cherkas, Fran Hochberg, Alex J MacGregor, Harold Snieder, and Tim D Spector. 2000. Happy families: A twin study of humour. Twin Research and Human Genetics, 3(1):17-22.
Cathrine Damgaard, Paulina Toborek, Trine Eriksen, and Barbara Plank. 2021. "I'll be there for you": The one with understanding indirect answers. Pro- ceedings of the 2nd Workshop on Computational Ap-
Thomas E. Ford, Shaun K Lappi, and Christopher J Holden. 2016. Personality, humor styles and hap- piness: Happy people have positive humor styles. Europe's 's journal of psychology, 12(3):320.
10

---

Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Bar- bara Plank, Dirk Hovy, and Massimo Poesio. 2021. Beyond black& & white: Leveraging annotator dis- agreement via soft-label multi-task learning. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2591-2597, Online. Association for Computational Linguistics.
Hannah Kirk, Abeba Birhane. Bertie Vidgen, and Leon Derezynski. 2022. Handling and presenting harmful text in NLP research. In Findings of the Associa- rion for Computational Linguistics: EMNLP 2022, pages 497-510, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Jan Kocoñ, Marein Gruza, Julita Bielaniewicz. Damian Grimling. Kamil Kanclerz. Piotr Milkowski, and Przemyslaw Kazienko. 2021. Learning personal hu- man biases and representations for subjective tasks in natural language processing. In 2021 IEEE Inter- national Conference Data Mining (ICDM), pages 1168-117 1173, IEEE.
Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an inves- tigation of annotator bias in natural language under- standing datasets. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural I Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1161-1166, Hong Kong, China, Association for Computational Linguistics.
Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo. Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021. Designing toxic content classification for diversity of perspectives. In Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021), pages 299-318.
Mitchell L Gordon, Michelle S Lam, Joon Sung Park. Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022. Jury learning: Integrat- ing dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Hu- man Factors in Computing Systems, pages 1-19.
Richard Lazarus. 1991. Emotion and adaptation. Ox- ford University Press.
Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021. Agrecing to disagree: Annotating offensive language datasets with annotators' dis- agreement. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Process- ing, pages 10528-10539, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3; Improving deberta using electra-style pre- training with gradient-disen disentangled embedding shar- ing. arXiv preprint arXiv:2111 (tt::2/11.0995. 11.0943..
Miles Hewstone, Mark Rubin, and Hazel Willis. 2002. Intergroup bias. Annual review of psychology,
53(1):575-66) 604.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.
Michael A Hogg. 2016. Social identity theory. Springer.
Nan-Jiang Jiang and
de Marneffe. 2022. Investigating reasons for disagreement in natu- ral language inference. Transactions of the Associa- tion for Computational Linguistics, 10: 1357-144.
Stuart Lloyd. 1982. Least squares quantization in pem. IEFF transactions on information theory, 28(2):129- 137.
Rod A Martin and Thomas Ford. 2018. The psychology of humor: An integrative approach. Academic press.
Nanjiang Jiang and Marie-Catherine de Marneffe. 2019a. Do you know that florence is packed with vis- itors? evaluating state-of-ther art models of speaker commitment. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 4208-422 4213, Florence, Italy, Associa- tion for Computational Linguistics.
Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276.282.
Johannes Mario Meissner, Napat Thumwanit, Saku Sug- awara, and Akiko Aizawa, 2021. Embracing ambi- guity: Shifting the training target of II.I models. Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing ( (Volume 2: Short Papers). pages 862-869 Online. Association for Computational Linguistics.
Nanjiang Jiang and Marie-Catherine de Marneffe. 2019b. Evaluating BERT for natural language in- ference: A case study on the CommitmentBank. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing DMMNLP-IJCNLPP pages 6086 6091, Hong Kong, China. Association for Computa- tional Linguistics.
Sewon Min, Julian Michael, Hannameh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering am- biguous open- domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 5783- 5797, Online. Association for Computational Lin- guistics.
Tonglin Jiang, Hao Li, and Yubo Hou. 2019. Cultural differences humor perception, usage, and implica- tions. Frontiers in psychology, 10:123.

---

Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on nat- ural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131-9143, Online. Association for Computational Linguistics.
Maarten Sap. Swabha Swayamdipta, Laura Vianna. Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5884-5906, Seattle, United States. Association for Computational Linguistics.
Rehecca J Passonneau, Vikas Bhardwaj, Ansaf Salleb- Aouissi, and Nancy Ide. 2012. Multiplicity and word sense: evaluating and learning from multiply labeled word sense annotations. Language Resources and Evaluation, 46:219-252.
Edwin Simpson, Crik-Lån Do Dinh, Tristan Miller, and Iryna Gurevych. 2019. Predicting humorousness and metaphor novelty with Gaussian process preference learning. In Proceedings of the 57th Anmual Meet- ing of the Association for Compulational Linguistics, pages 5716-5728, Florence, Italy. Association for Computational Linguistics.
Silviu Paun and Edwin Simpson. 2021. Aggregating and learning from multiple annotators. In Proceed- ings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Tu- torial Abstracts, pages 6-9, online. Association for Computational Linguistics.
Henri Tajfel and John C Turner. 2004. The social iden- tity theory of intergroup behavior. In Political psy- chology, pages 276-293. Psychology Press.
Ellie Pavlick and Chris Callison-Buree 2016. Most "babies" are "little and most "problems" are "huge": Compositional entailment in adjective-nouns In Pro- ceedings of the 54th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 2164-2173, Berlin, Germany, Associ- ation for Computational Linguistics.
Auke Tellegen, David Lykken. Thomas J Bouchard, Kimerly J Wilcox. Nancy L Segal, and Stephen Rich, 1988. Personality similarity in twins reared apart and together. Journal of personality and social psychol- ogy, 54(6):1031
Alexandra N Uma. Tommaso Fornaciari, Dirk Hovy, Sil- viu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 12:1385-1470
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transac- tions of the Association for Computational Linguis- fics, 7:677-695 694.
Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. 2021. Den v2: Improved deep &amp; cross network and practical lessons for web-scale learning to rank sys- tems. In Proceedings of the Web Conference 2021, WWW 221, page 1785-1797, New York, NY, USA. Association for Computing Machinery.
Barbara Plank. 2022. The "problem" of human label variation: On ground truth in data, modeling and evaluation. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 10671-10682. Abu Dhabi, United Arab Emirates, Association for Computational Linguistics.
Joan Plepi. Béla Neuendorf, Lucie Flek, and Charles Welch. 2022. Unifying data perspectivism and per- sonalization: An application to social norms. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 7391- 7402, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Zeerak Waseem. 2016. Are you a racist or am I seeing things? annotator influence on hate speech detection Twitter. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 138- 142, Austin, Texas. Association for Computational Linguistics.
Shujian Zhang, Chengyue Gong, and Eunsol Choi. 2021. Learning with different amounts of annotation: From zero to many labels. Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing. pages 7620-7632, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Morris Rosenberg. 2017. The self-concept: Social prod- uct and social force. In Social psychology, pages 593 624. Routledge.
David C Rowe. 1997. Genetics, temperament, and per- sonality. In Handbook of personality psychology, pages 367-386. Elsevier.
Xinliang Frederick Zhang and Marie-Catherine de Marneffe. 2021. Identifying inherent disagree- ment in natural language inference. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 4908 4915, Online, Association for Computational Linguistics.
Marta Sabou, Kalina Bontcheva, Leon Derczynski, and Arno Scharl. 2014. Corpus annotation through crowdsoureing: Towards best practice guidelines. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' 14), pages 866, Reykjavik, Iceland. European Lan- guage Resources Association (ELRA).
12

---

Xiang Zhou, Yixin Nie, and Mohit Bansal. 2022. Dis- tributed NLE: Learning to predict human opinion dis- tributions for language reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 972-987, Dublin, Ireland. Association for Computational Linguistics.
A Author Contributions
Naihao Deng led the project, wrote the codebase, conducted all the experiments, and drafted the en- tire manuscript. Rada Mihalcea, Lu Wang, Siyang Liu, Xinliang Frederick Zhang provided sugges- tions, and belped with the brainstorming and paper writing refining. Siyang Liu provided her experi- ence with experiment design, including improving baseline setting and significance testing, helped fine the writing Scction 5 and and contributed significantly during the rebuttal period. Rada Mi- halcea came up with the core idea of modeling an- notators through their annotations, which was later refined in team discussions. Winston Wu provided feedback and suggestions for the project, refined writing, and contributed significantly during the rebuttal period.
B More about Selected Datasets in TID-8
B.1 Dataset Examples
Table 7 shows examples in TID-8 where annotators disagree with each other.
B.2 Selected Datasets
We provide the description of the eight datasets we selected for TID-8:
FIA Friends QIA (Damgaard al., 2021) is
corpus of classifying indirect answers to polar ques- tions.
PEJ Pejorative (Dinu et al., 2021) classifies whether Tweets contain words that are used pejora- tively. By definition, pejorative words are words or phrases that have negative connotations or that intended to disparage or belittle.
HSB HS-Brexit (Akhtar et al., 2021) is an abu- sive language detection corpus on Brexit belonging to two distinct groups: target group of three Mus- lim immigrants in the UK, and control group of three other individuals.
MDA MultiDomain Agreement (Leonardelli et al., 2021) 1) a hate speech classification dataset English tweets from three domains of Black Lives Matter, Election, and Covid-19, with a particular focus on tweets that potentially lead to disagree- ment.
GOE
Go Emotions (Demszky et al., 2020) is a fine-grained emotion classification corpus of care- fully curated comments extracted from Reddit. We
13

---

Friends QIA
Humor
Question: Did Rachel tell you we hired a male nanny?
Text A: Being crushed by large objects can very de- pressing.
Answer: I think that's great!
ANN ANSWER (1), Not THE AKsWER (2), ANEWER SURJECT To Sowe.
Text B: As make your bed, so you will sleep on it.
CONDITIONS (3), Narree (4), OTHER (3): 1, 1,4
ANN wHICH is FUNNIER. X MEASS A TI: A, A, B, X,
Pejorative
CommitmentBank
Text: WORSTRAPPLYICSS Everything Jay-Z writes
Meg realized she' been a complete fool, She could have said differently. If she' 'd said Carolyn borrowed a book Clare and wanted to return they have given her the address.
Premise:
trash.
ANN (U) <-> Now PErourive (o): 0,
HS-Brexit
Hypothesis: Carolyn had borrowed book from Clarc.
Text: RT <user>: Islam has no place in Europe #Brexit.
ANN ENTALL (3) SCoNCRADRY 3, 3. -3,
ANN No HAre (1) <-> Hare 10 1, I, 0, 0,
Sentiment Analysis
MultiDomain Agreement
Text: Even hotel bar food is good in Califormia.. avo- cados, old chicken, and reasonably recent greens. Mmmm. Really.
Text: Please lost you yelling insanely at the sky on Nov 3
losers
ANN (1) <-> Not Orrensive (0): 1, 1, 0,
ANN Posruive SNeaAnv (-2): 2, 2, 0,
Go Emotions
Text: This is how feel when use crosswalk busy
street
ANN PosmivE (1), Neviial (0), AmBuuaus (-1), (12:: 1,
Table 7: Examples in TID-8 8 where annotators disagree with each other.
group emotions into four categories following sen- timent level divides in the original paper.
et al. (2020) to group the sentiments into positive, negative, ambiguous, and neutral.
HUM Humor (Simpson al., 2019) is a corpus of online texts for pairwise humor comparison.
B.4
Quality Control from the Selected Datasets
COM CommitmentBank (De Marneffe et al., 2019) is an NLI dataset. It contains naturally oc- curring discourses whose final sentence contains clause-embedding predicate under entailment canceling operator (question, modal, negation, an- tecedent of conditional). We note that, unlike the standard NLI setting where "Entailment", "Neu- tral", and "Contradiction" are used, we use the ratings from -3 to that reflect the different de- grees of "Entailment" and "Contradiction" and to represent "Neutral" particular.
FIA For Friends QIA, Damgaard al. (2021) provide the raw agreement distribution of the an- notations, which include the agreement for each of the final three categories. addition, there is a high inter-annotator agreement rate as shown Table 8
PEJ For Pejorative, Dinu et al. (2021) recruited specialists in linguistics for the annotation. The Cohen's k agreement score was 0.933.
HSB For HiS-rrxxit, Akhtar et al. (2021) briefed and trained the annotators that the annotators have similar understanding of the abusive cate- gories. For the purpose of their rescarch, Akhtar et al. (2021) selected three volunteers who were first or eccond-generatie immigrants and students from developing countries to Europe and the UK. of Muslim background; and three other volunteers who were researchers with western background and experience in the linguistic annotation.
SNT Sentiment Analysis (Díaz al., 2018) is sentiment classification dataset originally used to detect age-related sentiments.
B.3 More Details about Dataset Pre-Processing
For annotation split MultiDomain Agreement (MDA), we filtered 4746 examples (around 12% of the total examples) the annotators of those do not appear the training time for the annotation split. We use the entire dataset in the annotator split.
MDA For MultiDomain Agreement, Leonardelli et al. (2021) selected a pool of tweets from the three domains of interest and asked three expert linguists to annotate them to ensure high-quality
For Go Emotions (GOE), we follow Demszky
14

---

annotations. The tweets with the perfect agreement are used as the gold standard. Leonardelli al.
Data
2
5
6
COM
25
154
310
367
231
113
(2021) then included a gold standard tweet in ev- ery HIT (group of tweets to be annotated). If crowd worker fails to evaluate the gold tweet, the HIT is discarded. Moreover, after the task com- pletion, Leonardelli et al. (2021) removed all the annotations done by workers who did not reach a minimum overall accuracy of 70% with respect to the gold standard.
SNT
892
6.7k
5.7k
809
10
GOE
24k
25k
7.0k
941
HUM
2.4k
14k
11k
MDA
4.9k
5.8k
HSB
774
345
PFJ
856
76
FIA
5.6k
4
Table Label disagreements in TID-8.
GOE For Go Emotions, Demszky et al. (2020) used a checkbox for raters to indicate if labeling particular example was difficult, in which case raters could select "no emotions". Examples where emotion was selected were removed from the dataset. In addition, Demszky et al. (2020) con- ducted various data analyses such interrater cor- relation, correlation among emotions, principal pre- served component analysis, etc. These various data analyses demonstrate the quality of the data, as the results are coherent and reasonable.
ity control, we contacted the authors directly for the information.
B.5 Annotation Distribution
Figure 7 shows the number of examples annotated by each annotator among the eight datasets. In Fig- ure 7g, each annotator annotates similar amount of examples. In Figures 7d to 7f and 7h, a small group creates most of the dataset examples similar to the pattern spotted by Geva al. (2019), though more than 2/3 of the annotators annotate more than 2,000 examples in Figure 7h. Figures 7a to 7 there are just few annotators and each annotates the entire datasets, except for one in Figure 7b who only annotates six examples.
HUM For Humor, Simpson et al. (2019) com- puted the mean inter-annotator agreement (Krip- pendorff's a)) across instances. The result, 0.80, indicates a decent level of agreement among the annotators.
B.6 Label Distribution
COM For CommitmentBank, De Marneffe et al. (2019) collected annotations through question- naires that included eight discourses of interest and two constructed control discourses. These con- trol discourses were used to evaluate the annota- tors' attentiveness, with one clearly indicating the speaker's certainty of the truth of the content of the complement (CC) and the other indicating certainty of the negation of the CC. Responses of +2 +3 were accepted for the "true" control items, and re- sponses of -3 or -2 were accepted for the "false" ones. Any data from annotators who gave other responses to at least one control item was excluded from the analysis.
Table shows the number of examples correspond- ing to the number of unique labels of single ex- ample TID-8. Friends QTA a dataset with little to no disagreement, as only examples have two labels while the remaining 5.6k examples have a single label. Although there are examples Senti- ment Analysis and CommitmentBank where there is high disagreement, because of the rigorous qual- ity control protocol described in Appendix B.4, we attribute them as hard examples or ambiguous examples that naturally lead to disagreement. We include all these examples in our modeling process.
C More about Experiments
SNT For Sentiment Analysis, Díaz et al. (2018)
adopted the quality controls provided by )ualtrics an annotation platform. Díaz et al. (2018) checked on completion time and whether there is straight- lining. Specifically, the completion time check involved discarding any responses that were com- pleted more quickly than half of the median speed. Note that the original paper did not talk about qual-
C.1 Experiment Set-Ups
We adopt a learning rate of 1e-5 for all of our ex- periments. We set epochs for TID-8. In terms of batch size, we find that a larger batch size helps stabilize the model performance. Therefore, we set the batch size from 8, 16. 32, 64, 128, and 256 based on the capacity the GPUs. All of our experiments are run on the A40 GPU.
15

---

5000
1003
100
930
1000
300
403
1000
231
200
(a) Friends QIA (FIA)
(b) Pejorative (PEJ)
(c) HS-Brexit (HSB)
200
000
400-
1000
533
190
200
300
203
990
200
400
500
86c
1000
20c
40c
200
(d) Commitmenthank (COM)
(c) Humor (HUM)
(f) Momain Agreement (MDA)
10000
1600
(g) Sentiment (SNT)
(h) Go Emotions (GOE)
Figure The number of examples annotated by each annotator in TID-8. We categorize top three as Annotators Datasets, and the bottom five Many Annotators Datasets.
C.2
Baseline Models
for the weighted annotation embedding, where M the number of unique labels in the dataset.
Dawid and Skene (1979) introduced probabilistic approach that allows to reduce the influence of unreliable annotators during predictions. Yet, the quality assurance methods in the TID-8 datasets ensure the labels are genuine and not spam. Instead of diminishing consolidating labels, our interest lies in understanding how models can be trained on data with inherent disagreements. Consequently, we choose not to incorporate the models presented by Dawid and Skene (1979).
In TID-8, Sentiment Analysis has the most an- notators of 1481, and BommitmentBank has the most unique labels of 7 (Table 3). Take the BERT base model as example, H - 768, therefore, NH + 2H² + MH + 2H² ~ 3H2 + MH 1Millonn at its maximum, which is around 1% of BERT base's parameter size (110 Million parame- ters).
C.4 Experiment Results
Tables and 10 report the EM accuracies and macro Fl scores across models for the annota- tion split the eight datasets, respectively. We observe similar performance patterns between Ta- bles and 10.
C.3 Size of the Added Parameters
For the annotator embedding, the learnable ma- trix EA N is of size N H. where N is the number of annotators, H the hidden size of the model. For its associated weight, Oas we in- troduced 2 x H weight for the W, and W. matrices. Therefore, we introduced NH + 2/1²2 parameters for the weighted annotator embedding. Similarly, we introduced M H 2H² parameters
We report the BERT base performance for the annotator split on the eight datasets in Tables 6, and 12. Note that because for annotator split, we are testing on a different set of annotators from the train set, MV. ind cannot make any prediction based
16

---

MV
MV mamoo
NC
E,,
E.
B, E..
75.06
64.81
76.70
75.72
75.76
75.67
65.88
76.13
74.67
74.97
75.65
68.36
76.02
75.14
75.28
MDA
50.03
61.71
63.58
76.24
69.73
77.18
75.99
75.68
76.45
70.43
77.78
76.93
77.26
76.38
73.02
77.22
74.75
77.19
63.04
60.88
68.49
69.98
69.90
62.90
62.12
68.39
69.92
66,32
63.22
60.49
67.42
69.22
68.54
25.05
41.27
36.71
63.19
58.86
64.41
65.71
68,46
63.59
62.28
68.58
69.70
69.60
62.94
58.60
65.18
66.52
69.74
54.26
$2.05
56.72
$8.15
53.89
54.11
51.07
56.67
58.19
54.35
54.43
47.16
55.07
56.31
53.31
HUM
33.30
45.65
41.55
54.40
$2.55
54.26
51.97
50.02
54.71
53.63
56.33
57.70
53.31
54.67
54.81
57.18
$8.76
51,86
40.83
40.78
44.00
44.22
44.41
40.47
40.08
43.11
44.09
43.86
41.61
40.81
42.62
43.00
COM
14.02
25.58
18.26
40. 66
40.32
40.42
40.34
39.75
40.54
38.14
42.37
42.82
42.59
40.57
33.82
44.02
44.33
44.15
47.09
39.20
62.88
60.23
64.61
47.32
36.91
61.88
56.20
63.65
46.40
43.32
60.30
45.57
59.65
SNT
20.04
49.47
37.49
47.88
43.82
58.19
46.50
56.
45.75
43.62
61.21
52.57
60.83
48.76
43.78
67.37
68.39
69.77
61.76
60.49
61.68
61.51
61.96
65.31
62.64
65.66
63.56
62,86
66.78
66.04
63.22
62.01
61.70
FIA
20.22
45.67
45.67
68.03
62.91
65.31
70.38
62.50
67.61
67.51
68.73
68.92
68.77
71.73
72.32
72.69
69.56
72.20
67.48
64.84
65.28
65.42
65.77
68.59
67.22
64,93
62.84
62.94
71.46
71.20
61.29
62.32
60.82
PEJ
33.78
51.90
51.90
71.93
70.39
59.89
57.66
59.23
70.26
63.19
64.70
64.69
65.07
74.51
73.98
73.20
73.05
72.91
86.87
86.01
86.90
87.80
87.68
86.35
85.75
86.61
87.83
87.10
86.77
86.65
86.61
87.03
86.69
50.04
86.90
86.83
87.04
86.68
85.76
87.22
86.90
86.71
87.16
87.78
87.87
86.87
86.31
87.49
88.75
88.04
Table 9: EM accuracy scores for annotation split all eight datasets, where the annotators appear in both train and test sets. We average the results across 10 runs. The best results are in bold if they yield statistically significant difference from the baselines or the other way around (t-test, p < 0.05). For each dataset, the rows correspond to the scores from BERT base, BERT large, RoBERTa base, RoBERTa large, DeBERTa V3 base, and DeBERTa V3 large.
17

---

MV
mamto
NC
E,,
B, E..
72.13
64.65
75.43
74.70
74.62
73.07
65.73
75.09
73.97
74.22
73.26
67.92
75.01
73.60
73.98
MDA
49.08
59.70
38.87
73.67
69.50
75.73
73.34
73.16
74.06
69.85
76.32
75.55
75.82
74.22
71.69
76.27
71.69
76.23
59.4
56.98
66.00
67.55
67.46
59.23
58.54
65,84
67.39
61.71
59.28
55.00
64.41
66.50
65.70
24.02
13.43
59.52
52.33
59.27
60.86
65.69
59.85
58.14
65.78
67.07
66.87
59.15
52.34
60.35
61.83
67.15
46.32
41.79
52.97
$5.06
48.58
46.15
42.98
53,55
$5.66
49.74
46.35
40.76
51.29
52.37
47.33
HUM
32.52
43.82
19.57
46.66
43.71
48.45
42.23
37.58
46. 17
43.64
53.22
54.30
47.86
47.04
47.17
54.15
56.12
45.27
32.21
31.30
36.38
36.87
36.76
31.41
31.39
35.02
37.39
36.64
32.32
32.60
31.85
33.42
33.41
COM
13.86
22.15
4.41
32.34
32.29
32.13
31.10
30.82
29.71
25.92
31.89
32.78
32.50
23.29
36.64
38.71
37,93
32.71
28.39
58,16
48.99
59,40
34.30
30.63
57.40
43.78
58.68
34.40
32.90
55.40
33.64
54.
SNT
18.29
38.85
10.91
35.45
33.54
50.47
33.30
46.22
33.58
27.78
55.72
40.16
55,
37.45
36.08
64.36
64.
67.14
45.22
44.91
44.30
45.65
45.36
49.06
42.80
47.23
44.23
45.55
50.82
48.40
46.19
44.75
45.58
FIA
17.07
12.54
12.54
51.24
41.15
44.92
56.94
41.12
49.04
53.02
52.32
52.78
53.68
55.75
56.95
56.49
53.72
56.87
45.23
43.70
43.97
43.53
43,97
46.
45.34
43.29
41,10
41.92
48.26
48.06
40.42
40.68
40.24
PEJ
28.54
22.78
22.78
48.35
47.34
39.09
36.56
38.71
47.28
40.49
43.32
42.94
43.22
50.48
50.08
49.47
49.39
49.33
56.99
61.23
60.03
66.34
64.40
58.80
64.99
65.88
66.14
69.24
64.60
61.54
56.03
60.36
57.37
42.08
46.50
46,50
63.22
67.37
58.12
61.24
58.94
56.01
54.82
66.21
72.27
71.36
60..4
69.86
71.35
74.31
73.72
Table 10: Macro Fl scores for annotation split on all eight datasets, where the annotators appear in both train and test sets. We average the results across 10 runs. The best results are in bold if they yield a statistically significant difference from the baselines the other way around (t-test, < 0.05), For each dataset, six rows correspond the scores from BERT base, BERT large. RoBERTa base, RoBERTa large, DeBERTa V3 base, and DeBERTa V3 large.
18

---

marro
NC
E,, + E..
MDA
49.88
64.15
74.91
67.26
73.55
73.90
74.24
GOE
24.96
36.46
62.86
61.01
61.33
61.98
61.96
HUM
33.34
42.37
54.33
52.24
53.15
53.53
53.51
COM
14.22
19.88
40.78
40.49
40.80
40.30
40.28
SNT
19.98
42.56
43.93
38.56
36.99
40.82
37.90
FIA
20.22
46.41
84.55
83.00
77.48
83.70
83.23
PEJ
33.78
51.57
70.43
66.46
$2.45
63.71
59.47
HSB
49.88
95.09
87.77
89.30
80.78
89.34
88.99
Table 11: EM accuracy for the BERT base model annotator split, where different of annotators appear in train and test sets. We average the results across 10 runs. The best results are in bold if they yield statistically significant difference from the baselines the other way around (t-test, p < 0.05).
MV mazro
NC
E, + E.
MDA
48.86
39.08
72.35
66.76
72.04
71.59
71.95
GOE
23.98
13.36
59.33
56.91
56.68
58.13
58.05
HUM
32.50
19.84
46.59
42.54
47.64
47.09
47.39
COM
13.97
4.74
32.08
31.31
32.58
32.12
31,81
SNT
17.92
11.94
31.18
32.82
23.99
29.35
26.82
FIA
17.07
12.68
75.59
73.27
66.42
73.70
72.62
PEJ
28.54
22.68
47.33
44.60
24.80
41.75
36.99
HSB
37.13
48.74
65.23
65.10
49.45
62.57
63.14
Table 12: Macro Fl scores for the BERT base model annotator split, where different set of annotators appear in train and test sets. We average the results across 10 runs. The best results are in bold if they yield statistically significant difference from the baselines other way around (1-test, p S 0.05).
on its mechanism. Therefore, we omit the MVind baseline in Tables 6, 11 and 12.
achieves the best performance.
Table 10 in Appendix C.4 shows the macro Fl scores with similar trends.
C.5 Performances Patterns
Table 9 shows the EM accuracy scores in different settings TID-8 for the annotation split. Here mainly deseribe the performance patterns for the BERT base model. The improvement across different settings varies on TID-8. On Commit- mentBank and Sentiment Analysis, adding either the annotator or annotation embeddings improves the model performance, and adding the two embed- dings together further improves the model perfor- mance. On Go Emotions, HS-Brexit, and Humor, both embeddings improve the model performance, but adding the two embeddings yields less improve- ment than simply using annotator embeddings. On Multi-Domain Agreement, both annotator and an- notation embeddings improve the model perfor- mance, but adding annotation embeddings yields the most performance gain. Additionally, adding both embeddings together yields less performance gain than annotation embedding only. On Pejora- tive, there are no significant improvements after adding the annotator or annotation embeddings. On Friends QIA, however, adding either embed- ding hurts the performance, and the baseline setting
Our embeddings yield insignificant perfor- mance gains performance losses datasets with too few annotators or with few disagree- ments. For Friends QIA and Pejorative, adding annotator or annotation embeddings yields insignif- icant performance gains or performance losses. For Friends QIA, every annotator annotates examples in the dataset as shown in Figure 7. More- over, only 4 examples in Friends QIA have 2 dif- ferent labels, while the remaining 5.6k examples have only single label as shown in Table 8. For Pejorative, the proportion of examples with which annotators disagree is relatively small, as less than 10% of examples have 2 distinct labels (Table 8). Morcover, Pejorative only has 3 annotators and 2 of them annotate the whole dataset while the other annotates fewer than 10 examples. In these scenar- with such a high or almost perfect agreement rate and few annotators, adding extra annotator information may be a burden to the model, as there is not much the model can do to accommodate different annotators.
the
19

---

Cases when annotator embeddings alone im- prove performances. Apart from Go Emotions and Humor discussed in Section 8, on HS- HS-Brrxxit incorporating annotator embedding yields a perfor- mance gain. HS-Bus Brexit is annotated by six annota- tors: three are Muslim immigrants in the UK while the other three are not. As all of the annotators an- notate the entire dataset, we are able to calculate inter-annotaton agreement using the Cohen Kappa scores (McHugh, 2012) 2) and examine the agreement between annotators belonging to the same group (Muslim immigrants not). Figure 8 shows the Cohen Kappa scores, where annotators 4 to 6 are Muslim immigrants and are Though the inter-group agreement is higher (2 0.40). both the inter-group and overall inter-annotator agreements lie in the range of 0.20 to 0.60, which suggests a fair or moderate agreement. Table 13 shows two examples where annotators from a Muslim back- ground or no Muslim background disagree within their own groups. In such case, annotator embed- ding might better capture the individual variance.
Lo
1.00
0.9
0.41
1.00
0.8
0.7
5
0.45
0.44
1.00
0.6
0.22
0.22
0.28
1.00
0.5
0.25
0.4
0.2
Figure 8: Cohen Kappa scores between each annotator of HS-Brexit. Annotators 4 6 are Muslim immigrants and 1 to are not.
Cases when annotation embeddings alone im- prove model performances. As discussed in Section 8, on MultiDomain Agreement, adding annotation embeddings alone improves model per- formances. And the reason might be because the annotation is good reflection of their political beliefs and attitudes towards topics in the dataset domains. Therefore, annotation embeddings may reflect the collective tendencies of individuals shar- ing similar political belief's and attitudes.
Our findings align with social identity theory (Tajfel and Turner, 2004) which proposes that in- dividuals within the same group exhibit similari- ties, while differences exist between groups due to variation in attitudes, behaviors, and self-concepts (Hewstone et al., 2002; Hogg, 2016; Rosenberg, 2017).
No HATE (NH) <-> HATE (H)
Annotator ID)
4
5
6
Immi- grants in the UK
Muslim
Group
Non Muslim
Text: RT <user>: Islam has no place Europe #Brexit <url>
Annotation
NH
NH
Cases when adding annotator and annota- tion embeddings together yield the best perfor- mances. For the Sentiment Analysis, adding both annotator and annotation embedding yields the best performance. The Sentiment Analysis dataset is emotion classification dataset with the added spe- cific goal of studying age-related bias, as shown in Tables 5 and Apart from individual differences in emotional feelings, annotation embeddings can also capture tendencies as a group. Thus, consider- ing both individual and group tendencies, we find that two embeddings together yield better results
Who let this clown into the US? Deport now. <url> Aunotation
Tex::
NH
NH
II
II
II
Table 13: Label disagreements within the same demo- graphic group for HS- IIS-Brexit. Annotator 4 to 6 are Mus- lim immigrants and to 3 are not.
20

---

for the "Somewhat Negative" and "Somewhat Pos- itive" labels. This suggests that group of annota- tors may prefer use the a "moderate" extent their labeling process.
1.30
0.5
Samoowhal Negalive-
0.03
1.00
0.4
C.6
The Effects of Weights in Annotator and Annotation Embeddings
Sanewhut Posttive
4..1
0.21
.0.03
1,30
02
Tables 14 and 15 present a comparison between the weighted annotator and annotation embeddings versus the embeddings that are without the weight matrix.
Very Postive
0.21
44
4.15
100
-04
Vey Nepatiee
Soncebal Nipusia
Nomnal
Some Somewth Feadtr
Vay Ponticc
We notice that for the eight datasets, the weighted version generally performs better or about the same as its unweighted counterpart. The weight Oa. On may capture the relations between the text and the annotator annotation embeddings, there- fore the weighted annotator or annotation embed- dings may be integrated better with the embed- dings.
Figure 9: Pearson correlation of each label for Senti- ment Analysis. There is weak positive relationship between "very negative" and "very positive", as well as "somewhat negative" and "somewhat positive"..
than using one alone on three out the six models we tested.
One exception is the accuracy patterns for Mul- tiDomain Agreement. However, the weighted em- beddings still achieve better or similar macro F1 scores on MultiDomain Agreement.
Another reason why the group tendency is im- portant for Sentiment Analysis is that it has fine- grained labels that indicate different intensities of the same feelings or judgments. For instance, un- like Go Emotions, which has three labels indicat- ing positive, negative, or neutral, Sentiment Analy- sis has five labels represent different extents of positive and negative emotion. Therefore, certain groups of annotators may have their own interpre- tations or preferences of the scale on Sentiment Analysis. We calculate the Pearson correlations across labels by:
The other exception is Friends QIA in Few An- notators Datasets, where the unweighted embed- dings outperform the weighted ones. The reason could be that Friends QIA only has six annotators and the disagreement is minimal. Therefore, the unweighted embeddings alone may be enough capture individual preferences.
C.7 Take-Away Messages
We first obtain the matrix of for annotators (A.) who annotate more than 50 examples:
Single-dimensioned demographic features not enough. We find that individuals belonging to the same demographic groups may hold contrast- ing opinions. For instance, on HS- Brexit, where the text might contain hate speech towards the Muslim community, being from the Muslim community or not does not necessarily determine an individual's 's stance or opinion. Table 13 shows several examples where people from the same cultural background disagree with each other. Our findings are similar to Biester et al. (2022), who studied annotation across genders for datasets of sentiment analysis, natural language inference, and word similarity, and found lack of statistically significant differences in anno- tation by males and females on three out of the four datasets. Thus, relying on single demographic feature for analyzing perceptions oversimplifies the intricate nature of individual stances and opinions. We advocate for considering multiple dimensions to gain a more comprehensive understanding
A1
A2
AN
label
V11
V12
VIN
labelg
V21
V22
V2N
label,
V51
V52
VaN
We then calculate the Pearson correlation scores based on the row vectors. For instance, we want to calculate the Pearson correlation score for label, and labelz, we would calculate with respect to [V11. V12. VLvJ and [V21, V22. v2Nl.
Because the examples are randomly assigned by default, we assume that there would not be any obvious correlation between labels. However, Fig- ure 9 shows a moderate Pearson correlation score
21

---

Ens.. weight
Enw. weight
E Eawa. weight
E weight
En+ Eawo. weight
E Eau weight
MDA
75.17
76.70
76.86
75.72
77.19
75.76
GOF
66.68
68.49
69.00
69.98
69.20
69.90
HUM
54.13
56.72
55.37
58.15
54.65
53.89
COM
41.20
44,00
40.89
44.22
41.10
44.41
SNT
57.57
62.88
46.08
60.23
57.96
64.61
FIA
61.86
61,68
61.68
61,51
61.35
61.96
PEJ
66.43
65.28
65.78
65544
66.67
65.77
HSB
87.03
86.90
87.19
87.80
87.25
87.68
Table 14: EM accuracy scores for embeddings with without weights annotation split. We obtain the results from the BERT base model. We average the results across 10 runs. The best results are in bold they yield a statistically significant difference from the baselines (I-test, p S 0.05).
En w.o. weight
Enw. weight
Eawa weight
E w. weight
Eg+ Eawo. weight
Eu Eav. weight
MDA
74.88
74.70
72.42
75.43
75.34 35.34
74.62
GOF
66.47
67.55
63.57
66.00
66.71
67.46
HUM
48.96
55.06
45.69
52.97
47.54
48.58
COM
31.43
36.87
31.53
36.38
31.81
36.76
SNT
32.63
48.99
47.99
58.16
48.67
59.40
FIA
44.59
45.65
44.81
45.36
45.61
44,30
PEJ
44.71
43.99
44.08
43.53
44.80
43.97
HSB
55.48
66.34
54,30
60.03
54.80
64.40
Table 15: Macro FI scores for our embeddings with without weights annotation split. We obtain the results from the BERT base model. We average the results across 10 runs. The best results are in bold they yield a statistically significant difference from the baselines (I-test, p S 0.05).
diverse viewpoints.
Details of the Group Alignment with Demographic Features
Diversifying the data. Because of the inherent individual differences, it is crucial to incorporate diversity in the data collection process. Collecting data from wide range of sources, including indi- viduals from diverse backgrounds and demograph- ies, is imperative. There could be disagrcements involved in the process, eight datasets we studied in this paper. However, by gathering annotations from diverse populations, we can capture the richness and complexity of human experiences, perceptions, and opinions. Failing to account for these individual differences data collection could lead to biased or incomplete repre- sentations, limiting the validity and generalizability of research findings.
Dimension Demographic Features. We se- lect seven dimensions that are sequential or can be regarded as sequential data here, including age, grew-up area, current living area, annual household income, education, political identification, and gen- der. Table 17 shows examples of the different de- mographic features for each dimension. Note that the values for each dimension are the most preva- lent ones in the group, they are necessarily coming from single person in that group.
have seen in the
We first conduct the K-means cluster- ing cluster the embeddings. For an annotator a; corresponding the data point in the cluster, we find his or her demographic features for all the dimensions the metadata for each dimension {d1, d2. ... ;....,dppp ,11)-- There could be imbalances for the number people from dy corresponding to the dimension Dj. for instance, there might be more females than males in terms of the gender dimen- sion. Therefore, when counting its frequency in cluster/group, we give multiplier for dj
Methods
C.8 Performance of Our Methods on Other Datasets
Kumar al. (2021) introduced the Toxic Ratings dataset. We obtained the complete dataset from Kumar al. (2021) since the version available to the public lacks annotator identification. Table 16 shows the performance our methods with the BERT base model.
N
a
N Σ
D, = d,1
(4)
22

---

R
MV ind
MV macro
T
NC
En
E
E, Eo
TOR
20.06
55.39
52.36
53.17
48.08
58.07
57.37
58.63
Table 16: Performance of our methods on Toxic Ratings (TOR) dataset (Kumar et al., 2021) for annotation split.
where N is the total number of annotators (every annotator has their demographic features), ||] returns 1 if the dimension Dj for that annotator is dj. otherwise, it returns 0.
We then calculate the frequency of dj for the demographic dimension Dj for each cluster, where the frequency f for the demographic feature Dj d in cluster e
f- a -dyle
(5)
We show the most prevalent demographic feature for dimension Dj for each cluster in Figures and 10a to 10e, and list the demographic features for each cluster Table 17.
Normalization of Demographic Features
Current Live Area: Rural: 0.0, Suburban: 0.5, Urban: 1.0.
Grew Up Area: Rural: 0.0, Suburban: 0.5,
Urban: 1.0.
Age: 50-59: O, 60-69: 1, 70-79: 2, 80-89: 3, 90-99: 4, 100+:
Gender: Female: 0.0, Nonbinary: 0.5, Male: 1.0.
Political Identification: Very liberal: 0.0, Somewhat liberal: 0.25, Moderate: 0.5, Some- what conservative: 0.75, Very conservative: 1.0.
Less than high school: 0.0, High school graduate, GED, equivalent: 0.25, Some college associate's degree: 0.5, Bach- elor's degree: 0.75, Graduate professional degree: 1.0.
Education:
Annual Household Income: Less than $10,000: 0.0. $10.000 $14,999: 0.11, $15,000
$24,999; 0.22, $25,000 $34,999: 0.33.
$35,000 $49.999: 0.44, $50,000 $74,999:
0.56, $75,000 $99,999: 0.67, $100,000
$149,999: 0.78, $150,000 $199,999; 0.89,
More than $200,000: 1.0.
23

---

CA
CA
CA
GU
GU
GU
da
lo
0. 1.0
1.0
a6
06
0.0
014
0.4
AG
AG
AG
ED
ED
ED
UE
GE
CE
(a) Group
(b) Group
(c) Group
GU
GU
AIV
1.0
AB
1.0
0.6
0.6
0.4
AG
AG
ED
GE
GE
Pl
PL
(d) Group
(c) Group
Figure 10: Sentiment Analysis group alignment with demographic features. For the shorthands, CA: current living area, GU: grew area, AG: age, GE: gender, PI: political identification, ED: education, AH: annual household income.
Group ID)
0
2
3
4
Curr Area
Suburban
Urban
Urban
Rural
Suburban
Grew area
Suburban
Rural
Urban
Rural
Suburban
Age
70-79
90-99
90-99
100+
60-69
Gender
Male
Female
Male
Nonbinary
Female
Poli Identifi
Somewhat Conser- vative
Very lihcral
Modcrate
Somewhat Conser-
Moderate
vative
Edu
College/Associate
College/Assnciate
Bachelor's degree
Less than high school
High School
GED/ / equiv-
-alent
Annual Income
$1.00 S200k
S10k
S10k S15k
$35k S50k
$150k $200k
Table 17: The most prevalent demographic features each dimension for the five groups Appendix D gives details each demographic dimension. Note that the values for each dimension are the most prevalent ones each group. they are necessarily coming from single person in that group
24


Reviews:
[{"review_id": "oanWLPgRmU", "reviewer": "Reviewer_Ejmf", "summary": "", "questions": "Question A: How are unknown annotators (e.g., last paragraph of section 8) modeled? Is a zeroed embedding vector produced? Is there an embedding for all unknown annotators?\n \nQuestion B: In section 3, the problem is framed as that of maximizing the annotator-specific correct label. Am I correct that this is how the models are evaluated in Section 6? \n\nQuestion C: While I understand the paper's focus is on the analyses rather than on improving performance per se (a focus that I appreciate), I think the experiment considering unknown annotators should be given more relevance. This is because the paper still emphasizes “practical” aspects, e.g., performance improvement and model efficiency. While it is nice to show that the performance improves when knowing the specific annotator’s ID, in the majority of practical use cases, the set of annotators of the training set and those of the model target “users” are disjoint; however, it might be possible to collect the user’s opinion on a subset of instances, similarly to a cold start scenario in a recommender system. I would give more relevance to this experiment, for which results are currently relegated to the Appendix. I would also be interested in knowing the relationship between the number of available annotations for unknown annotators and the performance.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! We are grateful to see that you recognized the significance of the analysis in our work. \n\n- For Rejection Reason 2 & Question A:\n    - From an application perspective, the performance gains for known annotators show that if we have accumulated a certain amount of data on certain people, we can leverage those to gain insights into individual preferences and accommodate them effectively.\n    - For unknown annotators, we predict their labels by using an annotator embedding that is randomly initialized like the rest of the model parameters, without any tuning. As there is no corresponding annotated data in the training set for the unknown annotators, there will be zero annotation embeddings for these annotators. \n    - We acknowledge that future efforts are needed and we specifically consider pursuing a direction where we derive the embeddings for unknown annotators from embeddings of known annotators who exhibit similar demographic features, especially given the correlations between the embeddings and demographic features discussed in Section 8.\n- For Question B:\n    - You are correct, the metrics, namely exact match accuracy and macro F1 scores, are measured on the annotator-specific correct label.\n- For Question C:\n    - Thank you for your suggestion! We ran into space constraints for the submitted version, but we will consider moving the relevant part from the Appendix to the main body of the paper for the camera-ready version.\n- For presentation improvements:\n    - Thanks for your suggestion, and we are considering adding [this diagram](https://anonymous.4open.science/r/annotator-embeddings-9038/Concept-diagram.pdf) to our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper addresses an interesting and, in my opinion, very important problem clearly. \n\nIt has two main focuses: on the one hand, it shows that performance improves when adding annotators and annotations embedding to a simple transformer model (interestingly, the method increases the number of parameters by 1% only). \n\nOn the other hand, the paper performs a deep analysis of what the embeddings convey, their contributions to the performance, and many other interesting aspects. \n\nMany different datasets are explored (different tasks, number of annotators, number of instances, etc). References to previous work are clear and frame the problem well. ", "reasons_to_reject": "In my opinion, the paper's contribution comes more from the performed analysis than from the increase in performance per se. \n\nThis is because the paper assumes a closed set of annotators for which classification will be performed. This is far from most practical scenarios, where models are trained once and then used by different target users. ", "questions_to_authors": "Question A: How are unknown annotators (e.g., last paragraph of section 8) modeled? Is a zeroed embedding vector produced? Is there an embedding for all unknown annotators?\n \nQuestion B: In section 3, the problem is framed as that of maximizing the annotator-specific correct label. Am I correct that this is how the models are evaluated in Section 6? \n\nQuestion C: While I understand the paper's focus is on the analyses rather than on improving performance per se (a focus that I appreciate), I think the experiment considering unknown annotators should be given more relevance. This is because the paper still emphasizes “practical” aspects, e.g., performance improvement and model efficiency. While it is nice to show that the performance improves when knowing the specific annotator’s ID, in the majority of practical use cases, the set of annotators of the training set and those of the model target “users” are disjoint; however, it might be possible to collect the user’s opinion on a subset of instances, similarly to a cold start scenario in a recommender system. I would give more relevance to this experiment, for which results are currently relegated to the Appendix. I would also be interested in knowing the relationship between the number of available annotations for unknown annotators and the performance.", "paper_topic_and_main_contributions": "While the general practice is to force annotators’ agreement through aggregation strategies (e.g., majority voting), many tasks are legitimately subjective, so the disagreement among annotators can be due to their different points of view rather than, e.g. noise or bad guidelines. \n\nThis paper proposes a method to leverage and study annotators’ disagreement by embedding annotators and their annotations. \n\nExperiments are performed on a large set of diverse datasets, and the results are analyzed from a wide variety of points of view. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I think that adding a picture depicting the architecture and how the embedding and the matrices described in section 4 fit in it would make the understanding of the sections much more intuitive. \n"}, "extra_text": {}}, {"review_id": "wyjsvIcsM8", "reviewer": "Reviewer_nSUZ", "summary": "", "questions": "- why did you put the results and the findings for the \"few annotators\" datasets in the annex?\n\n- in 42-44 you mention \"under-represented\" groups whose opinions may not agree with the majority. I am very puzzled by this wording since a given annotator can be in one case in the majority, and in another case in the minority. So, what does this concept of \"under-represented\" group mean here? Wouldn't  it make sense to speak more of subjective tasks and opinions that might emerge from many factors?\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, we are glad to hear you found our experiments and the interpretation of the results to be thorough. We greatly appreciate your suggestion, and we intend to incorporate the \"few annotators\" results (Table 9) into the main body of our paper in the camera-ready, where an additional page is allowed. This inclusion will allow interested readers to gain a more comprehensive understanding of our methods' performance across various scenarios.\n- For Question 1:\n    - Because of the space limit, the results for \"few annotator\" datasets did not make it into the main text. \n    - We will include the results on \"few annotator\" datasets in the camera-ready version.\n- For Question 2:\n    - Thank you for engaging in a nuanced discussion about the term \"under-represented\".\n    - We concur that while general guidelines exist, determining hate speech is fundamentally influenced by subjective perceptions.\n    - We will change our way of framing the example and rewrite the text along the lines of: Words deemed inoffensive by white annotators may carry offensive connotations for black or Asian annotators, reflecting the cultural intricacies that contribute to the subjective nature of hate speech detection. \n- For presentation improvement:\n    - Thanks for your suggestions! In the updated version of our paper, we will make sure to be consistent in the naming of the dataset, we will move Table 9 to the main text and drop Table 4, and we will add a paragraph in the introduction section to specifically list  our contributions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n- the paper is clearly written and provides sufficient level of detail to understand the carried-out research\n\n- a novel method of integration of annotator idiosyncracies into the modelling task is presented\n\n- experiments are thorough in terms of the number of models and datasets explored, which make the findings relatively generic\n\n- a very thorough interpretation of the results and findings is provided\n", "reasons_to_reject": "\n- the paper is somewhat wrongly balanced, i.e. the annex is longer than the main body of the paper and includes some relevant results, e.g., the results of the experiments on the \"few annotators\" datasets, which would better fit to be placed in the main body of the article\n", "questions_to_authors": "- why did you put the results and the findings for the \"few annotators\" datasets in the annex?\n\n- in 42-44 you mention \"under-represented\" groups whose opinions may not agree with the majority. I am very puzzled by this wording since a given annotator can be in one case in the majority, and in another case in the minority. So, what does this concept of \"under-represented\" group mean here? Wouldn't  it make sense to speak more of subjective tasks and opinions that might emerge from many factors?\n", "paper_topic_and_main_contributions": "The paper reports on experiments focusing on the integration of annotator characteristics in modeling some subjective NLP tasks through creating annotator and annotation embeddings which are incorporated into the model. The evaluation on six NLP datasets revealed that the inclusion thereof leads to significant performance gains when learning from annotation disagreements.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- the datasets are referenced sometimes with full names, sometimes with the acronyms introduced earlier. Therefore, it would be better to use a consistent naming convention\n\n- Table 4 and Table 9 are partially redundant\n\n- maybe enumerating the main contributions in the introduction would improve the presentation"}}, {"review_id": "CTjNKKxOT0", "reviewer": "Reviewer_Yi2j", "summary": "", "questions": "A The method Dawid & Skyene is not considered as a baseline. Any specific reason as to why? \nB The datasets SBIC/Toxic Ratings (Kumar et al.) is also not included for experiments. They consist of large amounts of data points with annotations and annotator information that can be essential. \nC The datasets as a whole other than GOE is relatively small. Does this impact the generalization of your work?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, and for your appreciation of the importance of the direction we pursued and the extensive testing. \n- For Question A:\n    - Thanks for your suggestion. Upon closer examination, we found that the baseline in the suggested paper **is equivalent to the Majority Vote (MV$_{macro}$)**, one of the baseline models in our paper (with the results reported in Table 9). \n    - Conceptually, the suggested paper addresses the observation error, presupposing a single \"ground truth\" from which the observations may deviate. This is a different setup than the one used in our framework, where we assume that each label assigned by an annotator reflects their genuine intent.\n    - Mathematically, the difference between their settings and ours leads to that for the first step in the EM algorithm (step (2) after the initialization) from the suggested paper, equation (2.3) or (1.1) always gives us a probability of $1$ because the number of records equals to the number of the correct label in our setting. Specifically, let us take a closer look at the equation (1.1):\n        - $\\hat{\\pi}_{jl}^{(k)}$ = number of times observer $k$ records $l$ when $j$ is correct / number of patients seen by observer $k$ where $j$ is correct. \n        - In our setting, the observers are annotators, $l$ is the label assigned by the observer or the annotator, and $j$ in the original paper is the correct label, which in our setting is the same as $l$. Therefore, $\\hat{\\pi}_{jl}^{(k)}$ will always be $1$.\n        - Now consider (2.5), which is the probability of that we observe patient $i$ with label $j$, is $p(T_{ij} = 1 | data) = \\Pi_{k=1}^{K}\\Pi_{l=1}^{J}(\\pi_{jl}^{(k)})^{n_{il}^{(k)}}p_j / \\sum_{q=1}^J\\Pi_{k=1}^K\\Pi_{l=1}^J(\\pi_{ql}^{(k)})^{n_{il}^{(k)}}p_{q}$ can be reduced to $p(T_{ij} = 1 | data) = p_j / \\sum_{q=1}^Jp_q$ because the $\\pi$s are $1$. \n        - $p(T_{ij} = 1 | data) = p_j / \\sum_{q=1}^Jp_q = p_j$ ($j$ is the label or the observation replies) means that when the annotator or the observer labels, the probability distribution follows the overall label distribution.\n        - Therefore, if say $p_{j_1} > p_{j_2}$, we will predict label $j_1$ for that example. This **is equivalent to the Majority Vote (MV$_{macro}$) in our paper**.\n    - There are also differences between the application scenarios of the suggested paper and ours. The suggested paper pertains to domains that are subject to errors of measurement such as patient records. In contrast, our setup suits tasks that involve subjectivity, or in other words where there is no single \"ground truth\" label since different people have their own standards and therefore their own “ground truth” labels.\n- For Question B: the reason that we do not include the toxicity rating dataset (Dataset proposed in [Kumar et al.]( https://kumarde.com/papers/designing.pdf)) :\n    - We acknowledge the relevance of the dataset but want to clarify our reasons for its exclusion. The dataset includes annotations for each example alongside annotator demographic information. However, its data format is:\n```json\n{\n  \"comment\": text,\n  \"ratings\": [\n      {toxic_rating_1, demographic_feature_1},\n      {toxic_rating_2, demographic_feature_2},\n      ...\n      {toxic_rating_5, demographic_feature_5}\n   ]\n}\n```\nand **it lacks annotator identification, making it impossible to associate specific labels with corresponding annotators**. Unfortunately, this information is crucial for constructing the annotator and annotation embeddings. Therefore, our methods are incompatible with this dataset format. We nonetheless acknowledge that their work is highly relevant, and we will include this paper in the related work section in the updated version.\n- For Question C:\n    - We respectfully disagree with the statement that the datasets other than GOE (Go Emotions) are relatively small. Regarding the scale of the GOE dataset, which comprises 194k examples, it is worth noting that the other datasets also consist of a substantial number of examples, as illustrated by the following statistics (we include GOE here for a better comparison):\n| Dataset    | Number of Examples |\n| -------- | ------- |\n|GOE (Go Emotions)  |  194k |\n|HUM (Humor) | 141k | \n|SNT (Sentiment Analysis) | 60.4k |\n|MDA (MultiDomain Agreement) | 44k |\n|COM (CommitmentBank) | 11.5k |\n    - As shown in Table 4 and 9, our methods have demonstrated effectiveness across these datasets, spanning a diverse array of tasks and domains. Therefore, we are confident that our methods can generalize well.\n- Presentation improvements: \n    - Thank you for your feedback about improving the presentation. We greatly appreciate your suggestion to include additional content in the paper. In terms of the “key elements”, we ran into space issues that forced us to put content in the Appendix. As stated in our rebuttals to Reviewers 1 and 2, we will move several sections from the Appendix to the main content in the camera ready given the extra page. We will certainly enhance the comprehensive description of our findings in the updated version.\n- Lastly, here is [the anonymized repo of our project](https://anonymous.4open.science/r/annotator-embeddings-9038/). We will also make our Github repository publicly available upon acceptance.\n- We hope to have convinced you of our contributions, and we are happy to engage in further discussion if any clarifications are still needed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces an important direction in the domain of modeling annotators. \n2. The authors have extensively stress tested their methods across different datasets and different methods (more in the long appendix).", "reasons_to_reject": "The reasons to reject are in the questions and improvements. Open to hear feedback from the authors on these aspects.", "questions_to_authors": "A The method Dawid & Skyene is not considered as a baseline. Any specific reason as to why? \nB The datasets SBIC/Toxic Ratings (Kumar et al.) is also not included for experiments. They consist of large amounts of data points with annotations and annotator information that can be essential. \nC The datasets as a whole other than GOE is relatively small. Does this impact the generalization of your work?", "paper_topic_and_main_contributions": "The authors discuss about improving new methods for modeling annotators. They have introduced those methods by adding in an extra set of layers for embedding the annotators and their annotations. The authors have field tested their methods with publicly available datasets in the domain of modeling annotations. The authors have also evaluated the methods across different methods/metrics which is useful for understanding the limits of their work.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "A baseline -> https://www.jstor.org/stable/2346806\nDataset -> https://kumarde.com/papers/designing.pdf"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "This paper is 22 pages long with all the appendix and extra results but the main 8 pages are missing key elements that is crucial for an EMNLP submission. The language aspect of it, the authors have done a good job of having lot of tables and numbers on the readers but they should think about how to include the empirical results on how their methods perform better (or not) than other baselines. The #s on tables is one thing, but a challenge in this domain is that numbers are not the full story, and individual data points do tell a different interesting story. \n\nFor sharing/uploading code for papers https://anonymous.4open.science/"}}, {"review_id": "qmu1nK9e8C", "reviewer": "Reviewer_6CyZ", "summary": "", "questions": "A. How dependent is your method on what instances the annotators were assigned to label? If we were to recreate the dataset but switch up who annotates what, would you still get similar embeddings and results?\n\nB. Besides quality, what requirements did you have for your dataset?\n\nC. What are you trying to tell us through the TSNE plots?", "limitations": "", "rating": 1, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. As you have pointed out, “this paper tackles an important issue”. What we have done here is “representing annotators and annotations with embeddings”, and such an idea “is interesting and could potentially be used in other tasks as well”.\n- We respectfully suggest that you may not have fully understood our work. We note that **multiple other reviewers praised the clarity of our paper, as well as the broadness and concreteness of our experiments**.  \n    - In terms of clarity, Reviewer 1 (Ejmf) wrote  “The paper addresses an interesting and, in my opinion, very important problem clearly” and Reviewer 2 (nSUZ) wrote, “The paper is clearly written and provides sufficient level of detail to understand the carried-out research”. \n    - In terms of the methods and experiments, Reivewer 2 (nSUZ) wrote “a novel method of integration of annotator idiosyncracies into the modeling task is presented; experiments are thorough in terms of the number of models and datasets explored, which make the findings relatively generic”. And Reviewer 3 (Yi2j) wrote, “The authors have extensively stress tested their methods across different datasets and different methods”.\n- To help with the understanding of our work, we created a [concept diagram](https://anonymous.4open.science/r/annotator-embeddings-9038/Concept-diagram.pdf) that illustrates our approach, which we plan to add to the camera-ready version of our paper.\n- For your concern: “what the annotator has labeled most likely are more influenced by what the annotator has labeled and not necessarily by the annotator's preferences.”:\n    - To make it clear, this is **not a limitation of our work**, and we **did not discuss it in the limitation section**. The reasons are as follows:\n    - Annotators are **randomly** assigned examples in the datasets we experiment on (except for those \"few annotator\" datasets in which each annotator may annotate the entire dataset). Therefore, the very example that the annotator annotates has nothing to do with the assignment of the rest of his examples.\n    - Let’s consider a concrete (but extreme) example. For a task with two labels, say annotator A has annotated P, P, P for the first 3 examples. For the fourth example, he may annotate P or N. He may have a tendency of P, but he very well might label it as N if the text is indeed negative. Therefore, we combine the annotation embedding together with the text embedding, with a weight parameter to balance these two. In other words, we balance the effects of annotators’ preferences versus the text.\n    - The power of capturing the annotator’s preference by using annotation embeddings lies in when the text itself is not very extreme. In those cases, the annotator’s preference may dominate how he chooses his label.\n    - The performance improvement by using annotation embeddings on various tasks and models proves the point that labels assigned by the annotator to other examples are correlated with the label on the current example. But again, **the example itself is individually independent**.\n- For Question A:\n    - Our methods work as long as the annotators are assigned examples randomly. \n    - In addition, the consistent improvements we have observed across various datasets as shown in Table 4 and Table 9, reinforce the notion that our methods can generalize well across diverse scenarios. Therefore, we anticipate similar outcomes even if we were to recreate the dataset but switch up who annotates what.\n- For Question B:\n    - In addition to quality, the other requirement is to randomly assign annotators with examples, which is how most of the datasets are collected.\n- For Question C:\n    - The TSNE plots show the existence of clusters of embeddings. For TSNE plots Figure 3 and Figure 4, there are notable clusters that emerge for annotation embedding compared to the annotator embedding. These patterns remain consistent across various hyperparameters when we run TSNE plots.\n    - Furthermore, through Figure 4 and Figure 5, we see that these emerged groups can be grounded with real-world demographic features, which underscores the practical meaning of these emerged clusters we observed in our TSNE plots.\n- For 3b, We will ensure clarity by adding a sentence that we want to compare the annotation embedding (3a) to annotator embedding (3b) in the updated version.\n- We hope to have convinced you of our contributions, and we are happy to engage in further discussion if any clarifications are still needed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of representing annotators and annotations with embeddings is interesting and could potentially be used in other tasks as well.", "reasons_to_reject": "It's not entirely clear to me what is being done here. Perhaps more organization and a diagram of the experimental setup would help. \n\nIf my understanding is correct, I'm also not convinced that the way the annotator embeddings were created is appropriate. Unless we are dealing with a very dense matrix, embeddings based on what the annotator has labeled most likely are more influenced by what the annotator has labeled and not necessarily by the annotator's preferences. For example, if an annotator is only given negative sentences to label in a sentiment analysis task, it doesn't make sense to say that that annotator has a tendency to label things as negative. (You mention this issue in the limitations section and argue that it isn't the main focus of the paper, but I argue that how you deal with missing information is crucial in your work if you're proposing a new method)", "questions_to_authors": "A. How dependent is your method on what instances the annotators were assigned to label? If we were to recreate the dataset but switch up who annotates what, would you still get similar embeddings and results?\n\nB. Besides quality, what requirements did you have for your dataset?\n\nC. What are you trying to tell us through the TSNE plots?", "paper_topic_and_main_contributions": "This paper tackles an important issue in crowdsourced data, which is the handling of disagreements. They suggest that representing annotators and data instances as embeddings helps models learn better without additional computing resources.", "excitement": 1, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Make sure all the figures are referred to in the paper. For example, for Figure 3, you refer to Figure 3a, but never mention Figure 3b. In that case, do you really need that plot?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the problem of learning subjective tasks where annotators might have different perspectives, and proposes to capture those perspectives with "annotator embeddings." Results show that these embeddings improve performance, and the paper additionally carries out a large number of analyses to understand the learning process.

The main strength of this paper is the analysis of the approach, which is incredibly detailed and very interesting to read. I suspect that the analysis will be the thing remembered about this paper.

The main weakness of this paper is the setting - it assumes that the training and test data contain the same annotators. As a result, this almost reads more as a "personalization" paper than an "annotator modeling" paper. Reading it under that view does not - to me - detract from the paper, and in some ways makes it more interesting. The paper does not make a case for why the "same annotator" setting is realistic/interesting.

The "unknown annotators" experiment somewhat addresses this question (in the personalization language, this is the "cold start problem"). However, as AC, I do not understand this result. Based on the paper and the discussion with reviewer Ejmf, my understanding is that none of the reviewers at test time are seen at training time, that they're given a random embedding, and yet, even in that situation, there is no large degradation in performance from the Text Only baseline. But all this says is that having random embeddings doesn't hurt over having no embeddings, which is not saying much. The more interesting comparison is to the results from Table 4. As an attempt to reconstruct them (comparing to E_n+E_a for instance), we have:

            E_n+E_a                E_n+E_a
            (known annots)      (unnkown annots)
  MDA        75.76                      74.24
  HUM        53.89                      53.51
  COM        44.41                      40.28
  GOE        69.90                      61.96
  SNT         64.61                      37.90

Here, MDA and HUM show basically no difference, COM shows a reasonably big difference (4%) and GOE and SNT show large differences (8% and 27% respectively). However, in Table 4, MDA and HUM are the ones where E_n+E_a doesn't really improve over Text Only, and GOE and SNT are the ones where they do.

Basically all this seems to be saying is that when this approach provides big gains (in Table 4), it no longer does (in Table 6).

This is not at all surprising - there's no reason to even expect that this would work.

This is all to say that I agree with this weakness - though thinking of this as personalization rather than annotator disagreement mitigates this - and I don't think the experiment described really attenuates the concern.
Decision: Accept-Findings