When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks
Authors: Eve Fleisig
Rediet Abebe
Dan Klein
Venue: EMNLP 2023 Main (2023)
Keywords: annotator disagreement
hate speech
toxicity detection
offensive content detection
AI fairness
TL;DR: We model individual annotators to predict opinions of the target groups of hate speech & examine ways to maximize privacy while doing so.

Abstract: Though majority vote among annotators is typically used for ground truth labels in machine learning, annotator disagreement in tasks such as hate speech detection may reflect systematic differences in opinion across groups, not noise. Thus, a crucial problem in hate speech detection is determining if a statement is offensive to the demographic group that it targets, when that group may be a small fraction of the annotator pool. We construct a model that predicts individual annotator ratings on potentially offensive text and combines this information with the predicted target group of the text to predict the ratings of target group members. We show gains across a range of metrics, including raising performance over the baseline by 22% at predicting individual annotators' ratings and by 33% at predicting variance among annotators, which provides a metric for model uncertainty downstream. We find that annotators' ratings can be predicted using their demographic information as well as opinions on online content, and that non-invasive questions on annotators' online experiences minimize the need to collect demographic information when predicting annotators' opinions.

When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks
Eve Fleisig' UC Berkeley
Rediet Abebe Harvard University
Dan Klein UC Berkeley
Abstract
Task: Label offensive text
Text: 'Women should just stay kitchen."
Though majority vote among annotators is typ- ically used for ground truth labels in machine learning, annotator disagreement in tasks such hate speech detection may reflect systematic differences in opinion across groups, not noise. Thus, a crucial problem in hate speech detec- tion is determining a statement is offensive the demographic group that targets, when that group may be small fraction the anno- tator pool. We construct model that predicts individual annotator ratings on potentially of- fensive text and combines this information with the predicted target group of the text to predict the ratings of target group members. We show gains across range metrics, including rais- ing performance predicting individual annotators' ratings and by 33% at predicting variance among annotators, which provides metric for model uncertainty downstream. We find that annotators' ratings can be predicted using their demographic infor- mation as well as opinions on online content, that non-invasive questions on annotators' online experiences minimize the need col- lect demographic information when predicting annotators' opinions.
3
Not
Not
Not
OK
OK
OK
Aggregate
Target group label: Not OK
OK
Figure Majority vote aggregation obscures disagree- ment among annotators due to their lived experiences and other factors. Modeling individual annotator opin- ions helps to determine when the group targeted by possibly- hateful statement disagrees with the majority on whether the statement harmful.
the baseline by 22% at
single, fixed ground truth; that the aggregated judgments of group of annotators will accurately capture this ground truth; that disagreement among annotators noise, not signal; that such disagree- ment is not systematic; and that aggregation is equally suited to very straightforward tasks and more nuanced ones (Larimore et al., 2021; Palo- maki et al., 2018; Pavlick and Kwiatkowski, 2019; Prabhakaran et al., 2021; Sap et al., 2022; Waseem. 2016).
1 Introduction
For many machine learning tasks in which the ground truth is clear, having multiple people la- bel examples, then averaging their judgments, is an effective strategy: nearly all annotators agree on a label, the rest were likely inattentive. However, the assumption that disagreement noise may no longer hold if the task is more subjective, in the sense that the ground truth label varies by person. For example, the same text may truly be offen- sive to some people and not to others (Figure 1). Recent work has questioned several assumptions of majority-vote aggregated labels: that there is
In tasks such as hate speech detection, annotator disagreement often results not from noise, but from differences among populations, e.g., demographic groups political parties (Larimore al., 2021; Sap et al., 2022). Thus, training with aggregated annotations can cause the opinions of minoritized groups with greater expertise or personal experi- ence to overlooked (Prabhakaran et al., 2021).
In particular, annotators who are members of the group being targeted by possibly offensive statement often differ their opinions from the majority of annotators who rate the text. Because these annotators may have relevant lived experi-
*Corresponding author: Flleisi@bberkeee...

---

ence or greater familiarity with the context of the statement, understanding when and why their opin- ions differ from the majority can help to capture cases where the majority is wrong or opinions are truly divided over whether the example offensive.
White annotators than those of Black annotators because Black annotators are underrepresented in typical annotation pools (Prabhakaran et al., 2021).
This motivates our first objective, identifying crucial examples where target group members disagree. Disagreement among annotators may stem from variety of sourees, including level of experience, political background, and background experience with the topic on which they are anno- tating. That is, only some disagreement stems from the annotators who disagree being better-infolr informed annotators or key stakeholders. We argue that if statement may stereolype, demean, or otherwise harm demographic group. members of that de- mographic group typically well-suited to de- termining whether the statement is harmful: both from a normative standpoint, as they would be the group most affected, and because they likely have the most relevant lived experience regarding the harms that the group faces.
We construct a model that predicts individual annotators' ratings the offensiveness of text, as well as the potential target group(s)' of the text, and combines this information to predict the rat- ings of target group members on the offensiveness of the example. Compared to a baseline that does not model individual annotators, we raise perfor- mance by 22% at predicting individual annotators' ratings, 33% at predicting variance among annota- tors, and 22% at predicting target group members' aggregate ratings. We find that annotator survey responses and demographic features allow ratings be modeled effectively, and that survey questions on annotators' online experiences allow annotators' ratings to be predicted while minimizing intrusive collection of demographic information.
However, majority vote aggregation often ob- scures the opinion of the demographic being tar- geted by a statement, and so explicitly modeling the opinion of the target group provides a useful source of information for downstream decisions. Thus, finding cases where members of the group targeted by a harmful statement disagree with the majority opinion is crucial to determining whether the majority opinion is actually useful label on a particular example, and provides a vital source of information for downstream decisions whether content should be filtered. We address this by con- structing model that predicts the target group(s) of statement, then predicts the individual ratings of annotators who are target group members (Section 3.1).
2 Motivation and Related Work
Our work draws recent studies that investi- gate the causes of disagreement among annotators. Pavlick and Kwiatkowski (2019) and Palomaki al. (2018) note that disagreement among human annotations not necessarily noise because there is a plausible range of human judgments for some tasks, rather than a single ground truth. Waseem (2016) finds that less experienced annotators are more likely to label items as hate speech than more experienced ones. When evaluating gang-related tweets, domain experts' familiarity with language use in the community resulted in more informed judgments compared to those of annotators without relevant background (Patton et al., 2019). Anno- tator perception of racism varies with annotator race and political views (Larimore et al., 2021; Sap et al., 2022) and awareness of speaker race and dialect also affects annotation, such that annotators were less likely find text in African-American English (AAE) offensive if they were told that the text was in AAE or likely written by Black author (Sap et al., 2019).
Drawing earlier approaches such Dawid and Skene (1979), newer work has used disagree- ment between annotators as source of information. Foraaciai et al. (2021) use probability distributions over annotator labels as an auxiliary task to reduce model overfitting. Davani et al. (2022) use multi- task models that predict each annotator's rating of an example, then aggregate these separate predic- tions to produce a final majority-vote decision as well mcasure of uncertainty based on the vari- ance among predicted individual annotators' rat- ings of the example. Wan et al. (2023) directly pre- dict the degree of disagreement among annotators on an example using annotator demographics, and simulate potential annotators to predict variance in
In addition, aggregation obscures crucial differ- ences in opinion among annotators. For example, aggregated labels align more with the opinions of
'Throughout the paper. we use "target group" or "target groups" to mean the demographic groups to which a text may cause harm, i.c., the groups that are the subject of the text's stereotyping, demeaning, or otherwise hateful content.

---

the degree of disagreement. Gordon al. (2022) predict individual annotator judgments based on ratings linked to individual annotators and annota- tors' demographic information. For each example, their system returns a distribution and overall ver- dict from a "jury" of annotators with demographic characteristics chosen by evaluator in an inter- active system.
environments (e.g., sexual orientation). Thus, an open question is whether annotator ratings may be predicted while using less invasive questions and minimizing unnecessary demographic data collec- tion. We find that survey information about online preferences provides an extremely useful alternate source of information for predicting annotator rat- ings, while remaining less invasive and thus easier to obtain than extensive demographic questions (Section 4.4).
These approaches motivate our second objective, using disagreement as an independent source of information. Previous work, such as Gordon et al. (2022), uses -oman-o supervision to find alternatives to aggregated majority vote. Since hu- man judgments depend on the person's 's background and level of experience with the language usage and content of a text, not all evaluators may be able correctly identify the target group to select an appropriate jury; their own background may also influence their jury selection (e.g., through confir- mation bias) and thus the final system judgment. Gordon et al. (2022) investigate the benefits of the interactive approach; we extend on this work to examine the question of how to account for an- notator disagreement without human-in-ther loop supervision. Our model directly identifies the get group(s) of a statement and predicts the ratings of target group members, providing an indepen- dent source of information about possible errors by the majority of annotators. Our model's pre- dictions can be used as source of information by human evaluators in conjunction with human- in-the-loop approaches, or function independently (Section 3.2).
3 Approach
Our approach consists of two connected modules that work in parallel. Given information about the annotator and the text itself, the rating prediction module predicts the rating given by each annotator who labeled piece text (Figure 2, in red). For this prediction task, we used a RoBERTa-basse module (Liu et al., 2019) initially fine-tuned on the Jigsaw toxicity dataset, which we then fine-tuned on the hate speech detection dataset from Kumar et al. (2021). The target group prediction module predicts the demographic group(s) harmed by the input text (Figure 2, in blue). For this prediction task, we fine-tuned a GPT-2 based module on the text examples and annotated target groups from the Social Bias Frames dataset (Sap et al.. 2020). At test time, our model predicts the target group for the input text, then predicts the rating that the members of the target group would give that text (Figure 2, in purple).
3.1 Individual Rating Prediction Module
Our third objective considers how to minimize collection of annotators' demographic or iden- tifiable information. Using individual annotator IDs that map annotators to their ratings permits fine-grained prediction of individual annotator rat- ings. However, this information is often unavail- able (e.g., in the data used by Davani et al., 2022) and its collection may raise privacy concerns for sensitive tasks. Our method uses demographic in- formation and survey responses regarding online preferences to predict annotator ratings without the need to individually link responses to annotators (Section 4.1).
Motivated by the question whether information about participants besides directly tracking their judgments (e.g. with IDs) can help to predict partic- ipant judgments, we experimented with inclusion of two kinds of annotator information: (1) demo- graphic information and (2) responses to survey indicating annotator preferences and experiences with online content.
We obtained this information from Kumar et al. (2021)'s 1'ss 's dataset, in which each example labeled by five annotators and each annotator labeled 20 examples. For each annotator label, demographic information is provided the annotator's race, gender, importance of religion, LGBT status, ed- ucation, parental status, and political stance. The annotators also completed a brief survey on their preferences regarding online content, including the types of online content they consume (news sites,
In addition, approaches to predicting annota- for opinions that rely on extensive collection of demographic information also raise privacy con- cerns, particularly when asking about characteris- ties that are the basis for discrimination in some

---

GPT-2 Target Group Prediction Modile
Predicted target group of text
Moderator assistance; evidence for Incorrect labelling
Predicted opinion target group
Possibly Offensive Text
Annotator's predicted rating of the text
Rating Prediction Module
Annotator Demographics
Annotator Survey Responses
Figure 2: Structure of our approach. Given piece of text and the annotator's demographic information and survey responses, the rating prediction module predicts the rating given by each annotator who labeled a piece of text (red). The target group prediction module predicts the demographic group(s) harmed by the input text (bluc). At test time, our model predicts the target group for the input text, then predicts the rating that the members of the target group would give to that text (purple).
social media, forums, email, and/or messaging); their opinion how technology impacts people's 's lives; whether they have seen been personally targeted by toxic content; and their opinion on whether toxic content problem.
At test time, we evaluated performance of this module the Kumar et al. (2021) dataset using the mean absolute error (MAE) of predicting in- dividual annotators' ratings and MAE of predict- ing aggregate ratings for an utterance (following Gordon et al., 2022, who evaluated on the same dataset). We also evaluated the MAE of predicting the variance among individual annotator ratings, which provides a measure of uncertainty for model predictions (Davani al., 2022): if there is high variance among the labels that different annotators are predicted assign piece of text, this signal that there is low model confidence about whether the text harmful, or perhaps enough dis- agreement among the general population that the text should be rerouted to manual content modera- tion.
For these predictions, the input is formatted as:
81...8p [SEP] d1 d1..dd [SEP] w1 .......0 Wp
where 81 .... Sn is a template string describing the annotator's survey responses (examples in Ap- template string contain- ing the annotator's demographic information (e.g.. "The reader is 55-64 year old white female who has a bachelor's degree, is politically independent, is parent, and thinks religion is very important. The reader is straight and cisgender"), wi .... ton is the text being rated, and [SEP] is a separator token. We use a template string instead of cate- gorical variables in order to best take advantage the model's 's language pretraining objective (e.g., underlying associations about the experiences of different demographic groups).
pendix A), dd.,,
For each of these metrics, we hypothesized that modeling individual annotators' ratings with added survey and demographic information would reduce error relative to the bascline, which predicts anno- tators' aggregate ratings of a piece text using the average rating as the ground truth (the typical task setup for hate speech detection).
The model is then trained the regression task of predicting every annotator's rating (from 0=not at all offensive to 4=very offensive) on each ex- ample that they annotated. We use MSE loss with each annotator's 's rating on an example treated as a separate training point. To compare with the infor- mation provided by individual annotator IDs, we additionally trained versions of the model where the input is prepended with assigned ID corre- sponding unique of user responses. 2
3.2 Target Group Prediction Module
The second module predicts the target group(s) of the input text.3 Following Sap et al. (2020), we used GPT-2 based module that takes in input:
wi Wp [SEP] t...
'In theory, it is possible to train both modules simulta- neously, end-to-end. We trained them separately because at the time of writing, to our knowledge, there was no single dataset that contained sufficiently extensive information on annotators' demographic information, ratings from individual annotators, and labels for the predicted target group of the text.
'The original responses in the dataset do not provide IDs, but we found that fewer than 9% of the unique sets of survey and demographic responses corresponded to more than one annotator, so we assigned IDs to unique sets of responses.

---

where wi ... .... Wn is the text being rated and 8 ..... in is a comma-separated list of target groups. During training, this module predicts each token in the sequence conditioned on the previous tokens, us- ing GPT-2's s standard language modeling objective. This model was fine-tuned on the text examples and target groups from the Social Bias Frames dataset of hate speech (Sap et al., 2020).
formation about annotators, while still accurately predicting their ratings? To examine this, we train models that omit some survey and demographic information and measure their performance to in- vestigate what information contributes most to the accuracy of model predictions (Section 4.4).
Most importantly, given a statement that may target specific demographic group, can we predict the ratings of larget group members on whether the statement is harmful? We evaluate the full model's performance at predicting the ratings of target group members on statement (predicting the target group + predicting the ratings of that group's members). To measure performance on this task, we examine the model error at predicting the ratings given by annotators who labeled examples where they themselves were members of the group targeted by the example (Section 4.3).
This module generates free-text string of pre- dicted target groups, used instead of a categorical prediction because there is a long tail of many spe- cific target groups. After the model produces this free-text string, it standardizes the word forms of the list of target groups using mapping from list of word forms or variants of a demographic group (e.g., "Hispanic people", "Latinx folks") to one standardized variant ("Hispanic"), taken from the list of standardized demographics in Fleisig et al. (2023). Our model then uses string matching between this list of target groups and the template strings describing annotator demographics to find the set of annotators who are part of any of these target groups.
4.1 Individual Rating Prediction Module
We evaluated the performance of our rating pre- diction module with different information about the annotators provided as input (combinations of demographic information, survey responses, and annotator IDs). Table I gives the mean absolute error on predicting annotators' individual ratings, the aggregate ratings for all annotators who labeled an example, and the variance between annotators labeling an example. We found that both demo- graphic information and survey responses are use- ful predictors of annotator ratings, as adding either demographic or survey information results in 10% improvement at predicting individual annotators' ratings. Combining both sources of information results in the most accurate predictions: the model that includes both survey and demographic informa- tion raises performance over the baseline by 22% at predicting individual annotators' ratings and 33% at predicting variance among all the annotators who rated an example.
To provide information about the opinion of the group potentially being harmed by a text example at test time, the model combines the outputs of the target group prediction module and the rating pre- diction module by (1) predicting the target group of text example and (2) predicting the individ- ual ratings for all annotators in the predicted target group (Figure 2, in purple). To evaluate model per- formance, given a text example x, we examine the predicted individual ratings for all annotators in the predicted target group of x who provided labels for that example.
Appendices A and B contain additional details on model training for reproducibility.
4 Results
The key hypotheses we aimed to test were: Can we predict the ratings of individual annota- tors, using only their demographic information and information about their opinions on online content, without a specific mapping from each annotator to the statements they annotated? To measure this, we examine how well the individual rating prediction module predicts individual annotators' information given demographic information and/or survey re- sponses, compared
Our best model for individual rating prediction also improves performance at predicting aggregate ratings (by taking the average of the predicted indi- vidual annotators' ratings), raising performance by 16% over the baseline. Because this model has ac- cess to demographic and survey information about the individual annotators who labeled a response, but the baseline has no information about which subset of annotators labeled an example, this result is unsurprising. However, it further supports the hypothesis that even when predicting the aggregate
baseline that does not use information about the annotators (Section 4.1).
Can we mininize unnecesarry collection of in-

---

Model
Individual Rating MAE
Aggregate Rating MAE
Variance MAE
Text only
0.88
0.49
1.16
+ IDs
0.92
0.51
1.11
+ demographics
0.79
0.44
1.01
+ surveys
0.79
0.45
1.00
+ demographics
0.85
0.46
1.02
+ demographics ID + survey
0.74
0.41
0.84
+ demographics survey 0.69
0.41
0.78
Mean absolute error of rating prediction module with different input features. Training with demographic and survey information results in the greatest reduction when predicting individual annotators' ratings; the aggregate ratings of all annotators who labeled an example; and the variance between the annotators who labeled an example. Validation set results can be found Appendix
Table 1:
rating of several annotators on an example, the ag- gregate rating of that small subgroup of annotators is sufficiently dependent on the subgroup's 's back- grounds and opinions that taking these into account substantially improves accuracy over predicting the rating of the "average" annotator.
group(s) targeted by the example, the accuracy on exactly matching the list of target groups, and accu- racy partially matching the list of target groups. The model achieves word movers' distance of 0.370 the dataset, exact match accuracy of 58%, and partial match accuracy of 81 (error analysis in Appendix C).
By contrast, inclusion of annotator IDs as a fea- ture may in fact hinder our model's performance. This may be due to the fact that the language model objective does not take full advantage of categori- cal features and is better suited textual features. This hypothesis is borne out by previous work on the same dataset: our model achieves lower error than previous work when using only demographic features* and the input text (individual annotator MAE 0.77, compared to 0.81 in Gordon et al., 2022), but not when using demographic features and annotator IDs (individual annotator MAE 0.80, compared to 0.61 in Gordon al.). Thus, compared to previous work, this approach appears make better use of text features, but not of cate- gorical features without meaningful text represen- tations.
4.3 Full Model Performance
To evaluate the overall performance of the model (the target group prediction and rating modules to- gether), we had the target group module predict the target groups of the examples in the test set, then had the rating prediction module predict the ratings of members of the target group who rated that example. This is a harder task than predicting individual ratings across the entire dataset, since target group members tend belong to groups al- ready underrepresented in the annotator pool, such that there is less training data about their behavior. We considered whether the model can accu- rately estimate the ratings of target group members, which is necessary to flag cases where the consen- sus among target group members differs from that of the majority, by defining target offense error metric. Given a set of examples X, where each example z; has a target demographic group" t,, we defined the target offense error as the mean abso- lute error across all T; E of the members of 4 who annotated example Ti, where the average rating of the members of f, who annotated example 2; is at least I on 0-5 scale (i.e., they found the example at least somewhat offensive))
4.2 Target Group Prediction Module
To evaluate the performance of the target group pre- diction module the Kumar et al. (2021) dataset, which does not list the demographic groups geted by the examples, we manually annotated 100 examples from the dataset's test set with the groups targeted by the text. We then measured the word movers' distance between the predicted text and the freeform text that annotators provided for the
We also evaluated the performance of the com-
*We exclude queer and trans status from the demographic information here for even comparisons with previous work, which did not include these features.
*When there are multiple target groups, we define is as the union of those groups

---

bined model at predicting the individual and aggre- gated ratings among annotators who are members of 44 as well as the variance among members of & when two or more of them labeled zy (Table 2).
rates. We found that individual rating MAE was lowest (under 0.35) at predicting ratings on text targeting people who were racist, Syrian, Brazil- ian, teenagers, or millennials, and individual rating MAE was highest (over 1.52) at predicting ratings on text targeting people who were well-educated, non-violent, Russian, Australian, non-believers. To understand effects on text targeting intersec- tional groups or multiple groups, examined the performance for different numbers of target groups and found that individual rating MAE changed by no more than 0.02 for text targeting up to four target groups.
We found that the model achieves an MAE of 0.59 at predicting the average rating of the target group (22% improvement over the baseline), 0.73 at pre- dicting individual annotator ratings for the target group (18% improvement) and target offense error of 0.8 (17% improvement). The model also has an MAE of 1.22 at predicting the variance among target group members (28% improvement). This suggests that the model not only captures differ- ences in opinion between target group members and non-members, but also variation in opinion be- tween members of the target group. This helps to prevent group members from being modeled monolith and, by providing sign of disagreement among experienced stakeholders, provides a partic- ularly useful measure of model uncertainty in hate speech detection.
Overall, we found that the model is able com- bine its prediction of the target group of a statement with predictions of the rating of individual annota- tors in order to estimate the ratings of target group members on relevant examples, and that it does so best when provided with both demographic in- formation about the annotators and responses to survey questions about their online experiences. We also found that the model accurately predicts target group ratings in the key cases where target group members find that the statement is offensive.
Metric (Target Group Only)
Baseline
Best Model (demographics survey)
Individual Rating MAE
0.89
0.73
4.4 Ablations
Aggregated Rating MAE
0.76
0.59
The effectiveness of information on annotator de- mographies and their preferences regarding online content for rating prediction suggests that predict- ing individual ratings is feasible even when track- ing individual annotators' ratings across questions is not possible or excessively obtrusive. However, an additional concern is that asking for some types of demographic information (e.g., sexual orien- tation, gender identity) or asking for many fine- grained demographic attributes can infringe on an- notator privacy, making such data collection both harder to recruit for and ethically fraught. For these reasons, we trained models that omit some of the survey and demographic information to investigate which are most necessary to effectively predict an- notator ratings.
Variance MAE
1.69
1.22
Target Offense Error
0.96
0.80
Table 2: Combined model performance at predicting opinions of annotators who are members of the target group.
To understand model performance on annota- tors from different demographics and text target- ing different demographics (see also Appendix C), we first examined the five annotator demographic groups with the highest and lowest error rates. We found that the model performs best predicting the ratings of annotators who are conservative, non- binary, Native American or Alaska Native, Native Hawaiian or Pacific Islanders, or had less than a high school degree (individual rating MAE under .66). The model performed worst at predicting the ratings of annotators who are liberal, politically independent, transgender, had a doctoral profes- sional degree, for whom religion was somewhat important (individual rating MAE over 1.24).
By training models on single features, we found that opinions on whether toxic comments are a problem in online settings are the most useful features in isolation for predicting annotator rat- ings, followed by annotator race (Table 3). Us- ing forward feature selection, we also found that a model trained on only three features- race, their opinion on whether toxic posts are a problem, and the types of social media they use-
In addition, to understand model performance on text targeting different groups, we examined the five target groups with the highest and lowest error

---

Feature
Individual
Aggregate Rating MAF
Variance MAE
finely tune this tradeoff:: survey questions that cor- relate with demographies (c.g. types social me- dia used may correlate with age) give partial in- formation without guaranteeing that any single an- notator is a member of a specific group. In other words, -ondomoggapphic survey questions could function a mechanism to tune the tradeoff between privacy protection and representation. A caveat is that such proxy variables may introduce skews other issues into data collection, an important issue for future work. Future extensions of this research could also consider whether other specific survey questions are more useful for predicting ratings, allowing for accurate predictions while avoiding unnecessary collection of annotator information.
Rating MAE
Gender
1.0
0.67
1.2
Race
0.88
0.48
LGBT status
1.0
0.67
1.2
Education level
0.57
Age
0.98
0.56
1.2
Political leaning
1.0
0.57
Importance of religion
0.92
0.51
1.2
Impact of tech
1.0
0.67
1.2
Social media use
1.0
0.67
Thinks toxic posts are a problem?
0.87
0.49
1.1
Personally seen toxic content?
0.95
0.52
5
Conclusion
Race
0.79
0.44
1.02
Thinks toxic posts are problem?
We presented model that predicts individual anno- tator ratings on the offensiveness of text, which we applied automatically flagging examples where the opinion of the predicted target group differs from that of the majority. We found that our model raises performance over the baseline by 22% at predicting individual annotators' ratings, 33% at predicting variance among annotators, and 22% at predicting target group members' aggregate ratings. Annotator survey responses and demographic fea- tures allow ratings to be modeled without the need for identifying annotator IDs to be tracked, and we found that a model given only annotators' race, opinion on online toxic content, and social media usage performs comparably to one trained with annotators' full demographic information. This suggests that questions about online preferences provide future avenue for minimizing collection of demographic information when modeling anno- tator ratings.
Social media use
Table 3:
Performance of models trained on single fea- tures. Race and opinion on whether online toxic com- ments are problem the most useful individual fea- tures for predicting annotator ratings. The model given only annotators' race opinion whether toxic posts are problem + social media usage gives near-identical performance to the model trained with annotators' full demographic information.
approximately matches the performance of the model trained with annotators' full demographic information at predicting annotator ratings (indi- vidual rating 79)...79). This result suggests that future studies could use more targeted information collection to predict individual annotator ratings, reducing privacy risks while minimizing effects on model quality. Another potential implication is that for studies with limited ability to recruit perfectly representative population of annotators, it may be most important to prioritize recruiting a representa- tive pool of annotators along the axes that have the greatest effect annotator ratings, such as race.
Our findings provide method of modeling indi- vidual annotator ratings that can support research the downstream of information about anno- tator disagreement. Predicting the ratings of target group members allows hate speech detection sys- tems to flag examples where the target group dis- agrees with the majority as potentially mislabeled or as examples that should be routed to human content moderation. Understanding the variance among annotators provides an estimate of model confidence for content moderation filtering, fa- cilitating a rapid method of spotting cases where model is uncertain.
Demographic information is useful both to pre- dict individual ratings, and to identify annotators who are relevant stakeholders for particular infor- mation. For the latter purpose, there is an inherent tradeoff between certainty that an annotator is member of a relevant demographic group, and pro- tecting privacy by not collecting that demographic information. However, the use of non non-demographie survey questions may allow practitioners to more
In addition, since opinions within demographic

---

group are not monolithic, accurately predicting the variance among target group members especially useful: text that generates disagreement among peo- ple who are typically experienced stakeholders may require particularly attentive care. Potential appli- cations of this work that have yet to be explored include improving resource allocation for annota- tion by identifying the labelers whose judgments it would be most useful to have for given piece of text, and routing content moderation decisions to experts in particular areas.
holders rather than automated decision making. In addition, improving methods for intersectional de- mographic groups is an important area for future work.
Modeling individual annotators raises new pri- vacy concern,, since sufficiently detailed collection of annotator information may render the annota- tors re-identifiable. We discuss our findings on some approaches to minimizing privacy concerns in Section 4.4, and as previous work has noted, investigation into techniques such as differential privacy may help to minimize these risks (Gordon et al., 2022).
Limitations
This work was conducted on English text that pri- marily represents varieties of English used in the U.S.: the annotators who labeled the Kumar et al. (2021) dataset were from the United States and the annotators who labeled the Social Bias Frames dataset (Sap et al., 2020) were from the U.S. and Canada. It remains unclear how to generalize this work to groups that face severe harms (e.g., legal discrimination) in the U.S. and Canada or groups that may not face severe harms in the U.S. and Canada, but do face severe harms in other coun- tries. For example, in a country where a group faces serious persecution, it may be unsafe for members of that group from that country to identify them- selves and annotate data. In addition, statements that pit perspectives from multiple groups against each other (e.g.. text stating that members of one group hurt another group) may be more difficult to analyze under this framework when multiple po- tential harms towards different groups are at play. Future work could extend this paradigm to uncov- cring disagreements between multiple groups and refining approaches to intersectional groups.
These questions intertwine with issues of how to collect demographie information, since coarse- grained questionnaires may erase the experiences of more specific or intersectional subgroups, as well as the convergent expertise of different demo- graphic groups that have had overlapping experi- ences. Further research into effective and inclusive data collection could unlock better ways of under- standing these complex experiences.
Acknowledgments
Many thanks Deepak Kumar for providing ac- cess to the data used in this paper and Maarten Sap for data and discussions that inspired this re- search. Thank you to the Berkeley NLP group and recommender systems discussion group for their feedback on this research, and extra thanks to Nicholas Tomlin, Kevin Yang, and Ruiqi Zhong for their especially helpful feedback earlier drafts.
References
Aida Mostafazadeh Davani, Mark Díaz, and Vinodku- mar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective an- notations. Transactions of the Association for Com- putational Linguistics, 10:92-110.
Ethical Considerations
Predicting opinions of annotators from different demographic groups runs the risk of imputing an opinion to a group as a monolith instead of under- standing the diversity of opinions within that group. Such modeling is no substitute for adequate repre- sentation of minoritized groups in data collection. To avoid risks of tokenism, we encourage that indi- vidual annotator modeling be used as aid rather than replacement for more diverse and participa- tory data collection. For example, the model can be used highlight examples for which it would be useful to collect more data from a particular com- munity or delegate decisions to community stake-
Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error- rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics). 28( 18:22.2-2.
Eve Fleisig, Aubric Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, and Hanna Wallach. 2023. Fair- Prism: Evaluating fairness-related harms in text gen- eration. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics. As- sociation for Computational Linguisties.

---

Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Bar- bara Plank, Dirk Hovy, and Massimo Poesio. 2021. Beyond black& & white: Leveraging annotator dis- agreement via soft-label multi-task learning. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2591-2597, Online. Association for Computational Linguistics.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668-1678, Florence, Italy. Asso- ciation for Computational Linguistics.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Juraf- sky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power im- plications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477-5490, Online. Association for Computational Linguisties.
Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. 2022. Jury learning: Integrat- ing dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Hu- man Factors in Computing Systems, CHI '22, New York, NY, USA. Association for Computing Machin- ery.
Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5884-590 5906, Seattle, United States. Association for Computational Linguistics.
Deepak Kumar, Patrick Gage, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021. Designing toxic content classification for a diversity of perspectives. In SOUPS. Usenix.
Savannah Larimore, Ian Kennedy. Breon Haskett, and Alina Jevenie--Kocheer 2021. Reconsidering anno- tator disagreement about racist language: Noise or signal? In Proceedings of the Ninth International Workshop on Natural Language Processing for So- cial Media, pages 81 81-90, Online. Association for Computational Linguistics.
Ruyuan Wan, Jachyung Kim, and Dongyeop Kang. 2023. Everyone's voice matters: Quantifying anno- tation disagreement using demographic information. arXiv preprint arXiv:2301.052:
Zeerak Waseem. 2016. Are you racist or am I seeing things? annotator influence on hate speech detection on Twitter. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 138- 142, Austin, Texas. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1997.1169)
A Input Formatting
Jennimaria Palomaki, Olivia Rhinehart, and Michael Tseng. 2018. A case for a range of acceptable anno- tations. In SAD/CrowdBiasse @ HCOMP.
Example of a sentence representing survey infor- mation: "The reader uses social media, news sites, video sites, and web forums. The reader has seen toxic comments, has been personally targeted by toxic comments, thinks technology has somewhat positive impact on people's lives, and thinks toxic comments are frequently a problem."
Desmond Upton Patton, Philipp Blandfort, William R. Frey. Michael B. Gaskell, and Svebor Karaman. 2019. Annotating social media data from vulnerable pop- ulations: Evaluating disagreement between domain experts and graduate student annotators. In Hawaii International Conference on System Sciences.
Example of a sentence representing demo- graphic information: "The reader is 55 64 year old white female who has a bachelor's 's degree, is politically independent, is a parent, and thinks reli- gion is very important. The reader is straight and cisgender."
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transac- tions of the Association for Computational Linguis- tics, 7:677-694.
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On releasing annotator-leve labels and information in datasets. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 133-138, Punta Cana, Dominican Republic. Association for Computational Linguistics.
B Further Model and Dataset Details
We divided the dataset from Kumar al. (2021) into train set of 97,620 examples and validation and test sets of 5,000 examples each (following
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language models are unsupervised multitask learners.
Gordon et al., 2022, who also use a test set of 5,000 examples for this dataset). We used the Social Bias

---

Frames dataset's existing split into training set of 35,424 examples, validation of 4,666 examples, and test set of 4,691 examples (Sap et al., 2020). The model uses RoBERTa-baze which has 123 million parameters, and GPT2-large, which has 1.5 billion parameters, for total of approximately 1.623 billion parameters. No hyperparameter search was conducted; the recommended hyperpa- rameters for RoBERTa-base and GPT2-large were used (Liu al., 2019; Radford et al., 2018).
For each training run, the model was trained on two NVIDIA Quadro RIX 8000 GPUs for approx- imately 12 hours.
Results: Details
Table 4 provides the validation results for the met- ries in Table 1. Tables and provide details on the demographic groups for which the model performs best and worst.
Overall, each annotator labeled 20 examples on average (stddev=9). For the final level of cate- gories in the ablations in Section 4.4, there are mean of 31 annotators in each possible bucket (stddev=142).
Examining patterns of error in the target model, we found that the model was most likely to miss target group when the text was actually targeting people on the basis of race (33% of missed tar- get groups), followed by religion (23%). In cases where the model returned target group that was not represented in the example, the model most of- ten predicted that the text targeting people the basis of race (25% of false positive predictions) gender (18%).

---

Model
Individual Rating MAE
Aggregate Rating MAE
Variance MAE
Text only
0.87
0.48
1.16
IDs
0.88
0.50
1.11
demographics
0.77
0.44
1.02
surveys
0.77
0.43
demographics IDs
0.85
0.47
demographies survey
0.72
0.41
0.89
demographies survey
0.69
0.40
0.89
Table 4: Validation set results for Table 1. Mean absolute error of rating prediction module with different input features.
Model
Individual Rat-
ing MAE
Politically conservative
0.37
Nonbinary
0.63
American Indian Alaska
0.64
Native
Native Hawaiian Pacific
0.64
Islander
Less than high school de- gree
0.66
Religion somewhat impor- tant
0.73
Doctoral degree
0.73
Transgender
0.73
Politically independent
0.83
Politically liberal
1.24
Table 5: Annotator demographic groups with highest and lowest individual mean absolute error (five best and five worst).
Model
Individual Rat- ing MAE
Racist
0.06
Syrian
0.26
Brazilian
0.28
Teen
0.33
Millennial
0.35
Non-believer
1.52
Australian
1.52
Russian
1.52
Non-violent
1.55
Educated
1.85
Table Predicted target demographic groups with highest and lowest individual mean absolute error (five best and five worst), Only words that could be mapped demographic groups are included monn-mappable words such connectors stopwords were excluded).


Reviews:
[{"review_id": "eLiTb4Pwfw", "reviewer": "Reviewer_Muv1", "summary": "", "questions": "A. The dataset by Sap et al., Social Bias Inference Corpus is a good test subject for the work presented since it does not have a lot examples but the content is severely subjective and toxic, any reason as to why it is not considered? \nB. How does your work go against a non demographic modeling method such as the Dawid & Skyene approach? \n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for providing helpful feedback on our paper.\n\n> “The results in the tables do support the model does work, but paper seems to lack empirical results as to where each case on what works and what doesn't.”\n\nWe will expand on the discussion of cases in which the model works and does not work in the camera-ready version, with details on the performance at predicting ratings on different targeted demographic groups, and at predicting ratings by members of different demographic groups:\n\nWe found that the model performs best at predicting the ratings of annotators who are conservative, nonbinary, Native American or Alaska Native, Native Hawaiian or Pacific Islanders, or had less than a high school degree (individual rating MAE ranging from .37 to .66). The model performed worst at predicting the ratings of annotators who are liberal, politically independent, transgender, had a doctoral or professional degree, or for whom religion was somewhat important (individual rating MAE ranging from .73 to 1.24).\n\nWe also found that individual rating MAE was lowest (.06 to 0.35) at predicting ratings on text targeting people who were racist, Syrian, Brazilian, or teenagers, and individual rating MAE was highest (1.2 to 1.85) at predicting ratings on text targeting people who were well-educated, non-violent, Russian, or Australian. To understand effects on text targeting intersectional groups or multiple groups, we examined the performance for different numbers of target groups and found that individual rating MAE dropped by no more than 0.02 for text targeting up to four target groups.\n\nWe will discuss these findings in the paper (with a table of the full demographic breakdowns in the appendix).\n\n> “The dataset by Sap et al., Social Bias Inference Corpus is a good test subject for the work presented since it does not have a lot examples but the content is severely subjective and toxic, any reason as to why it is not considered? B. How does your work go against a non demographic modeling method such as the Dawid & Skyene approach?”\n\nWe used the SBIC data as a test set for the target group model, but not the full model, because compared to the Kumar et al. dataset, the SBIC dataset has (1) significantly less demographic information on its annotators, (2) fewer annotators per example, and (3) a much more uneven distribution of annotators per example (many annotators only labeled a single example).\n\nWe were unaware of the Dawid-Skene model, but will cite it and discuss differences in the approaches for the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. The model architecture the authors have presented is quite interesting. The decision for train/fine-tune each head on a dataset that is relevant to it is an interesting approach. \nB. The paper is easy to read and follow, well motivated. ", "reasons_to_reject": "A. The results in the tables do support the model does work, but paper seems to lack empirical results as to where each case on what works and what doesn't. The authors have an interesting example at the beginning (Fig 1) to demonstrate their motivation, but this wasn't highlighted well enough in their later findings. Not even the appendix. I feel this was a missed opportunity for a submission to a venue like EMNLP. \n\nSee the questions. ", "questions_to_authors": "A. The dataset by Sap et al., Social Bias Inference Corpus is a good test subject for the work presented since it does not have a lot examples but the content is severely subjective and toxic, any reason as to why it is not considered? \nB. How does your work go against a non demographic modeling method such as the Dawid & Skyene approach? \n", "paper_topic_and_main_contributions": "The authors present an interesting dimension for being inclusive of the majority as well as the minority opinion. It is a critical line of work in the dimension of ethics and ML. The paper paper presents a differentiated model training for identifying the targeted demographics in hate speech and offensive language. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "A. The limitations and the ethics sections does not count towards the 8 page limit, feels like it could have been better used for more empirical results. "}, "extra_text": {}}, {"review_id": "cJcPpwp5OB", "reviewer": "Reviewer_7dom", "summary": "", "questions": "How can those who organize annotation sessions introduce this study in their future annotation sessions?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for providing feedback on our paper. Regarding your question, we will release our code with the final version of the paper so that the model can be more widely used by future researchers conducting annotation work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very well written, and it addresses a crucial and common sense topic in human data annotation, which leads to the fact that, at the end of the day, different annotators' backgrounds influence the quality of our machine learning models.", "reasons_to_reject": "-", "questions_to_authors": "How can those who organize annotation sessions introduce this study in their future annotation sessions?", "paper_topic_and_main_contributions": "The authors construct a model that predicts individual annotator ratings on possibly problematic language and then combine this knowledge with the text's projected target group to predict target group members' ratings. When comparing MAE (Mean Absolute Error) to a baseline that does not model individual annotators, the authors find that the new model outperforms the baseline by 22% in predicting individual annotators' ratings, 33% in predicting variance among annotators, and 22% in predicting aggregate ratings of target group members.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-", "Typos_Grammar_Style_and_Presentation_Improvements": "-"}, "extra_text": {}}, {"review_id": "TzdXpyXUH4", "reviewer": "Reviewer_aGsZ", "summary": "", "questions": "• I’d be curious how coarse-grained that final level of categories is from Section 4.4 that provides good performance with minimal responses too. How many annotators, on average, fall into each unique bucket in the cross product of those categorical variables? A related point - how many responses, on average, did each annotator label, and would we really expect it to be enough for the model to learn something about IDs?\n\n• I don’t understand the result starting at line 414 until the end of the paragraph. If your model is even closer to the aggregate average rating, how is it not even more of a “regression to the majority vote” prediction model? I think there’s an obvious answer here but it’d be helpful to have it spelled out for me, because I can’t quite figure this point out.\n\n• An error analysis of the target group model (Section 4.2) would be useful, even if it is just a few sentences, to get some intuition about where mistakes from the model trend and implications for the overall goal of the paper\n\n• Would be interesting to see Table 2 broken out by major target groups and how that relates to the findings in Table 3, and/or if the model suffers in any cases on particular demographic groups.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for providing helpful feedback on our paper.\n\n> \"I am a little surprised at how limited the limitations statement in this paper is. First, I think there’s a significant risk of tokenism here. How do we avoid the situations where we just say “well, we had some female annotators, so we are ‘all good on gender bias’”? Or even worse, “we have a model of what a woman would say about this message”? \nSecond, although less important (because I think the work implicitly could address this concern upon extension), the limitations section seems to emphasize cultural differences but only touches, implicitly, on intersectionality. Worth a more explicit nod about how this paper does or does not address it.\"\n\nThese are important points; thank you very much for raising them. Regarding the risk of tokenism: We agree that our model is no substitute for adequate representation of minoritized groups in data collection. We will highlight the risk of using our model to impute an opinion to a group as a monolith instead of making the effort to examine the diversity of opinions within that group. We will note this as a limitation, and emphasize that the model should be used as an aid rather than a replacement for more diverse and participatory data collection. For example, the model could be used to highlight examples for which it would be useful to collect more data from a particular community or delegate decisions to community stakeholders rather than automated decision making.\n\nRegarding intersectionality: we will (a) highlight this as a limitation of the paper and (b) expand on the discussion of cases in which the model works and does not work, with particular emphasis on intersectional groups, in the camera-ready version. We will discuss the performance at predicting ratings on different targeted demographic groups, and at predicting ratings by members of different demographic groups, outlined briefly here:\n\nWe found that the model performs best at predicting the ratings of annotators who are conservative, nonbinary, Native American or Alaska Native, Native Hawaiian or Pacific Islanders, or had less than a high school degree (individual rating MAE ranging from .37 to .66). The model performed worst at predicting the ratings of annotators who are liberal, politically independent, transgender, had a doctoral or professional degree, or for whom religion was somewhat important (individual rating MAE ranging from .73 to 1.24).\n\nWe also found that individual rating MAE was lowest (.06 to 0.35) at predicting ratings on text targeting people who were racist, Syrian, Brazilian, or teenagers, and individual rating MAE was highest (1.2 to 1.85) at predicting ratings on text targeting people who were well-educated, non-violent, Russian, or Australian. To understand effects on text targeting intersectional groups or multiple groups, we examined the performance for different numbers of target groups and found that individual rating MAE dropped by no more than 0.02 for text targeting up to four target groups.\n\nWe will discuss these findings in the paper (with a table of the full demographic breakdowns in the appendix).\n\n\n> “I'm a bit dubious about the idea that this method increases privacy, could make things easier for researchers, or could be used to reduce information about annotators, which seems to be one of the main claims of the article. Specifically, the authors state that a limit to prior work is the need for annotator IDs that “might not be available”, but then suggest adding new survey data that almost certainly is unavailable in prior work…Even more simply, annotator IDs are in any reasonably ethical research entirely anonymized. It’s not clear to me then how limiting use of annotator IDs creates more private data.”\n\nWe will clarify this further for the camera-ready, but we would like to separate out the issues of (a) creating a system that does not require annotator IDs and (b) the potential uses of including survey data.\n\nNot requiring annotator IDs is helpful because existing datasets in which annotator information has been removed for anonymity (e.g., the data used by Davani et al., 2022 or the Annotators with Attitudes dataset from Sap et al., 2022) often have the following format:\n\n- [example A] [label 1; annotator demographics] [label 2; annotator demographics] [label 3; annotator demographics]\n- [example B] [label 1; annotator demographics] [label 2; annotator demographics] [label 3; annotator demographics]\n\ne.g.:\n- [“I hate group X”] [5; 35 year old White man] [4; 55 year old Black man] [5; 25 year old Hispanic woman]\n- [“I hate group Y”] [1; 45 year old Hispanic man] [3; 35 year old White man] [5; 60 year old White woman]\n\nThis format leaves it unclear whether the 35-year-old white men who labeled “I hate group X” and “I hate group Y” are the same person. Unlike prior work, our approach allows annotators to be modeled without this information. We apologize for the lack of clarity on the distinction between questions of privacy, and questions of addressing current data limitations. We will clarify that modeling without annotator IDs is more related to limitations of how data collection is presently done, rather than privacy.\n\nMeanwhile, we emphasize the potential usefulness of survey information as an alternative to collecting demographic information (separate from the question of annotator IDs). Collecting demographic data improves modeling of annotators, but such data collection can be highly invasive. If we can model annotator opinions just as well by knowing a proxy variable from survey data, this could help to preserve annotator privacy.\n\n> “It’s also not entirely clear to me that the claim that their work can a priori limit invasive data collection is valid. If I want to ensure that I am giving adequate voice to target populations, it would seem that I have to have some way I can validate that I am doing so, or have a strong proxy variable for it (which negates the claims of privacy preservation).”\n\nYes, this is an excellent point and we will discuss this issue in the camera-ready version. There is an intrinsic tradeoff (not limited to our work, but in annotator modeling overall) between certainty that the data is representative of a group G, and privacy protection that limits knowledge of who is in G. At minimum, we would like to note that such a tradeoff exists, and that use of non-demographic survey questions allows practitioners to choose points along this spectrum besides “full demographic information” and “no demographic information.” For example, if labeling a transphobic comment ideally requires transgender annotators, but collecting such information is highly sensitive, then asking whether an annotator has faced gender-based discrimination online will result in a subset of annotators with a higher percentage of transgender annotators but no guarantee that any single annotator is transgender. In other words, non-demographic survey questions can be a mechanism to “tune” the tradeoff between privacy protection and representation. (Investigating whether the use of different proxy variables introduces other issues into data collection is an important question for future work.)\n\n> “I’d be curious how coarse-grained that final level of categories is from Section 4.4 that provides good performance with minimal responses too. How many annotators, on average, fall into each unique bucket in the cross product of those categorical variables? A related point - how many responses, on average, did each annotator label, and would we really expect it to be enough for the model to learn something about IDs?”\n\nFor the final level of categories in Section 4.4, there are a mean of 31 annotators in each possible bucket (stddev=142). Overall, each annotator labeled 20 examples on average (stddev=9).\n\n> I don’t understand the result starting at line 414 until the end of the paragraph. If your model is even closer to the aggregate average rating, how is it not even more of a “regression to the majority vote” prediction model? I think there’s an obvious answer here but it’d be helpful to have it spelled out for me, because I can’t quite figure this point out.\n\nSuppose a dataset has six annotators A,B,C,D,E,F, and example 1 is labeled by annotators A,B,C. The baseline model doesn’t know which subset of annotators labeled example 1, so its prediction for the aggregate rating of A,B,C would be the same as its prediction for the aggregate rating of A,B,C,D,E,F. However, our best model predicts the individual ratings of A,B,C and then averages them when predicting the aggregate rating. If the average of A,B,C is far from the average of A,B,C,D,E,F, then our model would outperform the baseline model at predicting the aggregate rating on example 1. We will clarify this in the paper.\n\n> “An error analysis of the target group model (Section 4.2) would be useful, even if it is just a few sentences, to get some intuition about where mistakes from the model trend and implications for the overall goal of the paper. Would be interesting to see Table 2 broken out by major target groups and how that relates to the findings in Table 3, and/or if the model suffers in any cases on particular demographic groups.\n\nThank you; we will expand on the error analysis of the target group model as suggested for the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ Thinking about an important problem - how do we perform annotation according to those who are most impacted by the decision?\n+ Well-written and intuitive experimental design\n+ Some promising results in terms of the selected outcome metrics\n+ I do like the *idea* of being about to anonymize annotators to just their responses on some very coarse, non-invasive survey questions, although I have some qualms with how this is framed in the paper (see below).  ", "reasons_to_reject": "-  I am a little surprised at how limited the limitations statement in this paper is. First, I think there’s a significant risk of tokenism here. How do we avoid the situations where we just say “well, we had some female annotators, so we are ‘all good on gender bias’”? Or even worse, “we have a model of what a woman would say about this message”? Second, although less important (because I think the work implicitly could address this concern upon extension), the limitations section seems to emphasize cultural differences but only touches, implicitly, on intersectionality. Worth a more explicit nod about how this paper does or does not address it. \n- I'm a bit dubious about the idea that this method increases privacy, could make things easier for researchers, or could be used to reduce information about annotators, which seems to be one of the main claims of the article. Specifically, the authors state that a limit to prior work is the need for annotator IDs that “might not be available”, but then suggest adding new survey data that almost certainly is unavailable in prior work.  It’s also not entirely clear to me that the claim that their work can a priori limit invasive data collection is valid.  If I want to ensure that I am giving adequate voice to target populations, it would seem that I have to have some way I can validate that I am doing so, or have a strong proxy variable for it (which negates the claims of privacy preservation).  Even more simply, annotator IDs are in any reasonably ethical research entirely anonymized. It’s not clear to me then how limiting use of annotator IDs creates more private data.\n", "questions_to_authors": "• I’d be curious how coarse-grained that final level of categories is from Section 4.4 that provides good performance with minimal responses too. How many annotators, on average, fall into each unique bucket in the cross product of those categorical variables? A related point - how many responses, on average, did each annotator label, and would we really expect it to be enough for the model to learn something about IDs?\n\n• I don’t understand the result starting at line 414 until the end of the paragraph. If your model is even closer to the aggregate average rating, how is it not even more of a “regression to the majority vote” prediction model? I think there’s an obvious answer here but it’d be helpful to have it spelled out for me, because I can’t quite figure this point out.\n\n• An error analysis of the target group model (Section 4.2) would be useful, even if it is just a few sentences, to get some intuition about where mistakes from the model trend and implications for the overall goal of the paper\n\n• Would be interesting to see Table 2 broken out by major target groups and how that relates to the findings in Table 3, and/or if the model suffers in any cases on particular demographic groups.", "paper_topic_and_main_contributions": "The paper presents a novel approach that jointly 1) infers the target group for a hateful/toxic comment, 2) infers the likely annotation of the message from an annotator in the target group, and 3) uses that information to improve annotation quality according to a number of important metrics", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies how modeling annotator individuality creates more accurate classifier as well as how to remedy the subsequent privacy concerns. on All reviewers reach consensus that this paper is sound and exciting. The authors engaged in an in-depth discussion with reviewer aGsZ on this paper's implications and made convincing arguments about its claims. The paper is well-written and easy to follow, and potentially an impactful contribution to the community.
Decision: Accept-Main