Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedback (2024)

Abstract: There is a growing body of work on learning from human feedback to align various aspects of machine learning systems with human values and preferences. We consider the setting of fairness in content moderation, in which human feedback is used to determine how two comments — referencing different sensitive attribute groups — should be treated in comparison to one another. With a novel dataset collected from Prolific and MTurk, we find significant gaps in fairness preferences depending on the race, age, political stance, educational level, and LGBTQ+ identity of annotators. We also demonstrate that demographics mentioned in text have a strong influence on how users perceive individual fairness in moderation. Further, we find that differences also exist in downstream classifiers trained to predict human preferences. Finally, we observe that an ensemble, giving equal weight to classifiers trained on annotations from different demographics, performs better for different demographic intersections; compared to a single classifier that gives equal weight to each annotation. Warning: This paper discusses examples of content that may be offensive or disturbing.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9403–9425 August 11-16, 2024 ©2024 Association for Computational Linguistics
Warning: This paper discusses examples of content that may be offensive or disturbing.

heading: 1 Introduction
text: With artificial intelligence (AI) and machine learning (ML) systems being deployed in wide-ranging scenarios, there are growing calls for safe, responsible and trustworthy development of these systems (White House, 2023; European Commission, 2021; UK Government, 2023). An idea in this space that is receiving a lot of attention lately is that of learning from human feedback for making these systems more aligned with human values and preferences (Ganguli et al., 2023; Bai et al., 2022). The idea is certainly promising, as human preferences about values like fairness can often not be
expressed as simple mathematical constraints (Jung et al., 2021; Dorner et al., 2022), but there also remain several important open questions about the idea (Kirk et al., 2023).
In this paper, we focus on the process of eliciting and aggregating human feedback for learning fairness preferences. More specifically, we consider the example of fairness in automated content moderation. Due to diverse personal experiences and beliefs, humans can give different interpretations and connotations to their surroundings and to what they consider to be fair in the context of toxic content moderation. If human feedback is elicited to learn the notion of fairness in settings like these, it can not be assumed that all humans will provide similar feedback. Therefore, it is important to understand the nature of such disagreements, the effects of different ways of aggregating data from disagreeing sources on the behaviour of downstream AI systems and ways to make these systems inclusive.
While previous work has studied the effect of an-
9403
notators’ demographic identities in toxicity labels (discussed further in Section 2), our work focuses on individual fairness preferences, i.e. whether or not two entities ought to receive similar treatment Dwork et al., 2012. We collect data from online crowdsourcing platforms (MTurk and Prolific) to understand these issues. Similar to Dorner et al., 2022, we present participants with pairs of semantically similar -often toxic- comments (in English language), each of which references a different sensitive attribute group. For each pair, we collect participants’ opinions on how both sentences should be treated by a content moderator in comparison to one another. In addition, we ask participants to predict the "average American" answer to the same question.1 Thus, human judgments in our data do not refer to the toxicity level of single sentences, but rather to the fair treatment of two comments relative to each another. Figure 1 shows our survey approach with some examples.
Using the collected data, we study how annotators’ demographic characteristics (gender, race, age, political stance, educational level, and LGBTQ+ identity) influence their fairness judgments. In addition, we also study the effects of references to different sensitive attribute groups in the sentence pairs. We find significant differences in fairness preferences depending on the demographic identities of annotators, as well as the sensitive attribute mentioned in the sentence pairs. Further, we show that training on different annotator groups’ preference data significantly impacts the behaviour of downstream models trained to predict fairness preferences. Perhaps surprisingly, models that are trained on data from a given annotator demographic group a do not usually outperform different demographic groups’ models when evaluated on data from a. Instead, models perform better when evaluated on data from certain annotator groups like age 38+, no matter which group’s data they were trained on.
Finally, motivated by prior work on dealing with disagreements in crowdsourcing (e.g., Davani et al., 2022; Gordon et al., 2022), we investigate whether ensembling models for different demographics can improve predictions of fairness preferences for intersections of demographic groups. We find that ensembling does help in this case too, perhaps by reducing the impact of skew-
1Please see Section 8 for a discussion about the limitations of using the term "average American" in our survey.
ness in the number of labels provided by different groups. While fundamental tensions remain, this approach can provide more representation to the marginalised groups.
Availability of Data The dataset collected for this work is of independent interest for future research in this area. The crowdsourced data is available at https: //github.com/EmiliaAgis/Differences-inFairness-Preferences-ACL-2024. Please note that the dataset contains examples of sentences that may be offensive and disturbing. Personal identifiable information of the annotators has been removed. We also provide a readme file and a license file in the repository.

heading: 2 Related Work
text: Individual Fairness in Machine Learning Dwork et al., 2012 describe individual fairness as the principle that two individuals or instances that are similar with regards to a task, should be treated similarly. This is operationalised in terms of Lipschitz continuity of the mapping from instances to treatments, which can be shown to reduce to robustness to a metric-dependent set of perturbations for binary classification (Dorner et al., 2022). For example, it would likely seem unfair for a content moderator to remove a comment written in AfricanAmerican Vernacular English (Rios, 2020), while not removing a semantically equivalent comment written in Standard American English. Garg et al., 2019 operationalise this idea regarding the referent rather than the writer of a text. They suggest to measure fairness by comparing model outputs for a -often toxic- sentence s and versions s′ of the same sentence that have been modified by replacing words on a list of demographic identifiers with each other. Dorner et al., 2022 extend this beyond word replacement by using large language models for unsupervised rewriting in terms of demographic references. Human feedback is then used to validate whether these generated pairs align with human intuitions on individual fairness. The pairs are then used to train a classifier that predicts fairness judgments, which can be used as fairness constraints for downstream models. In this work, we collect annotators’ fairness judgments on the pairs generated by Dorner et al., 2022 and investigate how annotators’ demographics affect their fairness judgments.
Influence of Demographics on Human Annotations Goyal et al., 2022; Larimore et al., 2021; Kumar et al., 2021 explore the effect of race and LGBTQ+ identity on the perception of negativity in online comments. The first finds that annotators are more sensitive when exposed to toxic comments that regard their own minority group, while the latter two find that minority annotators react more negatively to certain topics like police brutality and are more likely to flag comments as harassment. Other work has found differences of varying strengths between classifiers for toxic content that have been trained on data provided by annotators from different demographics (Al Kuwatly et al., 2020; Binns et al., 2017). Meanwhile, Fleisig et al., 2023 found that providing hate speech classifiers with additional information about annotator demographics greatly improves their ability to predict individual annotator judgments. Instead of predicting the majority label, predictions of these individual annotator can be combined to combat majoritarian biases in downstream applications (Fleisig et al., 2023; Gordon et al., 2022). While our analysis is inspired by these works and others such as (Sap et al., 2022), we are the first to study the effects of demographics on individual fairness preferences rather than the perception of toxicity.
Variability and Disagreement in Crowdsourced Data There is also relevant literature (Plank, 2022) on variation and disagreement in data collected from a crowd. This literature includes (but is not limited to) (Aroyo and Welty, 2015; Inel et al., 2014; Aroyo et al., 2019; Arhin et al., 2021; Denton et al., 2021; Fleisig et al., 2023; Aroyo et al., 2024). Unlike prior work, we focus on differences in individual fairness preferences and the effects on downstream classifiers. More recently, there has been work on exploring whether the large language models reflect the "opinions" of certain populations more than that of others (Santurkar et al., 2023; Durmus et al., 2023; Ryan et al., 2024).

heading: 3 Data
text: Dataset from Prior Work Comment pairs from Dorner et al., 2022 consist of a large set of often toxic social media comments from the Jigsaw Civil Comments dataset that mention different demographic groups, as well as automatically generated semantically similar versions of these comments that reference a different demographic group than the original comment. For a subset (6902) of these
comment pairs, annotations from crowdworkers are also available. We will refer to this annotated dataset as Do. Each of the pairs in Do has one to nine labels on annotators’ predictions on whether or not the average American believes that both comments ought to be treated the same. 720 sentence pairs in Do also contain nine annotators’ personal opinions on the fair treatment of the sentence pairs. Further, in Do, annotators could label as many comment pairs as they liked, potentially leading to a few active annotators providing many or most labels.
New Dataset To remedy the above issues with Do, we construct a new dataset for our research. We first select a subset of comment pairs from Do. As we are interested in investigating disagreements between different annotators, we focus on sentence pairs with particularly contentious fairness judgments: After manually filtering out low quality pairs, for which the original and modified sentences s and s′ are not semantically similar, we collect labels for the 1500 sentence pairs with the largest unlikeability (Kader and Perry, 2007; Perry and Kader, 2005), a measure for the variability in categorical variables. Unlikeability is defined as u2(s, s
′) = 1 − ∑i p2i ∈ [0, 1 − 1i ] where pi is the relative frequency of each of the i annotation options for s, s′. Clearly u2(s, s′) = 0 if and only if responses are unanimous, and large unlikeability coefficients correspond to high variability in ratings. Further details about our selection of sentence pairs to label can be found in Appendix A.1.
We then collect 1000 annotators’ personal fairness judgments as well as their predictions for the average American’s judgment, and data on annotators’ gender, race, age, political stance, education, and LGBTQ+ identity using MTurk and Prolific, with 500 annotators per platform. Each annotator labeled 20 sentence pairs, as well as an attention check question, and each sentence pair was labeled by at least 10 annotators. We provide annotators with the following four options to select from:
1. It would be unfair to delete one of the comments, but not the other.
2. Deleting comment a) would be more justifiable than deleting comment b).
3. Deleting comment b) would be more justifiable than deleting comment a).
4. It would be justifiable to delete either of the comments without deleting the other.
We grouped our sentence pairs into four "focus
categories", depending on the sensitive attribute mentioned in the two sentences: Gender, Race, Religion and Mixed. For example, in the religion focus category, sentences in a pair refer to different religions. In the mixed focus category, sentences refer to different sensitive attributes, for example, s might reference catholic people, while s′ references women. The sentence pairs shown to annotators were split, such that each annotator saw an equal number of comment pairs from each category. Details about our crowdsourcing task, instructions, payments, etc are provided in Appendix A.2. This new dataset required approximately 4000 USD to construct (only including the payment made to the annotators and MTurk and Prolific commission).
As the demographic distributions and unalikeability coefficients among the labels collected on MTurk and Prolific were very similar to each other, we merged them into one large dataset (henceforth referred to as Dn). Further information about platform differences can be found in Appendix B.
The sentence pair generation technique in Dorner et al., 2022 aims to automatically generate good candidate pairs that many humans are likely to consider as valid fairness constraints, and our manual filtering of low-quality (semantically dissimilar) pairs is expected to exacerbate this. Correspondingly, despite the additional filtering for high unlikeability, annotations are skewed towards label option 1 (i.e. stating that both sentences should be treated similarly). We often binarize our labels by aggregating options 2, 3, and 4 to ensure each considered class contains enough labels for both learning and statistical analysis. Henceforth, we refer to option 1 as the Unfair label (i.e. sentences should be treated similarly) and the union of options 2, 3 and 4 as the Not Unfair label (i.e. sentences should not be treated similarly).
Preliminary Data Exploration As our new dataset Dn contains the sentence pairs from Do with the highest unalikeability coefficients, we expect increased variability in responses. Indeed, while Do had an overall median unalikeability of 0.198 for both personal opinion and predictions of the average American opinion, Dn reached a median of 0.558 and 0.481 respectively. 2
For conducting robust analysis with enough samples, we will binarize the race, age, and educational level characteristics of the annotators in the rest of
2For a fair comparison, we only considered the subset of sentence pairs in Do labeled by nine different annotators.
the paper (unless otherwise stated). For the gender characteristics, we only obtained 12 participations for the Non-Binary category. In analysis involving gender, we therefore skip this category. This is a limitation of our work and should be addressed in future research.
For all demographic groups of annotators, we observe a high level of coincidence in the responses for personal opinions and predicted average American’s opinion. We measured coincidence as the fraction of times annotators’ personal opinion and their perception of average American opinion are the same. It ranges from 73% for LGBTQ+ annotators to 90% for annotators over 67 years of age; while all demographics believe their views relatively align with the average American’s, such belief is strongest for people over 67 years of age and weakest for LGBTQ+ people.
Finally, we look at the unalikeability u2(s, s′) for our four focus categories of sentence pairs. Figure 2 shows the unalikeability coefficients for the labels of sentence pairs in each category. Sentences in the mixed focus category are noticeably more skewed towards larger unalikeability coefficients, meaning that responses are considerably more variable for such sentence pairs. This is true for both personal opinions and predictions for the average American opinion. For a more detailed statistical analysis, consider Table 12 in Appendix C. We also observe higher variability in annotators’ personal opinions than in their perception of average Americans’ opinion. Figure 4 in Appendix C shows the variance in the two types of answers.

heading: 4 Differences in Fairness Preferences
text: Method In this section, we study the effect of annotators’ demographic identities on their own fairness judgments and their predictions of the average American’s fairness judgment. Following the standard analysis methods used in prior work (e.g. Goyal et al., 2022, Kumar et al., 2021, Larimore et al., 2021), we predict fairness judgments based on annotators’ demographic information using logistic regression. Because of the large class imbalance in our annotations, we focus on binarized labels, as explained in Section 3. More concretely, in a single regression we use all demographic variables xji , where i stands for gender, race, age, political stance, educational level, or the LGBTQ+ identity of annotator j to predict their fairness judgments yjk,m ∈ {unfair, not unfair}, where k ≤ 20
enumerates all fairness judgments by annotator j and m either represents their own judgment or their prediction for the average American’s judgment. More details about about this analysis are in Appendix D. We account for multiple hypothesis testing using Bonferroni’s correction (Armstrong, 2014) in all reported p-values.
Results Table 1 shows the per demographic odd ratios relative to a reference demographic category obtained from the logistic regression.
We observe significant effects of age, politics, and education on annotators’ personal fairness judgments, and significant effects of all considered demographic variables except for gender on the predicted average American’s opinion.
We generally obtain more significant results (i.e. lower p-values) for the prediction of average American’s opinion compared to annotators’ personal judgments. Thus, we have stronger evidence that their demographics affect annotators’ perception of the average American’s judgment rather than their own judgments.
We observe that White annotators have 0.94 and 0.91 times the odds of answering Not Unfair compared to Non-White annotators when asked about their personal fairness judgments or their predictions for the average American’s judgment respectively. Only the latter effect is significant at the standard level of p = 0.05, suggesting that White annotators are more likely to believe that the average American would prefer sentences within a pair to be treated similarly.
While we do not observe significant results for the personal opinions of Republican annotators, the odds relative to Democrats for personal and predicted average American opinion appear directionally consistent at 1.10 and 1.13. Interestingly, rather than being between Democrats and Republicans, independent annotators are more likely to believe that sentence pairs should be treated similarly (and that the average American believes the same) than supporters of both parties.
A more coherent picture emerges for education: Annotators without a High School education have 0.56 and 0.46 times the odds of answering "not unfair" compared to annotators with at least a Bachelor’s education. The same tendency exists for annotators with a High School degree (odds 0.79 and 0.72). That is to say, as annotators obtain further education, both their likelihood of believing that sentences within a pair should be treated similarly, and their perception of the average American’s likelihood of doing so appear to decrease.
Lastly, perhaps surprisingly, the odds are 0.85 and 1.27 respectively for age 38+ relative to the younger group. This means that younger annotators are more likely to state that they believe sentences receive similar treatment, compared to older annotators. However, they are also less likely to think that the average American would want a pair to receive similar treatment. The same pattern can be observed for the LGBTQ+ demographic, where the respective relative odds are 0.94 and 1.25 for LGBTQ+ people. While only the effect on the perceived average American’s opinion is significant
Personal opinion
in that case, it is quite interesting that both age and LGBTQ+ status show the same pattern of divergence between personal and perceived majority opinion between groups.

heading: 4.1 Further Analysis
text: Relation with Sensitive Attribute in Sentence Pairs We similarly train logistic regression models to predict annotators’ fairness judgments based on which of the three focus categories (gender, race, religion) a sentence pair belongs to.
As shown in Table 2, we find that the sensitive attribute mentioned in the sentence pairs have a significant impact on annotators’ fairness judgments. Specifically, if the sensitive attribute referred in the sentence pair is gender, the odds of classifying the pair (s, s′) as "not unfair" are reduced by a factor 0.68 and 0.81 respectively, compared to sentence pairs in which the sensitive attribute mentioned is race. For religion compared to race, the odds are 0.98 and 0.91. In other words, sentence pairs in which sentences differ along gender are most likely to be annotated as deserving similar treatment.

heading: 5 Effect on Downstream Models
text: Method In the previous section, we provided evidence for wide differences in the fairness preferences depending on annotators’ demographic identities. In this section, we study whether these differences matter for machine learning models trained to predict human fairness preferences. Such an intermediate model of human fairness preferences can be used to train fairer classifiers for a downstream task (here content moderation) (Dorner et al., 2022) by creating a large pseudo-labeled set of fairness constraints from unlabeled sentence pairs. Note that, beyond the specific settings considered in this paper, in the popular RLHF (Reinforcement Learning from Human feedback) framework, an intermediate reward model is trained from human feedback as well, which is then used to align an AI system such as a large language model (Bai et al., 2022). It is reasonable to assume that if the intermediate model learns preferences of specific demographics, it will also impose those learned preferences in the downstream tasks. We focus on intermediate model in our setting of fairness in content moderation, but perform analysis without restricting it to a very specific task setting. We train several neural networks based on labels provided by different demographic groups separately. We hypothesize that, when tested on the data from the entirety of the population, or on the data from different demographic groups, these classifiers will learn noticeably different fairness preferences.
As Dn was specifically selected to only include sentence pairs with high unalikeability coefficients, which are plausibly particularly hard to classify, we found that Dn alone was not sufficient for training classifiers that clearly beat the random baseline. For more details, consider Appendix E. Correspondingly, we decided to supplement Dn with all sentence pairs from Do. As most pairs in Do only contain annotations for the predicted average American’s opinion and not annotators’ personal judgments, we only focus on the former. We were able to obtain data on annotator demographics, ex-
cept for LGBTQ+ identity and education, from the authors of Do. Correspondingly, we restrict our analysis to the remaining other demographic categories (gender, race, age, politics). Similar to the previous section, we use binarized "unfair"/"not unfair" labels in the analysis.
We created subsets of Do+n, grouping by the demographic categories of annotators. For each demographic variable, we created l+1 subset datasets D(κ), where l is the number of categories in a demographic variable and κ represents the value of the demographic variable or the union of these values. (e.g. the gender variable has κ = Female and κ = Male and κ = Male and Female. Here, l = 2). For each demographic variable, each of the first l subsets thus contains data annotated solely by participants who belong to the lth category in that demographic variable. The l+1th dataset contained all datapoints, regardless of the annotators’ demographics, which we denote as the Control dataset. For each of theses subsets, we fitted pre-trained multiheaded BERT based classifiers (Devlin et al., 2019) building upon the code by Dorner et al., 2022. Instead of aggregating individual annotations (e.g. into majority vote) in the training set, we used all the annotations for training the classifiers (Wei et al., 2023). In Appendix E, we discuss that the accuracy of classifiers trained with only majority vote was worse and that was the reason for using all annotations for training.
We trained and tested multiple classifiers ϕκi for each subset D(κ), where i ∈ [1, ..., 20] corresponds to one of 20 different random seeds for the creation of the train, validation (10%), and test sets (10%) Dtraini (κ),D val i (κ),D test i (κ). We provide further details about training the classifiers and hyperparameter tuning in Appendix F. Then, as done by Al Kuwatly et al., 2020 Binns et al., 2017, and Akhtar et al., 2020, we compared the set of models ϕκi and ϕ
κ′ , trained with data labeled by annotators from different demographic categories κ and κ′ respectively (e.g. we compare the models trained with data annotated by Democrat participants with the models trained with data annotated by Republican and Independent participants separately). Each model ϕκi is evaluated on all D test i (κ
′) for κ′ in the same demographic group (e.g. gender). Before that, in order to account for label imbalances, we optimized the classification threshold used by ϕκi on D val i (κ
′) to maximize the models’ balanced accuracy (sklearn, 2007) on the demographic κ′.
Results Table 3 shows that the performance of different classifiers vary noticeably depending on the training and test data used. However, there is no clear trend towards classifiers performing best for the demographic they were trained on. With few exceptions, the test set containing data annotated by a single certain demographic κ ∈ δ in a group of demographics δ tends to show higher
levels of balanced accuracy 3 across all models ϕκ ′ , κ′ ∈ δ. This is consistent with similar observations made by Binns et al., 2017 in their search for differences in judgments of offensiveness across genders, where they found that all classifiers perform better when evaluated on data annotated by men rather than women. It appears that classifiers are able to replicate preferences of annotators in some demographic groups (women, White people, people over 38 years of age and politically independent people) better than those of other demographic groups, potentially regardless of the training data. The reasons for this are not entirely clear; different data set sizes and levels of disagreement within group may be contributing factors. We compare some of these factors across demographics in Appendix G, and leave an in-depth investigation for
3Figure 7 in the Appendix also shows specificity and sensitivity scores for different models.
future work. Despite the lack of a clear directional effect of the training demographics, pairwise KolmogorovSmirnov tests (Kolmogorov–Smirnov Test, 2008) comparing the population of trained models ϕκ ϕκ ′ for different demographics κ and κ′ in terms of balanced accuracy on the control test set yield statistically significant differences for gender (P<.05), race (P<.001), and age (P<.001). This suggests, that the differences in fairness judgments based on annotator demographics observed in Section 4 do affect downstream machine learning models, albeit perhaps not in a clearly interpretable manner.

heading: 6 Ensemble Classifier
text: Previous work has observed disproportionate performance drops for demographic intersections, for example black women, compared to both black people or women (May et al., 2019; Tan and Celis,
2019). We investigated how our Control model that gives equal consideration all data points, regardless of the demographics of annotators, performs when tested on the labels provided by different demographic intersections (e.g. black women). We hypothesize that demographic intersections with a smaller number of annotators will tend to have worse performance because they are underrepresented in the Control model’s training data. For example, in Do+n, we have annotations from 377 (14.4%) Non-White, over 38 years of age, Democrat men, but only 5 (0.02%) Non-White, over 38 years of age, Republican women. We evaluated each of the 20 random seed versions of the Control model using only test labels provided by a specific demographic intersection. We then regressed the models’ balanced accuracies on the number of annotators per intersection, finding a positive correlation (with P= .015).
To address this, we use an ensemble classifier ϕ̄ that aggregates the previously trained classifiers ϕκ via majority voting. We hypothesize that by giving the same weight to each demographic, such an ensemble model will provide better results for a larger number of demographic intersections than our Control model, which gives the same weight to all labels. We compare the predictions ϕ̄(s, s′) with each intersection’s majority vote annotations and calculate balanced accuracy.
Table 4 shows the balanced accuracy scores obtained for our ensemble classifier and the Control model per demographic intersection. The ensemble classifier ϕ̄ provides significantly better (p<0.05) results for 9 out of 24 demographic intersections while the Control model did not provide significantly higher scores for any demographic intersection (and non-significantly better results for 5 out of 24 intersections). For coarse demographics categories (Table 5), we observe a significant improvement in the performance of 8 out of 9 demographics.
Interestingly, while our approach provides much better scores for our smallest demographic intersection (Non-White, older, Republican women), improving its balanced accuracy score by more than 10 percentage points, we also observe a performance improvement for White, older, Democrat women (our largest demographic intersection). Our analysis for fairness thus complements the analysis from prior work on dealing with disagreements (Davani et al., 2022; Gordon et al., 2022).

heading: 7 Conclusions
text: Human feedback is often used to improve artificial intelligence systems by learning to approximate human preferences such as fairness constraints. Via a crowdsourcing study, we investigated the role of different demographic factors on human annotators’ fairness preferences. We focused on a content moderation task as example, where annotators were asked to compare how two sentences should be treated in comparison to one another. We found that demographic variables (age, politics, education and LGBTQ+ identity) play a significant role in the annotations. We observed this effect for annotators’ personal opinions as well as their perception of the average American opinion. Further, we found that differences in the fairness preferences also influence the performance of models trained to predict these preferences. For example, models trained on annotations from different demographic categories, when evaluated on a control test set, gave statistically different balanced accuracy scores. Our results highlight the ethical implications of using human feedback to align AI systems with fairness preferences since the preferences of some demographic groups may be more dominantly reflected in the downstream systems. Finally, we observed that an ensemble classifier that uses the majority vote of models trained on annotations from different demographic categories can improve accuracy scores for a number of demographic intersections.

heading: 8 Limitations
text: Since crowdsourcing platforms like MTurk and Prolific are frequently used for collecting data for training machine learning classifiers, we focus on crowd worker judgements. We acknowledge that the opinions of crowd workers are not necessarily representative of the wider population (within the US or beyond). Therefore, conclusions about opinions of different demographics should be interpreted with this difference in mind and only carefully be generalized to other settings.
Our survey did not include data on several kinds of personal attributes such as disability status. We also bring attention to the lack of data in our survey from non-binary gender group. Further, we also binarized several attributes like age, race and educational status in our analysis due to data constraints, but future research should examine this aspect further.
Our survey was formulated in English and the workers were expected to know English; the results can therefore not be generalized for other languages and cultures, without further data. We also note that our survey focused on sentence pairs with particularly contentious fairness judgements. Further, our data focuses on content moderation domain only.
Our experiments on determining the effect of demographic differences in annotations on downstream models focused on the models that were trained to predict fairness judgments. As explained in the paper, this is a reasonable assumption for task-agnostic analysis (within the broader domain of content moderation). In future work, it will be interesting to investigate the effects on downstream models trained to respect the learnt fairness judgments.
The term "average-American" used in our survey is not very specific (compared to for e.g. "other annotators on MTurk, located in the US and with such and such qualifications"). The intuition behind asking two separate questions (personal opinion and perception of opinion of average-American) was to elicit clear information, even if the participants believed that their own opinions would differ from what they might assume to be a more commonly held opinions in the US. But we have not explored in this work how a different term could affect the complexity of the survey, the experience of participants and the results of the experiment.
Finally, we note that the positive correlation be-
tween performance and size of the demographic intersections does not fully disappear with ensemble approach and more generally, differences in performance across groups continue to exist. Therefore, future research should continue to explore solutions to dealing with disagreements.
We also remind the reader, that because of data availability issues, the results in Sections 5 and 6 pertain to annotators’ perception of the average American’s fairness judgments, rather than their personal fairness judgments.

heading: 9 Acknowledgments
text: This work was partially supported by the Oxford Martin School’s programme on Ethical Web and Data Architectures in the Age of AI (EWADA). The emojis in Figure 1 were designed by OpenMoji – the open-source emoji and icon project (license: CC BY-SA 4.0).

heading: A Appendix A: Data Collection Details
text: A.1 Selection of Sentence Pairs for Dn The first task in our data collection process was to determine an appropriate size for dataset Dn; the amount of sentence pairs nn to be included and the amount of labels to be collected per sentence pair. We preliminarily used dataset Do to analyze the effects of decreasing the dataset size on the significancy of results, to find a lower threshold on the amount of data necessary to obtain statistically significant results in regards to the differences between the models trained with data from opposite demographic categories. Initially, we used the entirety of Do and ten different train/validation/test splits in our data to create ten iterations of the best-performing BERT or RoBERTa models ϕ when trained solely on data annotated by demographic category κ, ϕκi , i ∈ [1, ..., 10], κ ∈ {female,male,Non − White,White}. Then, each iteration ϕκi was evaluated once for every category κ′ from its own demographic variable δ using as test set the labels provided by annotators from such demographic category κ′. After the creation of 10 iterations of each model, we used a Student’s t-test (Student, 1908) to observe whether differences in the labeling patterns of different demographic categories were significant. After observing significant differences for models trained on all Do, we gradually reduced to number of sentence pairs with which we trained and tested the BERT and RoBERTa models and repeated such procedure until the differences in the perception of individual fairness across demographics were no longer significant. We found that the classifiers built with a dataset conformed by 1421 sentence pairs and at least 4 participations per sentence pair were still able to recognize the differences between demographic categories κ significantly (α = 0.05). Given the allocated budget, we were able to collect data on 1,500 sentence pairs, having at least 10 worker labels per pair. We opted for increasing the number of labels per sentence pair to have a larger and more diverse pool of annotators’ demographics per pair and therefore obtaining more robust responses when filtering by demographic traits.
To select the sentence pairs to be included in Dn, we analyzed dataset Do to give priority to sentence pairs with a higher level of disagreement among
participants, as to explore if the demographic characteristics of each worker influenced such differences. We handpicked the sentences with the highest response variability and manually corrected the sentences that either had spelling or grammatical errors (i.e. why should not the foster father be permitted to adopt the infant that has bonded with her?) or that were factually incorrect. On the other hand, certain sentence pairs were not considered because of the lack of relationship between sentences a and b or because of incoherence in sentence b. As stated before, some sentence pairs provided by Dorner et al., 2022 rely on word replacement, taking sentence a as a base and creating sentence b by replacing the words referencing a demographic. The authors note that methodology oftentimes returns incoherent phrases (i.e. replacing "The President’s White House" with "The President’s Black House" would not be correct). On the other hand, we also observe pairs that do not show much similarity within their sentences and lack a strong relation between them. Some examples of sentence pairs that were disregarded for such reasons, independently of the level of variability in their labels, can be seen in Table 6.
Regarding the sensitive attribute mentioned in each sentence pair, we ensured that all participants were shown an identical amount of sentence pairs from each focuc category- Gender, Race, Religion, and Mixed. Table 7 provides sample sentences of such sentence pairs. We did not add a section on sentences that solely refer to the LGBTQ+ community because dataset Do only contained around 120 pairs of this kind and we soon realized that the sentences related to such topic had a lower qualitative quality than the rest. Also, we did not include pairs where the first sentence regarded gender and the second sentence regarded a different demographic trait: we found that most words that refer to people are gendered (i.e. lady, guy, girl, son, etc.) and changing these to a word that references a person’s religion, race or sexuality often resulted in extremely offensive sentences.
A.2 Data Collection Process For our collection task, we recruited workers from Amazon Mechanical Turk and Prolific to later redirect them to our survey which was designed and hosted on Qualtrics. To ensure high-quality data and a speedy collection process, we added a Captcha at the beginning of the survey, enabled the flagging of participants who answered the sur-
vey at an unusually high speed, added a function to prevent workers from responding to the survey more than one time, and only allowed users to enter the survey if they were being redirected from either MTurk or Prolific. Although Google’s technology reCaptchav3 could also be implemented in Qualtrics for bot detection purposes, we refrained from its use because of its lack of GDPR compliance.
We required all participants to have a minimum amount of completed surveys, a minimum acceptance rate and be located in the US. In the case of Mechanical Turk, we requested workers to have completed at least 5,000 previous tasks, with an acceptance rate of 95% or higher, as Dorner et al., 2022 did in their data collection process. On the other hand, for Prolific we required a minimum of 1000 previously completed tasks and a 97% acceptance rate. We decreased the number of previously completed tasks because according to Prolific, if we were to make our survey available only to those who had completed over 5,000 tasks, we would only be able to access a pool of over 300 participants4. Nonetheless, to compensate for this, we increased the threshold for the acceptance rate.
Prior to their participation, participants were informed about the existence of offensive content in the survey and assured that they could withdraw from participating at any given moment, but that
4Information provided at the moment of creating a new survey via Prolific.com
completion would be required to receive monetary compensation for their work. We also provided additional information on the task, the type of data that would be collected from them, survey examples, and contact information in case of concerns. This disclaimer was originally presented to the ETH Zurich Ethics Committee in May 2023 and accepted under proposal number EK 2023-N133-A.
While data obtained through Prolific preserved higher quality levels, Mechanical Turk presented many more submissions from what seemed bots or other workers that did not put effort in the tasks. On the first iteration of the data collection task, MTurk only delivered 140 acceptable responses out of 500 participations. We were able to identify human-like responses by requesting a written explanation to one of the previously provided answers. Moving forward, to avoid such behavior we created a new MTurk Qualification in which workers had to correctly respond to two hand-crafted multiple-choice questions to gain access to the rest of the survey. Unfortunately, upon the installment of such Qualification, the speed at which the survey was answered decreased significantly. While the first 500 submissions -regardless of their quality or approval rate- were collected in a lapse of 4 hours, we required approximately 3 weeks to collect 203 more participations once the Qualification was installed. Such a filter increased the quality of the responses drastically, raising to a 65% approval rate instead
of the original 28%. Nonetheless, having one-third of all HITs being rejected was still an elevated rate of rejection, we opted to remove the Qualification once again for the sake of time efficiency. After doing so, we observed data quality and speed of response similar to the first released batch. Other authors have researched on the shift in the data quality provided by MTurk. While Kennedy et al., 2020 hypothesize on users recurring to fraudulent access to studies to improve the quality of their responses, Moss and Litman, 2018 notes that poor quality responses often originate from repeated geolocations, and found 60 locations where workers were submitting HITs tied to server farms.
After each collection batch, all cases where the attention checks were failed, where the survey was responded to under 6 minutes5, or where the written explanation did not seem coherent were manually inspected. Cases where the submission did not meet our quality standards were discarded.
Participants were provided with a pair of relatively similar sentences and then given the following queries:
• Without further context, would you consider it unfair for a context moderator to delete one of the above comments but not the other?
– It would be unfair to delete one of the comments, but not the other.
– Deleting comment a) would be more justifiable than deleting comment b).
5The median completion time rounded to 13 minutes.
– Deleting comment b) would be more justifiable than deleting comment a).
– It would be justifiable to delete either of the comments without deleting the other.
• What do you expect the average American to answer to the previous question?
– It would be unfair to delete one of the comments, but not the other.
– Deleting comment a) would be more justifiable than deleting comment b).
– Deleting comment b) would be more justifiable than deleting comment a).
– It would be justifiable to delete either of the comments without deleting the other.
The previously shown format was taken from Dorner et al., 2022, which they used for their entire crowdsourcing process. However, after a preliminary release of the survey, we came to believe that the original response options were causing confusion among workers. For this reason, we modified them to a simpler version and rearranged the order in which they are shown, as previously presented. The original options read as follows:
• Without further context, would you consider it unfair for a context moderator to delete one of the above comments but not the other?
– It would be unfair for one of the two comments but not the other to be deleted (Both comments ought to be treated the same way, no matter what)
– If comment a) was deleted, comment b) should be deleted as well, but not the other way around. (Deleting comment b) would be more justifiable)
– If comment b) was deleted, comment a) should be deleted as well, but not the other way around. (Deleting comment a) would be more justifiable)
– It would be acceptable to delete either of the comments without deleting the other. (Treating the comments differently either way would be fine)
Payment Independently of the platform used to hire workers, we compensate our participants with 2.40 USD per survey, which yields approximately 12 USD per hour, given that the median completion time rounded to 13 minutes. This exceeds the USA federal minimum wage of 7.25 USD per hour and is only exceeded by the state minimum salary of 13 states. To ensure the fairness of such compensation, we ran a preliminary smaller survey in which we analyzed the time required to respond to the survey and adjusted our compensation accordingly.
Analysis of Data Collected With and Without Timer A timer of 12 minutes was set during the first batch of our data collection task on MTurk which provided 140 approved participations. While the survey had a median completion time of 13 minutes in Prolific, some workers did complain about having too little time to complete the survey on MTurk. We removed the timer in susequent batches. We compare this batch to the rest of the data collected via MTurk to assess if there were significant differences and decide whether this data could still be useful for our research or not.
As shown in Table 8, the raw responses for both batches are quite similar for both the answers regarding personal opinions and the answers regarding average American opinions.
Furthermore, we analyzed the level of disagreement per sentence pair in both datasets, via the unalikeability coefficient (Kader and Perry, 2007). The timed dataset appeared to have much lower variability per response than the dataset without a timer, however this was caused by the size difference in both datasets. While there exist 1,500 sentence pairs in our dataset, in the batch collected with a timer 640 sentence pairs had one or fewer responses and 1,040 sentence pairs had two or fewer responses, which caused the low variability count
to rise. In comparison, only 20 sentence pairs had one or fewer responses and 100 sentence pairs had two or fewer responses in the dataset without a timer, which naturally raised the unalikeability coefficient of answers per sentence pair. For this reason, we compared the level of disagreement in both datasets by conducting a pairwise KolmogorovSmirnov test on the unalikeability values of both batches, by only taking into consideration the data from sentence pairs that had exactly 2 participations in the batch with timer and the data from pairs that had 2 participations in the batch without timer; we continued by comparing the data corresponding to sentence pairs with 3 participations and so on. There is no statistically significant evidence of both batches having different unalikeability distributions except for the responses on the average American opinions where each dataset had either 2 or 5 participations. All variability differences between responses regarding personal opinions are not significant. Table 9 shows the obtained p-values and statistics for all Kolmogorov-Smirnov tests.
Finally, given that some of the experiments performed in our work use the majority votes per sentence pair as labels, we analyze how such labels change when adding or removing the batch with a timer from the rest of the MTurk dataset. For the responses on the average American’s opinions, 55 out of 1500 majority votes change. In these 55 cases, the answers provided by the timer dataset represent on average 47% of the responses per sentence pair. In the case of the answers on personal opinions, 62 out of 1500 majority votes change. Here, the answers provided by the timer dataset represent on average 43% of the responses per sentence pair. This effect is even smaller when we also take into account the data obtained from Prolific. In this case, we have 11 and 16 majority vote changes for the answers on personal opinions and the answers on the average American opinions respectively.

heading: B Appendix B: Further Data Exploration
text: We successfully obtained higher label variability by selecting the sentence pairs that deemed higher unalikeability coefficients in Do. Figure 3 shows the count of unalikeability coefficients for the data collected through MTurk, Prolific and Do. In the figure we only include the sentence pairs from Do that have 9 labels, for adding sentence pairs with less responses would naturally skew the graph further left.
Regarding the demographic distribution of annotators, per Table 10, we observe quite different distributions for age, politics, education, and LGBTQ+ identity in Prolific’s and MTurk’s populations. Prolific’s sample is less conservative, less educated, and more sexually diverse than MTurk’s sample. Our sample’s demographic distribution does not align with that of the US population, which was expected; according to Levay et al., 2016, annotators on MTurk are on average 20 years younger than the ANES sample (Studies, 2021), with a larger proportion of Democrats, which results in an underrepresentation of the Republican population. Finally, Levay et al. report a more educated population than the ANES sample, which can also be observed in our dataset. We do not observe high correlations between demographic traits in Dn -the union of the data obtained from MTurk and Prolific. However, the female population is older than the male population6 and shows a higher level of sexual diversity7. Also, there is a much higher proportion of young, Non-White people than young White people8. On the other hand, the demographic distribution of Do+n is shown in Table 11. We observe a very similar distribution as the one presented for Dn, with the distinction that we cannot measure
660% of women and 50% of men had over 38 years. 720% of women and 13% of men identified as LGBTQ+. 861% of Non-White people and 40% of White people had
37 years or less.
for the education and LGBTQ+ identity variables. We observe a slight shift in the race variable, with an increment in the Black people category and a decrement in the White people category. On the other hand, the percentage of annotators under 38 years went from nearly 45% to 54%.
Figure 4 shows how a large amount of sentence pairs (s, s′) reach very high levels of disagreement; the average difference in the unalikeability levels between the responses for personal opinions and perceptions of American opinions being approximately 0.06. In other words, although the level of disagreement for questions regarding the annotators’ personal opinions is slightly higher, there tends to be a similar level of disagreement for both types of questions. Table 13 shows examples of sentence pairs with different levels of agreement for both types of questions. On other findings, only 4 out of 1500 sentence pairs show perfect agreement between both types of questions, and 8.5% of annotators state that all sentence pairs should be treated similarly. Often, the written explanation for this last scenario was that annotators did not believe in moderation, but rather preferred non-moderated freedom of speech. This finding is similar to the work by Kumar et al., 2021, who state that 30% of their crowdsourced participants stated they would never remove an online comment, regardless of its toxicity levels.
C Statistical Significance of Differences in the Unalikeability Coefficients by Sensitive Attribute Mentioned in Senstnece Pairs
We performed Kolmogorov-Smirnov tests comparing the unalikeability coefficients u2(s, s′) separating by the sensitive attribute mentioned in the sentence pairs, as described in The Concise Encyclopedia of Statistics (Kolmogorov–Smirnov Test, 2008), where the null hypothesis states that both sets of unalikeability coefficients have the same distribution. As displayed in Table 12, we observe low p-values across the board, demonstrating that the focus category likely affects the distribution of unlikeability coefficients.
D Implementation Details for Regression Analysis in Section 4
Using the glm function from the stats package in R (R Core Team, 2021), we create a binomial logistic classifier Cdembin (x1, ..., x6), where regressors xi, i ∈ [1, ..., 6] represent the gender, race, age, political stance, educational level, and LGBTQ+ identity of annotators and Cdembin (x1, ..., x6) represents the prediction of our labels in Dn as to observe if such variables are able to significantly explain part of our responses. Such procedure is performed twice,
accounting for personal opinions and perceptions of the average American opinion.

heading: E Challenges in the Implementation of Neural Classifiers
text: Before our data collection process, guiding ourselves with the experience reported by Dorner et al., 2022, we fit several BERT and RoBERTa models on dataset Do to find the best prospect models for our future data Dn. Table 14 and Table 15 show the architecture and performance of the top 4 classifiers with their performance. For these models, we used our binarized labels as previously described. Please note that dataset Do only contains the annotators’ perspectives on the average American opinions, and not their personal beliefs, which is why we only constructed models for such labels.
Once we finished the collection of dataset Dn, we proceeded to train and test the models Φ̂ with the new data. Unfortunately, as presented in Table 16, although these configurations had provided relatively high balanced accuracy scores in Do, in Dn they did not provide values much higher than 0.5, which implied practically no learning from our
models. For this reason, we decided to create a new grid with the values shown under Table 17, which we used to train and test several new architectures on Dn. As can be noted, by varying the batch size, learning rate, and optimizer, we obtain 36 different combinations of models that can be implemented in either BERT, RoBERTa, or DeBERTa classifiers. At the same time, since we are experimenting on Dn, each of these combinations must be fiton the labels representing the annotators’ personal opinions and on the labels on their perspective on the average American opinion. This resulted in a total of 864 models to be trained, which was unrealistic because of time and computational constraints. For this reason, we relied on heuristics to build the models that had the most promising results.
Unfortunately, none of the previously selected model architectures showed proper learning during the training phase with data Dn and simply classified the grand majority of the test examples as 0, even when performing oversampling, data augmentation or regularization methods, as dropout and weight decay.
After examining the performance of our models on different subsets of both datasets Dn and Do,
we found that our classifiers returned acceptable results when using the sentence pairs from Do that we purposefully excluded from Dn, but not otherwise. In that sense, since we had only included in Dn the sentence pairs that had higher levels of disagreement among annotators, we hypothesize that such sentence pairs are more difficult to learn by neural classifiers.
To test our hypothesis, we must observe if there are significant differences in the performance of models when tested on sentence pairs with different levels of disagreement among annotators. For this, we trained ten iterations of our best-performing model ϕ̂1 using the entirety of Do.
Then, given the nature of the labels in our models, we recalculated the unalikeability coefficients taking into account binary labels instead of their original four categories9. We created a set of unalikeability thresholds and for each, we evaluated
9Now u2(s, s′) ∈ [0.0, 0.5], since the new coefficient of unalikeability u2(s, s′) only considers two types of labels.
Model Modeltype Batch size Epochs Learning rate Optimizer
ϕ̂1 Roberta 16 5 1e-5 Adam ϕ̂2 Bert 16 5 1e-5 Adam ϕ̂3 Bert 32 4 1e-5 Adam ϕ̂4 Bert 8 7 1e-6 Adam
Table 14: Notes: Architecture of best-performing models on dataset Do
all iterations ϕ̂i, i ∈ [1, ..., 10] with the sentences from the ith test set which had an unalikeability coefficient equal or greater to the threshold. As shown in Figure 5, we observe a logarithmic decay in the performance of our models as unalikeability coefficients of the evaluated sentences increase, which could possibly explain the low performance of all tried architectures on dataset Dn. To support such a statement, we calculate the unalikeability coefficient when considering binary labels in datasets Dn and Do. For a fair comparison, we remove from Do all sentences that were only labeled by a
Model Gender Race
Female Male NonWhite
White
ϕ̂1 0.71 0.68 0.64 0.70 ϕ̂2 0.67 0.64 0.61 0.66 ϕ̂3 0.64 0.60 0.59 0.67 ϕ̂4 0.68 0.61 0.59 0.67
Table 15: Notes: Balanced accuracy scores of the bestperforming models on dataset Do, when trained and tested exclusively on data annotated by a specific category from the gender or race demographic.
single annotator therefore yielding an unalikeability coefficient equal to zero and skewing the graph further left. Figure 6 portrays the differences in the annotator’s disagreement in datasets Do and Dn.
Given the previously presented results, we create a new dataset Do+n, which is the union between Dn and DoC . Such dataset, apart from containing the data we collected, also takes into account the
datapoints from Do which refer to sentence pairs not seen in Dn. The original purpose of our data collection task was to be able to control for the influence exerted by each annotator in the labels of our dataset, given that the task performed by Dorner et al., 2022 did not require annotators to provide an equal amount of labels. With dataset Do+n, we are still able to control for the influence of each annotator in the case of the most controversial sentence pairs, for the sentence pairs with the highest unalikeability coefficients are contained in Dn. Unfortunately, there is a cost for well-performing models where the influence of each annotator is accounted for: we must lose the LGBTQ+ identity and Education data for our annotators, as well as their responses regarding their personal opinions, for this fields are not contained in Do. Dataset Do+n is used for all the neural classifiers presented in our Experiments section with the parameters shown in Table 18; its results and performance statistics may be found in Section 5.
Figure 5: Balanced Accuracy scores of ten iterations model ϕ̂1, when tested on data with different thresholds of unalikeability coefficients. The darker blue line represents the mean.
Figure 6: Distribution of unalikeability coefficients
u2(s, s ′) on the binarized labels.
F Implementation Details for Section 5
We created subsets of Do+n, grouping by the demographic categories of annotators. For each demographic variable, we created l + 1 subset datasets D(κ), where l is the number of categories in a demographic variable and κ represents the value of the demographic variable or the union of these values. (e.g. the gender variable has κ = Female and κ = Male and κ = Male and Female. Here, l = 2). For each demographic variable, each of the first l subsets thus contains data annotated solely by participants who belong to the lth category in that demographic variable. The l+1th dataset contained all datapoints, regardless of the annotators’ demographics, which we denote as the Control dataset. For each of theses subsets, we fitted pretrained multiheaded BERT based classifiers (Devlin et al., 2019) (building upon the code by Dorner et al., 2022) with Adam optimization by adjusting its number of epochs, learning rate, batch size, and classification threshold in the final layer. We refer
to the classification threshold as the threshold that is necessary for our last layer’s Sigmoid function to provide an output of 1 instead of 0. Conventionally, such a threshold would be set to 0.50, however, given the data imbalance in our dataset, our classifiers tend to favor labels 0. For this reason, in the final version of our models, we lowered such threshold in order to find a balance between the true positive rate and the true negative rate.
F.1 Compute
For the computation of our neural classifiers, we relied on a system equipped with GPU NVIDIA V100- SXM2 32 GB. All other results reported in this paper were obtained using a personal computer equipped with an 11th Generation Intel Core i7 processor, with 2.80GHz of speed, and 16GB of RAM. For the

heading: G Similarities Between Demographic Categories
text: We observe the distribution of raw answers per demographic category in Dn. As previously mentioned, all demographic categories show large levels of coincidence between their personal opinions and their perception of the average American opinion. Furthermore, across all demographics, we observe that Unfair is the most common label for both response types. For all demographics, stating that sentence s′ is worse than sentence s is slightly more common than its counterpart, which seems logical given that sentences s are real-world sentences while sentences s′ were generated by (Dorner et al., 2022) using different methodologies,
which often seemed to result in higher offensiveness levels.
The median duration of the survey for all participants across different demographic categories is quite similar. LGBTQ+ annotators were the fastest to provide their answers, having a median duration time of 12.2 minutes, while annotators over 38 were the slowest with a median of 13.4 minutes.
Table 10 also compares the proportions of different demographic groups in Dn.
Finally, Table 19 shows the unalikeability scores (for binarized answers: unfair/not unfair) by different demographic groups in D0+n. We observe that unalikeability scores do not differ strongly from one group to another, except for the case of annotators under 38 years of age and those over 38 years of age.
None of the above factors in isolation fully explain why the models perform better when evaluated on test data from specific demographic groups. It will be an interesting future work to investigate this observation further in detail. As mentioned in our paper, similar observation — models performing best on test data from specific demographic groups — have been made by others in different contexts (for e.g. by Binns et al., 2017); but, to the best of our knowledge, there is no conclusive explanation in the literature.

heading: Future Work
text: It will be an interesting future work to investigate this observation further in detail. As mentioned in our paper, similar observation — models performing best on test data from specific demographic groups — have been made by others in different contexts (for e. g. by Binns et al. , 2017); but, to the best of our knowledge, there is no conclusive explanation in the literature