Venue: NAACL 2024

# GRASP: A Disagreement Analysis Framework to Assess

Group Associations in Perspectives

 Vinodkumar Prabhakaran* &Christopher M. Homan* &Lora Arroyo* \\ Google Research &Google Research &Google Research &Google Research &Google Research &Google Research &Google Research &Google Research &Google Research &Google Research &Google Research &Google Research &Google Research &Google Research &Google Research &Google Research &Google

across intersectional race/ethnicity-gender groups. The lack of effective metrics that can capture such inter-group and intra-group cohesion at scale to determine group-level associations, is a critical issue.

In this paper, we propose GRASP (Group Associations in Perspectives), a framework to measure the magnitude and strength of systematic diversity of perspectives among rater subgroups. GRASP combines _a suite of metrics_ that measure group associations in annotations with _a permutation tests based significance testing approach_ that assesses the reliability of these associations without any independence assumptions. We apply GRASP to two datasets: DICES-350 (Aroyo et al., 2023)--350 chatbot conversations annotated for safety by 104 raters from a diverse pool across age, gender, and race; and D3 (Davani et al., 2023b)--social media comments annotated for offensiveness by 4000 raters balanced across cultural regions, gender, and age. GRASP reveals systematic disagreements in annotations along demographic lines, and shows that it picks up task-dependent group associations in an efficient and effective manner, furthering the objective of identifying meaningful diversity in annotator perspectives in subjective tasks.

## 2 Related work

Prior work on detecting harmful language, such as toxicity Pavlopoulos et al. (2020); Xenos et al. (2022), offensiveness Davidson et al. (2017), and hate speech Warner and Hirschberg (2012); Waseem and Hovy (2016), has led to curating datasets and developing models for social media content moderation Wulczyn et al. (2017); Founta et al. (2018); Vidgen et al. (2019). Recent advancements in conversational AI also increased attention to ensure safety and mitigate potential harms (e.g., Solaiman and Dennison, 2021; Xu et al., 2021; Shelby et al., 2022; Si et al., 2022; Bian et al., 2023; Huang et al., 2023; Santurkar et al., 2023). The latest generation of AI-driven language technologies OpenAI (2022; Google, 2022, 2023; Taori et al., 2023) is based on large language models OpenAI, 2023; Touvron et al. (2023) using reinforcement learning from human feedback (RLHF) Christiano et al. (2023); Ouyang et al. (2022). Studies show that on human alignment tasks, rater disagreement can be as high as 40% Ziegler et al. (2020). However, not much work has gone into developing scalable methods to meaningfully measure and tackle these high levels of rater disagreement.

Rater disagreement has a long history in NLP research as a challenge for crowd-sourced annotations and as a potential indication of human biases Arhin et al. (2021); Mathew et al. (2021); Sahoo et al. (2022); Wich et al. (2020). Though traditionally viewed as a mark of poor quality data, disagreement is increasingly seen as an important qualitative signal in its own right, one that is present in most tasks that requires human judgement Aroyo and Welty (2013); Hovy et al. (2013); Plank et al. (2014); Klenner et al. (2020); Basile et al. (2021); Weerasooniya et al. (2023).

Empirical analyses of inter-rater disagreements put forth raters' backgrounds and experiences as crucial to their annotations in such tasks, leading to systematic disagreements (e.g., Prabhakaran et al., 2021; Kumar et al., 2021; Denton et al., 2021; Sap et al., 2022; Biester et al., 2022; Deng et al., 2023; Homan et al., 2023; Pei and Jurgens, 2023). For instance, raters' demographics, including first language, age, and education, can significantly impact the performance of hate speech and abusive language detectors trained on that rater's behavior Al Kuwatly et al. (2020), and raters' stereotypes about different social groups and attitudes toward racism impact their annotations of hate speech and racist language targeting those groups Sap et al. (2022); Davani et al. (2023). Similarly, Davani et al. (2023) show that annotations of offensiveness vary across geo-cultural contexts.

Consequently, a large body of work has emerged to quantify, model, and measure rater disagreement e.g., Kairam and Heer (2016); Founta et al. (2018); Geva et al. (2019); Chung et al. (2019); Obermeyer et al. (2019); Liu et al. (2019); Weerasooriya et al. (2020); Uma et al. (2021); Weerasooniya et al. (2023). In early work, Hovy et al. (2013) introduce MACE, an unsupervised item-response model to capture raters' relative trustworthiness to more accurately aggregate annotations into a final label. Recently, Weerasooriya et al. (2020) proposed predictive models for rater disagreement that take into account sampling error, a common problem in datasets with very few annotations per item.

Novel modeling efforts have further incorporated raters' demographics and other background attributes to improve the predictions Hovy (2015); Garten et al. (2019); Hovy and Yang (2021). Using multi-task modeling frameworks, Fornaciari et al. (2021) add an auxiliary task to predict the soft label distribution over rater labels, Davani et al. (2022) model individual raters using a shared network to 

preserve their systematic disagreements until prediction, and Orlikowski et al. (2023) incorporates a group-specific layer to assess the benefits of socio-demographic attributes in modeling annotations. Hung et al. (2023) demonstrating the performance improvement when predicting raters' age and gender is coupled with language modeling objectives. Our work provides a framework that anchors on intra-group and inter-group cohesion to qualify the strength of disagreements within and across groups, and provide statistical tests to assess the reliability of these observed group-level patterns.

## 3 Group Associations in Annotations

As outline above, recent studies have established the need to account for systematic rater disagreement in subjective tasks by demonstrating socio-demographic differences in rater perceptions. However, systematic approaches to reliably assess _whether_ and _how much_ diversity axes impact disagreement for different tasks are still missing. To address this gap, we introduce a comprehensive analysis framework to measure statistically significant group associations within human annotations.

### Terminology

Let us represent a human-annotated dataset as a collection of _items_\({\bf X}\) with a corresponding collection of _annotations_\({\bf Y}\), obtained from a collection of _raters_\({\bf Z}\). Each row \({\bf X}_{i}\) is an item that is annotated, and each corresponding \({\bf Y}_{i}\) captures the annotations for \({\bf X}_{i}\). The columns in \({\bf Y}_{i}\) correspond to individual raters' annotations. In other words, \({\bf Y}_{ij}\) represent annotations by rater \(j\in{\bf Z}\) for item \(i\).1 In its simplest case, \({\bf Y}_{ij}\) can be a binary value, but it can be conceived as a vector capturing \(j\)'s responses to different questions pertaining to \(i\), or a one-hot encoding of \(j\)'s annotation in case of categorical values. Each row \({\bf Z}_{k}\) represents a rater \(k\) and the columns of \({\bf Z}_{k}\) contain _group attributes_ (e.g., demographic characteristics such as gender, race/ethnicity, and/or age associated with \(k\)). Let \(\Pi\) denote a set of demographic properties, e.g., \(\Pi=\{\text{gender = MALE, age = GenZ}\}\). Then, let \({\bf Z}[\Pi]\subseteq{\bf Z}\) denote the subpopulation of raters satisfying that property, and let \({\bf Y}_{{\bf Z}[\Pi]}\) denote the submatrix of \({\bf Y}\) that captures the annotations of that subpopulation of raters according to \(\Pi\).

Footnote 1: Note that \({\bf Y}_{ij}\) may be a sparse matrix if each item is labeled by only a handful of raters (which is often the case).

### Disagreement Analysis Framework

We aim to determine whether certain rater groups, defined in terms of their demographic attributes, systematically (and in statistically significant ways) differ from others in terms of their annotations for a given task. For this, we need to measure the (dis)agreement between raters within the group, as well as with those from outside the group.

**In-group Cohesion (\(C_{I}(Y)\))** captures how much cohesion a particular rarer group has among themselves. Formally, an _in-group cohesion_ metric is a mapping \(C_{I}:2^{\mathbf{Y}}\rightarrow\mathbb{R}\) where, for any subgroup of annotations \(Y\subseteq{\bf Y}\), higher values of \(C_{I}(Y)\) indicate higher levels of agreement among \(Y\). We are interested in \(C_{I}({\bf Y}_{{\bf Z}[\Pi]})\), the in-group cohesion among raters who satisfy the set of demographic properties \(\Pi\).

**Cross-group Cohesion (\(C_{X}(Y,Y^{\prime})\))** captures how much one rater group agrees with another rater group. Formally, a _cross-group cohesion_ metric is a mapping \(C_{X}:2^{\mathbf{Y}}\times 2^{\mathbf{Y}}\rightarrow\mathbb{R}\) where, for any pair of subgroups of annotations \(Y,Y^{\prime}\subseteq{\bf Y}\), higher values of \(C_{X}(Y,Y^{\prime})\) indicate higher levels of agreement between the annotations in \(Y\) and \(Y^{\prime}\). While cross-group cohesion could be calculated for any two given subsets of annotations, we are primarily interested in \(C_{X}({\bf Y}_{{\bf Z}[\Pi]},{\bf Y}_{{\bf Z}[-\Pi]})\), the cross-group cohesion between raters satisfying demographic properties \(\Pi\) and those who do not.

**Group Association Index (GAI):** Both _in-group_ and _cross-group cohesion_ are useful for assessing the strength of annotation patterns found in a demographic grouping \(\Pi\). For instance, high in-group cohesion within \({\bf Z}[\Pi]\) and cross-group cohesion between \({\bf Z}[\Pi]\) and \({\bf Z}[-\Pi]\) might just mean that the task has high agreement across the board. On the other hand, \({\bf Z}[\Pi]\) having both low in-group and cross-group cohesion might suggest that the raters in general have a hard time agreeing with one another, regardless of the specific grouping \(\Pi\). Inspired by graph-theoretic metrics for community detection in networks, such as _modularity_(Newman, 2006), we introduce a group association index that combines these two aspects into a single score:

\[GAI(\Pi)=\frac{C_{I}({\bf Y}_{{\bf Z}[\Pi]})}{C_{X}({\bf Y}_{{\bf Z}[\Pi]},{ \bf Y}_{{\bf Z}[-\Pi]})}\]

The baseline value of \(GAI\) is \(1\); i.e., when \(C_{I}\) and \(C_{X}\) are more or less the same, regardless of 

their magnitudes, the task annotation patterns have minimal or no group association with \(\Pi\). When \(C_{I}\) is larger than \(C_{X}\), the \(GAI\) values will be higher than 1, suggesting higher group association with \(\Pi\) for the task. On the other hand, for \(GAI\) values less than 1, raters agree more with raters outside the group than within the group, suggesting that there are potential patterns of systematic disagreement that are not captured by \(\Pi\).

**Diversity Sensitivity Index (DSI):** GAI indicates which groups significantly differ from others. There are numerous demographic axes (e.g., gender, age, race/ethnicity, sexual orientation, etc.) along which a rater pool can be diversified. When recruiting raters, which (if any) of these should be prioritized? It helps to know whether and by how much the subgroups within any axis have a significant \(GAI\). This is more insightful than the average \(GAI\) value. Hence, we define _diversity sensitivity index_ of a task w.r.t. a demographic axis with \(K\) groups as the \(\max\) of \(GAI(\Pi_{k})\) for \(k\in[1,K]\). Note that the statistical significance (see below) of the \(GAI\) value applies to the \(DSI\) value too; i.e., if the \(GAI\) value is not significant, the \(DSI\) is not either, and vice versa.

### Significance Testing

To ensure our diversity measurements are reliable, it is important to test their significance. Commonly used tests assume the data items are independently sampled, which doesn't hold in our case, since each annotation depends on all items with the same rater and all raters who annotated that item. So we use _permutation tests_ to control for these dependencies.

Null hypothesis:For any in-group cohesion (or cross-group divergence) metric \(C_{I}\) (or \(C_{X}\)), our null hypothesis \(\mathbf{H_{0}}\) is

\begin{tabular}{l l}   \(\mathbf{H_{0}}\): & Value of \(C_{I}\) (or \(C_{X}\)) for any (pair \\  & of) subpopulation(s) \(\mathbf{Y_{Z[II_{1}]}}\) (, \(\mathbf{Y_{Z[II_{2}]}}\)) \\  & is independent of demographic profile(s) \\  & of member(s) of \(\Pi_{1}\) (and \(\Pi_{2}\)). \\ \end{tabular} To test \(H_{0}\), we randomly shuffle the raters demographic profiles, measure the test statistic after each shuffle, and then count how many times the shuffled statistic exceeds the observed value. If the observed value is significant, then _only a small percentage of the measurements from random groups should exceed the observed value_. Formally, p-value of \(C_{I}\) is defined as:

\[p_{C_{I}}(\mathbf{Y_{Z[II_{1}]}})=_{\mathrm{def}}\\ \\ \left\{\begin{array}{rl}\|\{s^{*}_{i}:s^{*}_{i}<C(\mathbf{Y_{ Z[II_{1}]}})\}\|/N\\ \text{if $C(\mathbf{Y_{Z[II_{1}]}})<s^{*}_{[N/2]}$},\\ \|\{s^{*}_{i}:s^{*}_{j}>C(\mathbf{Y_{Z[II_{1}]}})\}\|/N\\ \text{otherwise.}\end{array}\right.\]

where \(N\) is a large number and \(s^{*}_{1},\ldots,s^{*}_{N}\) are computed by the following pseudocode:

\begin{tabular}{l l}   \(i\gets 0\) \\   \(\mathbf{while}\) & \(i<N\)do \\   \(\mathbf{Z}^{*}\leftarrow\) randomly permute the rows of \(\mathbf{Z}\) (but fix the indices, so that the rows map to the same annotations even though their demographics have changed) \\   \(i\gets i+1\) \\   \(s^{*}_{i}\gets C(\mathbf{Y_{Z^{*}[II_{1}]}})\) \\   **end while** \\  & reorder \(s^{*}_{1},\ldots,s^{*}_{N}\) in ascending order. \\ \end{tabular} The p-value \(p_{C_{X}}(\mathbf{Y_{Z[II_{1}]}},\mathbf{Y_{Z[II_{2}]}})\) of \(C_{X}\) is defined as above, except that we replace \(C_{I}(\mathbf{Y_{Z[II_{1}]}})\) with \(C_{X}(\mathbf{Y_{Z[II_{1}]}},\mathbf{Y_{Z[II_{2}]}})\) (and \(C_{I}(\mathbf{Y_{Z^{*}[II_{1}]}})\) with \(C_{X}(\mathbf{Y_{Z^{*}[II_{1}]}},\mathbf{Y_{Z^{*}[II_{2}]}})\)).

Multiple test correction:If numerous tests are conducted and the null hypothesis is true, then by the Law of Large Numbers some of them are likely to have small p-values, making them falsely appear to be significant (type I error). There is no widely accepted best practice for dealing with this problem. Some researchers advocate never using p-values for exploratory research Hak (2014); Trafimov and Marks (2015) or to apply corrections such as Bonferroni Bonferroni Bonferroni (1936); Holm (1979) against the family-wise error rate. Other researchers see those approaches as too restrictive, which can lead to important discoveries being missed Gaus et al. (2015); Goeman and Solari (2011); Rubin (2017). We adopt a mixed approach and report two levels of significance: significance with no correction whatsoever and with Benjamini-Hochberg _false discovery rate (FDR)_ correction Benjamin and Hochberg (1995).

### Metrics

The concepts introduced in SS3.2 are _metric-agnostic_, and the choice of metric must be justified for each experiment. Here, we describe the three kinds of metrics we use in this paper for both \(C_{I}\) and \(C_{X}\); we compare and contrast what these metrics are sensitive to and what they reveal 



#### 3.4.1 In-group Cohesion Metrics

IRR:We use IRR (Inter-rater reliability, particularly, Krippendorff's alpha Krippendorff 2004) to measure within-group agreement while controlling for class imbalance. Krippendorff's alpha has an advantage over other IRR metrics: it can handle an arbitrary number of raters, answer options and items at one time, and it unifies and generalizes a number of other IRR metrics, including Scott's pi and Fleiss' kappa (Krippendorff, 2004). It is formulated as \(1-\frac{o_{d}}{e_{d}}\), where \(o_{d}\) is the mean observed disagreement between pairs of distinct raters, and \(e_{d}\) is the class-imbalance-controlling term. The \(o_{d}\) term is, effectively, hamming distance and \(e_{d}\) is the expected amount of disagreement, under the assumption that each rater's responses are randomly distributed among the conversations they label (but each rater's marginal distribution of annotations is fixed), independent of the other raters' responses.

Plurality size:IRR and our many other metrics are based on counting the (dis)agreements between pairs of raters. But in practice, raters are often seen as populations whose annotations are taken as _votes_, where the most popular annotation (i.e., majority vote) becomes the gold standard response. Thus, a very natural measurement of agreement is the fraction of raters who belong to the most popular choice (similar to Prabhakaran et al. 2021's approach). This metric is less sensitive to class imbalance than metrics that count pairwise disagreements. It is computed by iterating over each item, taking the argmax over the distribution of responses, and then taking its mean over all pairs.

Negentropy:IRR measures pairwise agreement between raters and plurality size captures the impact of disagreement in the rating aggregation process. Another common way to measure disagreement in groups, used in polls and surveys, is to estimate the distribution of annotations associated with each item. Entropy is a common metric for measuring the randomness of a probability distribution, such as the annotations from multiple raters to a safety question about a conversation. It captures how evenly distributed the ranges of responses are. To orient all our metrics so that larger numbers mean more agreement, we report _negentropy_(Brillouin, 1953): for each conversation, we compute the entropy over the distribution of responses. Then we subtract this from the maximum value entropy can take over the response domain. For a domain with \(n\) possible responses, this number is \(\ln n\). Finally, we take the mean over all conversations.

#### 3.4.2 Cross-group Divergence Metrics

Analogous to our in-group cohesion metrics, we focus on three cross-group cohesion metrics.

XRR:_Cross-replication reliability_(Wong et al., 2021) is similar to Krippendorff's alpha, except that the pairs of raters being compared come from separate groups. Like alpha, XRR can handle arbitrary numbers of raters, answer options and items. And it also controls for class imbalance.

Voting agreement:For across-group agreement, it is equally natural, by analogy to plurality size, to compare two groups as if they were voting blocks. For each item, we compute the plurality choice for each group. To account for class imbalance, we compute Krippendorff's alpha over all conversations between the two groups, based on each group's plurality choices. Although straightforward, we are not aware of this method proposed as a group-level divergence metric.

Cross-negentropy:Cross-entropy is algorithmically similar to entropy but is computed over two distributions, not one. We define cross-negentropy in an analogous manner to negentropy.

## 4 Experiments

### Data

We apply our metrics to the two datasets: **DICES-350**(Aroyo et al., 2023),2 and **D3**(Davani et al., 2023b). The DICES-350 dataset is a curated sample of 8k multi-turn conversation corpus generated by human agents interacting with a generative AI-chatbot (Thoppilan et al., 2022) in an adversarial setting. These conversations were then annotated for safety by a diverse rare pool. The D3 dataset contains a curated sample of social media posts from Jigsaw datasets (Jigsaw, 2019, 2018), annotated for offensiveness in text. We choose the DICES-350 and D3 datasets as they both contain fully replicated annotations from a diverse rare pool along with their demographic details, enabling our in-depth and fine-grained group-level analyses.

Footnote 2: [https://github.com/google-research-datasets/diices-dataset/tree/main/350](https://github.com/google-research-datasets/diices-dataset/tree/main/350)

DICES-350contains annotations for safety along 16 dimensions for all 350 conversation by 123 unique raters based in the US. The authors of

DICES-350 aimed for an approximately equal numbers of raters in each of the 12 demographic groups (3 x 4 design) created by fully crossing age groups (GenZ, Millennial, GenX+) with race/ethnicity (Asian; Black; Latine/x; White). All raters annotated all 350 conversations. We limit our study to 104 raters after removing 19 raters who were deemed unreliable by the authors of DICES-350. See Table 2 for breakdowns of the demographic groupings along race, gender, and age.

The safety annotation dimensions cover a variety of safety violations, including harmful content, unfair bias, misinformation, and political endorsements, and raters may respond _Safe_, _Unsafe_, or _Unsure_. We compute a single safety response for each rater-conversation pair by aggregating the responses into a single, overall safety response. For any conversation, if _any_ of the safety annotations is _Unsafe_, then we label the entire conversation as unsafe. Otherwise, if any of the safety annotations is _Unsure_, then so is the aggregated response. Otherwise, the aggregated response is _Safe_. In other words, it only takes one reason for a conversation to be unsafe and, conversely, if a conversation is unsafe, it need only be unsafe for one reason.

D3is similarly annotated by a diverse pool of 4k raters across 8 geo-cultural regions and 21 countries. Each item in the dataset was annotated by at least three raters in each region (\(\sim\)24 annotations per item). The annotation effort aimed for capturing an approximately equal number of raters (\(\sim\)450) from each region and equal ratio of representation for various demographic group across age (18 to 30, 30 to 50, and more than 50 years old) and genders [7]. See Table 3 for the breakdown of the demographic groups across different regions, gender, and age groups.

Raters were asked to label the textual items' level of offensiveness on a 5-point Likert scale, 1 being _not offensive at all_ and 5 being _extremely offensive_, with the option of choosing _Unsure_. We treated a score of 3 or higher as being _Offensive_, in line with the dataset creators [6].

### Results

We report results of our analysis using IRR and XRR as the in-group and cross-group cohesion metrics in for both DICES-350 and D3 datasets in Table 4. We focus on IRR and XRR based analysis in this section, but the full results using all other metrics are presented in Tables 5, 6, and 7.

We investigate groupings along age, gender, and either race/ethnicity (DICES-350) or region (D3). For DICES-350, we also explore intersectional groups along race/ethnicity and gender (some of the intersections of age and race/ethnicity are too small to reasonably assess significance), while we explored the intersection of region with both age and gender groups in the D3 dataset. Results for all intersections and statistically significant intersections are reported in Tables 5-7 and 4, respectively.

 

women (0.213). Asian women have the lowest score (0.073), followed by White women (0.114). Latine women also have the highest cross-group cohesion (0.199), followed by Latine raters (0.189). Asian women have the lowest score (0.134), followed by White women (0.152) and White raters (0.159). White men have the highest GAI score (1.262) followed by Latine women (1.196), Latine raters (1.139), and Black women (1.130). Some groups have GAIs significantly lower than baseline; Asian women have the lowest GAI (0.540), followed by White women (0.752), suggesting that these groups have constituent subgroups that have more agreement with raters outside this group.

The DSI metric looks at what is the highest GAI for each diversity axis (including intersectional axes) we consider. In the DICES-350, we observe the higher DSI for the intersectional axis of gender and race (1.262 for White men), followed by race considered alone (1.139 for Latinre raters). These numbers suggest that it is crucial to prioritize recruiting raters with a diverse representation along race and gender, while diversifying along age may be less crucial based on our results for this task. Note that, although unlikely, applying our framework along other intersectional axes including age may reveal other group associations.

D3 results:Here, 18-to-30-year-old Western Europeans have the highest IRR (0.177), followed North American women (0.153) and Western European women (0.151). Lowest scores are reported for 30-to-50-year-old raters from Sinosphere (0.033) and Indian Cultural Sphere (0.060), followed by 18-to-30-year-old (0.063), and women (0.070) groups from Indian Cultural Sphere. 18-to-30-year-old Western Europeans also have the highest XRR (0.126) followed by non-significant scores for Western European women (0.118) and North American women (0.116). Lowest XRR is reported for 50+-year-old raters of Sinosphere (0.061), followed by Sinosphere women (0.081) and 30-to-50-year-old Sinosphere raters (0.082), all significant after BH corrections. In terms of GAI scores, 50+-year-old raters of Sinosphere (2.225), and raters identifying with non-binary genders (2.172) report the highest GAI, followed by 18-to-30-year-old groups in Western European (1.402) and SubSaharan Africa (1.365); all significant after BH correction. Notably, unlike the DICES-350, different age and region groups have significantly high GAI scores; 18-to-30-year-old (1.068), North America (1.307), Sub Saharan Africa (1.361), and Western Europe (1.222). Interestingly, intersectional results demonstrate that while women in general did not report high GAI, subgroups of women in different regions show more in-group agreement.

\begin{table}
\begin{tabular}{l l c c c} \hline  & \multicolumn{4}{c}{**DICES-350**} \\ \hline Dimension & Group & IRR & XRR & GAI \\ \hline age & gen x+ & \(\downarrow\)0.166 & \(\downarrow\)0.171 & \(\downarrow\)0.975 \\ age & gen z & \(\downarrow\)0.166 & \(\downarrow\)0.172 & \(\downarrow\)0.966 \\ age & millenial & \(\uparrow\)0.189 & \(\uparrow\)0.179 & \(\uparrow\)1.052 \\ \hline gender & Man & \(\uparrow\)0.187 & \(\uparrow\)0.175 & \(\uparrow\)1.071 \\ gender & Woman & \(\downarrow\)0.160 & \(\uparrow\)0.175 & \(\downarrow\)0.916 \\ \hline race & As. & \(\downarrow\)0.145 & \(\downarrow\)0.166 & \(\downarrow\)0.872 \\ race & Bl. & \(\uparrow\)0.193 & \(\uparrow\)0.181 & \(\uparrow\)1.063 \\ race & Lat. & **0.215*** & **0.189*** & \(\uparrow\)1.139* \\ race & Multi. & \(\downarrow\)0.153 & \(\uparrow\)0.168 & \(\downarrow\)0.916 \\ race & Wh. & \(\downarrow\)0.145 & **0.159*** & \(\downarrow\)0.908 \\ \hline \multicolumn{4}{c}{Statistically Significant Intersections} \\ \hline race, gender & As., Woman & \(\downarrow\)**0.073*** & \(\downarrow\)**0.134*** & \(\downarrow\)**0.540*** \\ race, gender & Bl., Woman & \(\uparrow\)**0.213*** & \(\uparrow\)0.188 & \(\uparrow\)**1.130*** \\ race, gender & Lat., Woman & \(\uparrow\)**0.238*** & \(\uparrow\)**0.199*** & \(\uparrow\)**1.196*** \\ race, gender & Wh., Man & \(\uparrow\)**0.218*** & \(\downarrow\)0.173 & \(\uparrow\)**1.262*** \\ race, gender & Wh., Woman & \(\downarrow\)**0.114*** & \(\downarrow\)**0.152*** & \(\downarrow\)**0.752*** \\ \hline \multicolumn{4}{c}{**D3**} \\ \hline Dimension & Group & IRR & XRR & GAI \\ \hline age & (18,30) & **0.115*** & \(\uparrow\)0.107 & \(\uparrow\)**1.068*** \\ age & (30,50) & **0.089*** & \(\downarrow\)0.104 & \(\downarrow\)**0.850*** \\ age & 50+ & \(\uparrow\)0.110 & \(\uparrow\)0.111 & \(\uparrow\)0.999 \\ \hline gender & Woman & \(\uparrow\)0.110 & \(\uparrow\)0.108 & \(\uparrow\)1.024 \\ gender & Man & \(\downarrow\)0.105 & \(\uparrow\)0.107 & \(\downarrow\)0.976 \\ gender & Other & \(\uparrow\)0.209 & \(\downarrow\)0.096 & \(\uparrow\)**2.172*** \\ \hline region & AC. & **0.133*** & \(\uparrow\)0.113 & \(\uparrow\)**1.174*** \\ region & ICS. & \(\downarrow\)0.103 & \(\downarrow\)**0.099*** & \(\uparrow\)1.043 \\ region & LA. & \(\uparrow\)**0.129*** & \(\uparrow\)0.112 & \(\uparrow\)**1.152*** \\ region & NA. & \(\uparrow\)**0.143*** & \(\uparrow\)0.110 & \(\uparrow\)**1.307**** \\ region & Oc. & \(\uparrow\)0.118 & \(\downarrow\)0.103 & \(\uparrow\)**1.145*** \\ region & Si. & \(\downarrow\)**0.087*** & \(\uparrow\)**0.102*** & \(\uparrow\)**1.002*** \\ region & SSA. & \(\uparrow\)**0.142*** & \(\downarrow\)0.104 & \(\uparrow\)**1.361*** \\ region & WE. & \(\uparrow\)**0.135*** & \(\uparrow\)0.111 & \(\uparrow\)**1.222*** \\ \hline \multicolumn{4}{c}{Statistically Significant Intersections} \\ \hline region, age & ICS., (18,30) & **0.063*** & \(\downarrow\)0.100 & \(\downarrow\)**0.634*** \\ region, age & Ics., Os., (90) & **0.060*** & \(\downarrow\)0.100 & \(\downarrow\)**0.601*** \\ region, gender & ICS., Woman, & **0.070*** & \(\downarrow\)0.106 & \(\downarrow\)**0.655*** \\ region, age & LA., (18,30) & \(\uparrow\)**0.143*** & \(\uparrow\)0.118 & \(\uparrow\)**1.216*** \\ region, gender & LA., Woman & **0.143*** & \(\uparrow\)0.111 & \(\uparrow\)**1.290*** \\ region, gender & No., Woman & **0.153*** & \(\uparrow\)0.116 & \(\uparrow\)**1.314*** \\ region, age & Oc., (0.50) & \(\uparrow\)0.112 & \(\uparrow\)**0.089*** & \(\uparrow\)**1.1525*** \\ region, gender & Oc., Woman & \(\uparrow\)**0.133*** & \(\uparrow\)0.110 & \(\uparrow\)**1.208*** \\ region, age & Si., (30,50) & **0.033*** & \(\downarrow\)**0.082*** & \(\downarrow\)**0.405*** \\ region, age & Si., (30,50) & \(\uparrow\)0.137 & \(\downarrow\)**0.061*** & \(\uparrow\)**2.225*** \\ region, gender & Si., Women & \(\downarrow\)0.100 & **0.081*** & \(\uparrow\)**1.237*** \\ region, age & SSA., (18,30) & \(\uparrow\)**0.146*** & \(\downarrow\)0.107 & \(\uparrow\)**1.365*** \\ region, age & WE., (18,30) & \(\uparrow\)**0.177*** & \(\uparrow\)**0.126*** & \(\uparrow\)**1.402*** \\ region, gender & WE., Woman & \(\uparrow\)**0.151*** & \(\uparrow\)**0.118** & \(\uparrow\)**1.284*** \\ \hline \end{tabular}
\end{table}
Table 4: Results for in-group and cross-group cohesion, and GAI. Significant results are in **bold**: * for significance at \(p<0.05\), **for** significance after Benjamini-Hochberg correction. A \(\downarrow\) (or \(\uparrow\)) means that the result is less (or greater) than expected under the null hypothesis. GAI results based on \(C_{X}=\text{XRR}\) and \(C_{I}=\text{ 

We observe the highest DSI for the intersectional axis of region and age (Sinosphere, 50+) at 2.225, followed by a high DSI for gender (Other) at 2.172. This shows the importance of prioritizing raters from non-binary gender groups and specific subgroups along region and age to capture important diverse perspectives in assessing offense.

## 5 Discussion

We propose the GRASP framework that provides a means to assess the cohesion and strength of group associations along different axes of diversity that matter for a given task, identifying different groups, including intersectional groups, that are relevant for specific tasks. GRASP is generic and versatile, however different task and data settings can lead to different results in group associations, depending on the metrics we use. Our intent in presenting this breadth of results using different metrics is both to show the versatility of the framework w.r.t. underlying metrics, and also to demonstrate the differences that different underlying metrics yield. In practice, one should choose the metrics suitable to the specific task and data characteristics (e.g., number of raters, replication factor, and data skew).

**Task specific insights:** Our analysis provides insights about specific rater groups for each task. For instance, in the conversational safety task (DICES-350), White men having the highest and Asian women the lowest in-group cohesion. Interestingly, White women and Asian men had opposite cohesion trends from their alter-genders. This suggests that men are driving the high cohesion observed in White raters, and that women and men counteract each other in the weak effects observed in Asian raters overall. High coherence among White men is due to their strong tendency to prefer _Safe_ to _Unsafe_ annotations by a nearly \(3:1\) ratio. On the other hand, for the offense annotation task (D3), most regional groups show significant group associations. Notably, Indian cultural sphere and Sinosphere shows no significant in-group cohesion (or GAI), although 50+ groups within Sinosphere show high in-group cohesion. Age is a notable factor across board, both individually and within intersectional groups, suggesting the need for diversification of rater pools around age groups.

**Flexibility of group granularity:** Our analysis is generic enough that it can be applied groups defined by any subset of demographic characteristics, enabling it to easily reveal intersectional group associations. For instance, although age and gender groups revealed no association for safety, intersectional analysis revealed that gender plays a substantial role in driving race-level group tendencies.

**Flexibility of metrics**: Our framework is extensible to any (comparable) underlying in-group cohesion and cross-group divergence metrics. We observe that the values across our metrics vary (see Table 5 & 6); IRR numbers are relatively low (around \(0.2\)) while other metrics report much higher agreements. These disparities may point to potential overcompensation for class imbalance (\(2:1\) for _safe_ to _unsafe_) in the IRR metric. IRR is typically used to compare small groups of raters. With larger groups of raters there are quadratically more pairs of raters, and the high dimensionality of the response vectors (350 responses per rater) means that all pairs can potentially be very different from each other: there is both more space to disagree and more disagreements to count. Negentropy and plurality size are less sensitive to these effects, since they are both based on the distributions of all raters, not on the pairwise relationships between all raters. Future work should look into which metrics may be more suitable in specific task and data settings (e.g., number of raters, replication factor, etc.).

**Versatility across dataset characteristics**: The two datasets we applied our framework to differ not only on the underlying tasks, but also on other dataset characteristics/structure. DICES-350 contains fully parallel annotations (i.e., all 104 annotators annotated all 350 items), whereas D3 contains batches of annotations where sets of 35 items contain fully parallel annotations from 24+ raters. These differences did not hinder the applicability of the analysis framework. In fact, the D3 analysis provides a potential pathway where such highly parallel annotations by broadly diverse rarer pools could be performed in early phases, that can then inform more streamlined data collection through curated rater pools representing selected diversity axes based on this analysis, essentially saving cost while ensuring diversity in data.

**Exploratory Analysis:** Our approach also illustrate the usefulness of significance testing to exploratory analysis. We see the role of significance testing in exploratory research as a compass that provides perspective in light of conflicting results that lack inherent scales for interpretation. While they impose a hefty computational burden, the permutation tests control for joint dependencies in the data between raters and conversations that simpler 

tests do not. However, we believe the extra computational effort is well worth the trouble, especially in informing rater recruitment decisions.

## 6 Conclusion

We introduced GRASP, an analytical framework to measure systematic diversity in annotations among rater subgroups, to better understand the socio-cultural leanings of subjective tasks. We proposed a group association index that combines in-group and cross-group cohesion, along with statistical significance using permutation tests. Applying this framework to two datasets of subjective annotations, we demonstrated how it reveals systematic disagreements across various intersectional subgroups. Our work contributes to the efforts on bringing in diverse perspectives in data in an efficient and effective manner, furthering the goal of robust socio-technical evaluations of AI models.

Future work could explore adopting our framework to measure systematic disagreements between rarer groups in other subjective tasks (e.g., Kumar2022) to further validate its utility. Furthermore, our analysis framework provides actionable insights for practitioners to help prioritize demographic axes when diversifying rater pools. Future work could investigate how the framework may enable dynamic data collection that can adapt to emergent group associations among raters across different types of content and tasks. While our framework identifies systematic disagreements between groups, further investigation is needed to understand the underlying reasons that cause these disagreements, for instance, individual moral values Davani et al. (2023b).

## 7 Limitations

We acknowledge that the demographic breakdown in both datasets is a simplified representation of the population at large. We assume this was done to facilitate recruitment of raters in each group and to allow for less complexity in analysing intersecting groups. However, our analysis framework was applied on two independent datasets with different rater pools, demographic breakdowns and data collection designs, which points to its generalizability. Provided more granular demographic data, we are confident the frameworks can be readily applied.

We recognize that further research is needed to extend such analysis to other intersectional groups that we have not been investigated in this paper. For example, we believe that further slicing the ethnicity, native languages and age groups is likely to reveal further insights and provide additional evidence of systematic differences between different groupings of raters. Due to page limit this paper focuses on introducing the disagreement analysis framework, and provide initial analysis to demonstrate its utility in revealing significant group associations along socio-demographic lines.

Finally, we recognize more work is needed to distinguish _good_ from _bad_ disagreement. We focused on revealing statistically significant cohesion within groups (and lack of it across groups), which may weed out noisy disagreements. However, more work is needed to disentangle disagreements that are important to retain in the interest of retaining diverse perspectives, vs. those that are undesirable from a practitioners' perspective (e.g., lack of training in a particular rater platform/pool).

While the use of significance tests in exploratory analysis is controversial Balluerka et al. (2005), there is usually a degree of arbitrariness in their use, for instance, in the choice of level (e.g., \(p=0.05\), in our case), if nothing else. In the case of exploratory research such as ours, one must be careful not to abuse significance testing. For instance, we deliberately held back on a deeper exploration of intersectionality to reduce the risk of p-hacking (see discussion in SS 3.3, 4.2). We also note that we have many more significant results at the \(p=0.05\) level than chance would predict. There is also arbitrariness in the metrics used. For instance, there isn't uniform agreement on how to interpret well-established metrics such as Krippendorff's alpha.

## 8 Statement of Ethics

Collecting and analyzing socio-demographic information of annotators raise significant ethical considerations. Hence, all the demographics data present in DICES-350 Aroyo et al. (2023) and D3 Davani et al. (2023b) datasets are self-declared. Raters were presented a consent form before signing up for both studies to inform them about the gathering of personal demographics and that the content to be rated is adversarial (i.e., would possibly contain offensive content). All demographics questions had the option "Prefer not to answer". All data was collected in an anonymized fashion. Raters were allowed to quit the study at any time. Similar precautions should be taken while building new datasets with socio-demographic information 



## Acknowledgements

We thank Chris Welty and Katherine Heller for valuable feedback on early drafts of the manuscript. We also thank anonymous reviewers for their constructive feedback during the peer review process.

## References

* Al Kuwatly et al. (2020) Hala Al Kuwatly, Maximilian Wich, and Georg Groh. 2020. Identifying and measuring annotator bias based on annotators' demographic characteristics. In _Proceedings of the Fourth Workshop on Online Abuse and Harms_, pages 184-190.
* examining the challenges with annotating toxic text datasets.
* Aroyo et al. (2023) Lora Aroyo, Alex S. Taylor, Mark Diaz, Christopher Michael Homan, Alicia Parrish, Greg Serapiogarcia, Vinodkumar Prabhakaran, and Ding Wang. 2023. DICES Dataset: Diversity in conversational AI evaluation for safety.
* Aroyo and Welty (2013) Lora Aroyo and Chris Welty. 2013. Crowd truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard. _WebSci2013. ACM_, 2013(2013).
* Aroyo and Welty (2015) Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. _AI Magazine_, 36(1):15-24.
* Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Das Sarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Ellage, Zae Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Krawec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback.
* Balluerka et al. (2005) Nekane Balluerka, Juana Gomez, and Dolores Hidalgo. 2005. The controversy over null hypothesis significance testing revisited. _Methodology_, 1(2):55-70.
* Basile et al. (2021) Valerio Basile, Michael Fell, Tommaszo Fornaicari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. We need to consider disagreement in evaluation. In _Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future_, Online. Association for Computational Linguistics.
* Benjamini and Hochberg (1995) Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. _Journal of the Royal statistical society: series B (Methodological)_, 57(1):289-300.
* Bian et al. (2023) Ning Bian, Peilin Liu, Xianpei Han, Hongyu Lin, Yaojie Lu, Ben He, and Le Sun. 2023. A drop of ink may make a million think: The spread of false information in large language models. _arXiv preprint arXiv:2305.04812_.
* Biester et al. (2022) Laura Biester, Vanita Sharma, Ashkan Kazemi, Naihao Deng, Steven Wilson, and Rada Mihalcea. 2022. Analyzing the effects of annotator gender across NLP tasks. In _Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022_, pages 10-19, Marseille, France. European Language Resources Association.
* Bonferroni (1936) Carlo Bonferroni. 1936. Teoria statistica delle classi e calcolo delle probabilita. _Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commericiali di Firenze_, 8:3-62.
* Brillouin (1953) Leon Brillouin. 1953. The negentropy principle of information. _Journal of Applied Physics_, 24(9):1152-1163.
* Christiano et al. (2023) Paul Christiano, Jan Leike, Tom B. Brown, Miljan Marcite, Shane Legg, and Dario Amodei. 2023. Deep reinforcement learning from human preferences.
* Chung et al. (2019) John Joon Young Chung, Jean Y Song, Sandhu Kutty, Sungsoo Hong, Juho Kim, and Walter S Lasecki. 2019. Efficient elicitation approaches to estimate collective crowd answers. _CSCW_, pages 1-25.
* Davani et al. (2023a) Aida Mostafazadeh Davani, Mohammad Atari, Brendan Kennedy, and Morteza Dehghani. 2023a. Hale speech classifiers learn normative social stereotypes. _Transactions of the Association for Computational Linguistics_, 11:300-319.
* Dayani et al. (2022) Aida Mostafazadeh Davani, Mark Diaz, and Vinoddukumar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. _Transactions of the Association for Computational Linguistics_, 10:92-110.
* Davani et al. (2023b) Aida Mostafazadeh Davani, Mark Diaz, Dylan Baker, and Vinodkumar Prabhakaran. 2023b. Disentangling perceptions of offensiveness: Cultural and moral correlates.
* Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In _Proceedings of the international AAAI conference on web and social media_, volume 11, pages 512-515.
* Deng et al. (2023) Naihao Deng, Xinliang Zhang, Siyang Liu, Winston Wu, Lu Wang, and Rada Mihalcea. 2023. You are what you annotate: Towards better models through annotator representations. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 12475-12498, Singapore. Association for Computational Linguistics

Emily Denton, Mark Diaz, Ian Kivlichan, Vinodkumar Prabhakaran, and Rachel Rosen. 2021. Whose ground truth? accounting for individual and collective identities underlying dataset annotation. _arXiv preprint arXiv:2112.04554_.
* Commission (2020) European Commission. 2020. The digital services act: Ensuring a safe and accountable online environment. _The Digital Services Act: Ensuring a safe and accountability online environment_.
* Fornaciari et al. (2021) Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. 2021. Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2591-2597.
* Founta et al. (2018) Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kovutellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. In _Proceedings of the international AAAI conference on web and social media_, volume 12.
* Garten et al. (2019) Justin Garten, Brendan Kennedy, Joe Hoover, Kenji Sagae, and Morteza Dehghani. 2019. Incorporating demographic embeddings into language understanding. _Cognitive science_, 43(1):e12701.
* Gaus et al. (2015) Wilhelm Gaus, B Mayer, and R Muche. 2015. Interpretation of statistical significance-exploratory versus confirmative testing in clinical trials, epidemiological studies, meta-analyses and toxicological screening (using Guiskgo biloba as an example). _Clinical & Experimental Pharmacology_, 5(4):182-187.
* Geva et al. (2019) Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 1161-1166, Hong Kong, China. Association for Computational Linguistics.
* Glaese et al. (2022) Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoeeb Thacker, Lucy Campbell-Gillingham, Jonathan Uessato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Datathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soha Mokra, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Isson Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. 2022. Improving alignment of dialogue agents via targeted human judgements.

* Google (2022) Google. 2022. Pathways language model (PaLM): Scaling to 540 billion parameters for breakthrough performance.
* Google (2023) Google. 2023. PaLM 2 technical report.
* Hak (2014) Tony Hak. 2014. After statistics reform: Should we still teach significance testing? _ERIM Report Series Reference No. ERS-2014-001-ORG_.
* Holm (1979) Sture Holm. 1979. A simple sequentially rejective multiple test procedure. _Scandinavian journal of statistics_, pages 65-70.
* Homan et al. (2023) Christopher Homan, Greg Serapio-Garcia, Lora Arroyo, Mark Diaz, Alicia Parrish, Vinodkumar Prabhakaran, Alex S. Taylor, and Ding Wang. 2023. Intersectionality in conversational AI safety: How Bayesian multilevel models help understand diverse perceptions of safety.
* Hovy (2015) Dirk Hovy. 2015. Demographic factors improve classification performance. In _Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (Volume 1: Long papers)_, pages 752-762.
* Hovy et al. (2013) Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with mace. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1120-1130.
* Hovy and Yang (2021) Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 588-602.
* Huang et al. (2023) Fan Huang, Haewoon Kwak, and Jisun An. 2023. Is ChatGPT better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. In _Companion Proceedings of the ACM Web Conference 2023_, WWW '23 Companion, page 294-297, New York, NY, USA. Association for Computing Machinery.
* Hung et al. (2023) Chia-Chien Hung, Anne Lauscher, Dirk Hovy, Simone Paolo Ponzetto, and Goran Glavas. 2023. Can demographic factors improve text classification? revisiting demographic adaptation in the age of transformers. In _Findings of the 2023 Association for Computational Linguistics_.
* Higgsaw (2018) Jigsaw. 2018. Toxic comment classification challenge. Accessed: 2021-05-01.
* Jigsaw (2019) Jigsaw. 2019. Unintended bias in toxicity classification. Accessed: 2021-05-01.

 

Sanjay Kairam and Jeffrey Heer. 2016. Parting crowds: Characterizing divergent interpretations in crowd-sourced annotation tasks. In _CSCW_.
* Klanner et al. (2020) Manfred Klanner, Anne Gohring, and Michael Amsler. 2020. Harmonization sometimes harms. _CEUR Workshops Proc._.
* Krippendorff (2004) Klaus Krippendorff. 2004. Reliability in content analysis: Some common misconceptions and recommendations. _Human communication research_, 30(3):411-433.
* Kumar et al. (2021) Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021. Designing toxic content classification for a diversity of perspectives. In _Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021)_, pages 299-318.
* Kumar (2022) Sawan Kumar. 2022. Answer-level calibration for freeform multiple choice question answering. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 665-679, Dublin, Ireland. Association for Computational Linguistics.
* Liu et al. (2019) Tong Liu, Akash Venkatachalam, Pratik Sanjay Bondgale, and Christopher M. Homan. 2019. Learning to predict population-level label distributions. In _HCOMP_.
* Mathew et al. (2021) Binny Mathew, Punyajoy Saha, Seid Muhie Yimamm, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021. Hatexplain: A benchmark dataset for explainable hate speech detection. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 35, pages 14867-14875.
* Newman (2006) Mark EJ Newman. 2006. Modularity and community structure in networks. _Proceedings of the national academy of sciences_, 103(23):8577-8582.
* Obermeyer et al. (2019) Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. _Science_.
* OpenAI (2022) OpenAI. 2022. Introducing ChatGPT.
* OpenAI (2023) Matthias Orlikowski, Paul Rottger, Philipp Cimiano, and Dirk Hovy. 2023. The ecological fallacy in annotation: Modeling human label variation goes beyond sociodemographics. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, Toronto, Canada. Association for Computational Linguistics.
* Ouyang et al. (2021) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
* Pavlopoulos et al. (2020) John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithun Thain, and Ion Androutsopoulos. 2020. Toxicity detection: Does context really matter? _arXiv preprint arXiv:2006.00998_.
* Pei and Jurgens (2023) Jiaxin Pei and David Jurgens. 2023. When do annotator demographics matter? measuring the influence of annotator demographics with the POPQUOR dataset. In _Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII)_, pages 252-265, Toronto, Canada. Association for Computational Linguistics.
* Plank et al. (2014) Barbara Plank, Dirk Hovy, and Anders Sogaard. 2014. Linguistically debatable or just plain wrong? In _ACL_.
* Prabhakaran et al. (2021) Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On releasing annotator-level labels and information in datasets. In _Proceedings of The Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop_, pages 133-138.
* Rubin (2017) Mark Rubin. 2017. Do p values lose their meaning in exploratory analyses? it depends how you define the familywise error rate. _Review of General Psychology_, 21(3):269-275.
* Sahoo et al. (2022) Nihar Sahoo, Himanshu Gupta, and Pushpak Bhattacharyya. 2022. Detecting unintended social bias in toxic language datasets.
* Salminen et al. (2019) Joni Salminen, Hind Almerekhi, Ahmed Mohamed Kamel, Soon-gyo Jung, and Bernard J Jansen. 2019. Online hate ratings vary by extremes: A statistical analysis. In _Proceedings of the 2019 Conference on Human Information Interaction and Retrieval_, pages 213-217.
* Santurkar et al. (2023) Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? _arXiv preprint arXiv:2303.17548_.
* Sap et al. (2022) Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5884-5906, Seattle, United States. Association for Computational Linguistics.
* Shelby et al. (2022) Renee Shelby, Shalaleh Rismani, Kathryn Henne, Ajung Moon, Negar Rostamzadeh, Paul Nicholas, YILLA-AKBARI N'MAH, Jess Gallegoos, Andrew Smart, and GURLEEN VIRK. 2022. Identifying sociootechnical harms of algorithmic systems: Scoping a taxonomy for harm reduction. _arXiv preprint arXiv:2210.05791_ 

Wai Man Si, Michael Backes, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, Savvas Zannettou, and Yang Zhang. 2022. Why so toxic? measuring and triggering toxic behavior in open-domain chatbots. In _Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security_, CCS '22, page 2659-2673, New York, NY, USA. Association for Computing Machinery.
* Solaiman and Dennison (2021) Irene Solaiman and Christy Dennison. 2021. Process for adapting language models to society (palms) with values-targeted datasets. _Advances in Neural Information Processing Systems_, 34:5861-5873.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsumi B. Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model.
* Thoppian et al. (2022) Romal Thoppian, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. LaMDA: Language models for dialog applications. _arXiv preprint arXiv:2201.08239_.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models.
* Traifnow and Marks (2015) David Traifnow and Michael Marks. 2015. Editorial. _Basic and Applied Social Psychology_, 37(1):1-2.
* Uma et al. (2021) Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poseio. 2021. Learning from disagreement: A survey. _Journal of Artificial Intelligence Research_, 72:1385-1470.
* Vidgen et al. (2019) Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tremble, Scott Hale, and Helen Margetts. 2019. Challenges and frontiers in abusive content detection. In _Proceedings of the third workshop on abusive language online_. Association for Computational Linguistics.
* Warner and Hirschberg (2012) William Warner and Julia Hirschberg. 2012. Detecting hate speech on the world wide web. In _Proceedings of the second workshop on language in social media_, pages 19-26.
* Waseem (2016) Zeerak Waseem. 2016. Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. In _Proceedings of the first workshop on NLP and computational social science_, pages 138-142.
* Waseem and Hovy (2016) Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In _Proceedings of the NAACL student research workshop_, pages 88-93.
* Weerassooriya et al. (2020) Thirindu Cyril Weerassooriya, Tong Liu, and Christopher M. Homan. 2020. Neighborhood-based pooling for population-level label distribution learning. In _ECAI_.
* Weerassooriya et al. (2023a) Thirindu Cyril Weerassooriya, Sarah Luger, Saloni Poddar, Ashiqur KhudaBukhsh, and Christopher Homan. 2023a. Subjective crowd disagreements for subjective data: Uncovering meaningful CrowdOpinion with population-level learning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 950-966, Toronto, Canada. Association for Computational Linguistics.
* Veersacooriya et al. (2023b) Thirindu Cyril Weerassooriya, Alexander Ororbia, Raj Bhensadadia, Ashiqur KhudaBukhsh, and Christopher Homan. 2023b. Disagreement matters: Preserving label diversity by jointly modeling item and annotator label distributions with DisCo. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 4679-4695, Toronto, Canada. Association for Computational Linguistics.
* House (2023) White House. 2023. FACT SHEET: President Biden Issues Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence.
* Wich et al. (2020) Maximilian Wich, Hala Al Kuwatly, and Georg Groh. 2020. Investigating annotator bias with a graph-based approach. In _Proceedings of the Fourth Workshop on Online Abuse and Harms_, pages 191-199, Online. Association for Computational Linguistics.
* Wong et al. (2021) Ka Wong, Praveen Paritosh, and Lora Aroyo. 2021. Cross-replication reliability-an empirical approach to interpreting inter-rater reliability. _arXiv preprint arXiv:2106.07393_.
* Wulczyn et al. (2017) Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In _Proceedings of the 26th international conference on world wide web_, pages 1391-1399.
* Xenos et al. (2022) Alexandros Xenos, John Pavlopoulos, Ion Androut-sopoulos, Lucas Dixon, Jeffrey Sorensen, and Leo Laugier. 2022. Toxicity detection sensitive to conversational context. _First Monday_.
* Xu et al. (2021) Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2021. Recipes for safety in open-domain chatbots.
* Ziegler et al. (2020) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020. Fine-tuning language models from human preferences 

Appendix

Figures 1-8 report, for each metric and demographic group, the score of the metric as a horizontal black line and, subimposed beneath each horizontal line, a histogram of the metric scores under the permutation sampling determined by our null hypothesis. Result are significant when the horizontal is at the extreme end of the histograms. Histograms are also color-coded by the significance of the results they support: red histograms indicate that the result is significant at the \(p=0.05\) level, but only before adjusting for the false positive rate (FPR); green indicates significance at the \(p=0.05\) level, even after FPR adjustment. Given the exploratory nature of the work, both kinds of significance are meaningful and merit attention. But we can feel more confident that the FPR adjusted results are likely more robust and repeatable.

 

Figure 1: Within-group agreement metrics, by race/ethnicity. Negentropy and plurality size indicate that White raters have significantly more, and Multiracial significantly less, agreement than other races/ethnicities. IRR indicates that Latine raters have significantly more agreement than other races/ethnicities

Figure 2: Within-group agreement metrics, by race/ethnicity and gender. Histograms represent the distribution of agreement values under the null hypothesis. Black horizontal bars represent the observed values. These results show that white men have significantly less agreement than other groups, according to negentropy and plurality size, neither of which control for class imbalance. IRR shows that with controlling for class imbalance between _safe_ and _unsafe_ annotations, the amount of agreement is more moderate. Asian women show nearly the opposite results, with less agreement than other groups unless class imbalance is controlled.

 

Figure 4: Across-group agreement metrics, by race/ethnicity and gender. Histograms represent the distribution of agreement values under the null hypothesis. Black horizontal bars represent the observed values. Here, white men show signs of significantly low plurality agreement. With other groups. Yet safety agreement is significantly high (though will a small effect size). This seeming disparity is due to the high class imbalance within safety reasons and white men’s tendency to favor _safe annotations_. And so for specific safety reasons they appear more agreeable. However, when these reasons are aggregated into an overall safety score, differences between which men and other groups reveal themselves.

Figure 3: Across-group agreement metrics, by race/ethnicity. Histograms represent the distribution of agreement values under the null hypothesis. Black horizontal bars represent the observed values. White and multiracial voters show less overall agreement with others. Latine voters show more agreement with others.

 

Figure 5: Within-group agreement metrics, by age. Histograms represent the distribution of agreement values under the null hypothesis. Black horizontal bars represent the observed values. None of these groups show significant amounts of difference in disagreement.

Figure 6: Within-group agreement metrics, by gender. Histograms represent the distribution of agreement values under the null hypothesis. Black horizontal bars represent the observed values. None of these groups show significant amounts of difference in disagreement.

 

Figure 8: Across-group agreement metrics, by gender. Histograms represent the distribution of agreement values under the null hypothesis. Black horizontal bars represent the observed values.

Figure 7: Across-group agreement metrics, by age. Histograms represent the distribution of agreement values under the null hypothesis. Black horizontal bars represent the observed values.

 



\begin{table}
\begin{tabular}{l l|l l l l l l l l} \hline  & Dimension & Group & IRR & XRR & Negentropy & Cross & Plurality & Plurality \\  & Dimension & Group & IRR & XRR & Negentropy & Negentropy & size & agreement & GAI \\ \hline
0 & [age] & gen x+ & 1.066 & 1.071 & 1.042 & 1.0365 & 1.0693 & 1.0731 & 1.0975 \\
1 & [age] & gen z & 1.066 & 1.0172 & 1.0386 & 1.0392 & 10.698 & 1.0766 & 1.0966 \\
2 & [age] & millennial & 1.0189 & 1.0179 & 1.0415 & 1.0381 & 10.703 & 1.0751 & 1.1052 \\ \hline
3 & [gender] & Man & 10.187 & 10.175 & 10.419 & 10.394 & 10.707 & 10.800 & 11.071 \\
4 & [gender] & Woman & 10.160 & 10.175 & 10.362 & 10.404 & 10.685 & 10.800 & 10.916 \\ \hline
5 & [race] & Asian & 10.145 & 10.166 & 10.368 & 10.323 & 10.675 & 10.740 & 10.872 \\
6 & [race] & Black & 10.193 & 10.181 & 1.0411 & 10.361 & 10.705 & 10.796 & 11.063 \\
7 & [race] & Latinze & 10.215* & 10.189* & 10.467 & 10.412 & 10.716 & 10.747 & **1.139*** \\
8 & [race] & Multiracal & 10.153 & 10.168 & **0.355**** & **10.250*** & **10.661*** & 10.592 & 10.916 \\
9 & [race] & White & 10.145 & **10.159*** & **10.498*** & **10.417*** & **10.744*** & **0.552*** & 10.908 \\ \hline
10 & [race, gender] & Asian, Man & 10.193 & 10.188 & 10.495 & 10.417 & 10.733 & 10.722 & 11.024 \\
11 & [race, gender] & Asian, Women & **10.073*** & **10.134*** & **10.332*** & **10.193*** & **10.633*** & -0.543 & **10.540*** \\
12 & [race, gender] & Black, Man & 10.139 & 10.167 & 10.502 & 10.371 & 10.710 & 10.604 & 10.831 \\
13 & [race, gender] & Black, Woman & **10.213*** & 10.188 & 10.441 & 10.349 & 10.718 & 10.749 & **1.139*** \\
14 & [race, gender] & Latino, Mann & 10.195 & 10.183 & 10.491 & 10.383 & 10.716 & 10.687 & 11.069 \\
15 & [race, gender] & Latine, woman & **10.238*** & **10.199*** & 10.530 & 10.437 & 10.745 & 10.704 & **1.196*** \\
16 & [race, gender] & Multivical, Man & 10.190 & 10.182 & 10.432 & 10.273 & 10.688 & 10.562 & 11.043 \\
17 & [race, gender] & Mainrecal, Woman & 10.041 & 10.131 & **10.470*** & 10.184 & 10.674 & 10.438 & 10.312 \\
18 & [race, gender] & White, Man & **10.218*** & 10.173 & **10.724*** & **10.505*** & **10.833*** & **10.464*** & **1.262*** \\
19 & [race, gender] & White, Woman & **10.114*** & **10.152*** & 10.454 & 10.381 & 10.702 & 10.663 & **10.752*** \\ \hline \end{tabular}
\end{table}
Table 6: Results for in-group and cross-group cohesion, and GAI for demographic groups of **D3** raters. Significant results are in **bold**: * for significance at \(p<0.05\), ** for significance after Benjamini-Hochberg correction. A single asterisk (*) means significant at the \(p=0.05\) level. A double asterisk (**) means the results are significant after Benjamini-Hochberg correction. A \(\downarrow\) (or \(\uparrow\)) means that the result is less (or greater) than expected under the null hypothesis. GAI results based on \(C_{X}=\text{XRR}\) and \(C_{I}=\text{IRR}\). The DSI results are based on variable that minimized each dimension, and they are as follows. Age: 1.052 (millennial), gender: 1.071 (men), race/ethnicity: 1.139 (Latine raters), (gender, race/ethnicity): 1.262 (White men).

\begin{table}
\begin{tabular}{l l l l l l l l l l} \hline  & Dimension & Group & IRR & XRR & Negentropy & Class & Plurality & Plurality \\  & Dimension & Group & IRR & XRR & Negentropy & Negentropy & size & agreement & GAI \\ \hline
0 & [age] & (18,30) & 10.115** & 10.107 & 10.631** & 10.297 & 10.405 & 10.689** & 11.068** \\
1 & [age] & (30,50) & 10.089*** & 10.104 & 10.571** & 10.340 & 10.377* & 10.720** & 10.850** \\
2 & [age] & 50+ & 10.110 & 10.111 & 10.480 & 10.389 & 10.356 & 10.754 & 10.999 \\ \hline
3 & [gender] & Woman & 10.110 & 10.108 & 10.634** & 10.267** & 10.424 & 10.692** & 11.024 \\
4 & [gender] & Man & 10.105 & 10.107 & 10.612*** & 10.307* & 10.423 & 10.702* & 10.976 \\
5 & [gender] & Other & 10.209 & 10.096 & 10.030 & 10.605 & 10.192 & 10.978 & 11.2172* \\ \hline
6 & [region] & Arab Culture & **10.133*** & 10.113 & **0.452*** & 10.413 & 10.272 & 10.759** & **11.174*** \\
7 & [region] & Indian Cultural Sphere & 10.103 & 10.099* & **10.457*** & 10.418 & 10.280 & 10.760** & 11.043 \\
8 & [region] & Latin America & **10.129*** & 10.112 & **10.499*** & **10.400*** & 10.317 & 10.764*** & **11.152*** \\
9 & [region] & North America & **10.134*** & 10.110 & **10.443*** & **10.393*** & 10.316 & 10.772 & 11.307* \\
10 & [region] & Oceania & 10.118 & 10.103 & **0.372*** & 10.411 & 10.303 & 10.797** & **11.145*** \\
11 & [region] & Sinosphear & **10.087*** & **10.087*** & 10.405 & **10.381*** & **10.223*** & 10.788 & 11.002 \\
12 & [region] & Sub Salaran Africa & **10.142*** & 10.104 & 10.418 & **10.385*** & **10.262*** & 10.777 & **11.361*** \\
13 & [region] & Western Europe & **10.135*** & 10.111 & **10.448*** & **10.383*** & **10.236*** & 10.768 & **11.222*** \\ \hline \end{tabular}
\end{table}
Table 5: Results for in-group and cross-group cohesion, and GAI for demographic and intersectional groups within **DICS-350**. Significant results are in **bold**. A single asterisk (*) means the result is significant at the \(p=0.05\) level. A double asterisk (**) means the results are significant after Benjamini-Hochberg correction. A \(\downarrow\) means that the result is less than expected under the null hypothesis. A \(\uparrow\) means the result is greater. We report GAI based on \(C_{X}=\text{XRR}\) and \(C_{I}=\text{IRR}\). The DSI results are based on variable



\begin{table}
\begin{tabular}{l l l l l l l l l l l} \hline  & Dimension & Group & IRR & XRR & Negentropy & Cross & Plurality & \multicolumn{1}{c}{Plurality size} & \multicolumn{1}{c}{Plurality agreement} & \multicolumn{1}{c}{GAI} \\ \cline{2-9}

[MISSING_PAGE_POST]

ender] & NA., Man & \(\uparrow\)0.113 & \(\uparrow\)0.112 & \(\downarrow\)**0.188***** & \(\downarrow\)**0.454*** & \(\uparrow\)**0.278*** & \(\uparrow\)**0.885***** & \(\uparrow\)1.005 \\
19 & [region, gender] & NA., Woman & \(\uparrow\)**0.153***** & \(\uparrow\)0.116 & \(\uparrow\)0.299 & \(\downarrow\)0.449 & \(\downarrow\)0.239 & \(\downarrow\)0.825 & \(\uparrow\)**1.314***** \\ \hline
20 & [region, age] & Oc., (18,30) & \(\uparrow\)0.113 & \(\uparrow\)0.121 & \(\downarrow\)0.155 & \(\uparrow\)0.510 & \(\uparrow\)0.230 & \(\uparrow\)0.900 & \(\uparrow\)0.932 \\
21 & [region, age] & Oc., (30,50) & \(\uparrow\)0.112 & \(\downarrow\)**0.089***** & \(\bf\textbf{0.173*****}\) & \(\bf\textbf{0.455*****}\) & \(\bf\textbf{0.218}\) & \(\bf\textbf{0.900*****}\) & \(\bf\textbf{1.255*****}\) \\
22 & [region, age] & Oc., 50+ & \(\downarrow\)0.081 & \(\uparrow\)0.115 & \(\downarrow\)0.140 & \(\downarrow\)**0.481***** & \(\bf\textbf{0.286*****}\) & \(\bf\Phi.914\) & \(\downarrow\)0.699 \\
23 & [region, gender] & Oc., Man & \(\downarrow\)0.090 & \(\bf\textbf{0.091*****}\) & \(\bf\textbf{0.170*****}\) & \(\bf\textbf{0.448*****}\) & \(\bf\Phi.219\) & \(\bf\textbf{0.899*****}\) & \(\bf\Psi.988\) \\
24 & [region, gender] & Oc., Woman & \(\uparrow\)**0.133*** & \(\uparrow\)0.110 & \(\underline{\textbf{0.120*****}}\) & \(\bf\textbf{0.164*****}\) & \(\bf\Phi.266\) & \(\bf\Phi.853*****}\) & \(\bf\textbf{1.208*****}\) \\ \hline
25 & [region, age] & Si., (18,30) & \(\uparrow\)0.112 & \(\downarrow\)0.108 & \(\downarrow\)0.190 & \(\bf\textbf{0.456*****}\) & \(\bf\geq\)0.217 & \(\bf\Phi.883\) & \(\bf\textbf{1.029}\) \\
26 & [region, age] & Si., (30,50) & \(\downarrow\)**0.083***** & \(\bf\textbf{0.082*****}\)