SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Expert annotations (DICES-350 only)—Conversations contain safety annotations from trust and safety experts, accompanied by a motivation for the rating. These ratings come from in-house experts who define rater guidelines and oversee safety evaluations for machine learning systems. Each gold rating is first provided by one rater and then verified by a pool of five expert raters. When comparing these labels with the labels from the diverse crowd workers, we observed disagreements between the gold label and crowd majority vote label in  $34\%$  examples.  $30\%$  of the conversations were labelled as unsafe by the gold rater, but were labelled as safe by the crowd;  $4\%$  of the conversations were labelled as unsafe by the crowd, but safe by the gold raters. We also provide expert annotations for the "degree_of_harm", indicating the severity of safety risk, and "harm_type" indicating whether the conversation is of "Benign", "Debatable", "Extreme", or "Moderate" in adversariality (Fig. 3). These labels were used to identify rating patterns in response to more or less harmful conversations or conversations aligned with particular issues or topics. Labels were assigned and independently
Figure 4: Within-group agreement metrics, by race. IRR shows that Latine raters have significantly more agreement than other races. Negentropy (i.e. negative of entropy) and plurality size (i.e. the fraction of raters who choose the most popular response) show that White raters have significantly more, and Multiracial significantly less, agreement than other races.
Figure 5: Illustrative comparison between demographic sub-groups. The left graph shows rating counts for male and female annotators. The right graph shows counts for the 5 racial/ethnic groups.
checked by two members of the research team. Approximately one third of the conversations in DICES-990 also contain expert annotations for "degree_of_harm" and "harm_type".
Conversation evaluation—Raters' evaluation of the entire conversation for its legibility (Q1, §3.4).
Granular safety ratings—Raters' answers to the 16 (DICES-350) or 24 (DICES-990) safety questions spread across five categories of safety: "harmful content" (Q2.1–Q2.9), "unfair bias" (Q3.1–Q3.5), "misinformation" (Q4), "political affiliation" (Q5) and "safety policy violations" (Q6.1–Q6.3). For both brevity and illustrative purposes, we refer to the data from the aggregated overall safety ratings in this paper. However, it is worth emphasising that the DICES dataset provides further opportunity for extensive, detailed analysis of specific safety-related categories and specific rated conversations.
Aggregated ratings—Aggregated ratings, were generated from all granular safety ratings. They include a single aggregated overall safety rating ("Q_overall"), and aggregated ratings for the three safety categories that the 16 more granular safety ratings correspond to: "Harmful content" ("Q2_harmful_content_overall"), "Unfair bias" ("Q3_bias_overall") and "Safety policy violations" ("Q6_policy GUIDelines_overall"). To aggregate the overall safety rating, we considered a single rater to judge a conversation as unsafe if they responded with at least one "Yes" to any of the sub-rating questions (i.e., Q2.1–Q2.9, Q3.1–Q3.5, Q4, Q5, and Q6.1–Q6.3). If there were no "Yes" answers, but at least one "Unsure", we judged the overall rating as unsure. If the answers to all the individual questions were "No", we judged the rating of the conversation to be safe. The same was done for the three questions grouped by harm (Q2.1–Q2.9), bias (Q3.1–Q3.5) and safety policy violations (Q6.1–Q6.3). Just over  $60\%$  of the 36,400 ratings were labelled safe,  $33\%$  unsafe and  $6\%$  unsure. Figure 4 also shows the distributions across demographic groups.
5 Discussion and Limitations
We introduced the DICES dataset as a means to evaluate the safety of conversational AI systems, with a particular focus on subjective opinions from diverse raters. The combination of safety ratings by individual raters, along with their fine-grained demographic details, provides a grounding to further study how various subgroups differ in their perception of safety, and how to incorporate these diverse perspectives in safety evaluation and interventions. Rather than solving the ambiguity in the space of subjective safety opinions, DICES—with its over 2.5 million ratings from a diverse pool of almost 300 raters combined with expert-based ratings for safety, harm and degree of harm—is a unique and large resource that enables us to study, among other themes:
- ambiguity in safety evaluations — both in specific subtasks and content;
- rater disagreement on safety across different rater groups, including intersectional groups;
- how to build fine-tuning approaches that account for diverse opinions on safety.
DICES allows for more thorough comparative studies on these three annotation sets—raters representative of "wider user perspectives" on safety and experts with domain-specific opinions of safety. This opens the path to new approaches for safety sense-making, where raters and experts co-create the notion of "truth" in diverse, closer-to-real-world scenarios. This is a unique aspect of the DICES dataset, as there is no other resource, to our knowledge, that captures a high rating replication, diverse crowd of raters and expert opinions—on all items—in one dataset.
It is important to note, however, that our analyses show that (1) not all demographics play equal role in influencing safety perceptions, (2) diverse rater pool presents much higher rate of disagreement among raters, which poses a challenge to the traditional definition of 'gold' labels, which also challenges the status quo in terms of defining quality of raters. DICES opens up opportunities for the research community to further study these issues and in doing so extend the DICES dataset.
First, despite the fact that DICES contains an unprecedented number of safety ratings (i.e., 1,730,472 ratings in DICES-990 and 688,800 ratings in DICES-350 and in total over 2.5 million ratings) from large diverse rater pools, still the total number of conversations could be considered small (i.e., 1,340 conversations in total across DICES-990 and DICES-350). While these conversations were chosen carefully to demonstrate the impact of diversity, larger datasets might reveal patterns in rater disagreements that show greater dependencies on content.
Another limitation is our selection of demographic characteristics. In order to manage the time spent on recruitment, decrease the complexity in data analysis, and increase the statistical power of our observations, we limited the number of demographic categories to four (locale and gender in DICES-990 and race/ethnicity, gender and age group in DICES-350). Within these demographic axes, we also limited the number of sub-groups for the same reasons (i.e., two locales, five main ethnicity groups, three age groups and two genders). We believe that further disaggregating the ethnicity, gender and age groups of raters is likely to extend the insights and provide additional evidence of systematic differences between different groupings of raters. Sharing DICES is the first step towards understanding the impact of rater demographics on safety perspectives and will allow for follow-up studies that drill down using more categories and more granular sub-categories.
Despite the high rater replication per item, we still observed a high number of disagreements, indicative of the high subjectivity of the task. We explored different disagreement metrics and Bayesian multilevel modelling of demographics to gain understanding of the disagreements, however more work is needed to determine how this is further translated to decision-making in safety evaluation.
Finally, we recognise that more work is also needed to help distinguish disagreement that reflects differences between sincerely held beliefs and disagreements that reflects noise rooted in rater error, poor task design, or low quality items. We correlated temporal data with other behavioural traits of raters to qualitatively discover outliers. Ultimately, this would extend our methodology to include an approach to study outliers and different perspectives.
As we propose a methodology for assessing the influences of rater diversity on safety labels, our future work will focus on determining the ideal number of raters per conversation and to what extent the impact of rater diversity can be captured in smaller numbers of raters. This line of research will improve dataset generation approaches aimed to address rater diversity.
6 Statement of Ethics
All the demographics data was collected with an optional Google form and was self-declared. Raters were presented a consent form before signing up for the study to inform them about the gathering of personal demographics and that the conversations to be rated are adversarial (i.e., would possibly contain offensive content). All demographics questions had the option "Prefer not to answer". All data was collected in anatomised way after the data collection tasks were completed by the raters. Raters were allowed to quit the study at any time.
All data analysis in this work is done on a Google Colab CPU. No greater compute was needed. The code used for this will be released with the dataset on github upon publication at https://github.com/google-research-datasets/dices-dataset
Google LLC licenses this data under a Creative Commons Attribution 4.0 International License. Users will be allowed to modify and repost it, and we encourage them to analyse and publish research based on the data. The dataset is provided "AS IS" without any warranty, express or implied. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset.