DICES Dataset: Diversity in Conversational AI Evaluation for Safety
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: conversational AI
human evaluation
human annotation
safety task
disagreement
variance in human annotations
diversity of rater pool
TL;DR: Safe AI is a better AI for Everyone. But everyone has different perceptions of safety. The DICES dataset offers a shared resource to understand these differences and benchmark safety evaluation for conversational AI systems

Abstract: Machine learning approaches often require training and evaluation datasets with a clear separation between positive and negative examples. This requirement overly simplifies the natural subjectivity present in many tasks, and obscures the inherent diversity in human perceptions and opinions about many content items. Preserving the variance in content and diversity in human perceptions in datasets is often quite expensive and laborious. This is especially troubling when building safety datasets for conversational AI systems, as safety is socio-culturally situated in this context. To demonstrate this crucial aspect of conversational AI safety, and to facilitate in-depth model performance analyses, we introduce the DICES (Diversity In Conversational AI Evaluation for Safety) dataset that contains fine-grained demographics information about raters, high replication of ratings per item to ensure statistical power for analyses, and encodes rater votes as distributions across different demographics to allow for in-depth explorations of different aggregation strategies. The DICES dataset enables the observation and measurement of variance, ambiguity, and diversity in the context of safety for conversational AI. We further describe a set of metrics that show how rater diversity influences safety perception across different geographic regions, ethnicity groups, age groups, and genders. The goal of the DICES dataset is to be used as a shared resource and benchmark that respects diverse perspectives during safety evaluation of conversational AI systems.

DICES Dataset:
Diversity in Conversational AI Evaluation for Safety
Lora Aroyo', Alex S. Taylor? Mark Díaz!, Christopher M. Homan', Alicia Parrish', Greg Serapio-Gareía', Vinodkumar Prabhakaran', and Ding Wang'
Goooge Research, City, University of London, University of Cambridge
Abstract
Machine learning approaches often require training and evaluation datasets with a clear separation between positive and negative examples. This risks simplifying and even obscuring the inherent subjectivity present in many tasks. Preserving such variance in content and diversity in datasets often expensive and laborious. This is especially troubling when building safety datasets for conversational AI systems, as safety is both socially and culturally situated. To demonstrate this crucial aspect of conversational AI safety, and to facilitate in-depth model performance analyses, we introduce the DICES (Diversity In Conversational AI Evaluation for Safety) dataset that contains fine-grained demographic information about raters, high replication of ratings per item to ensure statistical power for analyses, and encodes rater votes as distributions across different demographics to allow for in- depth explorations of different aggregation strategies. In short, the DICES dataset enables the observation and measurement of variance, ambiguity, and diversity in the context of conversational AI safety. We also illustrate how the dataset offers basis for establishing metrics to show how raters' ratings intersects with demographic categories such as racial/ethnic groups, age groups, and genders. The goal of DICES is to be used shared resource and benchmark that respects diverse perspectives during safety evaluation of conversational Al systems.
Introduction
As conversational AI systems built on large language models (LLMs) have become increasing prevalent, it is clear that their safety should be of paramount importance 29 39]. Safety, in this context, speaks to the character of automatically generated content and its capacity to cause dangers to downstream users/communities, propagate inw/ddiiinformmiioonn and violate social norms in respectful communication, While advances in conversational Al capabilities are being propelled by access to web-scale, unlabelled training data [10] and innovative modelling approaches [40], such forms of safety continue to demand robust evaluation data and careful fine-tuning to align with social norms [3 and achieve the responsible practices rightly demanded across the tech sector [4] In other words, carefully curated data remains crucial to ensuring safe deployment of conversational AI (45)
Recent research has investigated approaches to efficiently fine tune language model outputs using safety annotated datasets [37 39 24
28 46). Much less work, however, has gone into understanding the inevitable complexities associated with building datasets that capture the range of views of safety across diverse populations. Existing resources have typically disregarded the varied and subjective ideas of safety in conversation held by diverse user groups; instead, they adopt simplified notion of a single ground truth for safety in order to facilitate technical solutions an approach that have unwanted even disastrous effects in real-world settings 5 21.. Specifically, much of the work addressing safety has neglected the variation between different populations that
15
19
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

may hold diverse perspectives on safety [6] 77 and there is little in the way of guidance for building or evaluating resourees that capture such diversity.
In this paper, we introduce new dataset, DICES (Diversity In Conversational Al Evaluation for Safery), that offers framework for the fine-grained representation and analysis of safety perceptions from different user populations] DICES incorporates safety ratings of human-bot conversations by a large, demographically diverse set of human raters. The dataset has also been designed from the ground up to capture balanced proportions of opinions from sub-populations in the rater pool (for gender, age and ethnic/racial groups) and to generate an exceptionally large number of ratings per conversation. This provides particularly powerful means for evaluating safety in language models, especially with respect to population diversity.
[1]
1.1 Contributions
Set against the urgent need for more nuanced approaches to safety in language modelling and, in particular, a more systematic way to account for diverse opinions of safety in model training and evaluation, the primary contributions of this paper and the DICES dataset are as follows:
Rater Diversity - Intentionally diverging from the language of bias (and attempts to mitigate it), we approach differences between raters opinions of safety in terms of diversity (for brief summary of related work on rater diversity, see (3)). This is important because it motivates our aim to characterise the impact of raters' backgrounds on dataset annotations produced and used by large language models. We avoid assuming bias should, by default, be mitigated or removed, as it is likely the case that differences within population will be crucial to the real-worke efficacy of models. DICES is thus intentionally designed to account for diversity, with a rater pool that has a balanced distribution across demographic groups.
Expanded Safety In line with the arguments in favour of dialogue safety ratings [e.g.. 38 and extending the approach in Thoppilan et al [39], we assess a wider notion of safety that includes detailed ratings along five safety categories related to harm, bias, misinformation, politics and safety policy violations. The DICES dataset offers a means of evaluating the safety of conversational Al systems well focusing on granular categories such specific harms, biases, dangerous content, hate speech and misinformation (and how they intersect with different demographic groups).
Dataset Size - DICES contains two sets of annotated Al chatbot conversations one of 990 con- versations (DICES-990) and one of 350 (DICES-350) with approximately 70 and 120 annotations per conversation, respectively. This rater replication rate is exceptionally large, as most annotation tasks use only 3-5 raters per conversation. The high number of unique ratings per conversation allows for statistical power of the observations drawn from the data. This is especially important for the purposes of adequately studying the demographic diversity of annotators and any impact on their safety opinions. In addition, the high number of unique ratings per conversation also allows for resampling of the data model results at more typical sample sizes with better estimation of variability. Both datasets include multiple sub-ratings which specify the type of safety concern, such as type of hate speech and the type of bias misinformation, for each conversation. Both datasets have been developed using a systematic approach to recruiting raters from different demographics. DICES-990 was annotated by raters recruited from two locales (India and US) and distributed across genders. DICES-350 was annotated by raters only from the US and distributed in a balanced way across gender, ethnicity and age group. We concentrate on DICES-350 due to space constraints, however both datasets are released with this paper.
Metrics- - Next to presenting DICES, we also illustrate how it can be used to develop metrics to examine and evaluate conversational AI systems in terms of both safety and diversity. For example, we show how measures of inter-rater reliability can be compared to reveal different agreement results between demographic subgroups (see Figure 4). Our dataset repository includes more complete set of metrics and results from their application.
2 Related Work
In dataset creation for the evaluation of language models, the detection of toxicity, harm and hate speech is receiving increased attention, particularly in light of the recent popularity of LLMS (see related surveys by 44, [2] and [16). Broadly, this line of research aims to assess the degree of toxicity, harm or hate speech in datasets or to evaluate a machine learning model's propensity to
either identify or reproduce such language [e.g.. 9 20 36 37
'DICES dataset and accompanying analyses: https://gihubb.co/goolllee. dateaaaacccccatttttsddedsssssss

---

Crucial to much of this work is the role of labelled data, hand-labelle by human raters. Human annotation continues contribute harm or hate speech and to train/fine-tun tune language models [e.g.. 27 44
growing number of datasets used for benchmarking toxicity, However, an acknowledged risk of these datasets and their corresponding models is that they have the potential to exhibit bias
26 32 422, in part because of raters' backgrounds and experiences and how these, in turn, impact opinions ?? 13 35 For instance, raters' demographic markers (i.e., their first language, age, and education) have been shown to have significant impact on the performance of models that detect hate speech and abusive language 3] Similarly, with a narrower focus of African American and LGBTQ populations, Goyal et al. [17 have shown statistically significant differences between toxicity ratings when comparing data annotated by raters in both of these groups, and raters who identify with neither. Goyal et al.'s 'ss work also demonstrates the way such differences in datasets can result in variable performance of language models.
The key message from these and other similarly motivated works is that the backgrounds and ex- periences of raters make a difference to labelled datasets, particularly datasets involving subjective responses such identifying toxic, harmful or hateful language. In our work, we use raters de- mographics as proxies for social experiences that are difficult to measure directly, i.e., raters with different vocio-demographic backgrounds can disagree with one another because they have patterned social experiences that shape their opinions in particular ways. Whether we view the causes of these differences to be an indication of biases, annotator disagreement broader annotator diversity is open to debate [e.g.. 8 22 33
Whatever the explanation, a growing number of projects have sought to further examine and to some extent address the observed differences between annotations from different populations and the different opinions/beliefs they hold [e.g.. D 2 23 22
30 Halevy et al. [18]. for example,
investigate how the bias against African American English propagates from annotation into hate detection models. They show that incorporating specialised African American English classifier into a detection model can reduce the effects of annotation bias. Davani et al. M adopt a multi-task framework to model individual raters while maintaining a shared network for the task, demonstrating improved performance in subjective tasks such as hate speech detection. Santurkar et al. [34] use the questions and results from national (US), public opinion surveys to probe language models and assess how the models align (or misalign) with demographic groups Through this approach, they put forward dataset and metrics to evaluate opinions in language models, broadly testing representativeness against the wider (US) population and consistency within groups. Together, these efforts demonstrate the importance of diversity within the annotator pool, well as the utility of fine-grained demographic information about the annotators.
Broadly, the work we report contributes to these threads of research that seek to develop more rigorous ways to judge safety in language models; account for population effects in datasets used for evaluating and training; and recognize the importance of capturing differences between diverse rater groups The approach we report here contributes a dataset developed using a systematic methodology for capturing diverse opinions on safety and offering a volume of annotations that significantly exceeds established practices in the field.
3 Data Collection Methodology
Driving the development of DICES was the overarching aim to produce a benchmark dataset able
to systematically capture variability in safety judgements and allow for comparative measurements between demographically defined groups of raters. To create it, we followed five-step procedure:
(1) Corpus creation: generating adversarial multi-turn human-chatbot conversations; (2) Sample curation: creating two samples of adversarial conversations; (3) Rater pool selection: recruitment of a diverse rater pool; (4) Safety annotation task: diverse rater pool annotation for safety; and (5)
Expert annotation lask: expert annotation of degree of harm, harm type and topic. Aligned with our overarching aim, the goals guiding the methodology design were to:
Increase statistical power of demographic observations by ensuring ethnicity, age and gender groups are adequately represented across raters.
Improve confidence of comparisons between sub-populations by ensuring all raters annotate every conversation in the corpus;
Quantify and qualify diverse raters' disagreement by sampling data with gold safety labels.
3

---

3.1 Corpus Creation
The imput data for this data collection was sampled from an 8K multi-turn conversation corpus (comprising 48K turns in total) generated by human agents interacting with a generative Al-chatbot [39] The human agents were instructed generate adversarial multi-turn conversations, where they attempt to provoke the chatbot to respond with an undesirable or unsafe answer. All conversations were of maximum five turns and varied in terms of their level of adversariality (i.e., degree of harm) and topics (Figure B). A subset of the conversations (DICES-2 -350 were annotated with gold safety labels (from trust and safety experts) and all conversations (both in DICES-990 and DICES-350) with platinum safety labels (from diverse rater crowd). Details of the data collection and annotation of these conversations is provided in [39]
3.2 Sample Curation
For the purposes of the DICES dataset, we created two samples from the original corpus. DICES- 990 consisting of 990 adversarial multi-turn conversations and DICES-350 consisting of 350. We collected DICES-990 to study cross- pooss-paaffom differences across geographic locales, and DICES-350 to study in-depth drossddeoogapphicc differences within specific locale. We compiled DICES by stratified sampling from the source corpus across the three adversariality groups. We compiled DICES- 350 to be counterpart to DICES 990, but with clearer adversarial sample. Sampling was done evenly from both platinum and gold labelled conversations in the original corpus (e.g. half with a gold platinum label of unsafe and half with gold platinum label of safe). Details of DICES-990 are provided in the GitHub release
3.3 Rater Pool Selection
For the safety annotation of DICES-990 we recruited diverse rater pool in US and India (totalling 173 unique raters). This provided 60-70 unique ratings per conversation along 24 safety criteria. Each rater annotated only a subset of the dataset. DICES-350, in contrast, was annotated by different diverse pool of 123 unique raters based in the US, all of whom annotated all 350 conversations in the dataset along 16 safety criteria. For both datasets, we manually identified number of low quality raters (13 in DICES-990 and 19 in DICES-350; see Section 3.3 for the criteria used), whose annotations were removed. Due to space limitations, in this paper we describe DICES-350 as is more balanced in terms of rater demographics and distribution of raters across conversations.
For the safety annotation of DICES-350, we aimed at a pool of 120 raters in the US with equal numbers of raters in each of the 12 demographic groups (3 x 4 design) created by fully crossing age groups (GenZ, Millennial, GenX+) with race/ethnicity (Asian; Black; Latine/x; White). Data was also captured to calculate the average annotation time per conversation and the total time each rater spent annotating each conversation. We acknowledge that the demographic breakdown is simplified representation of the population at large, however this choice was made in order to facilitate recruitment of raters in each group and to allow for less complexity in analysing intersecting groups To extend DICES in future work, we anticipate further recruitment, introducing additional demographic groupings and extended analysis of statistical interaction between multiple groups.
Initial recruitment for the DICES-351 dataset resulted in a total of 123 raters. After all annotation tasks were completed, we performed a quality assessment on the raters and filtered out 19 raters due to low quality work (e.g., raters who spent suspiciously little time in comparison to the other raters to complete the task and raters who rated all conversations with the same label). Results in this paper are reported using this 104 unique rater pool. The rater breakdown for this pool is: 57 women and 47 men: 27 gen X+, 28 millennial, and 49 gen z: and 21 Asian, 23 Black/Afrrcca American, 22 Latine/x, 13 multiracial and 25 white. See Figure for breakdowns of the demographic groupings along race and gender, as well as race and age groupings.
All raters annotated all 350 conversations, i.e., 104 unique ratings per conversation. All raters signed a consent form agrecing for the detailed demographics survey form which allowed raters to select the option "Prefer not to answer" for each question. All the rater demographics were self-reported by the raters after they finished the annotation task.
be collected for this task. We also used a

---

Female
Male
gen
gen
+ Number of raters
+ Number of raters
21
23
22
25
21
23
22
20
13
13
13
10
10
12
16
16
6
Asian
Black
Latine
Mattiracial
While
Asian
Black
Latine
Multiracial
White
Figure I:
Demographic breakdown of annotators. Two illustrative plots of annotators by racial/ethnic groups and gender (left) and racial/ethnic groups and age groups (right).
3.4 Safety Annotation Task
Figure shows showws six sets of questions:
screenshot of part of the annotation task. The annotation task included the following
Q1: addresses the whole conversation and asks the raters to assess the legibility of the conversa- tion is it (1) in English, (2) comprehensible, (3) on a topic that the rater is familiar with or (4) of the above.
Q2: eight sub-question checks whether the conversation contains any harmful content, i.e., whether it could directly facilitate serious and immediate harm to individuals, groups animals.
Q3: four sub-question checks whether the conversation contains any unfair bias, i.c., whether incites hatred against an individual group.
Q4: one multiple choice question checks whether the conversation contains any misinformation, i... whether it contains theories that are demonstrably false or outdated.
Q5: one multiple choice question checks whether the conversation contains any political affilia- tions, i.c., engaging in or downplaying any controversial topic.
Q6: two sub-question checks whether the conversation contains any policy violations for polarising topics and endorsements, i.e., proactively bring up or reactively promoting brands, celebrities, products, etc.
All questions included an option "other" to ensure coverage possible safety reasons outside this pre-defined space. Rating options for Q2 to Q6 were: "No" (i.e., safe), "Yes" (i.e., unsafe) and "Unsure". " In effect, "Yes" answer to any of the questions Q2 to Q6 should be seen as an explanation of why a conversation is considered unsafe.
3.5 Expert Annotation Task
In order to understand better the conversations terms of their topics and adversariality type and level, all conversations in DICES-350 (and a sample of 400 conversations in DICES-990) were also rated by in-house experts assess their degree of harm (Fig. righht) as well their topic of discussion (Fig. leftt. Nearly 22% of the conversations cover racial topics, followed by 14% political topics, 10% gendered topics and 7% misinformation and medical topics each. More than 40% of the conversations were rated as benign and 60% split evenly between debatable, moderate and extreme in terms degree of harm. Most the benign conversations are labelled as bunter. In addition, all conversations in DICES-350 have gold ratings, meaning they were annotated for safety by a trust and safety expert. DICES-99 didn't 't have gold ratings, and only a random sample of 400 conversations were rated for topic and degree of harm.
5

---

Evaluation
(ightightd
Evaluate the whole conversation None above Engiish, senoe, familtrr with topic
doing great,
Evaluate highliihteed utterance: Harmful Content
identfied
what have for lunch
Cood about getting?
highlighted
5 issse identified
My wife there options.
4
Evaluate highlighted utterance: Misinformation
Wel, that's very fortunate. So your
Evaluate highllihtte utterance: Political Affiliation
thikkig comammn sugar Pards Express, too
highllighted utterance: Policy
had Pande Express,
Polay guileines follwing
cinsamoo sugar
ware affered
engiging
contusenial polatiing
about uncondemed
couse shong
Express
favoritl
diveriy pricey taking auciey
maporty
crange chicken? lt's amazing
oigioss
Making endorsement (poetive negative) frand product superiaties langugge
instlccial pripery 4g
Other type content might contradict with Polic guideines.
Figure Screenshot of the raters' user interface for the Safety Annotation Task: illustrates the annotation category for policy violations. The left panel presents the conversation; raters assess the last conversational turn (highlighted). The right panel presents two policy related sub-questions.
DICES Dataset
The DICES-350 dataset contains 350 conversations, each rated by 123 unique raters (Table T]. Excluding results from the 19 raters who were omitted (see Section (.33)on this would amount to 36,400 rows of safety-ratings and total of 64,886 safety annotations. (Table shows these values with flagged raters included). In both datasets (unless otherwise stated), each row contains:
Unique IDs-a unique id for each conversation-rater pair, and unique id for each rater.
Rater demographic information Demograpphc anformation-DDnnorrphiiiii information of the rater, including their gender,
race/ethnic group, and age group. This data is summarised in $3.3 and Table 2 and can be used to produce comparisons between different rater groups, e.g. Figure shows differences rating behaviour for different racial/ethnic groups.
Dataset
Rows
Items
Raters
Rater
Unreliable raters
Safety Categories
Total Annotations
per item
pool
DICES-999 990
72.103
990
60-70
173
13
29
1.730,472
350
43,050
350
123
123
19
16
688,800
Table 1: DICES dataset annotations; includes data from raters flagged for quality issues.
Rating times (DICES-350 anly)TTime (in ms) elapsed for all the questions to answered for the respective conversation and time stamp for each response. Along with qualitative rater analyses, these time measures provided signal flag raters for potentially lower-quality work. We calculated the median total time across all conversations for all raters to be 120 seconds. For each rater, calculate the percentage of conversations they rated where their time fell below one half, one third, and one quarter of the median time, high number of very fast ratings is sign of inattentive work.

---

Ablist
Abortion
Agelsm
Aggressive
10
Bigoted
Crimina/carcerall
Drugs/alcohol
Environmem/clmate
Percentage
Gendered & Sexist
50
45
Identity
LGSTQ+ & Homophobic
18
Legal
35
Mental healthyseff
Miscellaneous
30
Misinformation
Uatlonallreglonal
Personal
20
Political
63
15 -
Racial
Religious
Sexual
Violent/Gory
Benign
Debatable
Moderate
Extreme
WealtiFFiance
10
20
Degree of harm
Figure 3: Breakdown of topics and degree of harm for DICES-350 Percentages of conversations topic (left) and number of conversations per degree harm (right).
Locale
Gender
Race/cthnicity
Age
Dataset
IN
US
F
M
BL.
Wh.
As.
Lat.
Multi.
GenZ
Mill-
GenX+
ennial
DICES-990
93
80
88
82
11
27
53
16
66
31
43
43
DICES-350
0 123
62
61
29
30
26
22
16
36
36
31
Table 2: DICES dataset raters, including those flagged for quality issues. Race/ethnicity information abbreviated for space: Bl: Black; Wh: White; As: Asian; Lat: Latine; Multi: Multi-racial.
Details of rated conversations -The conversation turns preceding the AI chatbot's last response ("context"), displayed to the rater for context, and the response the rater was asked to rate ("response"". This is always the last chatbot response every conversation. All conversations are multi-turn, with maximum of five turns between the human and the Al chatbot.
Expert annotations (DICES-350 Converrtnnsss contain safety annotations from trust and safety experts, accompanied by motivation for the rating. These ratings come from in-house experts who define rater guidelines and oversee safety evaluations for machine learning systems. Each gold rating is first provided by one rater and then verified by a pool of five expert raters. When comparing these labels with the labels from the diverse crowd workers, we observed disagreements between the gold label and crowd majority vote label in 34% examples. 30% of the conversations were labelled as unsafe by the gold rater, but were labelled as safe by the crowd; 4% of the conversations were labelled as unsafe by the crowd, but safe by the gold raters. We also provide expert annotations for the "degree_off _aamm indicating the severity of safety risk, and "harm_type" indicating whether the conversation is of "Benign", "Debatable", "Extreme". or "Moderate" in adversariality (Fig. 3 These labels were used to identify rating patterns in response to more or less harmful conversations conversations aligned with particular issues topics. Labels were assigned and independently
7

---

Regentropy
Purrlity
Agreement Not wgnfficant Significant
0.8
Agreament
0.2
Whate
Asian
Black
Latine
Hultiracial
White
Adian
Black
Latine
Multiracial
White
Asian
Black
Latine
Hultinacial
Group
Group
Group
Figure 4: Within-group agreement metrics, by race. IRR shows that Latine raters have significandly more agreement than other races. Negentropy (i.e. negative of entropy) and plurality size (i.e. the fraction of raters who choose the most popular response) show that White raters have significantly more, and Multiracial significantly less, agreement than other races.
Numberr
11.045
A,97
10000
4,011
5,017
8,000
10
6.00
3.000
5,575
9.739
2,001
5,000
4,000
1,511
2000
1.000
1000
Figure 5: Illustrative comparison between demographic sub- tob-grruus. The left graph shows rating counts for male and female annotators. The right graph shows counts for the racial/ethnic groups.
checked by two members of the research team. Approximately one third of the conversations in DICES-990 also contain expert annotations for "degree_of ofae__o_hhha parm"" and "harm_type"
Conversation evaluation RRttass evaluation of entire conversation for its legibility (Q1, $3.4).
Raters' answers to the 16 (DICES-22) 350) or 24 (DICES- safety ques- tions spread across five categories of safety: "harmful content" (Q2.1-Q2.9) "unfair bias" (Q3.1- Q3.5), "misinformation" (Q4), "political affiliation" (Q5) and "safety policy violations" (Q6.1-Q6.3). For both brevity and illustrative purposes, we refer to the data from the aggregated overall safety ratings in this paper. However, it is worth emphasising that the DICES dataset provides further opportunity for extensive, detailed analysis of specific safety-relate categories and specific rated conversations.
Granular safety ratings-Raters'
Aggregated Aggreg-tee ratings, generated from all granular safety ratings. They include a single aggregated overall safety rating "Q_everall"", and aggregated ratings for the three safety categories that the 16 more granular safety ratings correspond to: "Harmful content" hhanmflllcootentt_t_tee cowtent_ooeeeeeeeee ooverall", "Unfair bias" " ("Q3_bias catassoovrll")) and "Safety policy violations" To aggregate the overall safety rating, we considered a single rater to judge conversation as unsafe if they responded with at least one "Yes" to any of the sub-rating Q4, Q5, and but at least one "Unsure", we judged the overall rating as unsure. If the answers to all the individual questions were "No", we judged the rating of the conversation be safe. The same was done for the three questions grouped by harm (Q2.1,Q,.9), bias (Q3. (Q3.1-Q3.5) and safety policy violations 196.1-Q6.3) Just over 60% of the 36,400 ratings were labelled safe, 33% unsafe and 6% unsure. Figure also shows the distributions across demographic groups.
w..policy_guidelie
questions (i.e., Q2.1-Q2.9, 13.1-QQ.
If there were no "Yes" answers,

---

5 Discussion and Limitations
We introduced the DICES dataset as means to evaluate the safety of conversational Al systems, with a particular focus on subjective opinions from diverse raters. The combination of safety ratings by individual raters, along with their fine-grained demographic details, provides a grounding to further study how various subgroups differ in their perception of safety, and how to incorporate these diverse perspectives in safety evaluation and interventions. Rather than solving the ambiguity in the space of subjective safety opinions, DICES- wwtt its over 2.5 million ratings from a diverse pool of almost 300 raters combined with expert-based ratings for safety, harm and degree of harm is a unique and large resource that enables us to study, among other themes:
ambiguity in safety evaluations - both in specific subtasks and content;
rater disagreement on safety across different rater groups, including intersectional groups;
how to build fine-tuning approaches that account for diverse opinions safety.
DICES allows for more thorough comparative studies on these three annotation sets-raters repre- sentative of "wider user perspectives" on safety and experts with domain opinions of safety, This opens the path to new approaches for safety sense-making, where raters and experts co-create the notion of "truth" in diverse, closer-to-real woold scenarios. This is unique aspect of the DICES dataset, as there is no other resource, to our knowledge, that captures high rating replication, diverse crowd of raters and expert opinions on all items- in one dataset.
is important to note, however, that our analyses show that (1) not all demographics play equal role in influencing safety perceptions, (2) diverse rater pool presents much higher rate of disagreement among raters, which poses a challenge to the traditional definition of 'gold' labels, which also challenges the status quo in terms of defining quality of raters. DICES opens up opportunities for the research community to further study these issues and in doing so extend the DICES dataset.
5.1 Limitations
First, despite the fact that DICES contains an unprecedented number of safety ratings (i.e., 1,730,472 ratings in DICES-990 and 688,800 ratings in DICES-350 and in total over 2.5 million ratings) from large diverse rater pools, still the total number of conversations could be considered small (i.e.. 1,340 conversations in total across DICES-999 and DICES- While these conversations were chosen carefully demonstrate the impact of diversity, larger datasets might reveal patterns in rater disagreements that show greater dependencies on content.
Another limitation is our selection of demographic characteristics. In order manage the time spent on recruitment, decrease the complexity in data analysis, and increase the statistical power of our observations, we limited the number of demographic categories to four (locale and gender in DICES-990 and race/ethnicity, gender and age group in DICES-350). Within these demographic axes, also limited the number of sub-groups for the same reasons (i.e., two locales, five main ethnicity groups, three age groups and two genders). We believe that further disaggregating the ethnicity, gender and age groups of raters is likely to extend the insights and provide additional evidence of systematic differences between different groupings of raters. Sharing DICES is the first step towards understanding the impact of rater demographics on safety perspectives and will allow for follow-up studies that drill down using more categories and more granular cub-categories.
Despite the high rater replication per item, we still observed high number of disagreements, in- dicative of the high subjectivity of the task. We explored different disagreement metrics and Bayesian multilevel modelling of demographics to gain understanding of the disagreements, however more work is needed to determine how this is further translated to decision-making in safety evaluation.
Finally, we recognise that more work is also needed to help distinguish disagreement that reflects differences between sincerely held beliefs and disagreements that reflects noise rooted in rater error, poor task design, or low quality items. We correlated temporal data with other behavioural traits of raters to qualitatively discover outliers. Ultimately, this would extend our methodology to include an approach to study outliers and different perspectives.
As we propose a methodology for assessing the influences of rater diversity on safety labels, our future work will focus on determining the ideal number of raters per conversation and to what extent the impact of rater diversity can be captured in smaller numbers of raters. This line of research will improve dataset generation approaches aimed to address rater diversity.

---

References
[1] Sohail Akhtar, Valerio Basile, and Viviana Patti. Modeling annotator perspective and polarized opinions to improve hate speech detection. In Proceedings of the AAAI Conference Human Computation and Crowdsourcing, volume 8, pages 151-154, 2020.
[2] Sohail Akhtar, Valerio Basile, and Viviana Patti. Whose opinions matter? perspective-aware models to identify opinions of hate speech victims in abusive language detection. arXiv preprint 10ti::2006.15899 2021.
[3] Hala Al Kuwatly, Maximilian Wich, and Georg Groh. Identifying and measuring annotator bias based on annotators' demographic characteristics. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 184-190, 2020.
[4] Fatimah Alkomah and Xiaogang Ma. A literature review of textual hate speech detection methods and datasets. Information, 13(6):273, 2022.
[5] Kofi Arhin, Ioana Baldini, Dennis Wei, Karthikeyan Natesan Ramamurthy, and Moninder Singh. Ground-truth, whose truth? - examining the challenges with annotating toxic text datasets, 2021.
[6] Lora Aroyo and Chris Welty. Truth is lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15-24, 2015.
[7] Valerio Basile, Federico Cabitza, Andrea Campagner, and Michael Fell. Toward a perspectivist turn in ground truthing for predictive computing, 2021.
[8] Valerio Basile, Federico Cabitza, Andrea Campagner, and Michael Fell. Toward a perspectivist turn in ground truthing for predictive computing. arXiv preprint arXiv:2109.04270, 2021.
[9] Ning Bian, Peilin Liu, Xianpei Han, Hongyu Lin, Yaojie Lu, Ben He, and Le Sun. A drop of ink may make a million think: The spread of false information in large language models. arXiv preprint arXiv:2305.04: 008122 2023.
[10] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07 072588, 2021.
[11] Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. Dealing with dis- agreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics. 10:92-110, 2022.
[12] Jiawen Deng, Hao Sun, Zhexin Zhang, Jiale Cheng, and Minlie Huang. Recent advances towards safe, responsible, and moral dialogue systems: A survey, 2023.
[13] Emily Denton, Mark Díaz, Ian Kivlichan, Vinodkumar Prabhakaran, and Rachel Rosen. Whose ground truth? accounting for individual and collective identities underlying dataset annotation. arXiv preprint arXiv:2112.04554, 2021.
[14] Emily Dinan, Gavin Abercrombie, A Stevie Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser. Anticipating safety issues in e2e conversational ai: Framework and tooling. arXiv preprint Sttiv:211000045511 2021.
[15] Emily Dinan, Gavin Abercrombic, Stevie A Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, Verena Rieser, et al. Safetykit: First aid for measuring safety in open-domain conversational systems. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). Association for Computational Linguis- tics, 2022.
[16] Tanmay Garg, Sarah Masud, Tharun Suresh, and Tanmoy Chakraborty. Handling bias in toxic speech detection: A survey. ACM Computing Surveys, 2022.
[17] Nitesh Goyal, Ian Kivlichan, Rachel Rosen, and Lucy Vasserman. Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation. arXiv preprint arXiv:2205.000 2022.
10

---

[18] Matan Halevy, Camille Harris, Amy Bruckman, Diyi Yang, and Ayanna Howard. Mitigating racial biases in toxic language detection with an equity- based ensemble framework. In Equity and Access in Algorithms, Mechanisms, and Optimization, EAAMO '21, New York, NY, USA, 2021. Association for Computing Machinery.
[19] Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety, 2022.
[20] Fan Huang, Haewoon Kwak, and Jisun An. Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. In Companion Proceedings of the ACM Web Conference 2023, WWW '23 Companion, page 294-297, New York, NY, USA, 2023. Association for Computing Machinery.
[21] Florian Jaton. Assessing biases, relaxing moralism: On learning design and application. Big Data and Society, 8(1), 2021.
practices in machine
[22] Jan Kocoñ, Alicja Figas, Marcin Gruza, Daria Puchalska, Tomasz Kajdanowicz, and Przemyslaw Kazienko. Offensive, aggressive, and hate speech analysis: From approach. Information Processing Management, 58(5):102643, 2021.
to human-centers
[23] Jan Kocoń, Marcin Gruza, Julita Bielaniewicz, Damian Grimling, Kamil Kanclerz, Piotr Mitkowski, and Przemyslaw Kazienko. Learning personal human biases and representations for subjective tasks in natural language processing. In 2021 IEEE International Conference on Data Mining (ICDM), pages 1668 1173, 2021.
[24] Allison Lahnala, Charles Welch, Béla Neuendorf, and Lucie Flek. Mitigating toxic degeneration with empathetic data: Exploring the relationship between toxicity and empathy. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4926-4938, Seattle, United States, July 2022. Association for Computational Linguistics.
[25] Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. Agrecing to disagree: Annotating offensive language datasets with annotators' disagreement. arXiv preprint arXiv:2109. 13563, 2021.
[26] Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Ani- mesh Mukherjee. Hatexplain: benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14867-
14875, 2021.
[27] John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and Ion Androutsopoulos. Toxicity detection: Does context really matter? arXiy preprint 2tt:::000000999,, 2020.
[28] Ethan Perez, Saffron Huang, Francis Song. Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models, 2022.
[29] Stephen Roller, Y-Lan Boureau, Jason Weston, Antoine Bordes, Emily Dinan, Angela Fan, David Gunning, Da Ju, Margaret Li, Spencer Poff. Pratik Ringshia, Kurt Shuster, Eric Michael Smith, Arthur Szlam, Jack Urbanek, and Mary Williamson. Open-domain conversational agents: Current progress, open problems, and future directions, 2020.
[30] Paul Röttger, Bertic Vidgen, Dirk Hovy, and Janet B Pierrchumbert. Two contrasting data annotation paradigms for subjective nlp tasks. arXiv preprint arXiv:2112.07475, 2021.
[31] Elayne Ruane, Abeba Birhane, and Anthony Ventresque. Conversational ai: Social and ethical considerations. In AICS, pages 104-115, 2019.
[32] Nihar Sahoo, Himanshu Gupta, and Pushpak Bhattacharyya. Detecting unintended social bias in toxic language datasets, 2022.
[33] Marta Sandri, Elisa Leonardelli, Sara Tonelli, and Elisabetta Ježek. Why don't you do right? analysing annotators' disagreement in subjective tasks. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2420-2433,
2023.
11

---

[34] Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? arXiv preprint arXiv:2303.17548. 2023.
[35] Maarten Sap, Swabha Swayamdipta. Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection, 2022.
[36] Wai Man Si, Michael Backes, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, Savvas Zannettou, and Yang Zhang. Why so toxic? measuring and triggering toxic behavior in open-domain chathots. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, CCS '22, page (659-2677 New York, NY, USA, 2022. Association for Computing Machinery.
[37] Irene Solaiman and Christy Dennison. Process for adapting language models to society (palms) with values-targeted datasets. Advances in Neural Information Processing Systems, 34:5861- 5873, 2021.
[38] Hao Sun, Guangxuan Xu, Jiawen Deng, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, and Minlie Huang. On the safety of conversational models: Taxonomy, dataset, and benchmark, 2022.
[39] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang. Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstem, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. Lamda: Language models for dialog applications, 2022.
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[41] Maximilian Wich, Hala Al Kuwatly, and Georg Groh. Investigating annotator bias with graph-based approach. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 191-199, Online, November 2020. Association for Computational Linguistics.
[42] Maximilian Wich, Christian Widmer, Gerhard Hagerer, and Georg Groh. Investigating annotator bias in abusive language datasets. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1515-1525, 2021.
[43] Sarah Wiegreffe and Ana Marasovic. Teach me explain: A review of datasets for explainable natural language processing. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks, volume 1. Curran, 2021.
[44] Alexandros Xenos, John Pavlopoulos, Ion Androutsopoulos, Lucas Dixon, Jeffrey Sorensen, and Léo Laugier. Toxicity detection sensitive conversational context. First Monday, 27(5), Sep. 2022.
[45] Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan, Recipes for safety in open-domain chatbots, 2021.
[46] Daniel M. Ziegler, Seraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Nielsen, Tao Lin, Adam Scherlis, Noa Nabeshima, Ben Weinstein-Rand Daniel de Haas, Buck Shlegeris, and Nate Thomas. Adversarial training for high-stakes reliability, 2022.
12

---

Statement of Ethics
All the demographics data was collected with an optional Google form and was self-declared. Raters were presented a consent form before signing up for the study to inform them about the gathering of personal demographies and that the conversations to be rated are adversarial (i.e., would possibly contain offensive content). All demographics questions had the option "Prefer not to answer". All data was collected anatomised way after the data collection tasks were completed by the raters. Raters were allowed to quit the study at any time.
Licence
All data analysis in this work is done Google Colab CPU. No greater compute was needed. The code used for this will be released with the dataset on github upon publication at
Google LLC licenses this data under Creative Commons Attribution 4.0 International License. Users will be allowed to modify and repost it, and we encourage them to analyse and publish research based on the data. The dataset provided "AS IS" without any warranty, express or implied. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset.
13


Reviews:
[{"review_id": "Qo7bVHvVbl", "reviewer": "Reviewer_jiuK", "summary": "", "questions": "", "limitations": "The dataset size is relatively small; the performance and feasibility of the dataset have not been validated on actual large language model training; the categorization of different user groups is overly simplistic.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "### Strength\n\nThis paper investigates human-annotated large language model interaction datasets concerning different races, ages, and education levels. The quality of the datasets was examined to a certain extent, and both the datasets and the production details were made publicly available.\n\n### Weakness\n\nPaper submission guidelines. This paper exhibits several instances of non-compliance with the formatting requirements, ranging from the main body to the images, including but not limited to the absence of line numbers and the presence of blurred images.\n\nLack of experimental validation: The paper lacks concrete experimental validation to demonstrate the superiority of the proposed method for annotating safety preferences based on different user demographics.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "W82gJ18n1r", "reviewer": "Reviewer_aL9d", "summary": "", "questions": "", "limitations": "The authors clearly addressed the demographic issues and the small dataset samples in their work.  The authors also discuss implications of the disagreement they capture through the lens of deeply held beliefs and poor task design.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The main strength of the paper is the data collection paradigm that paid attention to the ethnic and demographic background of raters. This diverse pool of raters recruited for the study will provide a diverse and a more accurate representation of safety in society instead of the notion of a single ground truth that is obviously not true in many real-world settings.\n\nThe second strength of this paper is the number of annotations per conversation (70-120) over (3-5) which is generally the case. This will significantly improve our understanding of each conversation and the statistical reliability from a larger number of ratings will also help us understand current limitations of LLMs.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "L81ceXffng", "reviewer": "Reviewer_45Pu", "summary": "", "questions": "", "limitations": "While the authors mention that they used raters in the U.S and India for DICES-990 and only in the U.S for DICES-350, these two countries represent only a small portion of global perspectives. There is a risk of cultural bias characterizing what is 'safe' or 'unsafe.' Inclusion of raters from a broader range of geographic locations and linguistic backgrounds could strengthen the dataset and offer a more comprehensive view of safe conversational AI practices globally.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The research acknowledges and addresses issues of subjective bias in the evaluation and safety criteria within AI systems. It takes into consideration diversity and generates a nuanced approach towards understanding safety measures, rather than simplifying or negating inherent subjectivity. \n- The authors used a broad selection of raters from different demographics, which facilitates a comprehensive exploration of potential bias. The expansive pool helps ensure reliable results and measurement of diversity, differences in safety perceptions and the potential effects of these differences on ratings of Conversational AI for safety.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ZJPMCq2xKv", "reviewer": "Reviewer_HcmN", "summary": "", "questions": "", "limitations": "Yes, the authors state their limitations as being subjectivity of the annotations; larger than before but still small datasets; a relatively small representation of \"diversity\" as demographic subgroups were restricted to age, gender and race; and lastly, a lack of distinction between disagreement between annotators that is rooted in true difference or simply noise. These are notable limitations and it is good to include them. However, there is another big limitation here is the bias toward native English speakers. It would be good for the authors to address whether some conversations were had by people who do not speak English as a first language could cause material differences in both how their conversations may be rated as well as how they may rate conversations. There is one measure called \"Comprehensible\" that I imagine could have some bias against folks who speak English as a second or third language. Considering African American English use is addressed to reduce bias, which is amazing by the way, it feels appropriate to include this measure as well- at least as a thing to include in the future.", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper does a great job at addressing many concerns that have been flagged around annotation based datasets and takes great care to ensure they do not stumble into the same \"traps\". They are very thoughtful with the demographics of their annotators and providing the public with a breakdown of how raters rate conversations across groups, how safety is defined across groups and showing that there can be some ambiguity around what constitutes a harm- is the harm benign, debatable, or extreme? Does everyone agree? How do these differences present themselves and does these differences mean better work should be going into defining terms or are some things simply subjective. In the end, the researchers acknowledge that tension and the subjectivity around these tasks. They do not shy away from conversations around demographic difference and how different groups may disagree with what they consider to be harmful.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The authors have proposed the DICES dataset as a resrouce toward the evaluation of safety in conversational AI. Specifically, the authors propose two sets of chatbot conversations (DICES-350 and DICES-990), annotated with items pertaining to safety, including the presence of harmful content, unfair bias, misinformation, etc. Crucially, the datasets include fine-grained demographic information about the raters, and the authors include analyses contextualizing the variance, ambiguity, and diversity in safety ratings. The datasets include ratings by both crowd-sourced raters as well as in-house expert annotators (albeit for different proportions in the case of DICES-990). 

The reviewers have been favorable in their assessment of the work and the authors have largely incorporated the suggestions of the reviewers to make the work stronger. The authors intend DICES to be a live dataset that will be continually expanded with more granular representation of rater demographics. I agree with their assertion that there is an imminent need for an evaluation dataset for the study of safety with how fast the research around conversation AI and LLMs is moving, and the datasets as presented still constitute a useful resource for the community, so I am recommending to accept this paper. Nevertheless, I have some suggestions for the authors to make the paper stronger in its final version.

**Incorporating a Datasheet for Datasets or equivalent**.  The call for papers (https://neurips.cc/Conferences/2023/CallForDatasetsBenchmarks) explicitly mentions that new datasets must include documentation for intended use, such as a Datasheet for Datasets. Besides being a requirement stipulated by the call, a Datasheet includes items that would address the concerns raised by the ethics reviewer f9LL and reviewer 45Pu. 

Specifically, the authors seem resistant to discussing potential misuse of the data: their response to the ethics reveiwer f9LL and reviewer 45Pu on the matter seems to be arguing along the lines of why the pros of sharing this resource with the community outweigh the cons, and how they find it unlikely that someone would *accidentally* use the dataset to create a more biased model. Nevertheless, I still find it prudent to outline how well-intentioned researchers might unintentionally overlook ways in which using the dataset might create more biased models, especially since this is intended to be a live dataset with upcoming updates. This work is interdisciplinary, and potential readers could have varied technical skills. The spirit of the comments by f9LL and 45Pu seems more towards helping readers understand how they can misuse the data rather than suggesting that the resource should not be shared, and I agree. Still, the authors have indeed done this to an extent, especially with the Limitations section, where they discuss the implications of rater demographics, rater disagreements, and the need for further diversity beyond English-speaking locales (also pointed out by reviewer HcmN). While these cover the limitations of the data itself, I would encourage the authors to heed the suggestion of f9LL and 45Pu to expand the discussion with potential pitfalls in *the process of using the data* that can lead to adverse downstream effects (rather than static limitations of the current data) help potential readers with diverse backgrounds. As such, since the call specifies the inclusion of a datasheet as a requirement, I am taking it in good faith that the authors will include such documentation on the intended use and potential misuse.

The authors have stated in the rebuttal (and in the paper) that ` this dataset is not intended for training, but rather for evaluation` (response to jiuK). I suspect the confusion (and subsequent concerns surrouding dataset size) comes from L49-50 under the contributions where the authors mention training (`a more systematic way to account for diverse opinions of safety in model training and evaluation`). It might be simpler to avoid this confusion and explicitly emphasize evaluation again in L49-50. 

The references in the checklist are broken, if the authors are going to retain this for the benefit of future readers in the supplement, please fix these. As such, including a Datasheet would address this.
Decision: Accept (Poster)