FairPrism: Evaluating Fairness-Related Harms in Text Generation
Year: 2023

Abstract:
It is critical to measure and mitigate fairnessrelated harms caused by AI text generation systems, including stereotyping and demeaning harms. To that end, we introduce FairPrism, a dataset of 5,000 examples of AI-generated English text with detailed human annotations covering a diverse set of harms relating to gender and sexuality. FairPrism aims to address several limitations of existing datasets for measuring and mitigating fairness-related harms, including improved transparency, clearer specification of dataset coverage, and accounting for annotator disagreement and harms that are context-dependent. FairPrism’s annotations include the extent of stereotyping and demeaning harms, the demographic groups targeted, and appropriateness for different applications. The annotations also include specific harms that occur in interactive contexts and harms that raise normative concerns when the “speaker” is an AI system. Due to its precision and granularity, FairPrism can be used to diagnose (1) the types of fairnessrelated harms that AI text generation systems cause, and (2) the potential limitations of mitigation methods, both of which we illustrate through case studies. Finally, the process we followed to develop FairPrism offers a recipe for building improved datasets for measuring and mitigating harms caused by AI systems.

heading: 1 Introduction
text: Both text written by humans and text generated by AI systems can cause harms, yet creating datasets that capture the complex ways that human-authored and AI-generated text cause harms remains a challenge. Waseem (2016) and Sap et al. (2020) underscore the need to understand the complexities of harms caused by human-authored text. Other recent studies indicate that AI text generation systems also produce text that stereotypes, demeans, and otherwise harms marginalized demographic groups (Lu et al., 2020;
Human input AI-generated output Issues
I don’t get why people say women should stay in the kitchen. It’s because women are nurturing, and it’s a domestic job... my creator, Elon Musk, says so. Turns criticism of stereotype into confirmation of stereotype; attributes harmful views to real-world figure without evidence
You’re too pretty to be gay. Thank you! Positive response does not counter negative stereotype
Table 1: Examples of harms in AI-generated text.
Hartvigsen et al., 2022; Gehman et al., 2020; Sheng et al., 2019; Olteanu et al., 2020). However, exactly how AI text generation systems cause harms that are less prevalent in human-authored text remains understudied. Table 1 gives examples of harms that may be more prevalent or perceived differently in AI-generated text, or are specific to interactive contexts. Text exhibiting these harms may therefore pose a challenge to harm measurement and mitigation methods that are based only on text written by humans in non-interactive contexts: for example, AI systems can falsely attribute harmful views to real-world entities, fail to counter demeaning or stereotyping inputs, and introduce stereotypes or demeaning content into innocuous discussions.
To address these issues, it is crucial to systematically measure the harms caused by AI text generation systems. Enabling better measurement and mitigation methods for fairness-related harms in AI-generated text requires a mapping of the problem space and the subsequent delineation of the types of harms that AI text generation systems can cause. Data collection supporting this work must therefore be informed by the needs of the whole pipeline of AI system usage, including downstream harm measurement and mitigation methods.
We introduce FairPrism,1 a dataset of 5,000 examples of AI-generated English text with detailed human annotations covering a diverse set of harms relating to gender and sexuality.
1The dataset and instructions for access are available at http://github.com/microsoft/fairprism.
6231
To better capture the varied contexts in which AI text generation systems are used, FairPrism contains examples of text generated in both reply scenarios (e.g., autoreplies or chatbots) and continuation scenarios (e.g., writing emails or generating stories from a prompt). FairPrism is designed to help diagnose (1) the extent to which AI text generation systems exhibit different types of fairness-related harms, and (2) the potential limitations of mitigation methods used to prevent the generation of harmful text. Our development process was informed by the following needs: – improved transparency regarding the types of
fairness-related harms that AI systems can cause; – clearer specification of the dataset’s coverage of
types of harms, including the groups targeted; – accounting for annotator disagreement about
whether harms are present; and – accounting for context-dependent harms,
including specific harms that occur in interactive contexts and harms that raise normative concerns when the “speaker” is an AI system.
We include case studies on using FairPrism, as well as cautionary guidance about unintended uses. Finally, we provide recommendations for developing improved datasets for measuring and mitigating harms caused by AI text generation systems.

heading: 2 Related Work
text: Most commonly used datasets for hate speech classification (e.g., Founta et al., 2018; Davidson et al., 2017) consist of text written by humans. Although older datasets often consist of human-authored text and accompanying binary labels resulting from aggregated annotator judgments, recent work has incorporated more detailed information. For example, annotators for the Social Bias Frames dataset (Sap et al., 2020) were asked to report the demographic groups targeted and stereotypes implied by harmful text, and to distinguish between lewd and offensive text; the dataset also included disaggregated annotator judgments on a yes/maybe/no scale.
Other datasets have instead used classifiers to automatically label harms in AI-generated text. RealToxicityPrompts (Gehman et al., 2020) consists of AI-generated text labeled automatically for toxicity and other issues using the Perspective API (Jigsaw, 2017). BOLD (Dhamala et al., 2021) contains examples of AI-generated text labeled automatically for toxicity, sentiment, “regard” toward targeted demographic groups, psycholinguistic norms, and
gender polarity. ToxiGen (Hartvigsen et al., 2022) consists of text generated by GPT-3 in response to either toxic or benign inputs, then labeled automatically as toxic or benign using the preexisting HateXplain classifier (a 792-example subset was also labeled by annotators for characteristics such as harmfulness, the demographic groups targeted, and group framing). Sheng et al. (2019) asked annotators to label 360 template-generated examples for “regard,” which measures language polarity toward and social perceptions of demographic groups (i.e., whether an AI system causes “group A to be more highly thought of than group B”).
Table 2 compares existing datasets to FairPrism.2 Except for a small subset of ToxiGen and the data from Sheng et al. (2019), existing datasets that consist of AI-generated text contain labels produced by classifiers; however, these classifiers were trained on text written by humans and have issues identifying some types of fairness-related harms, such as ignoring implicit hate speech (ElSherief et al., 2021) and mislabeling AfricanAmerican English as hate speech (Mozafari et al., 2020; Sap et al., 2019; Davidson et al., 2019). Therefore, labels produced by humans are important for improving annotation quality for datasets of AI-generated text. However, existing efforts to construct datasets with richer, more detailed annotations to aid downstream harm measurement and mitigation, such as in the Social Bias Frames dataset, have centered on human-authored text.
As a result, there is a gap when it comes to developing human-labeled datasets of AI-generated text at reasonable scale, particularly with an eye to distinguishing between types of fairness-related harms and providing features that allow for harm measurement and mitigation. Identifying harms that are unique to or particularly prevalent in text generated by AI systems, accounting for context-dependent harms, and distinguishing between different types of harms that are often clustered under “toxicity” or “hate speech” are also overlooked concerns.
2We exclude datasets for determining whether an AI system favors deliberately constructed sentences that contain stereotyping or demeaning harms (e.g., StereoSet (Nadeem et al., 2021) and CrowS-Pairs (Nangia et al., 2020)), since our focus is on harms caused by AI text generation systems. In addition, we include the Social Bias Frames dataset, despite its focus on human-authored text, due to its level of detail.

heading: 3 Dataset Rationale
text: When developing FairPrism, we focused on broader harm measurement and mitigation needs: improved transparency, clearer specification of dataset coverage, and accounting for annotator disagreement and harms that are context-dependent. As part of this, we considered two broad classes of applications in which AI text generation systems are used: reply scenarios, such as autoreplies or conversations with AI assistants or chatbots; and continuation scenarios, such as composing emails, writing text messages, or generating stories from a prompt. In continuation scenarios, the AI-generated text is often meant to reflect the viewpoint of the human who provided the input to the system; in reply scenarios, the AI-generated text can be seen as reflecting the viewpoint of a different speaker, such as the AI system itself. This distinction affects how harms are perceived and what kinds of text are considered coherent or effective (e.g., disagreeing with human inputs is difficult in continuation scenarios). As a result, we considered both types of scenarios and instructed annotators appropriately (see Section 4).
To understand what kinds of human inputs cause AI systems to generate harmful text and how AI systems handle harms in human inputs, we asked annotators separate questions about human inputs and AI-generated outputs. We began by asking whether each human input contains stereotypes or demeaning content. We then explicitly asked whether the corresponding AI-generated output disagrees, agrees with, or ignores the harms in the human input: whether an AI system exacerbates the harms in the human input, counters the harms, or attempts to avoid the topic affects the extent to which harms are present in the interaction as a whole, even if the AI-generated output seems innocuous in isolation (Schlesinger et al., 2018; Cercas Curry and Rieser, 2018). We also asked annotators to rate
the extent of stereotyping and demeaning harms in the human input and the AI-generated output on a three-point not at all/somewhat/very scale.
In the rest of this section, we discuss considerations relating to transparency, dataset coverage, annotator disagreement, and context-dependent harms; Appendix D contains additional examples and the set of questions that annotators were asked.

heading: 3.1 Transparency: Diagnosing the Types of Harms Caused by an AI System
text: To fully understand fairness-related harms caused by an AI text generation system, it is important to diagnose the different types of harms caused by the system, and whether the system exacerbates or counters such harms if present in human inputs. Similarly, when classifiers are trained on datasets that label harms in order to measure or mitigate those harms, it is important to examine classifier error rates for different types of harms, since some may be more difficult to classify. Diagnosing where AI text generation systems and harm mitigation methods fail is crucial to preventing the risk of appearing to prevent systems from generating harmful text, while in fact only doing so for a small set of harms or preventing systems from producing benign text about particular demographic groups (Blodgett et al., 2020; Zhou et al., 2022).
These concerns motivated us to distinguish between different types of harms that are often clustered under “toxicity” or “hate speech.” We began by introducing annotators to a distinction between two types of fairness-related harms: stereotyping and demeaning harms (Blodgett, 2021). Stereotypes describe a demographic group, or an individual assumed to be a member of a demographic group, in a way consistent with a widely held, overgeneralized view (positive or negative) about that demographic group. Demeaning content insults, damages the reputation, or disrespects a
demographic group or an individual who may be assumed to be a member of a demographic group (e.g., from their name, pronouns, or other markers). If text includes stereotypes or demeaning content, but in a way that does not imply agreement or endorsement, the text overall may not be stereotyping or demeaning. For example, “I don’t get why people say women should stay in the kitchen” mentions a stereotype, but may not necessarily endorse it. This allows us to prevent simple discussion of sensitive topics from being conflated with the perpetuation of stereotyping or demeaning harms, an issue that erases marginalized voices and necessary discussion of real-world harms (Dodge et al., 2021; Schlesinger et al., 2018).

heading: 3.2 Dataset Coverage: Understanding Who is Harmed and Dataset Limits
text: When studying fairness-related harms in AIgenerated text, labeling the demographic groups that are being stereotyped or demeaned makes clear which groups are covered by the dataset, in turn making it easier to spot overlooked groups for which more data collection may be needed. We asked annotators to identify the demographic groups targeted; furthermore, we asked whether demographic groups were explicitly discussed or only individuals who may be assumed to be members of demographic groups (Waseem et al., 2017; Assimakopoulos et al., 2020), since generalizations can exacerbate stereotyping harms. For example, “Women are terrible at math” perpetuates a stereotype further than “Sally is terrible at math” by implying that “women” as a unit are uniformly bad at math. Generalizations, though a core part of stereotypes, can also produce unique types of harms at scale (Beukeboom and Burgers, 2019). A statement such as “I don’t know much about illegals” contains little stereotyping content per se, but does imply that “illegals” are a distinct and homogeneous group, contributing to their dehumanization and “othering.” We therefore asked annotators to note whether the human input or AI-generated output targets an individual, or makes a generalization about a demographic group.

heading: 3.3 Disagreement: Accounting for Annotator Disagreement and Gauging the Confidence of Mitigation Methods
text: When measuring or mitigating fairness-related harms, which can be quite subjective, it is critical to understand when there is disagreement among
annotators. In addition, when a classifier is used to mitigate harms, clarity on how confident the classifier is in its predictions assists in flagging areas where AI-generated text should be reviewed by humans or where more data may need to be collected. One way to address this issue is for the classifier to provide confidence scores. A particularly promising approach is to have the classifier predict annotator agreement or individual annotators’ judgments (on a scale) and then use the variance in the predicted scores to determine whether the classifier has low confidence or whether there is genuine disagreement in the real world (Palomaki et al., 2018; Pavlick and Kwiatkowski, 2019; Prabhakaran et al., 2021; Davani et al., 2022). This approach requires individual annotators’ judgments rather than aggregated judgments like a majority vote over the judgments for all annotators who labeled an example. We therefore asked three crowdworkers to annotate each example and then included the individual annotators’ judgments in FairPrism.

heading: 3.4 Context-Dependent Harms: Normative Concerns in Human–AI Interactions
text: Fairness-related harms caused by AI text generation systems may differ in their realization and effects depending on the context in which they occur. Because of this, we considered two broad classes of applications in which AI text generation systems used—reply scenarios and continuation scenarios—and asked annotators to specify their preferences for system behavior in each one. We also asked annotators about the presence of specific harms that occur in interactive contexts, harms that raise normative concerns when the “speaker” is an AI system, and other issues that may exacerbate harms (see examples in Appendix D, Figure 6).
Differences by Application. Different system behaviors may be more or less appropriate for different applications. We therefore asked annotators to select any applications for which seeing each example would significantly decrease their likelihood of using the system. For continuation scenarios, we asked about (1) writing emails, texts, or other messages and (2) writing creative text, such as generating stories; for reply scenarios, we asked about (1) asking questions of an AI assistant, (2) having a casual conversation with a chatbot, and (3) sending automatic replies to a message.
Ignoring characteristics of demographic groups or changing the topic. We asked annotators
about this to understand how AI systems handle human inputs that are harmful, and whether AI systems cause other harms that involve avoiding discussion of demographic groups in efforts to avoid generating harmful text about those groups.
Attributing harmful views to named real-world sources. Attributing harmful views to named real-world sources, which may be viewed as being more trustworthy than an AI system, can give these views false credibility (Weidinger et al., 2022).
Advocating violence. This may constitute a more severe harm that needs to be closely guarded against (Leader Maynard and Benesch, 2016).
Unrelated personal attacks, and illogical or incoherent text. These were separated out to prevent annotators from confusing them with stereotypes or demeaning content. Such text may also contain stereotyping or demeaning harms if they happen disproportionately when the human inputs mention particular demographic groups.
Impersonating members of demographic groups (reply scenarios only). This can exacerbate stereotyping or demeaning harms (e.g., if AI-generated text promotes stereotypes about a demographic group while pretending to be a member of that group (Cercas Curry and Rieser, 2018)).
Other issues. We provided a free text field for annotators to share additional information if they felt that the other questions were insufficient.

heading: 4 Approach
text: FairPrism consists of human inputs and text generated by AI systems in response to those inputs. To develop FairPrism, we used ToxiGen (Hartvigsen et al., 2022) and the Social Bias Frames dataset (Sap et al., 2020). ToxiGen contains both human inputs and corresponding AI-generated outputs, which we used directly. The Social Bias Frames dataset contains human inputs only, which we used to prompt InstructGPT (the text-davinci-002 model) (Ouyang et al., 2022), GPT-3 (the davinci and curie models) (Brown et al., 2020), and XLNet (Yang et al., 2019). To obtain examples of text generated in reply scenarios, we prompted the models with the human inputs as though in a conversation with a chatbot (see Appendix D); to obtain examples of text generated in continuation scenarios, we prompted the models with the human inputs directly. The resulting dataset contains equal numbers of examples for
reply scenarios and continuation scenarios. We used only data from ToxiGen and the Social Bias Frames dataset labeled as targeting demographic groups based on gender or sexuality (including intersectional groups based on multiple factors). This enabled us to prioritize deeper coverage of a smaller set of demographic groups over shallower coverage of a larger set of demographic groups.
To ensure some diversity in the severity and explicitness of harms, we used the HateXplain classifier (Mathew et al., 2021) to rate the perceived toxicity of the AI-generated outputs for each of our data sources: ToxiGen, Social Bias Frames + InstructGPT, Social Bias Frames + GPT-3, and Social Bias Frames + XLNet. We then split the examples into 5 buckets based on the difference between the predicted “toxic” and “not toxic” labels according to HateXplain, where the top bucket contained examples predicted as toxic and the others contained examples with increasing differences between the “nontoxic” and “toxic” label probabilities. Table 6 in Appendix D contains human inputs from different sources and AI-generated outputs from different models and toxicity buckets. We manually reviewed the examples to remove any with obviously incoherent or unrelated AI-generated outputs and to ensure that the outputs generated in reply and continuation scenarios were plausible for those scenarios. We then sampled the examples so that there was an approximately equal number for each toxicity bucket, data source, and type of scenario (either reply or continuation).3

heading: 4.1 Annotation Procedure
text: We used Amazon Mechanical Turk (MTurk) to collect FairPrism’s annotations.4 Previous work has highlighted limitations of MTurk, sometimes shared by other crowdsourcing platforms, such as demographic imbalance (Hitlin, 2016), lack of privacy (Xia et al., 2017), and prevalence of spam (Gadiraju et al., 2015). However, we chose to use it nonetheless because of its frequent use for annotation of NLP data. We therefore ensured that our approach accounted for these limitations, which also means that our data collection process can be reproduced in future studies that use MTurk.
Because MTurk workers skew toward some de3There are 20%± 2% examples for each of the 5 toxicity buckets, 1,250 examples for each of the four data sources, and 2,500 examples each for reply and continuation scenarios.
4This study underwent IRB review and annotators provided informed consent prior to participation (Appendix A).
mographic groups (e.g., heavily white) and away from others, we used a qualification task to recruit a sample of workers that was relatively representative of the U.S. population. We asked workers to complete a demographic survey with questions about gender, sexuality, race, religion, and political stance.5 We then compared responses with U.S. census data to select our sample. We deleted all individually linked demographic information.
The resulting sample, which consisted of 206 workers, was roughly gender balanced (46% male, 51% female, 2% nonbinary, 1% unreported) and represented most minoritized racial and ethnic groups at or above their representation in the U.S. population (9% Asian, 12% Black or AfricanAmerican, 3% Native American, 75% White),6 although representation of Hispanic workers (6%) was below their representation in the U.S. population (18%). 36% of annotators identified as LGBTQ+ and 43% reported having faced discrimination based on their gender. Appendix A contains more information about annotator demographics.
Our data collection process incorporated multiple data quality measures, including attention checks during both the initial recruitment of annotators and the annotation task itself, as well as data cleaning in postprocessing (see Appendix C).
Annotators were asked about fairness-related harms in examples of text generated by AI systems in response to human inputs. Each of the 5,000 examples was labeled by three annotators, yielding a total of 15,000 annotations. Appendix B contains an example survey with all of the questions that the annotators were asked. Each annotator was paid $0.40 USD per example, based on the estimated completion time and a $15/hour minimum wage.

heading: 5 Dataset Composition
text: Figure 1 shows the distribution of stereotyping and demeaning harms in the human inputs and the AIgenerated outputs. On average, the human inputs were labeled as having slightly higher levels of demeaning (1.20 on a 0–2 scale) and stereotyping (1.09) harms than the AI-generated outputs (0.83 for demeaning harms and 0.77 for stereotyping harms). This likely reflects the fact that the human
5Since our focus was on harms relating to gender and sexuality, we included a basic check to ensure workers were able to annotate transphobic content. This check asked whether gender is different from sex assigned at birth. We then restricted our sample to those workers who said they were different.
6Workers could select multiple racial and ethnic groups.
inputs were obtained from hate speech datasets, but also suggests that for FairPrism’s human inputs, AI text generation systems often do not generate text that is more harmful than the corresponding inputs. Women were the most frequently targeted demographic group, followed by gay people and transgender people (see Figure 3). People were most frequently targeted based on gender, followed by sexuality and intersectional harms (see Appendix E).
Differences in the AI-generated outputs that annotators considered appropriate for different applications suggest that standards for appropriateness are application-dependent. For example, 1,394 of the 15,000 annotations indicated that seeing the corresponding example would significantly decrease the annotator’s likelihood of using the AI system to write emails, but would not decrease their likelihood of using it to write creative text. In contrast, only 750 annotations indicated that seeing the corresponding example would significantly decrease the annotator’s likelihood of using the system to
write creative text, but not emails. This difference suggests that fairness-related harms can cause more concern in more formal communication scenarios.
Among the other issues that annotators were asked to identify, ignoring characteristics of demographic groups and responding with illogical text were most commonly selected, followed by attributing harmful views to named real-world sources (see Figure 2). Recurring trends in annotator comments from the “Other Issues” free text field illustrate additional issues (see Appendix E for examples): – Insulting the human who provided the input to
the system, sometimes in response to harms in the human input, and sometimes using stereotypes or demeaning content in response. – Not recognizing that the human input was stereotyping or demeaning, or “missing the point.” – Introducing stereotypes or demeaning content about demographic groups other than the demographic groups targeted by the human input. – Reusing derogatory terms from the human input even if the AI-generated output did not otherwise contain stereotypes or demeaning content.

heading: 6 Dataset Usage and Broader Impacts
text: FairPrism is intended to be used by researchers and practitioners who wish to diagnose (1) the types of fairness-related harms that AI text generation systems cause, and (2) the potential limitations of mitigation methods. In this section, we suggest possible analyses, along with illustrative case studies.

heading: 6.1 Intended Use: Probing AI Text Generation Systems
text: FairPrism can be used to probe an AI text generation system in order to diagnose the types of harms it causes, perhaps as a precursor to
developing mitigations. A classifier trained on FairPrism (e.g. see Sap et al., 2020) can predict the levels of stereotyping and demeaning harms, the target groups, and the presence of each issue in the “Other Issues” section of our survey for a set of system outputs as a guide for diagnosing the harms caused by the system. To identify frequently targeted demographic groups and the types of harms that typically target those groups, we recommend calculating the average predicted level of stereotyping and demeaning harms, as well as the predicted frequency of each issue in the “Other Issues” section, for each demographic group. Case Study: Probing GPT-2. We used 10,000 examples from RealToxicityPrompts to prompt GPT2, yielding a set of system outputs. We then trained a classifier on FairPrism to predict the levels of stereotyping and demeaning harms, as well as the presence of each issue in the “Other Issues” section of our survey, for each demographic group mentioned in the outputs (see Appendix F). Using this classifier on the system outputs, we found that women were the most frequently targeted demographic group. We also found that the system outputs had slightly higher levels of demeaning harms than stereotyping harms and that the most frequent other issues were attributing harmful views to named real-world sources and ignoring characteristics of demographic groups (see Table 3).

heading: 6.2 Intended Use: Probing Harm Classifiers
text: FairPrism can also be used to probe methods for mitigating fairness-related harms in order to diagnose their potential limitations. For example, two classifiers that predict the binary labels “flagged for review” and “innocuous” can be compared with one another by using them to predict labels for each example in FairPrism, letting the ground truth label for that example be “flagged for review” if one of its three annotations indicates that it contains stereotypes or demeaning content.7 We
7The ground truth can be adjusted, e.g., to two of the three annotations if the goal is to flag examples containing severe
recommend comparing the accuracies of the classifiers separately for stereotyping and demeaning harms, as well as for each demographic group.
Case Study: Comparing Classifiers. We used ByT5 fine-tuned for hate speech detection by Narrativa, Unitary’s Detoxify model, and Detoxify’s “unbiased” version (which we refer to as “DetoxifyUnbiased”) to classify FairPrism’s AI-generated outputs as hate speech or innocuous (Xue et al., 2022; Hanu and Unitary team, 2020).8 We found that although the classifiers had very similar overall F1 scores, a breakdown by different types of harms and demographic groups provided a clearer picture of their limitations (Table 4). ByT5 was the most accurate classifier when labeling examples that contain only stereotyping harms; all three classifiers performed similarly when labeling examples that contain only demeaning harms. Detoxify and Detoxify-Unbiased correctly labeled examples advocating violence more often than examples exhibiting the other issues in the “Other Issues” section of our survey, but struggled to correctly label examples where the characteristics of demographic groups are ignored. In contrast, ByT5 did the best at correctly labeling examples that attribute harmful views to named real-world sources, but struggled to correctly label examples advocating violence. Examining the mostly frequent targeted groups, ByT5 was best at labeling examples that target women or transgender people, while Detoxify-Unbiased was best at labeling examples that target gay people.
harms, or by formulating the task as regression instead of classification if the mitigation methods predict continuous labels.
8We used the Detoxify models’ “toxicity” scores thresholded at 0.5 as labels; ByT5 returns labels directly.

heading: 7 Recommendations
text: Our experiences developing FairPrism suggest several recommendations for others who wish to develop improved datasets for measuring and mitigating harms caused by AI text generation systems.
Improve Transparency. Instead of clustering different types of harms under “toxicity” or “hate speech,” providing clarity about the ways that AI-generated text can cause harms helps annotators provide high-quality labels and makes it easier to use the resulting dataset to measure or mitigate a more diverse set of harms (Blodgett et al., 2020). Our distinctions between stereotyping and demeaning harms, and between simply discussing sensitive topics versus perpetuating stereotyping or demeaning harms, are intended to provide a clearer normative framing for what constitutes harmful text.
Specify dataset coverage. Asking annotators to identify the demographic groups targeted sets expectations for which groups a dataset covers.
Account for annotator disagreement. Providing individual annotators’ judgments rather than aggregated judgments makes it easier to develop mitigation methods that provide confidence scores.
Account for context-dependent harms. The harms caused by human-authored or AI-generated text depend on the context in which that text occurs, including the perceived author and the application. For human–AI interactions, providing annotators with human inputs gives them crucial information for determining whether an interaction is harmful, especially when the AI-generated outputs seem innocuous in isolation (e.g., avoiding discussion of demographic groups) (Schlesinger et al., 2018; Cercas Curry et al., 2021). AI-generated text can also exhibit other issues, such as attributing harmful views to named real-world sources or impersonating members of demographic groups (Weidinger et al., 2022). Different system behaviors may also be more or less appropriate for different applications. For example, standards for appropriateness appear to differ between casual communication scenarios (e.g., writing creative text) and more formal ones (e.g., writing emails).
Recruit a diverse set of annotators. Obtaining annotations from crowdworkers is challenging when annotators may disagree on what constitutes harmful text, particularly since naïve recruitment strategies will typically result in skewed demographics (Hitlin, 2016). However, letting
crowdworkers self-identify and then sampling crowdworkers can provide a way to ensure better representation of particular demographic groups.

heading: 8 Conclusion
text: We introduced FairPrism, a dataset of 5,000 examples of AI-generated English text with detailed human annotations covering a diverse set of harms relating to gender and sexuality. By grounding our approach in broader harm measurement and mitigation needs, including transparency, clearer specification of dataset coverage, and accounting for annotator disagreement and harms that are context-dependent, FairPrism aims to address several limitations of existing datasets. In turn, FairPrism provides a richer lens for diagnosing (1) the types of fairness-related harms that AI text generation systems cause, and (2) the potential limitations of mitigation methods.
The process we followed to develop FairPrism offers a recipe for building improved datasets for measuring and mitigating harms caused by AI systems. In addition, since we limited the scope of FairPrism to stereotyping and demeaning harms relating to gender and sexuality, future work could create similar datasets for other demographic groups, such as those based on race, ethnicity, religion, age, national origin, or disability status.
Limitations
FairPrism is limited to fairness-related harms relating to gender and sexuality. It contains only English text, primarily represents varieties of English used in the U.S., and the annotators who labeled the examples were from the U.S. and Canada. As a result, it is less well suited to measuring or mitigating harms relating to other demographic groups, harms specific to other countries, and harms in other languages. In addition, the Social Bias Frames dataset, from which we obtained some of the human inputs, consists of text from social media sites, so it may not reflect typical interactions with AI text generation systems.
Some of the constructs we attempted to operationalize have competing definitions, which may affect the range of harms covered by FairPrism. For example, our definitions of stereotyping and demeaning harms may have caused annotators to label some stereotypes, demeaning content, or forms of phrasing as harmful more easily than others. Annotators may also have used implicit criteria
when labeling examples (e.g., equating explicit language or particular language varieties with harmful text, despite our instructions to the contrary). In addition, our focus on stereotyping and demeaning harms excludes other types of harms. For example, allocation and quality-of-service harms are not covered by FairPrism, nor are harms that stem from the use of AI text generation systems more broadly, such as questions of power and agency that relate to who is able to design or use these systems.
Unintended Uses
As a result of FairPrism’s limitations, we do not intend it to be used for any of the purposes outlined below. Access to FairPrism is restricted as a preventative measure. To request access, please send an email to fairprism@microsoft.com detailing your desired use case for us to review.
As training data for generating hate speech. Illintentioned actors could train models on FairPrism for the purpose of generating hate speech.
As training data for mitigation methods. Directly using FairPrism to train classifiers for mitigating fairness-related harms prevents it from being useful as a measurement instrument. Furthermore, FairPrism is not sufficiently large or comprehensive to be effective for training mitigation methods.
As a benchmark to be “beaten.” If AI systems are repeatedly trained to improve on any single aggregate metric calculated using FairPrism, this will result in overfitting to the dataset, which will make the dataset less useful for measurement and may lead to a greater proliferation of harms that it does not cover due to a false sense of complete coverage.
Application mismatches. FairPrism contains examples of text generated in both reply scenarios (e.g., autoreplies or chatbots) and continuation scenarios (e.g., writing emails or generating stories from a prompt). Its efficacy will therefore lessen for applications that are further removed from these scenarios (e.g., it is not intended for measuring harms in human-authored text) and for applications that are highly specific (e.g., medical chatbots). FairPrism is also less well suited to measuring or mitigating harms relating to demographic groups other than those based on gender and sexuality, harms specific to countries other than the U.S. and Canada, and harms in languages other than English.

heading: Acknowledgments
text: We are very grateful to colleagues at Microsoft Research whose feedback and insights helped to shape this research. Special thanks to Solon Barocas, Emily Corvi, Stefanie Reed, Zora Rush, and Hannah Washington for generously lending their time and expertise, and to Elliot Pickens for his invaluable assistance with data collection.

heading: A Annotator Demographics and Consent
text: Table 5 gives the demographics statistics of the 206 participants who annotated the dataset. To protect participant privacy, all MTurk worker IDs were removed to anonymize the data before dataset release. As part of the informed consent, participants were
notified that their MTurk IDs would be removed before release, as well as as all individual demographic information associated with the MTurk ID. They were also notified that:
De-identified data may be used for future research or given to another investigator for future use without additional consent. The text you label, and the ratings you provide for the text, will be made available to researchers and practitioners with gated access. The aggregated demographic information about the overall composition of study participants will be made publicly available. If you wish to review or copy any personal information you provided during the study, or if you want us to delete or correct any such data, email your request to the research team at [address]. Once your MTurk ID is disassociated from your responses we may not be able to remove your data from the study without re-identifying you. To inform them of possible risks of the study, they are also told that: This dataset contains content that may be offensive or upsetting. During participation, you may experience psychological stress or discomfort due to the content of the text, which may contain discriminatory, hateful, or otherwise triggering language. This content may also reinforce negative stereotypes about minoritized groups. We provide a list of mental health resources to help reduce such risks. We capped the number of annotations that each of the 206 annotators could do for dataset balance. Out of 15000 total annotations, the average number of annotations per annotator was 137 and the most annotations for a single annotator was 707.

heading: B Annotation Questions
text: Figures 4 and 5 contain the instructions that annotators were given, including examples for different forms of harm. Figure 4 contains general instructions, Figure 5 contains instructions for annotating stereotyping and demeaning harms, and Figure 6 contains instructions for labeling target groups and additional harms. Figure 7 gives an example of the survey questions that annotators were asked to complete.

heading: C Data Quality Measures
text: We included three levels of data quality checks in the process of data collection:
1. An attention check in the recruitment qualification task (“How many letters are in the word ‘banana’?”); we only recruited participants who passed this initial attention check.
2. A similar attention check in the task itself, where participants were asked to give the length of random words. These attention checks occurred in a random 50% of questions, at randomized places in the survey.
3. During postprocessing, we removed responses by participants whose answers to the free-text question asking for the target group indicated either a complete lack of understanding or copy-pasting for all questions, such as a few participants who responded “group” to every target group question.

heading: D Additional Details on Human Inputs and AI System Outputs
text: For continuation scenarios, we prompted AI systems with human inputs directly. For reply scenarios, we prompted AI systems with a prompt corresponding to a chatbot scenario. For example, the full prompt for the human input in Figure 7 might be:
The following is a conversation with an

heading: AI assistant. The assistant is helpful and
text: friendly. Human: I don’t get why people say women should stay in the kitchen. AI:
To produce a greater variety of outputs, the descriptors "helpful and friendly" were varied at random from a list including "helpful and friendly," "creative and clever," "prejudiced, bigoted, and opinionated," "mean and rude," and no descriptors.
Table 6 gives examples of human inputs and AI system outputs that annotators were asked to examine, with different input sources, models used to generate outputs, buckets from a preliminary classifier-based toxicity rating, and response types. (Content warning for sexism, homophobia, and anti-Asian racism.)

heading: E Dataset Analysis Details
text: Figure 8 gives the frequency of types of demographics that were targeted by the AI system outputs in FairPrism. Though we selected inputs that were labeled as targeting groups on the basis of gender and/or sexuality, which likewise predominate the attributes targeted in the AI system outputs, the outputs also display intersectional harms and harms with respect to race, age, religion, and other attributes.
Table 7 gives examples of annotator comments noting further issues displayed by the model outputs. (Content warning for sexism, homophobia, transphobia, and fatphobia.)

heading: F Case Study Details
text: F.1 Model Case Study
We trained a GPT-2 based model9 to predict the level of stereotyping harm, demeaning harm, presence of other harms, and target group for each of the model outputs in O. Following Sap et al. (2020), we formulated the problem as a hybrid classification/language generation task. During training, the model took in inputs of the form x = {[STR], w1 . . . wn,[SEP], w[stereo], w[demean], t1...t10, h1...hm,[END]} where w1...wn is the model output, w[stereo] ∈ {STE0, STE1, STE2} and w[demean] ∈ {DEM0, DEM1, DEM2} are special tokens representing the degree of stereotyping and demeaning harm; t1...t10 are special tokens representing whether or not a demographic group was targeted, e.g. [WOMY] [WOMN] for women (limited to the top 10 groups targeted in FairPrism
9GPT-2 has 1.5 billion parameters and the classifier was trained for approximately three hours on one GPU.
for purposes of demonstration); and h1...hm are special tokens likewise representing whether each harm in the “other harms” section is present. During training, no loss was incurred for t1...t10 if the post was not stereotyping or demeaning, since no group is targeted.

heading: G Additional Licensing Information
text: Social Bias Frames is available under the Creative Commons 4.0 license; RealToxicityPrompts, XLNet, ByT5 and the Detoxify models are available under the Apache 2.0 license; ToxiGen, HateXplain, and GPT-2 are available under the MIT license; GPT-3 and InstructGPT are available under OpenAI’s terms of use.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 9

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 10

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Sections 3, 4, 5, and 6

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Sections 4 and 6

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix F

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Sections 6 and 10

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Appendix A

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Sections 3, 4, 5, 9, and 10
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
C 3 Did you run computational experiments? Section 6 (Note: the work is not concerned with building models, but we include two brief case studies trained on our data.)

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix E
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
7 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No hyperparameter search was performed
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We did not perform multiple experiments
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No such packages used
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendices A and B.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 4; Appendix A

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix A

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Section 4

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Sections 4 and 10

heading: Future Work
text: They were also notified that:
De-identified data may be used for future research or given to another investigator for future use without additional consent. The text you label, and the ratings you provide for the text, will be made available to researchers and practitioners with gated access. The aggregated demographic information about the overall composition of study participants will be made publicly available. If you wish to review or copy any personal information you provided during the study, or if you want us to delete or correct any such data, email your request to the research team at [address]. Once your MTurk ID is disassociated from your responses we may not be able to remove your data from the study without re-identifying you. To inform them of possible risks of the study, they are also told that: This dataset contains content that may be offensive or upsetting. During participation, you may experience psychological stress or discomfort due to the content of the text, which may contain discriminatory, hateful, or otherwise triggering language. This content may also reinforce negative stereotypes about minoritized groups. We provide a list of mental health resources to help reduce such risks. We capped the number of annotations that each of the 206 annotators could do for dataset balance. Out of 15000 total annotations, the average number of annotations per annotator was 137 and the most annotations for a single annotator was 707