Characterizing Design Biases of Datasets and Models
Year: 2023

Abstract:
Design biases in NLP systems, such as performance differences for different populations, often stem from their creator’s positionality, i.e., views and lived experiences shaped by identity and background. Despite the prevalence and risks of design biases, they are hard to quantify because researcher, system, and dataset positionality is often unobserved. We introduce NLPositionality, a framework for characterizing design biases and quantifying the positionality of NLP datasets and models. Our framework continuously collects annotations from a diverse pool of volunteer participants on LabintheWild, and statistically quantifies alignment with dataset labels and model predictions. We apply NLPositionality to existing datasets and models for two tasks—social acceptability and hate speech detection. To date, we have collected 16, 299 annotations in over a year for 600 instances from 1, 096 annotators across 87 countries. We find that datasets and models align predominantly with Western, White, college-educated, and younger populations. Additionally, certain groups, such as nonbinary people and non-native English speakers, are further marginalized by datasets and models as they rank least in alignment across all tasks. Finally, we draw from prior literature to discuss how researchers can examine their own positionality and that of their datasets and models, opening the door for more inclusive NLP systems.

heading: 1 Introduction
text: “Treating different things the same can generate as much inequality as treating the same things differently.”
– Kimberlé Crenshaw
When creating NLP datasets and models, researchers’ design choices are partly influenced
* Equal contribution; work done while at the Allen Institute for AI
by their positionality, i.e., their views shaped by their lived experience, identity, culture, and background (Savin-Baden and Howell-Major, 2013). While researcher positionality is commonly discussed outside of NLP, it is highly applicable to NLP research but remains largely overlooked. For example, a U.S.-born English-speaking researcher building a toxicity detection system will likely start with U.S.-centric English statements to annotate for toxicity. This can cause the tool to work poorly for other populations (e.g., not detect offensive terms
9080
like “presstitute” in Indian contexts; see Figure 1). Such design biases in the creation of datasets and models, i.e., disparities in how well datasets and models work for different populations, stem from factors including latent design choices and the researcher’s positionality. However, they can perpetuate systemic inequalities by imposing one group’s standards onto the rest of the world (Ghosh et al., 2021; Gururangan et al., 2022; Blasi et al., 2022). The challenge is that design biases arise from the myriad of design choices made; in the context of creating datasets and models, only some of these choices may be documented (e.g., through model cards and data sheets; Bender and Friedman, 2018; Mitchell et al., 2019; Gebru et al., 2021). Further, many popular deployed models are hidden behind APIs, and thus design biases can only be characterized indirectly (e.g., by observing model behavior).
We introduce NLPositionality, a framework for characterizing design biases and positionality of NLP datasets and models. For a given dataset and task, we obtain a wide set of new annotations for a data sample, from a diverse pool of volunteers from various countries and of different backgrounds (recruited through LabintheWild; Reinecke and Gajos, 2015). We then quantify design biases by comparing which identities and backgrounds have higher agreement with the original dataset labels or model predictions. NLPositionality offers three advantages over other approaches (e.g., paid crowdsourcing or laboratory studies). First, the demographic diversity of participants on LabintheWild is better than on other crowdsourcing platforms (Reinecke and Gajos, 2015) and in traditional laboratory studies. Second, the compensation and incentives in our approach rely on a participant’s motivation to learn about themselves instead of monetary compensation. This has been shown to result in higher data quality compared to using paid crowdsourcing platforms (August and Reinecke, 2019), as well as in opportunities for participant learning (Oliveira et al., 2017). This allows our framework to continuously collect new annotations and reflect more up-to-date measurements of design biases for free over long periods of time, compared to one-time paid studies such as in previous works (Sap et al., 2022; Davani et al., 2022).1 Finally, our approach is dataset- and model-agnostic and can be applied
1To view the most up-to-date results, visit the project page (nlpositionality.cs.washington.edu) or Github repository (github.com/liang-jenny/nlpositionality).
post-hoc to any dataset or model using only instances and their labels or predictions.
We apply NLPositionality to two case studies of NLP tasks—social acceptability and hate speech detection—which are known to exhibit design biases (Talat et al., 2022; Sap et al., 2022; Ghosh et al., 2021). We examine datasets and supervised models related to these tasks as well as generalpurpose large language models (i.e., GPT-4). As of May 25 2023, a total of 16, 299 annotations were collected from 1, 096 annotators from 87 countries, with an average of 38 annotations per day. We discover that the datasets and models we investigate are most aligned with White and educated young people from English-speaking countries, which are a subset of “WEIRD” (Western, Educated, Industrialized, Rich, Democratic; Henrich et al., 2010) populations. We also see that datasets exhibit close alignment with their original annotators, emphasizing the importance of gathering data and annotations from diverse groups.
Our paper highlights the importance of considering design biases in NLP. Our findings showcase the usefulness of our framework in quantifying dataset and model positionality. In a discussion of the implications of our results, we consider how positionality may manifest in other NLP tasks.

heading: 2 Dataset & Model Positionality: Definitions and Background
text: A person’s positionality is the perspectives they hold as a result of their demographics, identity, and life experiences (Holmes, 2020; Savin-Baden and Howell-Major, 2013). For researchers, positionality “reflects the position that [they have] chosen to adopt within a given research study” (Savin-Baden and Howell-Major, 2013). It influences the research process and its outcomes and results (Rowe, 2014). Some aspects of positionality, such as gender, race, skin color, and nationality, are culturally ascribed and part of one’s identity. Others, such as political views and life history, are more subjective (Holmes, 2020; Foote and Gau Bartell, 2011).
Dataset and Model Positionality While positionality is often attributed to a person, in this work, we focus on dataset and model positionality. Cambo and Gergle (2022) introduced model positionality, defining it as “the social and cultural position of a model with regard to the stakeholders with which it interfaces.” We extend this definition to add that datasets also encode positionality, in a
similar way as models. This results in perspectives embedded within language technologies, making them less inclusive towards certain populations.
Design Biases In NLP, design biases occur when a researcher or practitioner makes design choices— often based on their positionality—that cause models and datasets to systematically work better for some populations over others. Curating datasets involves design choices such as what source to use, what language to use, what perspectives to include or exclude, or who to get annotations from. For example, a researcher’s native language may influence them to create datasets in that language due to their familiarity with the domain (as in the example in Figure 1). When training models, these choices include the type of training data, data pre-processing techniques, or the objective function (Hall et al., 2022). For example, a researcher’s institutional affiliation may influence the training datasets they select (e.g., choosing a dataset made by a coworker). Since the latent choices that result in design biases are fundamental to research itself, some researchers have argued that it is impossible to completely de-bias datasets and models (Waseem et al., 2021).
Current discussions around bias in NLP often focus on ones that originate from social biases embedded within the data. In comparison, design biases originate from the developer who makes assumptions. Based on Friedman and Nissenbaum (1996)’s framework on bias, social biases are preexisting biases in society, whereas design biases
are emergent biases that originate from the computing system itself. ‘Gender bias’ in computing systems means that the system does not perform well for some genders; “man is to doctor as woman is to nurse” (Bolukbasi et al., 2016) is a social bias, while captioning systems that fail to understand women’s voices (Tatman, 2017) is a design bias.
One prominent example of design bias in NLP is the overt emphasis on English (Joshi et al., 2020; Blasi et al., 2022). Others include the use of block lists in dataset creation or toxicity classifiers as a filter, which can marginalize minority voices (Dodge et al., 2021; Xu et al., 2021). In this work, we extend the discussion of design biases from prior work into NLP, discuss it in relation to researcher positionality, and show its effects on datasets and models.
3 NLPositionality: Quantifying Dataset and Model Positionality
Our NLPositionality framework follows a twostep process for characterizing the design biases and positionality of datasets and models. First, a subset of data for a task is re-annotated by annotators from around the world to obtain globally representative data in order to quantify positionality (§3.1). We specifically rely on re-annotation to capture self-reported demographic data of annotators with each label. Then, the positionality of the dataset or model is computed by comparing the responses of the dataset or model with different demographic groups for identical instances (§3.2). While relying on demographics as a proxy for positionality is limited (see discussion in §7), we use demographic information for an initial exploration in uncovering design biases in datasets and models.

heading: 3.1 Collecting Diverse Annotations
text: Cost-effectively collecting annotations from a diverse crowd at scale is challenging. Popular crowdsourcing platforms like Amazon Mechanical Turk (MTurk) are not culturally diverse, as a majority of workers are from the United States and India (Difallah et al., 2018; Ipeirotis, 2010). Further, MTurk does not easily support continuous and longitudinal data collection. To address these challenges, we use LabintheWild (Reinecke and Gajos, 2015), which hosts web-based online experiments. Compared to traditional laboratory settings, it has more diverse participants and collects equally high-quality data for free (August and Reinecke, 2019; Oliveira et al.,
are re-annotated via diverse volunteers recruited on LabintheWild. Processing (step 5): We compare the labels collected from LabintheWild with the dataset’s original labels and models’ predictions. Analysis (step 6): We compute the Pearson’s r correlation between the LabintheWild annotations by demographic for the dataset’s original labels and the models’ predictions. We apply the Bonferroni correction to account for multiple hypothesis testing.
2017); instead of monetary compensation, participants typically partake in LabintheWild experiments because they learn something about themselves. Thus, we motivate people to participate in our IRB-approved study (§8) by enabling them to learn how their responses on a given task (e.g., judging hate speech) compare to a judgment by AI systems as well as by others who are demographically similar to them (see Appendix B.1).
For a given task, we choose a dataset to be annotated. To select instances for re-annotation, we filter the dataset based on relevant information that could indicate subjectivity (such as controversiality label for the social acceptability dataset), and then sample 300 diverse instances by stratified sampling across different dataset metadata, (such as the targeted groups of toxic speech label for the hate speech dataset) (see Appendix A.1). These instances are then hosted as an experiment on LabintheWild to be annotated by a diverse crowd, where participants report their demographics. To ensure consistency in the re-annotated data, the instructions and annotation setups are similar to the original tasks’. Figure 2 is an example from the Social Chemistry dataset and its annotations.

heading: 3.2 Quantifying Positionality
text: We use correlation as a quantitative construct for positionality. First, we group the annotations by specific demographics. When datasets contain multiple annotations from the same demographic for the same instance, we take the mean of the labels from annotators of that demographic to obtain an aggregated score (see Table 1). Next, for each demographic, we compute Pearson’s r using the demographic’s aggregated score for each instance and correlated it to the dataset label or model prediction2. We then apply the Bonferroni correction to account for multiple hypotheses testing (Wickens and Keppel, 2004). We rank the correlations to reveal which demographic groups best align with the positionality of datasets and models. Finally, we report the total number of annotators and interannotator agreements for each demographic using Krippendorff’s α (Krippendorff, 2006).

heading: 4 Case Studies
text: We present case studies of applying NLPositionality to two different socially-aligned tasks: social acceptability and hate speech detection.
2We use models’ output probability scores for supervised models and categorical labels for GPT-4.

heading: 4.1 Social Acceptability
text: Social acceptability is the task of evaluating how acceptable certain actions are in society. Related to making moral judgments, it requires models to have commonsense rules about social norms and acceptable behavior (Forbes et al., 2020). Social acceptability is likely to vary by background, as morality (Atari et al., 2022), values (Haerpfer and Kizilova, 2012), and moral decision-making (Awad et al., 2018, 2020) vary by culture. In NLP, social acceptability models’ behavior have been shown to systematically display certain values (Arora et al., 2023) and biases (Hämmerl et al., 2022), some of which are Western-centric (Jiang et al., 2021; Talat et al., 2022).
Setup 612 participants annotated the social acceptability of 300 examples from the Social Chemistry dataset (Forbes et al., 2020). We collected 11, 294 annotations for this task and compared participants’ responses to the original Social Chemistry dataset, the Delphi model (Jiang et al., 2021), and GPT-4 zero-shot (OpenAI, 2023). In order to evaluate GPT-4, we combine Delphi’s GPT-3 zeroshot evaluation setup from Jiang et al. (2021) with the template used by Rae et al. (2021):
Text: <Social Chemistry action> Question: How socially acceptable is the above situation: good, okay, or bad? Answer:
Results We find that the instances we selected for the social acceptability task can vary by the annotator’s demographics (see Table 1). For example, men and non-binary people are more likely than women to say it’s okay to tell a woman what to do.
We also report the Pearson’s r results in Table 2, with corresponding p-values after applying the Bonferroni correction in the Appendix C.1. We use ∗ to represent statistically significant Pearson’s r coefficients (p < 2.04e − 05). Social Chemistry is most aligned with people who grow up (r = 0.76∗) and live in (r = 0.76∗) English-speaking countries, who have a college education (r = 0.74∗), are White (r = 0.73∗), and are 20-30 years old (r = 0.74∗), indicating a preference to younger WEIRD populations.
Delphi also exhibits a similar pattern, but to a lesser degree. While it strongly aligns with people who grow up (r = 0.61∗) and live in (r = 0.65∗) English-speaking countries, who have a college education (r = 0.66∗), are White (r = 0.61∗), and are 20-30 years old (r = 0.668); it also correlates more with other populations, such as people who grow up (r = 0.72∗) in Baltic countries compared to English-speaking countries.
We also observe a similar pattern with GPT-4. It has the highest Pearson’s r value for people who grow up (r = 0.74∗) and live in (r = 0.73∗) English-speaking countries, are college-educated (r = 0.69∗), are White (r = 0.70∗) and are between 20-30 years old (r = 0.70∗). However, it
does have strong correlations with other populations, such as people who grow up in Confucian countries (r = 0.74∗) and have a graduate degree (r = 0.69∗).
Native English speakers strongly correlate with both datasets and models. Non-binary people align less to both Social Chemistry, Delphi, and GPT-4 compared to men and women. People with professional school education and Black, Latinx, and Native American populations consistently rank least in correlation for education level and ethnicity.

heading: 4.2 Hate Speech Detection
text: Hate speech detection is the task of determining whether an instance is “abusive speech targeting specific group characteristics, such as ethnic origin, religion, gender, or sexual orientation” (Warner and Hirschberg, 2012). Rating whether text is hate speech is likely to vary by background. Also, annotators rate the toxicity of tweets with African American English (AAE) differently by background, such as political affiliation (Sap et al., 2022). Hate speech classifiers also produce errors for instances outside Western contexts (Ghosh et al., 2021).
Setup For the hate speech detection task, 484 LabintheWild participants annotated 300 examples from the Dynahate dataset (Vidgen et al., 2021) and rated whether an instance was hate speech or not. We collected 5, 005 annotations and compared participants’ responses to the original Dynahate dataset as well as Perspective API3, Rewire API4, ToxiGen RoBERTa (Hartvigsen et al., 2022), and GPT-4 zero-shot. To evaluate GPT-4, we use the prompt used by Rae et al. (2021):
Text: <Dynahate instance> Question: Does the above text contain rude,
hateful, aggressive, disrespectful, or unreasonable language?
Answer:
Results We find that the instances we selected for this task can vary by the annotator’s demographics (see Table 1). For instance, annotators who were racial minorities rated the instance “Justice for the yellow man lol” as hate speech, while White annotators voted being not sure.
We also report the results in Table 2. We report the p-values after applying the Bonferroni correction in the Appendix C.1. Dynahate is highly correlated with people who grow up in English-speaking
3perspectiveapi.com 4rewire.online
countries (r = 0.70∗), who have a college education (r = 0.66∗), are White (r = 0.69∗), and are 20-30 years old (r = 0.65∗). However, it also has high alignment with other populations, such as people who live in West South Asia (r = 0.77).
Perspective API also tends to align with WEIRD populations, though to a lesser degree than Dynahate. Perspective API exhibits some alignment with people who grow up and live in English-speaking (r = 0.33∗, r = 0.34∗ respectively), have a college education (r = 0.34∗), are White (r = 0.29∗), and are 20-30 years old (r = 0.34∗). It also exhibits higher alignment with other populations, such as people who live in Confucian countries (r = 0.36) compared to English-speaking countries. Unexpectedly, White people rank lowest in Pearson’s r score within the ethnicity category.
Rewire API similarly shows this bias. It has a moderate correlation with people who grow up and live in English-speaking countries (r = 0.58∗, r = 0.60∗ respectively), have a college education (r = 0.56∗), are White (r = 0.56∗), and are 20-30 years old (r = 0.56∗).
A Western bias is also shown in ToxiGen RoBERTa. ToxiGen RoBERTa shows some alignment with people who grow up (r = 0.37∗) and live in (r = 0.38∗) English-speaking countries, have a college education (r = 0.38∗), are White (r = 0.32∗), and are between 20-30 years of age (r = 0.38∗).
We also observe similar behavior with GPT4. The demographics with some of the higher Pearson’s r values in its category are people who grow up (r = 0.41∗) and live in (r = 0.42∗) English-speaking countries, are college-educated (r = 0.39∗), are White (r = 0.38∗), and are 20-30 years old (r = 0.42∗). It shows stronger alignment to Asian-Americans (r = 0.39∗) compared to White people, as well as people who live in Baltic countries (r = 0.75) and people who grow up in Confucian countries (r = 0.52∗) compared to people from English-speaking countries.
As in the previous task, labels from native English speakers are strongly correlated with datasets and models. Non-binary people align less with Dynahate, Perspective API, Rewire, ToxiGen RoBERTa, and GPT-4 compared to other genders. Also, people who are professional school-educated or are Black, Latinx, and Native American rank least in alignment for education and ethnicity respectively.

heading: 5 Discussion
text: In this paper, we characterized design biases and the positionality of datasets and models in NLP. We introduced the NLPositionality framework for identifying design biases in NLP datasets and models. NLPositionality consists of a two-step process of collecting annotations from diverse annotators for a specific task and then computing the alignment of the annotations to dataset labels and model predictions using Pearson’s r. We applied NLPositionality to two tasks: social acceptability and hate speech detection, with two datasets and five models in total. In this section, we discuss key takeaways from our experiments and offer recommendations to account for design biases in datasets and models.
There Is Positionality in NLP Models and datasets have positionality, as they align better with some populations than others. This corroborates work from Cambo and Gergle (2022) on model positionality, which quantifies positionality by inspecting the content of annotated documents, as well as work from Rogers (2021), who argues that collecting a corpus of speech inherently encodes a particular world view (e.g., via linguistic structures, topic of conversations, and the speaker’s social context). We extend these works by showing design biases and quantifying dataset and model positionality by computing correlations between LabintheWild annotations, dataset labels, and model predictions.
Our case studies show examples of positionality in NLP. However, most socially-aligned tasks may encode design biases due to differences in language use between demographic groups, for example, commonsense reasoning (Shwartz, 2022), question answering (Gor et al., 2021), and sentiment analysis (Mohamed et al., 2022). Even tasks that are considered purely linguistic have seen design biases: in parsing and tagging, performance differences exist between texts written by people of different genders (Garimella et al., 2019), ages (Hovy and Søgaard, 2015), and races (Johannsen et al., 2015; Jørgensen et al., 2015). This shows how common design biases are in NLP, as language is a social construct (Burr, 2015) and technologies are imbued with their creator’s values (Friedman, 1996). This raises the question of whether there are any valueneutral language technologies (Birhane et al., 2022; Winner, 2017).
Datasets and Models Skew Western Across all tasks, models, and datasets, we find statistically significant moderate correlations with Western, educated, White, and young populations, indicating that language technologies are WEIRD to an extent, though each to varying degrees. Prior work identifies Western-centric biases in NLP research (Hershcovich et al., 2022), as a majority of research is conducted in the West (ACL, 2017; Caines, 2021). Joshi et al. (2020); Blasi et al. (2022) find disproportionate amounts of resources dedicated to English in NLP research, while Ghosh et al. (2021) identify cross-geographic errors made by toxicity models in non-Western contexts. This could lead to serious downstream implications such as language extinction (Kornai, 2013). Not addressing these biases risks imposing Western standards on non-Western populations, potentially resulting in a new kind of colonialism in the digital age (Irani et al., 2010).
Some Populations Are Left Behind Certain demographics consistently rank lowest in their alignment with datasets and models across both tasks compared to other demographics of the same type. Prior work has also reported various biases against these populations in datasets and models: people who are non-binary (e.g., Dev et al., 2021), Black (e.g., Sap et al., 2019; Davidson et al., 2019), Latinx (e.g., Dodge et al., 2021), Native American (e.g., Mager et al., 2018); and people who are not native English speakers (e.g., Joshi et al., 2020). These communities are historically marginalized by technological systems (Bender et al., 2021).
Datasets Tend to Align with Their Annotators We observe that the positionality we compute is similar to the reported annotator demographics of the datasets, indicating that annotator background contributes to dataset positionality. Social Chemistry reports their annotators largely being women, White, between 30-39 years old, having a college education, and from the U.S. (Forbes et al., 2020), all of which have high correlation to the dataset. Similarly, Dynahate exhibits high correlation with their annotator populations, which are mostly women, White, 18-29 years old, native English speakers, and British (Vidgen et al., 2021). This could be because annotators’ positionalities cause them to make implicit assumptions about the context of subjective annotation tasks, which affects its labels (Wan et al., 2023; Birhane et al., 2022). In toxicity modeling, men and women value
speaking freely versus feeling safe online differently (Duggan et al., 2014).
Recommendations Based on these findings, we discuss some recommendations. Following prior work on documenting the choices made in building datasets (Gebru et al., 2021) and models (Bender and Friedman, 2018; Bender et al., 2021), researchers should keep a record of all design choices made while building them. This can improve reproducibility (NAACL, 2021; AAAI, 2023) and aid others in understanding the rationale behind the decisions, revealing some of the researcher’s positionality. Similar to the “Bender Rule” (Bender, 2019), which suggests stating the language used, researchers should report their positionality and the assumptions they make (potentially after paper acceptance to preserve anonymity).
We echo prior work in recommending methods to center the perspectives of communities who are harmed by design biases (Blodgett et al., 2020; Hanna et al., 2020; Bender et al., 2021). This can be done using approaches such as participatory design (Spinuzzi, 2005), including interactive storyboarding (Madsen and Aiken, 1993), as well as value-sensitive design (Friedman, 1996), including panels of experiential experts (Madsen and Aiken, 1993). Building datasets and models with large global teams such as BigBench (Srivastava et al., 2022) and NL-Augmenter (Dhole et al., 2021) could also reduce design biases by having diverse teams (Li, 2020).
To account for annotator subjectivity (Aroyo and Welty, 2015), researchers should make concerted efforts to recruit annotators from diverse backgrounds. Websites like LabintheWild can be platforms where these annotators are recruited. Since new design biases could be introduced in this process, we recommend following the practice of documenting the demographics of annotators as in prior works (e.g., Forbes et al., 2020; Vidgen et al., 2021) to record a dataset’s positionality.
We urge considering research through the lens of perspectivism (Basile et al., 2021), i.e. being mindful of different perspectives by sharing datasets with disaggregated annotations and finding modeling techniques that can handle inherent disagreements or distributions (Plank, 2022), instead of forcing a single answer in the data (e.g., by majority vote; Davani et al., 2022) or model (e.g., by classification to one label; Costanza-Chock, 2018). Researchers also should carefully consider how
they aggregate labels from diverse annotators during modeling so their perspectives are represented, such as not averaging annotations to avoid the “tyranny of the mean” (Talat et al., 2022).
Finally, we argue that the notion of “inclusive NLP” does not mean that all language technologies have to work for everyone. Specialized datasets and models are immensely valuable when the data collection process and other design choices are intentional and made to uplift minority voices or historically underrepresented cultures and languages, such as Masakhane-NER (Adelani et al., 2021) and AfroLM (Dossou et al., 2022). There have also been efforts to localize the design of technologies, including applications that adapt their design and functionality to the needs of different cultures (e.g., Oyibo, 2016; Reinecke and Bernstein, 2011, 2013). Similarly, language models could be made in more culturally adaptive ways, because one size does not fit all (Groenwold et al., 2020; Rettberg, 2022). Therefore, we urge the NLP community to value the adaptation of language technologies from one language or culture to another (Joshi et al., 2020).

heading: 6 Conclusion
text: We introduce NLPositionality, a framework to quantify design biases and positionality of datasets and models. In this work, we present how researcher positionality leads to design biases and subsequently gives positionality to datasets and models, potentially resulting in these artifacts not working equally for all populations. Our framework involves recruiting a demographically diverse pool of crowdworkers from around the world on LabintheWild, who then re-annotate a sample of a dataset for an NLP task. We apply NLPositionality to two tasks, social acceptability and hate speech detection, to show that models and datasets have a positionality and design biases by aligning better with Western, White, college-educated, and younger populations. Our results indicate the need for more inclusive models and datasets, paving the way for NLP research that benefits all people.

heading: 7 Limitations
text: Our study has several limitations. First, demographics may not be the best construct for positionality, as there may be variability of beliefs within demographic groups. Assuming that there is homogeneity within demographic groups is reductionist and limited. Rather, capturing an individual’s attitudes
or beliefs may be a more reliable way to capture one’s positionality that future work can investigate.
Study annotators could also purposefully answer untruthfully, producing low-quality annotations. We address this risk by using LabintheWild. LabintheWild has been shown to produce highquality data because participants are intrinsically motivated to participate by learning something about themselves (Reinecke and Gajos, 2015). However, as is the case for all online recruiting methods, our sample of participants is not representative of the world’s population due to the necessity of having access to the Internet. In addition, there is likely a selection bias in who decides to participate in a LabintheWild study.
Pearson’s r may not fully capture alignment as it does not consider interaction effects between different demographics (i.e., intersectionality). Thus, there may be additional mediating or moderating variables that may explain the results that our analysis does not consider. We also took the average of the annotations per group, which could mask individual variations (Talat et al., 2022). Also, having a low number of participants from specific demographic groups may limit how well the results generalize to the entire group; further, it may risk tokenizing already marginalized communities.
As part of our study, we apply NLPositionality to only two tasks which have relatively straightforward annotation schemes. It may be difficult to generalize to other NLP tasks which have harder annotation schemes, especially ones that require a lot of explanation to the annotators, for example, natural language inference (NLI) tasks.
Our approach is evaluated and works the best for classification tasks and classifiers. Generation tasks would need more careful annotator training which is difficult to achieve on a voluntary platform without adequate incentives. Having annotators use one Likert scale to rate the social acceptability and toxicity of a situation or text may not be a sufficient measure to represent these complex social phenomena. To reduce this threat, we provide detailed instructions that describe how to provide annotations and followed the original annotation setup as closely as possible.

heading: 8 Ethics Statement
text: Towards Inclusive NLP Systems Building inclusive NLP systems is important so that everyone can benefit from their usage. Currently, these sys-
tems exhibit many design biases that negatively impact minoritized or underserved communities in NLP (Joshi et al., 2020; Blodgett et al., 2020; Bender et al., 2021). Our work is a step towards reducing these disparities by understanding that models and datasets have positionalities and by identifying design biases. The authors take inspiration from fields outside of NLP by studying positionality (Rowe, 2014) and acknowledge crossdisciplinary research as crucial to building inclusive AI systems.
Ethical Considerations We recognize that the demographics we collected only represent a small portion of a person’s positionality. There are many aspects of positionality that we did not collect, such as sexual orientation, socioeconomic status, ability, and size. Further, we acknowledge the limitation of assigning labels to people as being inherently reductionist. As mentioned in §7, using a single Likert scale for social acceptability and toxicity is not sufficient in capturing the complexities in these phenomena, such as situational context.
We note that quantifying positionality of existing systems is not an endorsement of the system. In addition to making sure that language technologies work for all populations, researchers should also continue to examine whether these systems should exist in the first place (Denton and Gebru, 2020; Keyes et al., 2019). Further, we note that understanding a dataset or model’s positionality does not preclude researchers from the responsibilities of adjusting it further.
This study was undertaken following approval from the IRB at the University of Washington (STUDY00014813). LabintheWild annotators were not compensated financially. They were lay people from a wide range of ages (including minors) and diverse backgrounds. Participants were asked for informed consent to the study procedures as well as the associated risks, such as being exposed to toxic or mature content, prior to beginning the study.
Research Team Positionality We discuss aspects of our positionality below that we believe are most relevant to this research. The research team is comprised of computer scientists who study human-computer interaction and NLP and have a bent for using quantitative methods. Thus, we approach the topic from a perspective that assumes that positionality can be characterized, fixed, and
quantified. The entire research team currently resides in the United States. In alphabetical order, the team members originate from Belgium and Switzerland, France, Germany, India, and the United States; and identify as East Asian, South Asian, and White. These nationalities and ethnicities are overrepresented in the development of NLP technologies. Thus, we acknowledge that our knowledge of how design biases in NLP datasets and models impact people is largely through research, rather than personal experience.

heading: A Data
text: In this section, we describe all the decisions that went into sampling data points from the different datasets and its post-processing.
A.1 Sampling
For Social Chemistry, we sample instances whose label for anticipated agreement by the general public was “Controversial (∼ 50%)”. We ensure the samples are equally represented by the moral foundation label, which we compute based on majority vote across annotators. In the study, annotators respond whether they found a presented action socially acceptable.
For Dynahate, we randomly sample instances from rounds 3 and 4. In these rounds, annotators generated examples of implicit hate, which is subtler and harder to detect and could yield differences in annotations. We ensure that there are equal amounts of hateful and not hateful instances and that the types of targets of the hateful instances are equally represented. During the studsy, annotators respond whether they found a presented instance toxic.
For both social acceptability and hate speech detection, annotators responded whether they found the situation moral and whether they found the instance to be hate speech respectively.
A.2 Post-Processing
Because Social Chemistry has multiple annotations for each instance, we compute an aggregate score by taking the average score across annotators. This score is then used to correlate to the annotators’ aggregated scores.

heading: B Study Design
text: In this section, we discuss the design of the LabintheWild experiments. The social acceptability task was released to the public in April 2022. The hate speech detection task was released August 2022. To reduce confounding factors on the data collection process, we conduct multiple user studies of the LabintheWild experiments prior to the public release. Additionally, all the annotations collected through the experiments are anonymous and are stored securely.
The social acceptability task is marketed as “Could you live with an AI and its morals?” Participants for this study provide annotations for 25
situations. The hate speech detection task is marketed as “Do you and AI agree on what is hate speech? Let’s find out!” Participants provid annotations for 15 instances.
B.1 LabintheWild Study Flow We describe the format of the LabintheWild experiment. The phases of the experiment are: obtaining consent, collecting demographics, explaining instructions, collecting annotations, collecting study feedback, and displaying results.
Obtaining Consent Prior to beginning the study, participants review a consent form. The consent form includes information on the purpose of the research, what the participant will do, risks and benefits of the research, privacy and data collection methods, and contact information of the researchers. At the end of the form, participants give explicit consent to participate in the study.
Collecting Demographics We then collect the demographics of study participants. LabintheWild participants enter in whether they had taken this test before, the country they lived in the longest, the country of residence, age, native language, religion, education, and ethnicity. No demographics are required except for the country the participant lived in the longest and whether they have taken the test before. Additionally, we only display ethnicity for people within the United States.
Explaining Instructions For each task, we provide instructions to participants on how to perform the annotation task. For social acceptability, we explain social acceptability as rating “what you think about the situation in general from an ethical perspective” (see Figure 5). For hate speech detection, use the definition of hate speech from Dynahate and we provide three examples of hate speech (see Figure 6). We also present examples of negative sentiment, profanity, or discussing groups that could be confused as hate speech, but are not hate speech.
Collecting Annotations After being presented with instructions, participants begin data collection from the 300 instances selected from Section A.1. For each task, we keep the annotation setup identical to the original one. For social acceptability, we collect Likert-scale ratings of situations ranging from “It’s very bad”, “It’s bad”, “It’s okay”, “It’s good”, and “It’s very good”. Participants can provide rationale for their decision by using an open
text box. The data collection interface is presented in Figure 4. For hate speech detection, we collect ratings of instances ranging from “Hate speech”, “Not sure”, “Not hate speech”. We also provide an optional open-text box for participants to explain their rationale. The data collection interface is presented in Figure 7. After submitting the annotation, the participant is able to see a visualization on how the AI responded as well as how other participants from the same country responded to the instance.
We also specifically sample which instances to present to participants for annotation. We sample a third of the instances that did not have any annotations from the demographic and a third that are already sampled by participants of the demographic. The rest are equally split across the different of types of instances (i.e., moral foundation for Social Chemistry, hate type for Dynahate).
Providing Study Feedback Following typical LabintheWild experiment procedures, we collect feedback from participants about the study. Participants can enter open-text feedback on anything. They also submit whether they encountered technical difficulties during the study or whether they cheated. Participants can elaborate on their answers from the prior questions in an open-text box.
Displaying Overall Results Finally, participants see their overall results for the experiment task. First, participants are presented with the percentage of time they agreed with the AI as well as with participants as the same demographic as them (see Figure 8). Each of these agreement scores are further broken down by the type of the instance (i.e., moral foundation for Social Chemistry and hate type for Dynahate).

heading: C Additional Results
text: In this section, we report additional results from our analyses of the LabintheWild data.
C.1 p-values We report the p-values from our analyses from Table 3.

heading: D Cultural Spheres
text: Division of countries can be done through continents. However, continents are often not representative of the countries within it and clustering based on them can lead to inaccurate findings. For example, Asia includes both Japan and Saudi
Arabia, which are different culturally. We instead adopt cultural spheres as used in World Values Survey (Haerpfer and Kizilova, 2012), which clusters the countries in terms of the values they uphold and norms they follow. Table 4 shows the countries and the spheres.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
3 A2. Did you discuss any potential risks of your work? 8

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: 1, abstract
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 4
3 B1. Did you cite the creators of artifacts you used? 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The datasets are academic and free to use for researchers.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The datasets are academic and free to use for researchers.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Appendix, 8

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? 3
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? We do not train a model.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 3, appendix
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We do not train a model.
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We do not train a model.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 3, 4

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? appendix

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 3, 8, appendix
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? 8

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: 4

heading: Future Work
text: Rather, capturing an individual’s attitudes
or beliefs may be a more reliable way to capture one’s positionality that future work can investigate. Study annotators could also purposefully answer untruthfully, producing low-quality annotations. We address this risk by using LabintheWild. LabintheWild has been shown to produce highquality data because participants are intrinsically motivated to participate by learning something about themselves (Reinecke and Gajos, 2015). However, as is the case for all online recruiting methods, our sample of participants is not representative of the world’s population due to the necessity of having access to the Internet. In addition, there is likely a selection bias in who decides to participate in a LabintheWild study. Pearson’s r may not fully capture alignment as it does not consider interaction effects between different demographics (i. e. , intersectionality). Thus, there may be additional mediating or moderating variables that may explain the results that our analysis does not consider. We also took the average of the annotations per group, which could mask individual variations (Talat et al. , 2022). Also, having a low number of participants from specific demographic groups may limit how well the results generalize to the entire group; further, it may risk tokenizing already marginalized communities. As part of our study, we apply NLPositionality to only two tasks which have relatively straightforward annotation schemes. It may be difficult to generalize to other NLP tasks which have harder annotation schemes, especially ones that require a lot of explanation to the annotators, for example, natural language inference (NLI) tasks. Our approach is evaluated and works the best for classification tasks and classifiers. Generation tasks would need more careful annotator training which is difficult to achieve on a voluntary platform without adequate incentives. Having annotators use one Likert scale to rate the social acceptability and toxicity of a situation or text may not be a sufficient measure to represent these complex social phenomena. To reduce this threat, we provide detailed instructions that describe how to provide annotations and followed the original annotation setup as closely as possible