SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sunipa Dev
sunipadev@google.com
Jaya Goyal
Circadian Connect
jaya@circadianconnect.com
Dinesh Tewari
dineshtewari@google.com
Shachi Dave*
shachi@google.com
Vinodkumar Prabhakaran*
vinodkpg@google.com
With rapid development and deployment of generative language models in global settings, there is an urgent need to also scale our measurements of harm, not just in the number and types of harms covered, but also how well they account for local cultural contexts, including marginalized identities and the social biases experienced by them. Current evaluation paradigms are limited in their abilities to address this, as they are not representative of diverse, locally situated but global, sociocultural perspectives. Our evaluation resources must be enhanced and calibrated by including people and experiences from different cultures and societies worldwide, to prevent gross underestimations or skewed measurements of harm. In this work, we demonstrate a socio-culturally aware expansion of evaluation resources in the Indian societal context, specifically for the harm of stereotyping. We devise a community engaged effort to build a resource that contains stereotypes for axes of disparity uniquely present in India. The resultant resource increases the number of stereotypes known for and in the Indian context by over 1000 stereotypes across many unique identities. We also demonstrate the utility and effectiveness of such expanded resources for evaluations of language models. CONTENT WARNING: This paper contains examples of stereotypes that may be offensive.
Generative Artificial Intelligence has seen immense progress in recent years [11, 42, 13, 39], leading to their accelerated integration into a myriad of applications, including writing assistance [21], and story telling [45]. Alongside, there is also growing awareness and concerns about the responsibility and safety aspects of these models [4, 17]. One particular challenge is that generative models may learn, perpetuate, and even amplify social stereotypes in the data [46], either reproducing them explicitly in generated language or images, or reflecting them in distributional differences in the outputs generated. While the exact harms of such propagation of stereotypes will depend on the specific application contexts, it is crucial to be able to reliably detect them in the generated outputs.
A growing body of work looks into detecting and mitigating these harms [8, 38], through interventions such as data augmentation [48] or adversarial training [38, 19]. While these techniques are useful, they are entirely reliant on underlying resources such as lists of words, phrases, and sentences, which are markers of the specific stereotype [12, 31, 32, 23]. These resources in turn are lacking in many ways: first, there is a Western bias in that, these resources mostly cover stereotype harms about and from the Western perspective[40, 35]. Secondly, they are built based on researchers' world views and rarely represent diverse community perspectives. In the pipeline towards the creation of such
data resources, individuals and communities are most often only able to annotate to the validity of a stereotype or statement shown to them. They are neither able to communicate on existing stereotypes and disparities known to them, nor the identities that face these stereotypes and the marginalization. In combination, this leads to under-representation of a broad range of locally situated, socio-cultural perspectives in the evaluation pipelines, resulting in miscalculation or exclusion of potential harms of a model in global settings[3, 25, 29, 5, 2, 36].
It is important to note that developing a resource in this varied societal context is challenging. Community engagements such as workshops, and focus groups are important to obtain a nuanced understanding of societies, in an open-ended manner. However, such qualitative methods pose limitations for this study. Such methods often generate generalisations that are valid for a particular context and not populations and may not be able to address the issue of broad coverage of perspectives [36]. On the other hand, recent work which used large language models themselves to scale the coverage of stereotype resources, captures only the data present in the training set and fails to include community insights [22]. To accomplish diversification of stereotype evaluation, we need resources that are locally situated, yet include broad perspectives.
In this work, we address both these challenges towards inclusive stereotype resource creation at scale. We build a dataset of stereotypes through community engagement, by adopting an open-ended survey based data collection strategy, and pooling stereotypes through free-form text from a carefully selected, diverse set of participants that represent a diverse slice of society. We focus specifically on communities and societal structures in India, and by engaging with them, we build a dataset of stereotypes that are socio-culturally situated in the Indian context. Our dataset - SPICE (Stereotype Pooling in India with Community Engagement) - uncovers identity axes and disparities unique to India and highlights stereotypes affecting intersectional communities as well. We also demonstrate the utility of this dataset in detecting stereotypical associations and inferences made by language models for the English language. While we work specifically with the Indian context, our methodology is generalizable and applicable to different socio-cultural contexts.
We start by discussing some related work on fairer and responsible AI that aims to bring a broader global lens, as well as work on including community perspectives in AI research and development.
2.1 Fairness across Global Cultures
While there is a sizeable body of work investigating the fairness of ML and NLP models and tools, a vast majority of them focus on the Western context. This is sometimes explicitly stated, but more commonly implicitly assumed as they focus on models trained on data scraped primarily in English data from the West [18], or examine interactions of models with Western axes of disparities [7, 17]. However, the concept of 'fairness' is inherently socially situated, with the regional histories, its prevalent cultures, languages, and socio-economic structures being pivotal in defining what is fair or who the marginalized groups are therein [40]. The applicability of such resources, narrowly scoped in and for the West, fail to generalize for global settings and misreport biases, and experienced harms [29, 2, 36]. Recent work has called for broadening the perspective to include global cultural contexts [35], especially in the Global South, and our work contributes to filling this gap.
Particularly for India, recent work emphasizes this socially situated nature of fairness and harms [40], and creates recontextualized data and model resources [29, 5]. India is a nation with over a billion people with a plurality of identities, with 22 official languages, hundreds of regional dialects, and diversity in ancestry, regional identity, religion, and more, leading to a wide range of cultural diversity. This diversity in turn, when coupled with social disparities along socioeconomic status, caste identity, gender, etc., results in biases that are unique to India that are often not represented in commonly used evaluation resources built in the West. Although there have been concerted efforts to address this gap in recent years [29, 5, 22], they still have limited coverage of local community insights. To address this, in this work, we engage with a diverse set of participants in India through an open-ended survey to build a stereotype evaluation resource that has increased coverage of local insights, thereby enabling fairness evaluations to be more representative of Indian society.
2.2 Community Engagements for AI Harm Estimations
Evaluations for model utility and safety require human judgement and several human-in-the-loop strategies are in place in the field of NLP that use annotations to build stereotype resources [5, 31]. However, these annotations do not leave scope for participants to share their perspectives on a subject, rather only attest to the validity of a given statement. Deeper engagements with specific underserved and marginalized communities, or culturally diverse communities are pivotal in preventing the perpetration of marginalization in the evaluations themselves. When unchecked, this can lead to underestimation of harms meted out in global contexts [36].
Participatory design or community engagements are not methods per se with prescriptive steps or defined processes. Instead, they consist of the ideology that the people destined to use a system should play a critical role in designing it [37, 41, 6]. These engagements can be through an extensive repertoire of techniques, and frameworks such as workshops, focus groups, surveys distributed to target communities, and more, that can be used to bring communities into the process of resolving responsible AI-related issues. Each method renders specific advantages, such as deeper engagement in small focus groups versus the scale of people engaged with surveys. For studies towards model evaluations, focus groups, and workshops can aid the identification of specific harm types [20, 36], whereas surveys can help gain broader engagement with people at large to gain diverse perspectives at scale. In this work, we use surveys as our tool to engage with a wider population slice of India.
3 Stereotype Resources in NLP: the Story so Far.
With the rapid growth of language technologies and applications, there has been increased efforts to evaluate them for unwanted social biases, and downstream harms, in particular, propagating societal stereotypes. A stereotype is a belief or generalization about the identity of a person such as their race, gender, and nationality. We use identity term or identities to refer to words or phrases used to describe a group of people with a common trait. Identity terms can be about gender (e.g. female), nationality (e.g. Indian), region (e.g. North Indian), state (e.g. Bihari), caste (e.g. Kshatriya), and more. Attribute term or attributes are characteristics or categories such as profession, adjectives, socio-economic status, and so on, that are often associated with certain social identities. E.g., "women are homemakers" is a common gender-based stereotype reflected in many language models [8].
A large proportion of stereotype evaluation systems in NLP consist of templated sentences [23, 47, 16, 27], generated at scale using lists of words consisting of identities of persons and attributes stereotypically associated with them. These word lists in turn are primarily sourced from social science literature [9, 24], existing NLP work [8, 25], and crowdsourcing annotations [31, 5, 22] regarding stereotypical nature of associations presented to each crowd worker. This pipeline of evaluation limits who gets to participate in the creation of these resources, and in turn limits which stereotypes and which identities they account for.
In recent work, we significantly expanded the scope and coverage of stereotype resources in a multi-pronged way. We first used a dictionary-based approach to generate all possible identity-attribute pairs for a list of locally relevant identities and attributes, followed by locally situated annotations to label the pairs that constitute a social stereotype [5]. This study was contextually situated in India and collected stereotypes for Indian axes of disparities such as region (as denoted by Indian states), and religion. However, it was limited by the identity and attribute lists that generated the tuples, thereby limiting which stereotypes could be identified. To expand this coverage and remove restrictions of possible attributes that can be recovered in stereotypes for each identity, we complemented this with another approach that leveraged generative capabilities of language models trained on vast amounts of English text, to extract stereotypes similar to what it sees in prompts [22]. This coupled with geographically diverse human validation, ensured broader coverage of stereotypes at scale.
However, both methods are still stunted in what identities and attributes are included. Annotators can only speak to the validity of an association presented to them (which is either generated from researcher-curated lists or by a language model), and cannot identify and contribute any-all stereotypes they know. This one directional channel of stereotype curation curbs representation of niche, yet potentially pernicious stereotypes in evaluation methods that use such resources. Participatory or community engaged methods are thus vital on the path toward inclusive evaluations of models. In this work, we implement a community engaged method for stereotype collection through surveys seeking free-form text, to ensure the inclusion of diverse identities, experiences, and stereotypes at scale.
4 Data Collection Methodology
4.1 Structure of survey
Population surveys, as a tool, is the mainstay of social science and sociology research. We use this method to collect open-ended stereotypes, without any prompts or pre-filled sentences or samples to annotate as stereotypical versus not. Following collection of informed consent, which includes information on monetary compensation of INR 1000 per participant, the survey is constructed of three major sections. We construct this study, consent form, and survey with a rigorous review process that takes into account the intended usage and storage of resultant data, privacy of PII and SPII, and more (details in Data Card in Appendix A.1).
The first section of the survey collects demographic information about the respondent, which is voluntary to provide includes gender, caste, religion, and languages known. For caste, we include as options constitutionally defined caste categories that are often required in government documents. These were: Scheduled Castes (SC), Scheduled Tribes (ST), Other Backward Classes (OBCs), and General or Open category. Respondents could also choose not to identify or self-identify.
The second section of the survey defines the concept of social stereotypes and provides further clarity using examples of commonly known stereotypes extracted from existing work on stereotypes in India [9, 5]. These include, for instance, "Jains are vegetarian," and "South Indians are intelligent."
Finally, in the third section, the respondents are asked to provide examples of stereotypes that they know of. We don't distinguish (nor ask for) whether the respondent believes in the provided stereotypes or not, as our intention is not to study the participants, rather to build a more comprehensive repository of possible stereotypes present in Indian society, so as to equip model evaluations better. In this section, the respondents see one example of how to write a stereotype into the tables provided in the survey interface, detailing how they are specifically asked to parse a stereotype into who the stereotype is about (or the identity term) and what the generalization or association made is (or attribute term). This enables the extraction of stereotypes in the format of tuples composed of an identity term and an attribute term, which is commonly used by NLP resources for model evaluations for the harm of stereotyping. The survey respondents can provide any number of stereotypes with a minimum of 5 and a maximum of 15. They can optionally also provide any additional feedback about the survey or information about stereotypes here.
The chosen language for the collection of data in this survey is English. This is motivated by several reasons. First, language models currently deployed in India are predominantly in the English language, but stereotype evaluations do not consider salient stereotypes in India. Safeguards in the English language are thus urgent to be developed. Secondly, this work is intended as ground work, leading to proof-of-concept of collecting such a resource at the grass root level using free-form text-based surveys. While challenging to complete in one language only due to manual cleaning critically required in the pipeline, the success of extracting a resource this way will act as the foundation for future work where we shift focus to the multitude of languages used in India. However, to ensure the participants understand and fill in the survey correctly, we provided video tutorials to describe sections two and three of the survey. We iterated over this element of survey explanation using text, and videos as media for communicating what the survey entails, and identified video explanations to be the easiest for respondents to grasp. These videos are in Hindi and English languages, which together are spoken, and understood by  $43.8\%$  Indians [1].
4.2 Distribution of surveys
Since the survey is composed of open-ended questions asking about stereotypes known to the respondent, it is important to seek out diverse perspectives and lived experiences within India. Distributing surveys online via social media, crowd working platforms such as Amazon Turks would reach a skewed segment of society, with low probabilities of gaining broader perspectives. We thus choose to engage with students at government colleges government-aided universities or colleges which have implemented mandatory reservations based on quotas for each social category in India. This ensures a higher degree of diversity for gender, caste, socio-economic status, etc. among our participants. This design choice was intentional to diversify the respondent pool for these axes of identities specifically. Our work however does not diversify for all possible identity axes in India that
could be important, including age, access to education, languages known, etc., which we leave for future work.
We also aim to collect diverse regional perspectives within India, to account for which we distribute surveys in Northern, Eastern, Western, and Southern states in India, in the states of Delhi and Haryana, Orissa, Maharashtra, and Andhra Pradesh respectively. Within each region, we distribute 100 surveys in a university located in an urban location or a city, and another 100 in a university at a suburban location. We manually reviewed the responses for quality and removed survey responses that were not usable (see Appendix A.4). The total number of valid submissions received across all regions is 655, where a submission is considered valid if it contains at least 5 valid identity-attribute pairs.
We used the Qualtrics platform to distribute surveys. We identified and hired a local coordinator for each of the four regions, who reached out to the various institutions and their students, and helped ensure wider reach locally. These coordinators were our points of contact who helped coordinate and send out the survey link to students via email or message groups.
4.3 Data Cleaning and Processing
Use of open-ended questionnaire for collecting stereotypes prevents a predefined set of identities or stereotypes to be collected. It also prevents respondents to become biased about the stereotypes to write about. We clean and process entries for spelling errors, mixed case words, representation of the same identity in different ways (singular, plural, different expressions, and languages), etc (details in Appendix A.5). We would like to note here that the step of data cleaning and processing into a specific format can be dependent on intended task usage, and should be further conducted as needed.
5 SPICE Dataset: Stereotype Pooling in India with Community Engagement
The data collected from each regional survey is aggregated to get the list of stereotypes in SPICE (Stereotype Pooling in India through Community Engagement) dataset. The dataset is comprised of over 2000 stereotypes (see Appendix A.1 for data), where each stereotype is listed along with the total number of persons who submitted the stereotype. We further list the number of persons from a specific identity group (region, gender, religion, and caste) that submitted each stereotype. The regions in the dataset are the regions we distributed surveys in (West urban, West suburban, East urban, etc.). The gender identities included are male and female, as respondents either identified as male, female or declined to identify. We note that no personally identifiable information of each individual respondent is available in the dataset, as we only provide aggregated information about individual stereotypes. The SPICE dataset is released at https://github.com/google-research-datasets/SPICE.
5.1 Dataset Characteristics
Diverse Participant Base As discussed in Section 4, in our attempt to maximize the diversity of our participant pool along different socio-economically salient categories in India, we distributed our surveys to students attending public universities. As such,  $92.5\%$  of our participant pool identified as being in the 18-24 years age range. While  $64.4\%$  were in undergraduate or graduate general degree programs (in commerce, science, psychology etc.),  $21.4\%$  were engaged in professional degree tracks such as engineering, and medical sciences.  $52\%$  participants identified as female and  $42\%$  identified as male, and the remaining chose not to answer the question or declined to self identify their gender. The range of languages known by the pool of participants included Punjabi, Oriya, Tamil, and Hindi.  $6\%$  participants self identified as Scheduled Castes,  $4\%$  as Scheduled Tribes,  $25\%$  as Other Backward Classes, and  $55\%$  as General Category. Further, the regional diversity of participants was ensured by design, since we distribute the surveys to urban and suburban regions in different parts of the country.
Increased Number and Diversity of Identities in Stereotypes Since we collect stereotypes in SPICE using free form text-based surveys, the stereotypes collected can be about any identity of a person. This community-engaged and open-ended approach uncovers stereotypes for about 500 additional identity terms over other complementary approaches using dictionary-based [5], and LLM-partnered strategies [22], which in total collect stereotypes of 27 identity terms. With such high diversity of identities, the number of stereotypes collected is also greater. We approximate the
1 https://www_qualtrics.com/
Table 1: Region based stereotypes and the distribution across regions in which they were collected. W, S, E, N: West, South, North, East and Su, U: Suburban and Urban, so WSu: West South Urban, NU: North Urban, and so on.
Identity	Attribute	WSu	WU	SSu	SU	ESu	EU	NSu	NU
gujaratis	business persons	8	16	0	0	0	0	0	0
north eastern	chinese	0	2	0	0	0	2	0	2
marathas	warriors	4	3	0	0	0	0	0	0
south indian	madrasi	1	2	0	0	0	0	1	0
mangala	barber	0	0	2	0	0	0	0	0
Table 2: Some caste and religion based stereotypes as volunteered by persons belonging to the same caste category, or religion.
Participant	Stereotype
Hindu	Hindu: devotional, vegetarian, casteist
Muslim	Muslim: daring, terrorist; Muslim woman: oppressed
Jain	Jain: business persons
Scheduled Castes	Dalits: uneducated, untouchable; Scheduled caste person: poor, get reservations
Scheduled Tribes	Scheduled tribe person: poor, uneducated, farming
General Category	Brahmin: vegetarian, superior; Brahmin woman: beautiful; Kshatriya: warrior, soldier
different axes of identity for a stereotype by tagging identity terms using sets of words for the same. For example, we use words "male", "woman", etc. for identifying stereotypes about gender, words "brahmin", "dalit", "baniya", etc., which are names of caste categories to identify stereotypes about caste identities. A full list of word lists used for this tagging is provided in Appendix B.1. It is important to note that these word lists are incomplete and can be made more granular and detailed. With this preliminary analysis however, we find that at least 476 stereotypes about gender disparity, 165 about religion, 121 about caste, and 327 about regions, states, and union territories in India. In addition, there are other stereotypes present about the age of a person, nationality, disability status, physical appearance and skin tone of a person, and more.