SlowGuess/NAACL_2022_OCR huggingface dataset:
Vijit Malik $^{1*}$  Sunipa Dev $^{2}$  Akihiro Nishi $^{2}$  Nanyun Peng $^{2}$  Kai-Wei Chang $^{2}$ $^{1}$ Indian Institute of Technology Kanpur (IIT-K)  
 $^{2}$ University of California, Los Angeles  
{vijitvm}@iitk.ac.in  
{akihiroishi}@ucla.edu  
{sunipa, violetpeng, kwchang}@cs.ucla.edu
Trigger warning: This paper contains examples of stereotypes and other harms that could be offensive and triggering to individuals.
Language representations are efficient tools used across NLP applications, but they are strife with encoded societal biases. These biases are studied extensively, but with a primary focus on English language representations and biases common in the context of Western society. In this work, we investigate biases present in Hindi language representations with focuses on caste and religion-associated biases. We demonstrate how biases are unique to specific language representations based on the history and culture of the region they are widely spoken in, and how the same societal bias (such as binary gender-associated biases) is encoded by different words and text spans across languages. The discoveries of our work highlight the necessity of culture awareness and linguistic artifacts when modeling language representations, in order to better understand the encoded biases.
Language models and representations (Pennington et al., 2014; Bojanowski et al., 2017; Devlin et al., 2019) are commonly used across the world in a variety of applications including machine translation (Kunchukuttan et al., 2017), information retrieval (Rao and Devi, 2018), chatbots (Bhagwat et al., 2019), sentiment classification (Kumar et al., 2019) and more. However, it is also known that these representations capture and propagate societal biases including gender (Bolukbasi et al., 2016), race (Caliskan et al., 2017), (Manzini et al., 2019), and nationality (Dev and Phillips, 2019) related stereotypes. This bias is present across representations for different languages. Each language reflects the culture and history of regions they are used popularly in, and as we go from one language
to another, the notion of bias, and the types of societal biases change accordingly. This key difference however, is not reflected in the effort made towards detecting, identifying, and mitigating biases in language representations, with the majority of efforts predominantly in English language and in the context of Western society (Bolukbasi et al., 2016; Nangia et al., 2020). Some recent work tackle the challenges of societal biases in language representations coming from various cultures and languages such as Arabic, French, Spanish, Chinese, and German (Lauscher et al., 2020; Chavez Mula and Spanakis, 2020; Kurpicz-Briki, 2020; Zhou et al., 2019; Zhao et al., 2020; Liang et al., 2020). Additionally, Ghosh et al. (2021) and Sambasivan et al. (2020) explore biases and algorithmic fairness concerning non-western contexts in machine learning and Pujari et al. (2019) focus on fairness in language technologies for Indic society by investigating binary gender associated bias in Hindi language representations. However, it is unclear if the representations capture other biases that are distinct to the Indian society and can cause harm, such as caste and religion related biases.
In this work, we make three main contributions: (i) with a careful study of the social and cultural composition of the Indian society, we highlight and devise measures to detect and quantify different societal biases present in Hindi language representations, and show gender, caste and religion bias in the language; (ii) we discuss the gendered nature of Hindi and the implications it has on the bias detection techniques for gender bias, highlighting the importance of leveraging linguistic knowledge when developing bias detection methods; and (iii) we demonstrate how the translation of word lists or one-to-one mapping of bias measures across languages is insufficient to detect biases meaningfully, indicating how bias measurement methods cannot be directly adapted from one language to another. Even when detecting the same societal types of
*Work done while interning at UCLA.
Human Language Technologies, pages 1041 - 1052
biases in a different language, translations of the words from English into the target language whose representations are being evaluated, does not suffice as the words may not exist, nor be commonly used or associated with the same sentiment or bias. Sentiments (good, bad, evil, etc) can also be encoded by distinct sets of objects and words (see Sec. 3 for discussion).
All these discoveries call for socio-cultural awareness, and attention to the differences in language structure, the changes of grammars, etc., in multilingual fairness studies. We hope this work can shed lights for future studies in these directions<sup>1</sup>.
2 Language and Culture Considerations
The perception and interpretation of biases are sensitive to societal construct, socio-cultural structures, and historical compositions of different regions of the world (Cheung and Chan, 2007). Since languages are culture and region-specific, there is a requirement to study the socio-cultural differences when trying to understand the biases and potential harms of language representations globally. Consider the example of the Hindi language where along with gender bias (Amutha, 2017), other biases like the ones based on caste and religion are also pervasive. Caste is unique to the culture in the Indian peninsula and is not usually considered when analyzing biases in languages in Western studies but remnants of caste based stereotypes are still prolific in the modern hindi literature (Gupta, 2021). Similarly, region and culture-specific biases also are present with respect to religion, occupation, or location (Thorat and Newman, 2007). Additionally, there are several key linguistic differences between English and Hindi languages such as pronouns which in Hindi do not indicate gender unlike in English. Instead, gender may be indicated by adjectives or verbs (Section 2.1), thus requiring distinct strategies for gender bias detection. Further, the word order in Hindi is distinct from that of English, and is similar to Japanese, Korean, Mongolian and Turkish $^{2}$ . Unlike English, which has fixed word order, Hindi does not has fixed word order.
2.1 Gender in Hindi
The syntactical use of gender in Hindi is layered and distinct from English (Hall, 2002) in different ways. These differences are reflected in the structure and composition of text, and is essential when interpreting the ways biases are likely to be encoded.
Gendered verbs: Verbs in Hindi can be gendered depending upon the tense of the sentence. In case of past tense and the perfect tenses which are built upon the past tense, the gender of the verb is determined by the gender of the subject. For example, 'went' is 'gaya' if male and 'gayi' if female.
Gendered Adjectives: Adjectives in Hindi can also be gendered. However, not all adjectives change form according to the gender of the subject. For example, the adjective 'gharelu' (Domestic) in Hindi is used the same whether a man is domestic or a woman is domestic, but adjectives like 'good' is 'achha' if male and 'acchhi' if female.
Gendered titles: Some titles for entities in Hindi can be gendered. For example: 'teacher' is 'adhyapak' if male and 'adhyapika' if female.
Gendered inanimate nouns: Instantiations of grammatical gender for inanimate nouns when used in a sentence is an important aspect of Hindi language. Note that these instantiations also depend upon the dialect spoken (Hall, 2002). The word 'dahi' (yogurt) is assigned feminine forms of verbs and adjectives in western dialects and masculine forms in Eastern dialects of Hindi.
2.2 Caste, Religion, and Occupation Biases
Historically, discrimination based on attributes of caste, religion, and occupation has been prominent in India (Banerjee and Knight, 1985; Deshpande, 2010). While illegal, some of these biases are still common and as a result, language resources reflect the same. Caste is a form of social stratification unique to Hinduism and Indian society. It involves the concept of hereditary transmission of the style of life in terms of occupation, status, and social interaction, and is commonly associated strongly with last names of persons<sup>3</sup>. Consequently, it is associated with strong biases pertaining to purity, goodness, intelligence, etc. of individuals, which is reflected commonly in Hindi corpora. Despite being a secular nation, due to historical clashes, there are biases in India against the relatively minority religious population practicing Islam as opposed
$^{1}$ Code is available at https://github.com/vijit-m/SocHindi  $^{2}$ Word order refers to positioning of subject, verb, and object in a sentence.
<sup>3</sup>https://en.wikipedia.org/wiki/Caste
to the majority religion of Hinduism. These biases are associated with perceptions (good, bad, evil, etc.) of the different populations. Although biases against other religions are also present, we especially focus upon Islam and Hinduism since these are the most prominent. Like caste, some last names are highly associated with religion in India, and can serve as a proxy for studying the bias. In addition to these biases, the gaps between rural and urban India in terms of the education and poverty has led to a discrepancy of perception (positive versus negative) between urban and rural occupations.
3 Measuring Biases in Hindi language Representations
To quantify bias in English embeddings, Caliskan et al. (2017) propose the Word Embedding Association Test (WEAT), which measures the stereotypical association between two sets of target concepts and attributes (See Appendix B), where a larger WEAT score indicates a larger bias. May et al. (2019) propose SEAT (Sentence Embedding Association Test) for measuring bias in sentence encoders. Similar to word lists for WEAT tests, SEAT comprises of sentences in which each sentence is a semantically neutral template which are completed with target words related to protected attributes and associated stereotypes. This puts focus on target words on which bias is to be measured.
3.1 Gender Bias
For evaluating binary gender-associated bias, we create WEAT tests (Caliskan et al., 2017) in two ways in Hindi, by creating (i) Translated word lists, and (ii) Language-Specific word lists.
For the Translated test, we directly translate each individual word in each test for career & family, arts & mathematics and arts & sciences tests in (Caliskan et al., 2017). Note that direct translations of some words like 'Shakespeare' and 'NASA' in Arts and Science lists, are not accurate, have ambiguous spellings, or are not populus in the literature. Also, some words from English like 'cousin' do not have a corresponding word in Hindi. Next, we develop a set of socially-aware Language-specific tests, where we curate word lists (both attribute and target) (Appendix C) based on popular word usage in Hindi and their associations, and word frequencies in Hindi Wikipedia text.
Table 1 highlights how translated sets capture lesser bias as compared to the WEAT tests specifically created for Hindi. In particular, the WEAT translated test for binary gender and science versus arts captures significantly low bias, unlike what is prevalent in the society as well as associated text and representations (Khadilkar et al., 2021; Madaan et al., 2018; Pundir and Singh, 2019). This, in turn, emphasizes the importance of creating language-specific tests.
For the WEAT Hindi test set, we create another test to quantify gender bias across neutral adjectives, based on societal biases in Indic society (Gundugurti et al., 2015). Appendix C lists all word sets used in each WEAT test. Further, in Section 2.1, we see that there are four specific gendered word groups in Hindi, all of which are meaningfully gendered and important to be encoded and retained in our representations. For each such group, we construct an independent "Meaningful Encoding (ME)" WEAT test (Dev et al., 2021a) (see word lists in Appendix C). A Meaningful Encoding WEAT test uses attribute lists of words having meaningful gendered information (like gendered verbs) which should be captured by representations. The importance of this is two-fold: (i) it allows us to verify if meaningful gendered information is encoded in our representations, and (ii) compare with biased associations (measured by WEAT) to gauge the overall magnitude of bias versus information about an attribute captured by our embeddings.
In Table 2 we observe that for 300 dimensional Hindi GloVe embeddings (Kumar et al., 2020), significant bias is observed using the three WEAT tests (Language-Specific) for binary gender and adjectives, science v/s arts, and maths v/s arts. Each score is over 1.00, and similar valued to WEAT tests for meaningful information encoding (ME scores in Table 2), which highlights how the magnitude of bias encoded is high. Of the four meaningful information encodings, the weakest association is seen among gendered entities, owing to how prone they are to ambiguous usage across different regions (Section 2.1).
To develop SEAT tests in Hindi, similar to (May et al., 2019), we construct a list of sentence templates and fits each target word from a WEAT target list to construct SEAT target lists for each part of the speech category. We used the Hindi translations of the semantically neutral templates provided in (May et al., 2019). However, we remove some am
4Google-Translate API used to obtain translations.
Attribute	Description	WEAT (GloVe)	SEAT (GloVe)
Translated	Lang-Specific	Translated	Lang-Specific
Gender	maths, arts vs male, female	0.94 (0.02)	1.12 (0.01)	0.87 (0.00)	1.14 (0.00)
science, arts vs male, female	0.27 (0.31)	1.13 (0.02)	0.18 (0.17)	1.03 (0.00)
Caste	adjectives vs caste	0.72 (0.00)	1.52 (0.00)	0.74 (0.00)	1.40 (0.00)
Religion	adjectives vs religion terms	1.05 (0.00)	1.28 (0.01)	1.04 (0.00)	1.20 (0.00)
adjectives vs lastnames	0.93 (0.00)	1.55 (0.00)	0.95 (0.00)	1.41 (0.00)
Occupation	adjectives vs urban, rural occupations	-0.08 (0.59)	1.58 (0.00)	-0.13 (0.88)	1.42 (0.00)
Table 1: WEAT and SEAT bias measurements (with p-values in parentheses) for tests with translated versus language-specific word lists. Highlighted values point towards the observation that more bias is captured in language-specific curated word lists.
biguous translations and we add other templates based on colloquial usage (Appendix D) and word lists created for WEAT (Appendix C). We conduct SEAT tests upon 300 dimensional Hindi GloVe embeddings and Hindi ELMo (Kumar et al., 2020).
Table 2 demonstrates that for GloVe, the SEAT scores report significant bias for all tests, while for ELMo, the bias is mainly measured in tests with binary gender and adjectives. From Table 1 we note that the Language-Specific word lists record higher amounts of bias than the translated test. This highlights the significance of constructing the word lists in WEAT tests while keeping language and cultural considerations in mind.
3.2 Caste Bias
To evaluate Hindi representations for caste bias, we build two WEAT tests and two corresponding SEAT tests using the last names that are statistically more associated with stereotypically lower and upper castes. For lower castes, we randomly sample lower caste names from the list of scheduled castes provided by the Department of Social Justice and Empowerment in India (Appendix C). Our first test is based upon detecting biased association of occupations with 'upper' castes (the upper strata of castes) and 'lower' castes. Note that, some caste names mean certain occupations themselves in Hindi language. For example 'kumhar' means both a lower caste and the occupation of pottery. We ensure that target word lists have no ambiguous entities. Another WEAT test we build is based upon positive and negative adjectives association with caste names. For the Translated version, we take Hindi translations of words from Caliskan et al. (2017) for detecting racial bias. For Language-specific test we curate a new word list of adjectives (Appendix C) based on words used popularly as positive or negative in Hindi (Thorat and Newman, 2007).
Table 2 highlights that there is significant caste
related perception bias. For both WEAT and SEAT tests, the measured biases are over 1.2 for GloVe embeddings. The results in Table 1 compare the Translated and Language-Specific adjective word lists. For WEAT test, the bias measured by Translated word lists are less than half of that measured by Language-Specific word lists, emphasizing the importance of creating socially-aware Language-Specific word lists which correlate better with the society and culture the language is associated with.
3.3 Religion Bias
We construct two WEAT and two SEAT tests to detect religion associated biases in Hindi embeddings. Our first test is based upon associating positive and negative adjectives to religious entities. One attribute list consists of Hindu religious entities and one consists of Muslim religious entities. In our second test, we associate adjectives with lastnames. This stems from the distinct last names commonly used by the two populations (see Appendix C). Similar to Caste bias detection, we experiment with the Translated and Language-Specific adjective lists. Further, we evaluate if religious information which is correctly associated is learnt by the representations (for example, mosque being the place of worship in Islam should be associated with it in representations). For this, we create a meaningful encoding (ME) test for religious information (see word lists in Appendix).
In Table 2 we see using the WEAT and SEAT scores for 300-dimensional GloVe embeddings that significant religious bias with respect to the positive and negative perception is detected. Table 1 compares the measured bias in case of Translated and Language-Specific adjective word lists, with the latter capturing significantly larger bias.
3.4 Rural v/s Urban Occupation Bias
Besides, we detect bias in urban and rural occupations, which is prevalent in Indic society - with
Attribute	Description	WEAT	SEAT
GloVe	GloVe	ELMo
Gender	BM	maths, arts vs male, female	1.12 (0.01)	1.14 (0.00)	0.17
science, arts vs male, female	1.13 (0.01)	1.03 (0.00)	0.14
adjectives vs male, female	1.21 (0.02)	1.19 (0.00)	1.37
ME	gendered verbs vs male, female	1.87 (0.00)	1.84 (0.00)	1.66
gendered adjectives vs male, female	1.70 (0.00)	1.63 (0.00)	1.78
gendered entities vs male, female	1.14 (0.01)	1.12 (0.00)	1.77
gendered titles vs male, female	1.92 (0.00)	1.86 (0.00)	1.64
Caste	BM	occupations vs caste	1.44 (0.00)	1.26 (0.00)	0.89
adjectives vs caste	1.52 (0.00)	1.40 (0.00)	0.48
Religion	BM	adjectives vs religion terms	1.28 (0.01)	1.20 (0.00)	0.75
adjectives vs lastnames	1.55 (0.00)	1.41 (0.00)	1.02
ME	religious entities vs religion	1.75 (0.00)	1.69 (0.00)	1.23
Occupation	BM	adjectives vs urban, rural occupations	1.58 (0.00)	1.42 (0.00)	1.12
Table 2: Bias measurements (with p-values in parentheses) for gender, caste, religion and occupation bias. These results are for Language-Specific word lists; BM: Bias Measuring test, ME: Meaningful Encoding test.
urban occupations seen as better, richer, more desirable, and even of a higher social status. We construct WEAT and SEAT tests where the attribute list consists of lists of urban occupations and rural occupations and the target lists consisted of polarized adjectives (Appendix C).
Table 2 illustrates with WEAT and SEAT scores the biased associations of perception between urban and rural occupations. For both GloVe and ELMo embeddings, we observe significant  $(>1.0)$  bias with the WEAT test, highlighting the presence of occupation associated bias.
Biases are inherently complex as are their encodings in language representations. Their detection needs to take into account a multitude of factors - the language and its grammar, the regions it is spoken in, as well as the history and culture of the region. We demonstrate here how a predetermined set of biases and a peripheral evaluation consisting of a narrow perspective of how biases manifest themselves is not sufficient to achieve fair representations across the globe.
Our work is limited by the scarcity of robust language models in Hindi language, as well as dedicated word lists for different language tasks in Hindi language. Hence, a number of extrinsic tests and experiments for bias evaluation could not be performed to evaluate bias more extensively. We thus focus here only on intrinsic measurements of bias which may not be correlated with bias expressed in downstream tasks (Goldfarb-Tarrant et al., 2019; Cao et al., 2022). We leave investigations regarding the same to future work. Furthermore, we acknowledge that our analysis of gender associated biases is limited to binary gender and our intrinsic evaluations require discrete categor
rizations (Dev et al., 2021b; Antoniak and Mimno, 2021). Finally, despite the limitations, we believe our work lays down some fundamentals with respect to evaluating biases across languages and associated cultures.
Language models, with their widespread applications impact people across the world. This makes it imperative that associated harms be understood not just for the Western world and with a focus on English language models, but also across languages and cultures. With this work, we highlight the importance of social and cultural awareness for the same. Bias detection methods need this cultural expertise and can then be followed by adapted mitigation methods (some possible adapted methods discussed in Appendix E). With this work, we demonstrate how translations of words is not sufficient for capturing biases across languages, and thus highlight the need for development of strategies with specific languages and cultures in mind.
This work was supported by NSF IIS-1927554 and NSF grant #2030859 to the Computing Research Association for the CIFellows Project. We thank the anonymous reviewers, and members of UCLA-NLP and Plus labs for their feedback.
D. Amutha. 2017. The roots of gender inequality in india.
Maria Antoniak and David Mimno. 2021. Bad seeds: Evaluating lexical methods for bias measurement. In Proceedings of the 59th Annual Meeting of the