SlowGuess/NAACL_2022_OCR huggingface dataset:
Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1889-1904, Online. Association for Computational Linguistics.
Biswajit Banerjee and John B Knight. 1985. Caste discrimination in the indian urban labour market. Journal of development Economics, 17(3):277-307.
Varad Bhagwat, Mrunali N Nagarkar, Pooja Paramane, and Shrikant Jindam. 2019. Review of chatbot system in Hindi language. In Review of Chatbot System in Hindi Language.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186.
Yang Trista Cao, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta, Varun Kumar, Jwala Dhamala, and Aram Galstyan. 2022. On the intrinsic and extrinsic fairness evaluation metrics for contextualized language representations.
Rodrigo Alejandro Chávez Mula and Gerasimos Spanakis. 2020. Evaluating bias in Dutch word embeddings. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 56-71, Barcelona, Spain (Online). Association for Computational Linguistics.
Hoi Yan Cheung and Alex W. H. Chan. 2007. How culture affects female inequality across countries: An empirical study. Journal of Studies in International Education, 11(2):157-179.
Manali S Deshpande. 2010. History of the indian caste system and its impact on india today. CalPoly Student Research.
Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Srikumar. 2021a. OSCaR: Orthogonal subspace correction and rectification of biases in word embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5034-5050, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Arjun Subramonian, Jeff Phillips, and Kai-Wei Chang. 2021b. Harms of gender exclusivity and challenges
in non-binary representation in language technologies. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1968-1994, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sunipa Dev and Jeff Phillips. 2019. Attenuating bias in word vectors. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 879-887. PMLR.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Sayan Ghosh, Dylan Baker, David Jurgens, and Vinodkumar Prabhakaran. 2021. Detecting cross-geographic biases in toxicity modeling on social media.
Seraphina Goldfarb-Tarrant, Haining Feng, and Nanyun Peng. 2019. Plan, write, and revise: an interactive system for open-domain story generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 89-97, Minneapolis, Minnesota. Association for Computational Linguistics.
Prasad Gundugurti, KL Vidya, and V Sriramya. 2015. The indian "girl" psychology: A perspective. Indian journal of psychiatry, 57:S212-5.
Khushi Gupta. 2021. Stereotypes in bollywood cinema: Does article 15 reinforce the dalit narrative? In *Inquiries Journal [Online]*.
Kira Hall. 2002. "Unnatural" Gender in Hindi, pages 133-162. Oxford University Press.
Kunal Khadilkar, Ashiqur R. KhudaBukhsh, and Tom M. Mitchell. 2021. Gender bias, social bias, and representation: 70 years of bhollywood. Patterns, page 100409.
Saurav Kumar, Saunack Kumar, Diptesh Kanojia, and Pushpak Bhattacharyya. 2020. "a passage to india": Pre-trained word embeddings for indian languages. In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), pages 352-357.
Yaman Kumar, Debanjan Mahata, Sagar Aggarwal, Anmol Chugh, Rajat Maheshwari, and Rajiv Ratn Shah. 2019. Bhaav-a text corpus for emotion analysis from Hindi stories. arXiv preprint arXiv:1910.04073.
Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. 2017. The iit bombay english-hindi parallel corpus. arXiv preprint arXiv:1710.02855.
Mascha Kurbicz-Briki. 2020. Cultural differences in bias? origin and gender bias in pre-trained german and french word embeddings. Proceedings of the 5th Swiss Text Analytics Conference (SwissText) & 16th Conference on Natural Language Processing (KONVENS).
Anne Lauscher, Rafik Takieddin, Simone Paolo Ponzetto, and Goran Glavaš. 2020. AraWEAT: Multidimensional analysis of biases in Arabic word embeddings. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 192-199, Barcelona, Spain (Online). Association for Computational Linguistics.
Sheng Liang, Philipp Dufter, and Hinrich Schütze. 2020. Monolingual and multilingual reduction of gender bias in contextualized representations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5082-5093, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Nishtha Madaan, Sameep Mehta, Taneea S. Agrawaal, Vrinda Malhotra, Aditi Aggarwal, Yatin Gupta, and Mayank Saxena. 2018. Analyze, detect and remove gender stereotyping from Bollywood movies. In FAccT.
Thomas Manzini, Lim Yao Chong, Alan W Black, and Yulia Tsvetkov. 2019. Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 615-621, Minneapolis, Minnesota. Association for Computational Linguistics.
Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 622-628, Minneapolis, Minnesota. Association for Computational Linguistics.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953-1967, Online. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference
on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Arun K. Pujari, Ansh Mittal, Anshuman Padhi, Anshul Jain, Mukesh Jadon, and Vikas Kumar. 2019. Debiasing gender biased Hindi words with word-embedding. In Proceedings of the 2019 2nd International Conference on Algorithms, Computing and Artificial Intelligence, ACAI 2019, page 450-456, New York, NY, USA. Association for Computing Machinery.
Ishita Pundir and Alankrita Singh. 2019. Portrait of women in indian fiction. Volume 09:137-141.
Pattabhi RK Rao and Sobha Lalitha Devi. 2018. Eventextract-il: Event extraction from newswires and social media text in indian languages@ fire 2018-an overview. In FIRE (Working Notes), pages 282-290.
Nithya Sambasivan, Erin Arnesen, Ben Hutchinson, and Vinodkumar Prabhakaran. 2020. Nonportability of algorithmic fairness in india. ArXiv, abs/2012.03659.
Sukhadeo Thorat and Katherine S Newman. 2007. Caste and economic discrimination: causes, consequences and remedies. Economic and Political Weekly, pages 4121-4124.
Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini, Kai-Wei Chang, and Ahmed Hassan Awadallah. 2020. Gender bias in multilingual embeddings and cross-lingual transfer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2896-2907, Online. Association for Computational Linguistics.
Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang, Muhao Chen, Ryan Cotterell, and Kai-Wei Chang. 2019. Examining gender bias in languages with grammatical gender. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5276-5284, Hong Kong, China. Association for Computational Linguistics.
As acknowledged in the paper, our work is severely limited by the scarcity of language models, dedicated word lists, and language tasks for Hindi language. A number of tests and experiments could not be performed to evaluate bias more extensively. Furthermore, the lack of established language tasks and datasets in Hindi made it difficult to analyze the extrinsic bias in downstream tasks. Although this limits our evaluations of bias in this work, with more work like this and development of more language tools for Hindi, this can be overcome. We further emphasize that while we have evaluated some biases, these are not the only biases present in the Indian society or Hindi language. We merely provide evaluations for some that are strongly present in the literature and thus in the language representations as well.
Since this work highlights various biases and the words commonly associated with it, it can potentially be triggering to persons. However, it is important to study these biases and their impact on language tools in order to mitigate their effect.
B WEAT
Let  $X$  and  $Y$  be equal-sized sets of target concept embeddings and let  $A$  and  $B$  be sets of attribute embeddings. Let  $\cos(a, b)$  denote the cosine similarity between vectors  $a$  and  $b$ . The test statistic is a difference between sums over the respective target concepts,
$$
s (X, Y, A, B) = \sum_ {x \in X} s (x, A, B) - \sum_ {y \in Y} s (y, A, B) \tag {1}
$$
where the quantity,  $s(w,A,B)$  is calculated using cosine similarity as follows:
$$
s (w, A, B) = \frac {\sum_ {a \in A} \cos (w , a)}{| A |} - \frac {\sum_ {b \in B} \cos (w , b)}{| B |} \tag {2}
$$
The amount of bias in WEAT is analyzed by effect size  $d$  calculated as:
$$
d = \frac {\operatorname {m e a n} _ {x \in X} s (x , A , B) - \operatorname {m e a n} _ {y \in Y} s (y , A , B)}{\operatorname {s t d d e v} _ {w \in X \cup Y} s (w , A , B)} \tag {3}
$$
In order to compute the test significance (p-value),  $X$  and  $Y$  lists are merged together, and 10,000 permutations,  $P$  of the combined list is generated.
For the  $i$ -th list in  $P$ , it is split in new pairs of  $X_{i}$  and  $Y_{i}$  lists. Then the test statistic equation is used to calculate the p-value in the following way:
$$
p = \frac {\sum_ {i \in P} [ s (X _ {i} , Y _ {i} , A , B) ] > s (X , Y , A , B) ]}{| P |} \tag {4}
$$
C WEAT word lists
In our bias detection methods using WEAT, we constructed WEAT tests for Gender, Caste, Religion and Occupation (Rural v/s Urban occupations) biases. Since the direct translations of word lists from (Caliskan et al., 2017) did not provide us with any significant evidence of bias, we constructed the word lists ourselves based upon popular Hindi words usage. Refer to Table 7 for the word lists used to detect gender bias. We obtain the science, maths, gendered words, gendered adjectives and occupations from online Hindi resources. We refer to Wikipedia for glossary of Hinduism entities and Islamic entities. For the list of castes, we refer to the list of scheduled castes provided by the Department of Social Justice and Empowerment in India. For the lastnames we refer to the popular Islamic lastnames and Hindu lastnames provided by online resources.
In addition to the bias measurement (BM), tests, we also provide meaningful encoding. (ME) tests used to capture meaningful gendered information in Hindi.
Table 8, Table 9 and Table 10 provides the word lists used in the measurement of caste, religion and occupation biases respectively.
D SEAT
We define a list of semantically neutral sentence templates for each part of speech type of words as follows:
- Hindi-SEAT-name: 'yeha _ hai', 'veha _ hai', 'vahan _ hai', 'yahan _ hai', '_ yahan hai', 'vahan hai', 'iska naam _ hai', 'uska naam _ h'
- Hindi-SEAT-common-nouns: 'yeha _ hai', 'veha _ hai', 'vahan _ hai', 'yahan _ hai', 'yahan hai', 'vahan hai', 'vo _ hai', 'ye _ hai'
- Hindi-SEAT-verbs: 'yeha _ hai', 'veha _ hai', 'vo _ hai', 'ye _ hai', 'vahan _ hai', 'yahan _ hai'
- Hindi-SEAT-adjectives: 'yeha _ hai', 'veha _ hai', 'vo _ hai', 'ye _ hai'
In other words, if the target word is adjective, we use Hindi-SEAT-adjective list of semantically bleached sentences with each WEAT target word.
E Debiasing
There have been notable advances towards debiasing embeddings along the direction of gender bias. Both Bolukbasi et al. (2016) and Dev and Phillips (2019) propose using linear projection to debias word embeddings, but the former in addition also equalizes word pairs about the attribute (e.g., gender) axis.
Although we tried and adapted several existing methods for debiasing, we could not evaluate the performance of the debiasing methods on the extrinsic tasks. This is because of the scarcity of reliable Hindi language datasets, which made any form of notable inferences harder. In addition, the deep learning models were already underperforming on these Hindi datasets.
In this work we use the more general approach of linear projection as it can be adapted to several biases apart from gender.
In the method of linear projection, all words  $w \in W$  are debiased to  $w'$  by being projected orthogonally to the identified bias vector  $v_{B}$ .
$$
w ^ {\prime} = w - \langle w, v _ {B} \rangle v _ {B} \tag {5}
$$
In case of hard debiasing, we required list of equalizing pairs and list of words to not debias in Hindi. However, direct translation of the word lists to Hindi did not always make sense. Since, some words like 'she' and 'he' had overlapping translations and both the pronouns are referred to as 'veha' in Hindi. This overlapping translation is true the other way round as well, the word grandfather can be either 'nana' (maternal grandfather) or 'dada' (paternal grandfather).
For languages with grammatical gender, Zhou et al. (2019) proposed to determine semantic gender direction. To obtain the semantic gender direction  $(d_{s})$ , the grammatical gender component  $(d_{g})$  in the computed gender direction (obtained from PCA over gendered word pairs,  $d_{PCA}$ ) to make the semantic gender direction orthogonal to grammatical gender.
$$
d _ {s} = d _ {P C A} - \langle d _ {P C A}, d _ {g} \rangle d _ {g} \tag {6}
$$
We use this orthogonalized gender direction to perform linear debiasing. We refer to this method as LPSG (Linear Projection with Semantic Gender).
E.1 Debiasing Binary Gender
The first step in debiasing using linear projection is to identify a bias subspace/vector. We experiment with different settings to identify the gender vector in Hindi, including (i) a single gender specific word pair direction of  $\{n\vec{a}ri - n\vec{a}r\}$ , (ii) PCA over a list of paired gender specific words (in the form  $\{\vec{m}_i - \vec{f}_i\}$ ). For more results with other gender directions, refer to the Appendix. Also, the word lists used in the experiment are provided in Appendix E.
For hard debiasing, we considered two types of gender definition word lists. In one list we included only the gender definitional pairs translated to Hindi from the original English lists (after some modifications to remove ambiguous translations). In another experiment, we added pairs of gendered verbs to the list as well.
Hindi is a language having grammatical gender. As introduced in Section 2.1, we have 4 special gender directions along which we want to preserve the information. The direction for gendered verbs  $(d_v)$ , adjectives  $(d_a)$ , titles  $(d_t)$  and entities  $(d_e)$  were calculated by conducting PCA over the word lists (App E). For LPSG method, we provide results of orthogonalizing the semantic gender with respect to verbs and adjectives directions. In another experiment, we orthogonalize the semantic gender with respect to all the 4 directions.
Table 3 demonstrates how different gender subspaces affect the WEAT effect sizes in both bias measuring and information retention tests. Note that the single direction of  $\{n\vec{a}ri - n\vec{a}r\}$  was able to debias the best upon the math and arts test. Pairwise PCA over gendered words debiased the science and arts test quite significantly with an effect size of only 0.001 after debiasing. Hard debiasing is not able to debias the first two tests in the WEAT setting, however, it reduces the effect sizes in case of SEAT (see Table 4). In both WEAT and SEAT tests for neutral adjectives vs gendered words, hard debiasing performs best against any other methods. Although hard debiasing works competitively, it comes with the downside that it does not retain the gendered information in our information retention (IR) tests. Both of the LPSG variants were able to debias competitively while at the same time were
best in retaining the gendered information of IR tests.
E.2 Debiasing Caste
Similar to debiasing gender, for caste we first begin with determining the caste direction with a two words, one stereotypically considered upper caste and one lower:  $\{\vec{g h a s i y a} -\vec{p a n d i t}\}$ . We also try with the direction  $\{\vec{g h a s i y a} -\vec{d e s a i}\}$ . Since castes do not occur in pairs, a set of word pairs cannot be meaningfully constructed as done with binary gender in English (Bolukbasi et al., 2016). Hence, we compose lists of stereotypically upper and lower castes, and conduct PCA over the combined list to obtain the vector of caste bias. Refer to Appendix E for the word lists used in the experiment. In Table 5 we can observe that the linear debiasing using the single direction of  $\{\vec{g h a s i y a} -\vec{d e s a i}\}$  is unable to debias competitively when compared with the other two methods. Note that the single direction of  $\{\vec{g h a s i y a} -\vec{p a n d i t}\}$  is able to debias better than PCA over list of caste names.
E.3 Debiasing Religion
In order to mitigate religious biases in Hindi, we acknowledge how in Indian culture, the religion of a person is generally identifiable by their last names. We thus, utilize last names to determine the direction of bias. We use both (i) a single set of common last names  $\{a\vec{ch}\vec{a} r\vec{a} y - n\vec{a} s\vec{r}\}$ , and (ii) a set of hindu and muslim entities.
Another religion direction is calculated by combining word lists of Hindu and Muslim lastnames and then conducting PCA over them, we call this religion bias direction as  $d_{last}$ . The words lists are provided in Appendix E.
In Hindi language, various religious entities are inherently associated with a particular religion, for example, "Bible is to Christianity as Bhagwad Gita is to Hinduism" is not bias. To accommodate for such cases, we again take motivation from (Zhou et al., 2019) to obtain a direction  $d_{ent}$  from the entities word lists (Appendix E) and keep the religion direction calculation calculated by Hindu and Muslim lastnames  $d_{last}$ , orthogonal to it.
$$
d _ {l a s t} ^ {\prime} = d _ {l a s t} - \langle d _ {l a s t}, d _ {e n t} \rangle d _ {e n t} \tag {7}
$$
We believe that if we debias words using  $d_{last}^{\prime}$  as bias direction, we should be able to preserve the knowledge of religion information retention test and debias competitively.
In Table 6, we see that linear debiasing by conducting PCA over a list of religious entities is not able to debias much in any of the tests. The same could be observed for linear debiasing using single set of common last names  $\{acharya - nasir\}$ . However, if we linear debias by PCA over a list of lastnames, we are able to debias significantly. Although the Information Retention WEAT effect size is less than the previous methods, they did not even affect the religion bias which is our primary goal. Zhou's variant for religion debias performs well since it is able to debias competitively as well as retains greater amount of necessary religion information. Refer to Appendix C for the word lists used in the test.
Description (vs male, female)	Original WEAT	Linear Projection	Hard Debiasing	LPSG
naari-nar	PCA	Gen. words	Gen. words,verbs	w/o verbs & adj	w/o all dir.
maths, arts	1.12 (0.01)	0.44 (0.20)	0.77 (0.06)	1.48 (0.00)	1.13 (0.00)	0.95 (0.03)	1.04 (0.02)
science, arts	1.13 (0.02)	0.50 (0.18)	0.00 (0.49)	1.66 (0.00)	1.57 (0.00)	0.24 (0.28)	0.42 (0.21)
adjectives	1.22 (0.02)	0.96 (0.06)	0.82 (0.08)	0.37 (0.27)	0.49 (0.20)	0.94 (0.05)	0.94 (0.049)
IR	gen. verbs	1.87 (0.00)	1.87 (0.00)	1.79 (0.00)	1.12 (0.01)	-1.18 (0.99)	1.85 (0.00)	1.85 (0.00)
gen. adj	1.70 (0.00)	1.63 (0.00)	1.66 (0.00)	1.19 (0.00)	0.78 (0.05)	1.71 (0.00)	1.75 (0.00)
gen. entities	1.14 (0.01)	1.01 (0.02)	0.99 (0.02)	0.66 (0.08)	0.27 (0.28)	1.13 (0.00)	1.18 (0.00)
gen. titles	1.92 (0.00)	1.91 (0.00)	1.89 (0.00)	1.19 (0.00)	1.59 (0.00)	1.92 (0.00)	1.91 (0.00)
Table 3: Debiasing results for gender across different debiasing methods of linear projection, Bolukbasi's hard debiasing and different variants of LPSG debiasing. We provide WEAT effect sizes with p-values of the test in parentheses. PCA for Linear Projection was done on gendered word pairs. IR stands for Information Retention.
Description (vs male, female)	Original SEAT	Linear Projection	Hard Debiasing	LPSG
naari-nar	PCA	Gen. words	Gen. words,verbs	w/o verbs & adj	w/o all dir.
maths, arts	1.14 (0.00)	0.64 (0.00)	0.78 (0.00)	0.76 (0.00)	1.09 (0.00)	0.96 (0.00)	0.99 (0.00)
science, arts	1.03 (0.00)	0.55 (0.00)	0.07 (0.33)	0.70 (0.00)	0.91 (0.00)	0.26 (0.06)	0.38 (0.02)
adjectives	1.19 (0.00)	0.98 (0.00)	0.80 (0.00)	0.23 (0.30)	0.34 (0.31)	0.94 (0.00)	0.92 (0.00)
IR	gen. verbs	1.84 (0.00)	1.83 (0.00)	1.67 (0.00)	0.31 (0.33)	-0.70 (0.70)	1.80 (0.00)	1.78 (0.00)
gen. adj	1.63 (0.00)	1.58 (0.00)	1.54 (0.00)	0.45 (0.17)	0.36 (0.33)	1.63 (0.00)	1.67 (0.00)
gen. entities	1.12 (0.00)	1.02 (0.00)	0.99 (0.00)	0.42 (0.14)	0.45 (0.17)	1.13 (0.00)	1.16 (0.00)
gen. titles	1.86 (0.00)	1.85 (0.00)	1.75 (0.00)	0.15 (0.41)	0.90 (0.22)	1.82 (0.00)	1.80 (0.00)
Table 4: Debiasing results for gender across different debiasing methods of linear projection, Bolukbasi's hard debiasing and different variants of LPSG debiasing. We provide SEAT effect sizes with p-values of the test in parentheses. PCA for Linear Projection was done on gendered word pairs. IR stands for Information Retention.
Test Type	Description (vs caste)	Original Score	Linear Projection
ghasiya - desai	ghasiya - pandit	PCA
WEAT	occupations	1.44 (0.00)	1.34 (0.00)	0.78 (0.09)	1.21 (0.02)
adjectives	1.52 (0.00)	1.51 (0.00)	1.31 (0.01)	1.33 (0.00)
SEAT	occupations	1.26 (0.00)	1.17 (0.00)	0.67 (0.00)	0.89 (0.00)
adjectives	1.40 (0.00)	1.36 (0.00)	1.18 (0.00)	1.18 (0.00)