SlowGuess/NAACL_2022_OCR huggingface dataset:
Victor Steinborn and Philipp Dufter* and Haris Jabbar and Hinrich Schütze
{vsteinborn, philipp, jabbar}@cis.lmu.de
Warning: This work deals with statements of a stereotypical nature that may be upsetting.
Bias research in NLP is a rapidly growing and developing field. Similar to CrowS-Pairs (Nangia et al., 2020), we assess gender bias in masked-language models (MLMs) by studying pairs of sentences that are identical except that the individuals referred to have different gender. Most bias research focuses on and often is specific to English. Using a novel methodology for creating sentence pairs that is applicable across languages, we create, based on CrowS-Pairs, a multilingual dataset for English, Finnish, German, Indonesian and Thai. Additionally, we propose  $S_{\mathrm{JSD}}$ , a new bias measure based on Jensen-Shannon divergence, which we argue retains more information from the model output probabilities than other previously proposed bias measures for MLMs. Using multilingual MLMs, we find that  $S_{\mathrm{JSD}}$  diagnoses the same systematic biased behavior for non-English that previous studies have found for monolingual English pre-trained MLMs.  $S_{\mathrm{JSD}}$  outperforms the CrowS-Pairs measure, which struggles to find such biases for smaller non-English datasets.
Pretrained language models (PLMs) have greatly benefited NLP (Raffel et al., 2020; Peters et al., 2018; Devlin et al., 2019; Zhuang et al., 2021). However, commonly used PLMs such as BERT have been shown to encapsulate social biases, including those relating to gender and race (Kurita et al., 2019; Nadeem et al., 2021; Nangia et al., 2020). The general consensus is that these biases are learned from the statistical distributional cooccurrence of words relating to a group (such as terms relating to men or women) with a context in which that group is often mentioned in corpora (Bolukbasi et al., 2016; Webster et al., 2021). For
Figure 1: Following Nangia et al. (2020), we assess multilingual gender bias in MLMs by matching gender-specific tokens (light blue) in the context of non-genderspecific tokens (dark blue) in sentence pairs. We develop a methodology for creating sentence pairs that we argue is applicable across languages in contrast to prior work. We mask unchanged tokens one at a time and calculate  $S_{\mathrm{JSD}}$ , a novel information-theoretic bias measure whose sentence-level average we show to be better behaved than competing measures.
example, "doctor" may co-occur with "man" more often than with "woman", leading to an internal representation in the model where a gender-neutral concept, such as being a doctor, is more closely associated with male-related terms than with female-related terms (Bolukbasi et al., 2016).
In this work we tackle this type of binary stereotypical representational gender bias (henceforth simply "gender bias") in MLMs in a multilingual setting. We propose a multilingual approach to study gender bias in MLMs, outlined in Figure 1, which, to the best of our knowledge, can in principle be extended to any language.<sup>1</sup>
The importance of developing AI systems that
$^{1}$ Code and dataset with additional languages available at: https://github.com/VSteinborn/s_jsd-multilingual-bias
*Now at Apple
Findings of the Association for Computational Linguistics: NAACL 2022, pages 921 - 932
are mindful of different societal groups, such as people of different genders, is a topic much discussed in the area of fairness research in NLP (Blodgett et al., 2020). However, a shortfall of this area is its almost exclusive focus on English. As far as we are aware, ours is the first study to attempt to create a truly multilingual approach to study gender bias in language models. Previous multilingual approaches were largely limited to sentences with fixed templates and grammar structures, which heavily constrains the range of languages that may be studied with a given template (González et al., 2020). Our approach builds on Nangia et al. (2020) and attempts to study natural sentences by comparing a pair of sentences that differ only by the gender of persons mentioned, a process which we will refer to as gender swapping.
To illustrate the problem of using templates, consider the following sentence pair and its German translation.
(1) a. He is the doctor here.
b. She is the doctor here.
(2) a. Er ist der Arzt hier.
b. SieistdieArztin hier.
In German the only parts that remain the same are "ist" and "hier" under gender swapping, as the German word for the profession "doctor" and its associated definite article change form depending on the gender of the person. Thus, template structures developed for English of the form
(3) [person] is the [profession] here.
have to be heavily modified and constrained to create grammatically correct sentences in German. The problem is exacerbated in multilingual studies, where appropriate templates need to be defined for each language.
We take inspiration from CrowS-Pairs (CPS) (Nangia et al., 2020), which studies pairs of crowdsourced sentences, for a range of social biases. It includes gender-swapped pairs for the diagnosis of gender bias. However, we found that we cannot simply translate CPS into other languages. The main problem is that English pronouns are clear indicators of gender – at least of binary gender, which we focus on in this paper. But this clear indication gets lost in translation for languages that have gender-neutral pronouns like Finnish and those that predominantly use null pronouns like
Thai.2 We could mandate that only words with "gender-inherent" meaning like "mother", "wife" and "sister" are used, but that would exclude many topics that we need to cover in a good diagnostic dataset, e.g., work life and sports.
The solution we propose is to simply use names to indicate gender. Our assumption here is that all languages have words for names and that there are two subsets of names that can only have female and male referents. Note that there are certainly "unisex" names, i.e., names that can refer to both men and women, even in English ("Jess", "Leslie"). But as far as we know there is no language that has no "monosex" names, i.e., names that can refer to only one gender. We rely on such monosex names to construct sentence pairs.
In English, we select a few frequent male and female names; we only use them for English. Before translating the sentence pairs into another language, we first identify corresponding frequent male and female names in the target language. The translators are then instructed to only use those names. This methodology should be applicable universally, so that we can construct a multilingual gender bias resource for any set of languages. In this paper, we translate the CPS dataset into German, Indonesian, Thai and Finnish. We edit the original CPS dataset before translation to heed the recommendations of Blodgett et al. (2021). A more detailed description of dataset creation will be given in §3.1. $^{3}$
The second contribution of this paper is  $S_{\mathrm{JSD}}$ , a novel measure based on the Jensen-Shannon divergence (Lin, 1991), to test MLMs for social biases by using sentence pairs that capture a binary contrast between two groups. The measure used in CPS (see §3.2) makes use of a binary decision process, which has the effect of removing information of the probability values from the MLM, which we show reduces the measure's predictive power. Our motivation for introducing  $S_{\mathrm{JSD}}$  is to retain as much information from the MLM output probabilities as possible in our final reported score in order to make effective use of the limited amount of human-translated sentences that are available.
Thus, our contributions are (1) developing a method for creating multilingual datasets for diagnosing gender bias in language models that is ap
2The English sentence "she ate it" is simply expressed as "ate" in many "pro-drop" languages as long as subject and object of "ate" are clear from context.
3Blodgett et al. (2021) argue against using names for race. Their arguments do not apply to gender in our setup. See §3.1.
plicable across the diverse set of human languages, (2) applying this method, taking the CPS dataset (Nangia et al., 2020) as a starting point, and creating a multilingual gender bias diagnosis dataset for English, German, Thai, Indonesian and Finnish, (3) proposing the  $S_{\mathrm{JSD}}$  measure, which retains information regarding the numeric output probabilities of MLMs.
Given this work focuses on multilingual methods to measure gender bias in MLMs, this discussion will focus on evaluation measures and techniques; a thorough discussion of debiasing methods is beyond the scope of this paper.
Bias Measures in MLMs. Recently, pretrained masked language models, such as BERT (Devlin et al., 2019), have significantly gained in popularity, which in turn has led to numerous studies analyzing their behavior, including their encapsulation and reproduction of social bias. Prior to the emergence of these models however, it was already well known that NLP models can learn social biases from corpora, as exemplified in work by Bolukbasi et al. (2016) who demonstrated that word embeddings contain societal gender biases. Subsequently, further tests, such as the word embedding association test (WEAT) by Caliskan et al. (2017), demonstrated that word embeddings also have other biases, including racial biases. May et al. (2019) extended WEAT to sentence encoders, including BERT, with the sentence encoder association test (SEAT), to study sentence-level social biases in these models using template constructed sentences. However, the results of this study were inconclusive, and Kurita et al. (2019) showed that the cosine-based methods used in WEAT and SEAT are not appropriate for contextualized embeddings, and instead use a scoring method based on the prediction probability of an attribute given a target in template sentences.
The evaluation method used in StereoSet (Nadeem et al., 2021) was inspired by SEAT while CPS (Nangia et al., 2020) uses pseudo-log-likelihood MLM scoring (Salazar et al., 2020). A contribution of CPS and StereoSet is to provide techniques that evaluate natural sentences instead of simple templates. One disadvantage of template approaches is that they have been shown to be highly dependent on the template chosen, as well as on the terms that are chosen to substitute into
the template (Delobelle et al., 2021; Antoniak and Mimno, 2021). Nonetheless, Kaneko and Bollegala (2021) criticize CPS and StereoSet for their evaluation measures, arguing that the act of masking tokens results in a systematic overestimate in measured biases. However, they also describe this effect as systematic, and thus we would expect systematic trends in bias scores between models to remain conserved when masking tokens.
Multilingual Studies of Bias in MLMs. As far as we are aware, there are no studies that have attempted to develop a multilingual method to test for gender bias in MLMs without template structures. However, there are several multilingual studies. For example, González et al. (2020) constructed sentence templates for languages with type B reflexivization (including Swedish and Russian), which can be used to construct challenge datasets to measure gender bias. Similarly, Câmara et al. (2022) used template structures to test MLMs for intersectional biases in English, Spanish and Arabic. Bartl et al. (2020) also constructed templates to study biases in German and English BERT models, but sometimes a different form of a template has to be used depending on the gender of a mentioned person. Liang et al. (2020) examined the case of English and Chinese using templates while focusing on the cross-lingual transfer of removing biases in Chinese using English training data.
Counter Factual Data Augmentation (CFA). Our work generally falls under the category of CFA. CFA has been used to train a model on an augmented training corpus by swapping target terms, which has been shown to be effective for debiasing in multilingual settings via zero-shot transfer learning (Lauscher et al., 2021). However, simple substitution methods employed in CFA fail at producing grammatical sentences in languages with gender agreement rules. For such languages, other strategies, such as machine translation (Jain et al., 2021), have to be employed.
Barikeri et al. (2021) create templates from real-world conversational text that can be used to evaluate language models for social biases. These templates then produce so-called "counterfactual pairs" Zhao et al. (2018) by substituting terms representing different social groups, resulting in sentence pairs similar to those in CrowS-Pairs (Nangia et al., 2020). Again, as we discussed in the introduction, templates are difficult to use for many languages.
In contrast to most work on CFA, we do not use
templates, we target non-English, we create data by crowdsourcing and our focus is measuring bias cross-lingually, as opposed to debiasing.
Bias From a Social Science Perspective. A critical survey of 146 NLP papers by Blodgett et al. (2020) outlines common pitfalls in NLP research, including the CPS study, when attempting to study social bias. We attempt to take into account their recommendations in this work.
A major obstacle in transferring existing techniques to measure gender bias in languages beyond English is that we need to adapt methods to the target language's gender agreement system. Methods for measuring gender bias in MLMs often rely on fixed sentence templates, where predefined words are inserted that test some aspect of bias, such as occupational gender bias (e.g., Kurita et al., 2019; Webster et al., 2021)). While these template structures can be modified and applied to a range of languages, once a template is chosen, the range of languages that can be studied is restricted (González et al., 2020).
Thus, to design a multilingual approach to gender bias, we want to move beyond the rigid artificial sentence structures that result from using templates. We also speculate that moving away from rigid sentence structures allows us to probe the language model more deeply for biases. It may be possible that superficially debiased language models can perform well on certain bias evaluation tasks that use templates, similar to the situation for linearly debiased word embeddings that perform well on some bias measures but still encapsulate significant distributional biases (Gonen and Goldberg, 2019).
Two evaluation datasets that go beyond templates are StereoSet (Nadeem et al., 2021) and CPS (Nangia et al., 2020). One important difference between them is the masking pattern. While StereoSet's context association test masks words that may be gendered in a different language (e.g., adjectives in Spanish), CPS consists of pairs of sentences and only masks tokens that are shared by the two sentences. Here we will only consider the CPS dataset, which also marks which of the two sentences is more stereotypical (Nangia et al., 2020).
For our dataset, we consider sentence pairs where people of the male and female gender are
being contrasted, for example:
(4) a. He is a pilot.
b. She is a pilot.
For this example, we assume each word is a separate token. The unmodified tokens common to both sentences are: "is", "a", "pilot". For each sentence, the unmodified tokens form a set  $U$  and the remaining modified tokens a set  $M$  ("He" for (4)a, for example). Thus, for each sentence, the set of all tokens is the union of  $U$  and  $M$ .
We will make the assumption that, for sufficiently long and complex sentences, when swapping the gender of a person reference in a sentence there remain sections of the sentence that remain unchanged and that this is true for all languages. From this observation, we found the masking pattern CPS implements to be appropriate for multiple languages and thus the sentences labeled with the "gender" tag in the CPS dataset were selected as the basis for subsequent translations.
The CPS dataset was recently criticized for lacking clear explanations of what types of social biases are being measured (Blodgett et al., 2021). For this reason the selected CPS sentences have been minimally modified to be mindful of the pitfalls outlined in (Blodgett et al., 2021). For example, some sentences were omitted because the contrasted groups were unrelated to the stated "gender" label, such as for sentences that contrasted two racial groups instead.
We will now outline the modifications of the CPS dataset for this study.
First, we ensured each sentence only compares binary gender. Non-binary gender adds a level of complexity in the multilingual context that we leave for future work. We also removed sentences that compare clothing items, most likely intended as a proxy for gender. Clothing items and their significance differ across cultures, so such sentences are difficult to translate.
Second, for sentences that only used a pronoun to identify gender, we exchanged the pronoun with a common name that is stereotypically associated with one gender in the English dataset. Subsequently, when translating the English dataset into other languages, the names were exchanged for others that are common gendered names in the target language. We limited the number of names in the English dataset to four to simplify the subsequent translation process. Names were introduced
because many languages do not have gendered pronouns, and thus information relating to gender may be lost in translation. For example, a typical translation of (4) into Indonesian results in two identical sentences, which makes the sentence pair useless for Indonesian. Using names as a proxy for identifying a social group is discouraged in (Blodgett et al., 2021) for race bias, but using stereotypically gendered names as a proxy for binary gender seems unproblematic to us. For example, whereas names only indirectly and ambiguously identify race (at least in English), we can easily find names that are "monosex", i.e., names that can only have either male or female referents. Thus, we would modify example (4) as follows for our dataset:
(5) a. Robert is a pilot.
b. Olivia is a pilot.
Finally, we removed sentences that did not correctly isolate a stereotype, an issue noted in the original paper (Nangia et al., 2020).
In this work we investigate binary gender stereotypes as a representational harm across languages, to use the terminology of Blodgett et al. (2020). The CPS dataset was created by US crowdworkers (Nangia et al., 2020). We make the assumption that most aspects of gender bias should be part of a diagnostic test across languages and cultures. For example, the associations of "doctor" with "male" or of "childcare" with "female" are biases that most cultures are at risk for. So we should test whether our language models exhibit these biases for all cultures. There probably are aspects of gender bias that are relevant to only a small subset of cultures (e.g., the association of "being eligible to drive a car" with "male"). We stress the importance of investigating gender bias multilingually. Given that our study is the first to do this, we feel justified to leave the issue of how to comprehensively test for all aspects of bias in gender diagnosis to future work.
Note that we do not make the assumption that gender bias is the same across languages! If "childcare" is strongly associated with "female" in (the training corpus of) language A, but not in (the training corpus of) language B, then (assuming we use models that pick up bias from their training corpora) our methodology will find less gender bias for language B - and this would be the intended result of our work.
For the translations, we hired translators to trans
De	En	Fi	Id	Th
#w	5470	5548	4151	4790	6693
#w/s	13	13	10	11	16
Table 1: Our multilingual bias diagnosis dataset consists of 212 sentence pairs in five languages. The table gives total number of words (#w) and words per sentences (#w/s) for each language. Thai was tokenized with Deepcut (Kittinaradorn et al., 2019).
late the modified English dataset into their native language. Translators were paid an agreed upon amount above the minimum wage in their respective country of residence and were informed of the intended use of their translations. Each translator was provided an instruction sheet, which exemplifies the translation process of CPS sentence pairs from English to German. The translation instructions can be found in the supplementary material and the target languages of the translations were German (De), Finnish (Fi), Indonesian (Id) and Thai (Th). We chose these languages to cover different language families and because translators for them were easily available to us.
An overview of the metadata of the edited and translated dataset is given in Table 1.
3.2 Bias Measure
Our aim is to create a bias measure that can retain meaningful information from the model output that is relevant for detecting multilingual gender bias. Before introducing our proposed measure, we will go over the CPS measure (Nangia et al., 2020).
CrowS-Pairs Measure. Given is a pair of gender swapped sentences. One sentence is judged to be socially more stereotypical than the other by the annotators in the CPS study (Nangia et al., 2020). We refer to the two sentences as "more" and "less".
The set of tokens that are shared (resp. are not shared, i.e., modified) between the two sentences is denoted as  $U$  (resp.  $M$ ) - see §3.1. For each sentence the tokens in  $U$  are masked one at a time. Each time a token is masked, the sentence is passed through the model and the model output probabilities are obtained. Following Nangia et al. (2020)'s notation, we denote the output probability of the model for the  $i^{\text{th}}$  correct token under the mask  $u_{G,i} \in U$  in the more stereotypical sentence as  $P_{\text{more}}(u_{G,i}) \equiv P(u_i|U_{\backslash u_i}, M, \theta)$ , where  $M$  are the unique tokens in the more stereotypical sentence and  $\theta$  are the model parameters. The output probability for the other sentence  $P_{\text{less}}$  is defined analog-
gously.
The score for a sentence in the pair is its pseudo-log-likelihood, calculated as the sum of  $\log P(u_{G,i})$  over all  $u$  in  $U$  where  $P$  is either  $P_{\mathrm{more}}$  or  $P_{\mathrm{less}}$ . The sentence pair is assigned a binary score of 1 (resp. 0) if the more stereotypical sentence has a larger (resp. smaller) score. A possible advantage of this binarization is that the numerical value of the pseudo-log-likelihood cannot be interpreted (hence "pseudo") (Nangia et al., 2020; Salazar et al., 2020), so one can only rely on the comparison of the scores, not on their absolute values. The final score is the percentage of sentences that have been assigned a score of 1.
According to Nangia et al. (2020), an ideal unbiased model would achieve a score of 50 on a dataset. However, it is important to keep in mind that each sentence pair contributes with equal weight to the final score, due to binarization. Consider as an example a language in which a small part of the sentence pairs are diagnosed as extremely biased, but most sentence pairs do not show bias, so their final score will be randomly 0 or 1. In such a case, CPS does not distinguish strong bias from weak bias and sentence pairs that are not biased contribute noise to the final measure. Hence, unusually biased behavior of the model may not be effectively captured by the measure, and in order to obtain meaningful results a large number of human-annotated sentence pairs is required.