SlowGuess/NAACL_2022_OCR huggingface dataset:
The following simulated scenario will illustrate the effect of dataset size. Let us ignore the internal mechanisms of the model and for simplicity assume that a biased model has a fixed probability of  $p = 0.55$  to assign a binary score of 1. This may be modeled as a Bernoulli process (Papoulis and Pillai, 2002). For such a model and for a set of  $n = 200$  sentence pairs, roughly the number of sentences we consider in our study, the expected dataset score is 55 and the standard error 3.5 (since the standard error is  $\sim \frac{1}{\sqrt{n}}$  for Bernoulli). Thus, the CPS measure must rely on a large number of sentence pairs to obtain statistically meaningful results because of the binary decision process that disregards information regarding the extent of the discrepancy between  $S_{\mathrm{more}}$  and  $S_{\mathrm{less}}$ . The measures of Nadeem et al. (2021) in StereoSet and of Kaneko and Bollegala (2021) also employ binarization and therefore do not make efficient use of the available data to measure bias.
The Proposed  $S_{\mathrm{JSD}}$  measure. Our goal in de
veloping the  $S_{\mathrm{JSD}}$  measure was to create a theoretically well founded measure that retains information regarding MLM output probabilities, avoiding the binary decision process in CPS. This is especially important for our study, where we had limited resources to create the translated dataset.
The  $S_{\mathrm{JSD}}$  measure is based on the Jensen-Shannon divergence (Lin, 1991), a quantity bounded to the range  $[0,1]$ , that measures the similarity between two probability distributions,  $P$  and  $Q$ , defined as follows:
$$
\operatorname {J S D} (P \mid \mid Q) = H \left(\frac {P + Q}{2}\right) - \frac {H (P) + H (Q)}{2} \tag {1}
$$
where  $H$  is entropy. If  $P$  and  $Q$  are unrelated and share no overlap  $\mathrm{JSD}(P||Q) = 1$  and if they are the same distribution (maximum overlap)  $\mathrm{JSD}(P||Q) = 0$ . The square root of the Jensen-Shannon divergence, the Jensen-Shannon distance, is a metric, i.e., it satisfies a range of properties intuitive to measures of distance, including the triangle inequality (Endres and Schindelin, 2003).
Define the gold distribution as a one-hot distribution  $G$  that identifies the correct token under the mask. We then define our measure  $S_{\mathrm{JSD}}$  as the difference of two distances: the Jensen-Shannon distance between  $P_{\mathrm{more}}$  (resp.  $P_{\mathrm{less}}$ ) and the gold distribution:
$$
S _ {\mathrm {J S D}} = \sqrt {\mathrm {J S D} \left(P _ {\text {m o r e}} \| G\right)} - \sqrt {\mathrm {J S D} \left(P _ {\text {l e s s}} \| G\right)} \tag {2}
$$
This definition may also be expressed purely in terms of the model output probability for the token under the mask  $P_{\mathrm{more/less}}(u_G)$ , as  $\mathrm{JSD}(P||G)$  may be expressed in the form shown in Eq. 3 for any distribution  $P$ . Thus only human annotated text is evaluated.
$$
\begin{array}{l} \operatorname {J S D} (P | | G) = \frac {1}{2} (P _ {G} \log_ {2} (P _ {G}) \\ - \left(P _ {G} + 1\right) \log_ {2} \left(P _ {G} + 1\right) + 2), \quad P \left(u _ {G}\right) \equiv P _ {G} \tag {3} \\ \end{array}
$$
The quantity  $S_{\mathrm{JSD}}$  is also bound to the range  $[-1, 1]$ , which limits the effect of outliers. The theoretically ideal non-biased model should yield a value of 0 for  $S_{\mathrm{JSD}}$  when the distance of  $P_{\mathrm{more}}$  to  $G$  is equal to the distance of  $P_{\mathrm{less}}$  to  $G$ . When  $P_{\mathrm{more}}$  is closer to  $G$  than  $P_{\mathrm{less}}$ , we take this as a sign of bias for the stereotypical sentence, thus we expect biased models to systematically generate negative  $S_{\mathrm{JSD}}$  scores.
Model	Lang.	\( S_{JSD} \times 10^{-3} \)	CPS	B.\( S_{JSD} \)
mBERT	En	-0.05±1	57±3	50 ±3
xlmR	En	-1	±2	62±3	54 ±3
mBERT	De	-1	±2	57±3	55 ±3
xlmR	De	-2	±2	51±3	50 ±3
mBERT	Id	-3	±1	46±3	51 ±3
xlmR	Id	-4	±2	51±3	54 ±3
mBERT	Th	-4	±2	60±3	60 ±3
xlmR	Th	-4	±2	57±3	57 ±3
mBERT	Fi	-0.2	±2	44±3	50 ±3
xlmR	Fi	-3	±2	51±3	53 ±3
To generate a score for a sentence pair, we take the average of  $S_{\mathrm{JSD}}$  scores. For the score of the entire dataset, we take the average of the sentence scores.
Error Analysis. For an analysis of the error of the reported score on the dataset, we bootstrap the sentence scores to determine an estimate for the standard error using SciPy (Efron and Tibshirani, 1993; Virtanen et al., 2020). For CPS we achieve this by bootstrapping the binary sentence scores.
For our experiments we make use of the Transformers library (Wolf et al., 2020). We use two multilingual models, multilingual BERT (mBERT) (Devlin et al., 2019), trained on Wikipedia, and base xlmRoBERTa (xlmR) (Conneau et al., 2020), trained on Wikipedia and filtered CommonCrawl data from the internet (Wenzek et al., 2020). We choose xlmR as it has been shown to significantly outperform mBERT on numerous cross-lingual tasks (Conneau et al., 2020). As of this writing, xlmR seems to be the best performing multilingual model in the Transformers library (Wolf et al., 2020; Conneau et al., 2020). The two models differ in training data by the CommonCrawl, which we assume to be more of a source of bias than Wikipedia, based on the results of the CPS study. Nangia et al. (2020) found RoBERTa, trained on Wikipedia and the CommonCrawl, among other datasets (Zhuang et al., 2021), to generally have higher bias scores,
Table 2: CPS and  $S_{\mathrm{JSD}}$  scores and standard errors on our multilingual bias diagnosis dataset. The  $S_{\mathrm{JSD}}$  scores systematically identify the stereotypical sentence as indicated by the negative scores. Some CPS scores are below 50, indicating the measure cannot capture the stereotypical behavior of the model for this dataset. The binarized version of  $S_{\mathrm{JSD}}$  (B.SJDS) also illustrates the effect of binarization. B.SJDS has scores of 50 in three cases where  $S_{\mathrm{JSD}}$  is negative, suggesting that binarization reduces the predictive power of the measure.
Model	Unperturbed	Perturbed
\( S'_{JSD} \)	CPS	\( S'_{JSD} \)	CPS
BERT	-6±1	60.5±1.3	-6±1	58.6±1.3
RoBERTa	-10±1	65.5±1.2	-10±1	63.5±1.2
ALBERT	-13±1	67.0±1.2	-11±1	64.5±1.2
mBERT	-4±1	53.6±1.3	-3±1	55.6±1.3
xlmR	-4±1	57.1±1.3	-4±1	56.6±1.3
Table 3: Scores and standard errors on the original CPS dataset (Nangia et al., 2020), for which BERT (Devlin et al., 2019), RoBERTa (Zhuang et al., 2021) and ALBERT (Lan et al., 2020) were used.  $S_{\mathrm{JSD}}' = S_{\mathrm{JSD}} \times 10^{-3}$ . Unperturbed and perturbed conditions where a sentence is perturbed by removing the final character. For this larger dataset both  $S_{\mathrm{JSD}}$  and CPS show the same systematic trends in bias scores between the models, in agreement with the results of Nangia et al. (2020). Under the effect of the perturbation, the dataset is of sufficient size that both measures are robust and retain their systematic trends. The number of significant figures for CPS was chosen to match the results of the original CPS study.
compared to BERT (Devlin et al., 2019), although this was not true for gender bias.
We run the two models on our translated datasets and calculate CPS and  $S_{\mathrm{JSD}}$  scores. Running a model on a single language using an Intel Xeon Processor E5-2680 v2 takes roughly 15 minutes.
We also test  $S_{\mathrm{JSD}}$  on the models and dataset used in the CPS study (Nangia et al., 2020).
Finally, we test the effect of model size on the scores by comparing the large and base xlm-RoBERTa models. See the appendix for a list of all models used.
Table 2 shows results for CPS and  $S_{\mathrm{JSD}}$  on the multilingual dataset. We observe that the CPS measure reports scores well under 50 for multiple languages. This goes against the intuition that MLMs learn stereotypical associations from data: it wrongly suggests that male stereotypes are associated with women and female stereotypes with men. We suspect this behavior of CPS comes from the binary decision problem outlined in §3.2, which is especially relevant for smaller datasets.
A first indication to suspect that we might be in this regime is that the CPS standard errors are close in value to the estimated standard errors assuming a Bernoulli process, as discussed in §3.2. Thus we cannot make a reliable inference regarding model bias. We can also observe a clustering of CPS sentence scores, before binarization, around
Figure 2: The difference  $S_{\mathrm{more}} - S_{\mathrm{less}}$  for our multilingual bias diagnosis dataset. The white points mark the averages and the box and whiskers plots mark the quartiles. Most of the scores cluster around the decision boundary denoted by the horizontal dotted line.
the decision boundary in Figure 2, indicating that slight variations in bias scores can substantially change the CPS score. Furthermore, the effect of binarizing  $S_{\mathrm{JSD}}$  (i.e., following the CPS method but replacing  $\log P_{\mathrm{more}}(u_{G,i})$  with the JSD distance to the gold token) is shown in Table 2. These binarized  $S_{\mathrm{JSD}}$  scores fail to detect bias by yielding scores of 50 in three cases – whereas the  $S_{\mathrm{JSD}}$  score predicts bias as expected. All this, coupled with the discussion in §3.2, reinforces our argument that binarization harms measure performance and that  $S_{\mathrm{JSD}}$  is numerically more suitable and theoretically justified as a measure compared to CPS, especially on smaller datasets. Note that we did not unbinarize CPS scores as they have no clear statistical interpretation (Nangia et al., 2020; Salazar et al., 2020); see discussion in §3.2.
Table 2 shows that  $S_{\mathrm{JSD}}$  has negative values, i.e., indicates bias consistently across all languages and models. Interestingly, xlmR consistently yields equal or more negative  $S_{\mathrm{JSD}}$  scores than mBERT; this supports our hypothesis that xlmR encapsulates more gender bias. However, we also note that some of the standard errors are large compared to their associated average. This, we suspect, is largely due to the limited number of sentences that are available in the analysis.
To test our measure on a larger dataset, we applied  $S_{\mathrm{JSD}}$  to the original CPS dataset (which is much larger since it covers many different types of bias) in Table 3. We also tested the stability of the measures by perturbing the dataset by removing the final character of each sentence, which most commonly resulted in the removal of a period. From this we observe that both the CPS and the
$S_{\mathrm{JSD}}$  standard errors decrease with dataset size, as expected.
Ideally, future MLM bias datasets would have a statistical justification that the calculated sample error is adequate to make judgments about a model's bias as probed by a particular measure.
Finally, we also tested for model size by comparing the base and large xlmR models (Conneau et al., 2020). While both models were found to be biased for the more stereotypical sentence, the scores between the two models were too similar and no pattern distinguishing them could be found using either bias measure. We suspect this is due to the small size of the dataset. More complex factors could be at play as well, such as multilingual models benefiting from more multilingual data, in terms of gender bias mitigation, a hypothesis supported by the more negative  $S_{\mathrm{JSD}}$  scores for monolingual models in Table 3.
Qualitative Analysis. In Table 2 the English CPS scores reproduce the systematic behavior from the CPS study (Nangia et al., 2020); however, this is not true for all languages, such as German. This may simply be a result of the much larger amount of English training data available to the models (Conneau et al., 2020). The nature of this additional data does not need to be biased in nature. A contributing factor may be that this additional data forces the tokenizer to adjust the tokens in its vocabulary to be whole words for English whereas for other languages tokens are more often subwords, which cannot be exchanged for others without forming grammatically incorrect sentences. To make this point, consider the following sentence pair from the dataset:
(6) a. Women are terrible at handiwork.
b. Men are terrible at handiwork.
Using mBERT, three tokens in the set  $U$  together form the composite word "handiwork" ("hand", "#i", "#work"); the remaining four are separate words and the period. However, it is almost trivial to predict any one of the subword tokens from the composite word, thus differences in MLM prediction probabilities may not be informative for detecting gender bias. In this case CPS assigns a sentence score of 1 and  $S_{\mathrm{JSD}} = .0075$ . The value of  $S_{\mathrm{more}} - S_{\mathrm{less}}$  for CPS is 1.29, placing it close to the decision boundary in Figure 2 and thus making CPS prone to noise.
For the German translation of the sentence, three
tokens in  $U$  are individual words or the period, while the remaining five form composite words. CPS assigns a sentence score of 0 and  $S_{\mathrm{JSD}} = .0135$ . In this case  $S_{\mathrm{more}} - S_{\mathrm{less}}$  for CPS is -.98, once again placing it close to the decision boundary in Figure 2.
Over the whole dataset, for German,  $57\%$  and  $75\%$  of tokens in the more stereotypical sentence were correctly predicted using mBERT and xlmR, respectively, whereas for English the prediction accuracy was lower at  $56\%$  and  $68\%$ , despite having more training data. Thus, compared to German, the CPS measure may be better suited for English, where individual tokens are not as trivial to predict and the CPS measure is not as prone to being influenced by noise from subword tokens.
6 Summary of Limitations
Our results indicate that  $S_{\mathrm{JSD}}$  is superior to the original CrowS-Pairs measure. But like the CrowS-Pairs measure,  $S_{\mathrm{JSD}}$  does not provide reliable measurements consistently. The most noticeable case of this is that for many models, "reverse bias" is well within the confidence interval of the bias measures, i.e., values below 50 are within the confidence interval for the CrowS-Pairs measure and positive values for  $S_{\mathrm{JSD}}$ . We use reverse bias to refer to bias that is the opposite from the stereotype. Examples would include that the model favors women to be doctors and men to cry easily. While we did not confirm this experimentally, it seems not possible that a language model would learn a (spurious) stereotype even though the reverse of the stereotype dominates in the training corpus. Thus, this finding suggests that the measures must be interpreted with caution.
One of our original goals was a cross-lingual quantitative comparison of subtypes of gender bias. For example, maybe the "doctors are men" subtype of gender bias is less prevalent in Sweden than in Germany. Or the subtype "childcare is women's business" is stronger in Russia than in Canada. However, the two measures are not reliable on a sentence-pair by sentence-pair basis, so that one would need hundreds of examples of a subtype to make such inferences. This would require a dataset two orders of magnitude larger than the one we created.
We hypothesize that the main reason for the unreliability of the measures for individual sentence pairs is that predicting subwords is easy and
not strongly linked to the difficulty of predicting a word; see “Qualitative Analysis” in the last section. Since most non-English languages will contain words broken into subwords in a given sentence pair, unrealistically high prediction accuracy and a lack of comparability of scores of a sentence pair across languages are the result.
In this paper, we developed a method for creating a multilingual gender bias diagnosis dataset that can be used across languages. Based on CrowS-Pairs (Nangia et al., 2020), we used this method to construct a multilingual gender bias diagnosis dataset for English, Finnish, German, Indonesian and Thai. Additionally, we proposed a new measure based on the Jensen-Shannon divergence from information theory,  $S_{\mathrm{JSD}}$ , to study bias in MLMs using sentence pairs that contrast two groups. Using this measure we found that all studied models showed signs of gender bias for more stereotypical sentences across all five languages. Our hope is that our methods can be used for better evaluation of bias and debiasing in MLMs. We also hope that our work will foster more multilingual work on bias in language models.
In the future, since most recent bias research focused on PLMs and word embeddings, we plan to develop measures for downstream tasks as recommended by Blodgett et al. (2020) and Delobelle et al. (2021), which may be incorporated in a development pipeline when releasing models (Nozza et al., 2022).
8 Ethical Considerations
The dataset presented in this paper aims to make progress in the evaluation of multilingual gender bias in MLMs, however we argue that it should not be used to train such models. As the presented dataset is intended as a test set, training on it would defeat its purpose as a test of gender bias in MLMs. The presented dataset is based on the CPS dataset, an English crowdsourced dataset aimed at evaluating social biases in the United States (Nangia et al., 2020). For the purpose of this study we made the assumption that the biases in the CPS dataset relating to gender can be extended to the other languages studied and are relevant in cultures where the languages are spoken, however we caution against the blind implementation of such systems without an understanding of the target culture.
This work also focused exclusively on binary gender. The non-trivial nature of representing non-binary people in languages with strong gender agreement rules, such as German, substantially complicates the process of creating natural sentences that could be used for evaluation. For this reason, and because it is an important area with its own challenges, we leave the representation of non-binary people in multilingual settings to future work, where it can be studied with care as a topic in its own right.
Additionally, we caution against concluding that models are completely bias free when they generate scores that theoretically unbiased models are expected to generate. It may be that these models still encode biases that cannot be captured using the proposed measure or dataset, which may later manifest once a model is implemented.
Acknowledgements. We thank Sheng Liang and David Lowell for their thoughtful feedback and suggestions and the translators for their valuable insights into their native languages. This project was funded by a 2020 Award for Inclusion Research from Google Research, the European Research Council (grant #740516) and the German Federal Ministry of Education and Research (BMBF, grant #01IS18036A).
Maria Antoniak and David Mimno. 2021. Bad seeds: Evaluating lexical methods for bias measurement. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1889-1904, Online. Association for Computational Linguistics.
Soumya Barikeri, Anne Lauscher, Ivan Vulic, and Goran Glavaš. 2021. RedditBias: A real-world resource for bias evaluation and debiasing of conversational language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1941-1955, Online. Association for Computational Linguistics.
Marion Bartl, Malvina Nissim, and Albert Gatt. 2020. Unmasking contextual stereotypes: Measuring and mitigating BERT's gender bias. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 1-16, Barcelona, Spain (Online). Association for Computational Linguistics.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454-5476, Online. Association for Computational Linguistics.
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1004-1015, Online. Association for Computational Linguistics.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29:4349-4357.
Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
António Camara, Nina Taneja, Tamjeed Azad, Emily Allaway, and Richard Zemel. 2022. Mapping the multilingual margins: Intersectional biases of sentiment analysis systems in english, spanish, and arabic. ArXiv.
Pieter Delobelle, Ewoenam Kwaku Tokpo, Toon Calders, and Bettina Berendt. 2021. Measuring fairness with biased rulers: A survey on quantifying biases in pretrained language models. ArXiv.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Bradley Efron and Robert Tibshirani. 1993. An introduction to the bootstrap. Number 57 in Monographs on statistics and applied probability. Chapman & Hall.