SlowGuess/NAACL_2022_OCR huggingface dataset:
Masahiro Kaneko<sup>1</sup> Aizhan Imankulova<sup>2</sup> Danushka Bollegala<sup>3,4*</sup> Naoaki Okazaki<sup>1</sup>
$^{1}$ Tokyo Institute of Technology  $^{2}$ CogSmart Co., Ltd.
<sup>3</sup>University of Liverpool <sup>4</sup>Amazon
masahiro.kaneko@nlp.c.titech.ac.jp
aizhan.imankulova@cogsmart-global.com
danushka@liverpool.ac.uk okazaki@c.titech.ac.jp
Masked Language Models (MLMs) pre-trained by predicting masked tokens on large corpora have been used successfully in natural language processing tasks for a variety of languages. Unfortunately, it was reported that MLMs also learn discriminative biases regarding attributes such as gender and race. Because most studies have focused on MLMs in English, the bias of MLMs in other languages has rarely been investigated. Manual annotation of evaluation data for languages other than English has been challenging due to the cost and difficulty in recruiting annotators. Moreover, the existing bias evaluation methods require the stereotypical sentence pairs consisting of the same context with attribute words (e.g. He/She is a nurse). We propose Multilingual Bias Evaluation (MBE) score, to evaluate bias in various languages using only English attribute word lists and parallel corpora between the target language and English without requiring manually annotated data. We evaluated MLMs in eight languages using the MBE and confirmed that gender-related biases are encoded in MLMs for all those languages. We manually created datasets for gender bias in Japanese and Russian to evaluate the validity of the MBE. The results show that the bias scores reported by the MBE significantly correlates with that computed from the above manually created datasets and the existing English datasets for gender bias.
Masked Language Models (MLMs) (Devlin et al., 2019), which are pre-trained on large corpora, have been used successfully in natural language processing tasks for various languages (Conneau and Lample, 2019; Martin et al., 2020; Conneau et al., 2020). Unfortunately, it has been reported that MLMs also
learn social biases regarding attributes such as gender, religion, and race (Kurita et al., 2019; Dev et al., 2020; Kaneko and Bollegala, 2021a; Bender et al., 2021). The bias in MLMs is evaluated by the imbalance of the likelihood between pairs of sentences associated with an attribute that has a common context (e.g. He/She is a nurse). Nadeem et al. (2021) masked the modified tokens (e.g. He, She), and Nangia et al. (2020) masked the unmodified tokens (e.g. is, a, nurse) one word at a time and calculated the likelihood from their predictions to evaluate the bias. Kaneko and Bollegala (2021c) evaluated the bias using the average of the likelihoods of all tokens without masking the MLM.
Despite the numerous studies of social bias in MLMs covering English, social biases in MLMs for other languages remain understudied (Lewis and Lupyan, 2020; Liang et al., 2020; Bartl et al., 2020; Zhao et al., 2020). To realise the diverse and inclusive social and cultural impact of AI, we believe it is important to establish tools for detecting and mitigating unfair social biases in MLMs, not only for English but for all languages. However, the significant manual annotation effort, the costs and difficulties in recruiting qualified annotators remain major challenges when creating bias evaluation benchmarks for target languages. For example, existing bias evaluation benchmarks such as CrowS-Pairs (CP; Nangia et al., 2020) and StereoseSet (SS; Nadeem et al., 2021) require human-written sentences (or pairs of sentences) eliciting different types of social biases expressed in the target language. However, scaling up this approach to all languages is challenging because recruiting a sufficiently large pool of annotators to cover the different types of social biases in those languages is difficult. Because of the above-mentioned challenges, bias evaluation datasets and studies outside English remain under-developed.
To address this problem, we propose Multilin-
*Danushka Bollegala holds concurrent appointments as a Professor at University of Liverpool and as an Amazon Scholar. This paper describes work performed at the University of Liverpool and is not associated with Amazon.
Human Language Technologies, pages 2740 - 2750
Figure 1: The bias evaluation method using a parallel corpus between English and the target language and an English female and male words list. Matrix values in Step 2 are the similarities between male and female sentences.
gual Bias Evaluation (MBE) score $^{1}$ , a social bias evaluation method that can be used to evaluate biases in pre-trained MLMs for a target language without requiring significant manual annotations for that language. MBE can perform equivalent bias evaluation using only existing parallel corpora and lists of English masculine (e.g. he, his, father, son etc.) and feminine (e.g. she, her, mother, daughter, etc.) words, without requiring any manually annotated sentences for social biases in the target language. Although MBE require parallel corpora, such sources already exist for numerous language pairs $^{2}$  or can be automatically mined with less effort compared to annotating bias evaluation data (Artetxe and Schwenk, 2019b,a). As a concrete example, we evaluate the proposed method for gender bias, which exists in many languages. Extending the proposed method to other types of social biases beyond gender biases is deferred to future work. As shown in Figure 1, MBE first (shown in Step 1) extracts target language sentences containing female words (female sentences) and sentences containing male words (male sentences) from a parallel corpus between English and the target language using a list of gender words in English. Second, (shown in Step 2) MBE calculates the likelihoods for each of the extracted female and male sentences in the target language using the given MLM under evaluation. Finally, (shown in Step 3) MBE compares the likelihoods between each female sentence and all male sentences considering all pairwise combinations, and increment a count by 1 if the likelihood of the male sentence is greater than that of the female sentence, and 0 otherwise.
As for Step 1, we do not require any knowledge
about the target language or manual annotations because we use only the existing English attribute word lists and parallel corpora between English and the target language. This is attractive from a data availability point of view, which makes our proposed method easily extendable to different languages. Kaneko and Bollegala (2021c) found that the frequency of the words associated with the male gender to be significantly higher than that for the female gender in the data used to train MLMs. They showed that these frequency-related biases are encoded into MLMs, and independently of whether a sentence contains a stereotypical or antistereotypical context, an MLM that is biased towards the male gender, on average, assigns higher likelihood scores to sentences that contain masculine words than feminine words<sup>3</sup>. Inspired by this finding, in Step 2, we calculate the likelihood scores assigned by an MLM under evaluation to sentences that contain male and female related words in different contexts (e.g. He is a baseball player and She is a nurse). Step 3 of our proposed method performs the computation of the bias score considering the similarity between the contexts in sentence pairs that contain male and female related words. The more similar the sentence pairs are, the more similar the estimates would be compared to the bias evaluation measures that require identical contexts. Therefore, we weight the bias score estimates by the similarity of the sentence pairs using the sentence representations obtained from the MLM under evaluation. We ignore dissimilar sentence pairs and compute the bias score from the similar sen
Nangia et al. (2020) define stereotypical sentence to be a case where an advantaged group (in the case of gender bias male is considered as the advantaged group, whereas female is the disadvantaged group) is associated with a pleasant attribute (e.g. The man is intelligent) or a disadvantaged group is associated with an unpleasant attribute (e.g. The woman is careless).
$^{1}$ Our code and dataset: https://github.com/kanekomasahiro/bias_eval_in_multiple_mlm  $^{2}$ https://www.clarin.eu/resource-families/parallel-corpora
tence pairs, which is defined as the percentage of sentence pairs where the sentence containing the masculine words is assigned a higher likelihood than the sentence containing the feminine words.
Bias in MLM is thought to depend on both the MLM and the evaluation data, so in this paper, we are investigating both using two corpora for English MLMs. Using the proposed method, we evaluated gender bias in MLMs in eight other languages: German, Japanese, Arabic, Spanish, Portuguese, Russian, Indonesian, and Chinese. Prior work investigating social biases in MLMs for English have shown that different types and levels of biases are shown by different MLMs even for the same language (Kaneko and Bollegala, 2021a,c; Dev et al., 2020). We defer covering different MLMs across multiple languages to future work and focus on establishing MBE as an evaluation measure that can be used for such a study.
Our evaluations show that all MLMs learn gender-related biases in all languages studied. To further validate MBE, we conduct a meta-evaluation where we use an existing manually annotated English bias evaluation dataset and two additional datasets we annotate in this work covering gender-related biases in Japanese and Russian languages. The bias scores computed using MBE show significantly high correlations with human bias annotations on both datasets (CP and SS), showing its validity for multiple languages as a gender bias evaluation method. Furthermore, we show that MBE is superior to methods using machine translations to evaluate bias in non-English languages. We also show that bias evaluation methods based on templates and word lists significantly overestimate the bias in MLMs due to the unnaturalness of the created templates. Our analyses on the effects of English names on gender information and preservation of gender information in parallel corpora suggest that bias can be evaluated reasonably even with some loss of gender information.
In the study of bias in English MLMs, May et al. (2019) and Kurita et al. (2019) use a pair of artificial sentences created using manually written templates. However, template-based evaluation is problematic because it uses an artificial context that does not reflect the natural usage and distribution of words in the target language. To solve this problem, Nadeem et al. (2021) and Nangia et al.
(2020) manually created bias evaluation datasets, SS and CP, respectively, with stereotypical and antistereotypical sentence-pairs with identical contexts, except the attribute words. However, recent work has pointed out various issues in CP and SS datasets and has argued that they may not provide effective measurements of stereotyping (Blodgett et al., 2021). In this study, (social) bias is defined as the tendency towards outputting sentences about a particular advantageous or disadvantageous group, such as males or females, given the same context by an MLM. However, these benchmarks are currently the most commonly used benchmarks for bias evaluation in MLMs, so we also use them in this work. We note that MBE is independent of any bias evaluation benchmark datasets. Our focus in this paper is on evaluating gender bias in multiple languages and not on comparing or proposing novel debiasing methods. However, for the completion of the discussion, we note that methods for debiasing MLMs using sentence vectors from MLMs (Bommasani et al., 2020) and lists of English male and female words has been studied (Sedoc and Ungar, 2019; Kaneko and Bollegala, 2021a; Dev et al., 2020; Zhou et al., 2022).
In prior work on MLMs, social biases for languages other than English have rarely been investigated. Ahn and Oh (2021) investigated ethnic bias in monolingual MLM in six languages by extending the templates to other languages using machine translation. The biases of MLMs have been evaluated using templates for English and Chinese (Liang et al., 2020) and for English and German (Bartl et al., 2020). Zhao et al. (2020) investigated the gender bias of a classifier that predicts the occupation from resumes using multilingual word embeddings and multilingual MLM embedding in Spanish, German and French. They evaluated bias by using machine translation on the English data, when an MLM is used to create feature representations in a specific task. However, this setting is different from that of our study, where we evaluate the bias of MLMs independently of a specific task. Moreover, the above studies do not discuss or propose methods on how to create evaluation data that can be applied to many languages.
Following the pioneering work by Bolukbasi et al. (2016) that proposed a bias evaluation and debiasing methods, various studies have investigated social biases in English (Caliskan et al., 2017; Zhao et al., 2018; Kaneko and Bollegala, 2019, 2021b;
Dev and Phillips, 2019). Unlike the contextual word embeddings produced by MLMs, evaluating social biases in static word embeddings is relatively less complicated because it can often be done using word lists without requiring annotated sentences. In static word embeddings, bias has been investigated in various languages besides English due to this ease of annotating evaluation data. Lauscher and Glavaš (2019) translated the English word lists into six languages and evaluated the bias of the word embeddings. Zhou et al. (2019) proposed an evaluation metric for languages that require gender morphological agreement, such as in Spanish and French. Friedman et al. (2019) quantified the gender bias of word embeddings to understand cultural contexts with large-scale data, and used it to characterize the statistical gender gap in education, politics, economics, and health in US states and several countries. Bansal et al. (2021) proposed a debiasing method by constructing the same bias space for multiple languages, and adapted it to three Indian languages. Other bias studies have been conducted for specific languages (Takeshita et al., 2020; Pujari et al., 2019; Sahlgren and Olsson, 2019; Chávez Mulsa and Spanakis, 2020), but they are not easily transferable to novel languages.
3 Bias Evaluation for Multiple Languages
Our proposed MBE score evaluates the gender bias of the target language under evaluation in three steps (see Figure 1). In Step 1, we first define the set of English sentences  $\mathcal{E}$  and the set of target language sentences  $\mathcal{T}$  of the parallel corpus, where  $N$  is the data size, and  $(e_i,t_i)$  is a parallel sentence pair. Let  $\mathcal{V}_f$  (e.g. she, woman, female) be the list of female words and  $\mathcal{V}_m$  (e.g. he, man, male) be the list of male words in English. We then extract sentences that contain a female or a male word from  $\mathcal{E}$ . Sentences that contain both male and female words are excluded. Let us denote the set of sentences extracted for a female or a male word  $w$  by  $\Phi(w)$ . Let  $\mathcal{E}_f = \bigcup_{w\in \mathcal{V}_f}\Phi(w)$  and  $\mathcal{E}_m = \bigcup_{w\in \mathcal{V}_m}\Phi(w)$  be the sets of sentences containing respectively all of the male and female words. The set of sentences in the target language of the source sentences included in  $\mathcal{E}_f$  and  $\mathcal{E}_m$  is denoted by  $\mathcal{T}_f$  and  $\mathcal{T}_m$ , respectively. It is assumed that gender information is retained in the parallel corpus, and whether this is actually the case is verified later.
In Step 2, we compute the likelihood for the full sentences in  $\mathcal{T}_f$  and  $\mathcal{T}_m$ . Let us consider a
target sentence  $T = w_{1},w_{2},\ldots ,w_{|T|}$ , containing length  $|T|$  sequence of tokens  $w_{i}$ . We calculate the likelihood with All Unmasked Likelihood with Attention weights (AULA; Kaneko and Bollegala, 2021c) which evaluates the bias by considering the weight of MLM attention as the importance of tokens. Given an MLM with pre-trained parameters  $\theta$ , which we must evaluate it for its gender bias, let us denote the probability  $P_{\mathrm{MLM}}(w_i|T;\theta)$  assigned by the MLM to a token  $w_{i}$  conditioned on all the tokens of  $T$ . AULA predicts all of the tokens in  $T$  using the attention weights to evaluate social biases considering the relative importance of words in a sentence, which is given by (1).
$$
A (T) := \frac {1}{| T |} \sum_ {i = 1} ^ {| T |} \alpha_ {i} \log P _ {\mathrm {M L M}} \left(w _ {i} \mid T; \theta\right) \tag {1}
$$
Here,  $\alpha_{i}$  is the average of all multi-head attentions associated with  $w_{i}$ .
In Step 3, by comparing the likelihoods of female and male sentences returned by AULA, we calculate the bias score as the weighted average of the similarities of contexts using the sentence representations produced by the MLM under evaluation. Specifically, We use the percentage of male  $(T_{m})$  sentences (e.g. He is a baseball player) preferred by the MLM over female  $(T_{f})$  ones (e.g. She is a nurse) to define the corresponding multilingual bias evaluation measure (MBE bias score) as follows:
$$
1 0 0 \times \frac {\sum_ {T _ {m} \in \mathcal {T} _ {m}} \sum_ {T _ {f} \in \mathcal {T} _ {f}} C \left(T _ {m} , T _ {f}\right) \mathbb {I} \left(A \left(T _ {m}\right) > A \left(T _ {f}\right)\right)}{\sum_ {T _ {m} \in \mathcal {T} _ {m}} \sum_ {T _ {f} \in \mathcal {T} _ {f}} C \left(T _ {m} , T _ {f}\right)} \tag {2}
$$
Here,  $\mathbb{I}$  is the indicator function, which returns 1 if its argument is True and 0 otherwise.  $C(T_m, T_f)$  uses the average of the last layer in MLM for all tokens except special tokens to compute the sentence embeddings of  $T_m$  and  $T_f$  respectively and computes the cosine similarity of these embeddings. According to this evaluation measure, values close to 50 indicate that the MLM under evaluation is neither females nor males biased, hence, it can be regarded as unbiased. On the other hand, values below 50 indicate a bias towards the male group and above 50 towards the female group. We report a statistically significant difference comparing to the model with randomly assigned results of the indicator function  $\mathbb{I}$  in Equation 2 with the McNemar's test  $(p < 0.05)$ . For each sentence, the presence or absence of bias is predicted by two methods, MLM
Lang	TED	News
German	4.7K	2.1K
Japanese	6.2K	1.8K
Arabic	7.0K	1.7K
Spanish	7.1K	17.3K
Portuguese	5.7K	2.2K
Russian	6.7K	3.9K
Indonesian	2.9K	0.5K
Chinese	6.8K	3.4K
and Random. The McNemar's test was used by classifying into four categories: only MLM was biased, only random was biased, both were unbiased, and both were biased. We use the statistically significant difference to determine if there is a bias.
4 Gender Bias in Masked Language Models
We use two parallel corpora, the TED2020 v1 corpus in the spoken language domain (TED) and the GlobalVoices corpus in the news domain (News). Table 1 shows the total number of extracted male and female sentences for each language. Except for Spanish, the News corpus is smaller than the TED corpus for all languages. In particular, the Indonesian news corpus is an extremely low resource. For the list of female and male words in English, we use the list created by Bolukbasi et al. (2016) in addition to the female and male names in CP (Nangia et al., 2020). The extracted male and female sentences were downsampled to create sets of an equal number of sentences. We experimented on the GeForce RTX 2080 Ti using the transformers implementation with default settings (Wolf et al., 2020). All evaluations are completed within 10 minutes.
We used Masked Language Models (MLMs) in eight languages for our experiments: Japanese $^{8}$ , German $^{9}$  (Chan et al., 2020), Arabic $^{10}$  ( Antoun
Table 1: The total number of male and female sentences extracted from the parallel data for each language.
Lang	MBE(TED)	MBE(News)
German	54.69‡	55.12‡
Japanese	54.52‡	50.99
Arabic	55.72‡	54.39‡
Spanish	51.44‡	51.69‡
Portuguese	53.07‡	54.99‡
Russian	54.59‡	51.00
Indonesian	52.38‡	50.52
Chinese	52.86‡	51.80‡
Table 2: The bias score of MLMs using MBE in different languages.  $\ddagger$  indicates statistically significant difference at  $p < 0.05$ .
et al., 2020), Spanish $^{11}$  (Canete et al., 2020), Portuguese $^{12}$  (Souza et al., 2020), Russian $^{13}$ , Indonesian $^{14}$  and Chinese $^{15}$  (Cui et al., 2020).
Table 2 shows the bias scores of the proposed MBE method for the TED and News corpora for the MLMs considered. Here, the significant difference is evaluated against the MBE score of a randomly assigned indicator function. Overall, we see gender-related biases are reported in all cases. In particular, significant biases are shown in the News corpus for all languages except Japanese, Russian and Indonesian. Moreover, the different levels of biases reported for Russian and Japanese between TED and News corpora indicate that bias evaluations are affected not only by the MLMs but also the corpora used. It is known that the bias tendency of MLMs changes depending on the training data (Babaeianjelodar et al., 2020), and similarly, the bias evaluation of MLMs is affected by the evaluation corpus. Because MBE can evaluate bias in various domains as long as there are parallel corpora. It can also capture corpus-dependent biases, unlike existing methods requiring manually created domain-specific sentence pairs.
5 Meta-Evaluation
We perform a meta-evaluation to validate MBE scores against human bias ratings. In §5.1 we measure the correlation between MBE scores and existing measures on CP and SS, which are manu
Shf	MBE
CP	Spearman	0.06	0.41
Pearson	0.05	0.63†
Direction	0.54	0.72
Diff	4.06	2.36
SS	Spearman	0.21	0.41
Pearson	0.04	0.62†
Direction	0.54	0.72
Diff	6.66	5.04
Table 3: Bias scores computed using Shf and the MBE methods for English MLMs in CP and SS. Correlation between the original and proposed evaluation represented by Spearman and Pearson correlation coefficients.  $\dagger$  indicates significant correlation at  $p < 0.05$ . Direction is the percentage of agreement for direction of the bias score between original and proposed evaluations. Diff is the mean of the difference between the bias scores of the original and proposed methods.
ally annotated bias evaluation benchmarks for English. In §5.2 to compare the evaluation methods in the target languages using MBE and manually annotated data, we manually translate the CP into the Japanese and Russian, which demonstrate high corpus-specific biases according to Table 2.
5.1 Gender Bias Evaluation Using Manually Annotated Data in English
To validate MBE scores using human bias ratings, we use CP and SS datasets for English. As baseline method we use Shf, which shuffles the sets of male and female sentences and randomly pair sentences from this set. Shf is used to show the usefulness of comparing the likelihoods of male and female sets. In the existing evaluation method using manually annotated sentence pairs, the bias score is calculated for stereotypical  $S_{s}$  (e.g. He is a doctor) and anti-stereotypical  $S_{a}$  (e.g. She is a doctor) sentences with identical contexts as follows:
$$
\frac {1 0 0}{N} \sum_ {S _ {s}, S _ {a}} \mathbb {I} \left(A \left(S _ {s}\right) > A \left(S _ {a}\right)\right) \tag {3}
$$
where  $N$  is the total number of sentences. We use this bias score as an upper bound score to compare against it the results for Shf and MBE using the rank correlations (Spearman and Pearson), the agreement of the direction of bias between female and male directions (Direction), where the bias scores above 50 indicate a bias towards the male direction and that below 50 towards the female direction, and the difference of the bias scores (Diff) from the results of the method using manual annotation. In the proposed method and Shf, for the
gender bias data of CP and SS, we extract sentences containing male and female words for each sentence, instead of sentence pairs, and use them for evaluation using Equation 3.