Cross-lingual Transfer Can Worsen Bias in Sentiment Analysis
Authors: Seraphina Goldfarb-Tarrant
Björn Ross
Adam Lopez
Venue: EMNLP 2023 Main (2023)
Keywords: bias
fairness
multilingual
sentiment analysis
counterfactual
contrastive
TL;DR: We do the first test for whether cross-lingual transfer can increase/decrease gender and racial biases in sentiment analysis, as compared to monolingual systems. We find that it can.

Abstract: Sentiment analysis (SA) systems are widely deployed in many of the world's languages, and there is well-documented evidence of demographic bias in these systems. In languages beyond English, scarcer training data is often supplemented with transfer learning using pre-trained models, including multilingual models trained on other languages. In some cases, even supervision data comes from other languages. Does cross-lingual transfer also import new biases? To answer this question, we use counterfactual evaluation to test whether gender or racial biases are imported when using cross-lingual transfer, compared to a monolingual transfer setting. Across five languages, we find that systems using cross-lingual transfer usually become more biased than their monolingual counterparts. We also find racial biases to be much more prevalent than gender biases. To spur further research on this topic, we release the sentiment models we used for this study, and the intermediate checkpoints throughout training, yielding 1,525 distinct models; we also release our evaluation code.

Cross-lingual Transfer Can Worsen Bias in Sentiment Analysis
Seraphina Colrfarb-Tarrrnt
Björn Ross
Adam Lopez
School of Informatics University of Edinburgh s t.rrrantteed... ac uk Cohere
School of Informaties University of Edinburgh b rosssedd........... ac. uk
School of Informatics University Edinburgh
alopezêinf. ed. ac uk
seraphina@cohere
Abstract
Monolingual
Cross-lingual
Sentiment analysis (SA) systems widely de- ployed many of the world's languages, and there well-documented evidence of demo- graphic bias in these systems. In languages beyond English, scarcer training data is often supplemented with transfer learning using pre- trained models. including multilingual mod- els trained on other languages. In some cases, even supervision data comes from other lan- guages. Does cross-lingual transfer also import new biases? To answer this question, we use counterfactual evaluation to test whether gen- der or racial biases are imported when using cross-lingual transfer, compared gual transfer setting, Across five languages, we find that systems using cross- ponsu-l transfer usually become more biased than their monolin- gual counterparts. We also find racial biases to much more prevalent than gender biases. To spur further research on this topic, release the sentiment models we used for this study, and the intermediate checkpoints throughout training, yielding 1,525 distinct models; we also release our evaluation code.
Corpus (japanese)
Corpus (multiingual)
pretrained model
pretrained model
Sentiment data (japanese)
Sentiment data (english)
sentiment model R
sentiment model R
monolin-
Sa
The conversation with that person is annoying.
Sp # A & 0) - t カ ft
The conversation with that Korean person is annoying.
Bias
R(sa) R(S,)
Figure 1: We use counterfactual evaluation to evaluate how bias is differs in monolingual cross-lingual systems. Counterfactual pairs (e.g. sentences b) vary a single demographic variable (c.g. race). We measure bias the difference in scores for the pair. An unbiased model should be invariant to the counterfactual, with difference of zero.
1 Introduction
Sentiment analysis (SA) has many practical appli- cations, leading widespread interest in using for many languages. SA naturally framed as su- pervised learning problem, but substantial amounts of supervised training data exist in only a handful of languages. Since creating supervised training data in new language is costly, two transfer learning strategies commonly used to reduce its cost, or even to avoid it altogether. The first, which reduces cost, is monolingual transfer: we pre-train an un- supervised model on large corpus in the target language, fine-tune on a small amount of supervi- sion data in that language, and apply the model in that language (Gururangan et al., 2020). The
second, which avoids annotation cost altogether, is zero-shot cross-lingual transfer: pre-train an unsupervised model on large corpus in many languages, fine-tune on already available supervi- sion data high-resource language, and use the model directly in the target language (Eisenschlos et al., 2019; Ranasinghe and Zampieri, 2020).
While transfer learning strategies be used to avoid annotation costs, we hypothesised that they may incur other costs in the form of bias. It is well-known that high-resource SA models exhibit gender and racial biases (Kiritchenko and Moham- mad, 2018: Thelwall, 2018; Sweeney and Najafian,
'https: / thub. www..com/ee wtw...om/eeraphnnatararrto Inttrrrantt multil i nngual sent ment, meent_analyeee analysis

---

2020). Less is known about bias in other languages. A recent study found that SA models trained with monolingual transfer were less biased than those trained without any transfer learning (Goldfarb- Tarrant et al., 2023). As far as we are aware, there is no work that studies the effect of cross-lingual transfer on bias.
guages cover three different language families, that all have fewer sentiment analysis resources then English; including English in the study enables us to compare to previous work. We find that:
Zero-shot multilingual transfer generally in- creases bias compared to monolingual models. Racial bias in particular changes dramatically.
But there is good reason to hypothesise that cross-lingual transfer may introduce new biases. Specific cultural meanings, multiple word senses, and dialect differences often contribute to errors in multilingual SA systems (Mohammad et al., 2016; Troiano et al., 2020), and are also sources of bias (Sap et al., 2019). For example, the English word foreigner translates to the Japanese word gaifin (3)- λ) which has approximately the same meaning, but more negative sentiment. Bias may also arise from differences in what is explicitly expressed. For example, there evidence that syntactic gen- der agreement increases gender information in rep- resentations (Gonen et al., 2019a; McCurdy and Serbetci, 2017), and there is also evidence that gender information in representations correlates with gender bias (Orgad et al., 2022). From these facts, we hypothesise that multilingual pre-training on languages with gender agreement will produce more gender bias in target languages without gen- der agreement, while producing less bias in target languages with gender agreement.
2. The increase in bias in cross-lingual transfer is largely, but not entirely attributable to the mul- tilingual pre-training data, rather than cross- lingual supervision data.
3.
As hypothesised, gender bias is influenced by multilingual pre-training in directions that are predictable by the presence or absence of syntactic gender agreement in the target language.
4. Compressing models via distillation often re- duces bias, but not always.
We conclude with set of recommendations test for bias in zero-shot cross-lingual transfer learn- ing, to create more resources to allow testing, and to expand bias research outside of English. We re- lease all models and code used for our experiments, to facilitate further research.
Background
2.1 Cross-lingual Transfer
In this paper, we conduct the first investigation of biases imported by cross-lingual transfer, an- swering the following research questions: (RQ1) What biases are imported via cross-lingual transfer, compared to those found in monolingual transfer? When using cross- -inguul transfer, are observed bi- ases explained by the pre-training data, or by the cross-lingual supervision data? Since practical sys- tems often use distilled models, we also ask: (RQ2) Do distilled transfer models show the same trends standard ones?
The aim of transfer learning is to leverage a plen- tiul resource bootstrap learning for a task with few resources. Cross-lingual transfer learning (Ruder et al., 2019; Pires et al., 2019; Wu and Dredze, 2019) extends this idea to transferring across languages. It works by pre-training a model on text in many languages, including both the tar- get language and one more additional languages with substantial resources in the target task. For example, we pre-train a model on a multilingual web crawl containing both English and Japanese, and fine-tune on many English reviews (plentiful resource). We then assume that since the model knows about both Japanese and polarity detection, it can be applied to the task even though it has never seen examples of polarity detection in Japanese. We call this zero-shot cross- lingull transfer (ZS- XLT). An alternative approach is few-shot transfer, where we also use a very small amount of target- language supervision. We focus on zero-shot trans- fer because it makes clear any causal link between multilingual training and bias transfer.
We investigate these questions via counterfac- tual evaluation, in which test examples are edited to change a single variable of interest-such as the race of the subject- so that any change in model behaviour can be attributed to that edit. We use the counterfactual evaluation benchmarks of Kir- itchenko and Mohammad (2018) and an extension et al., 2023) to test for gen- der, racial, and immigrant bias in five languages: Japanese (ja), simplified Chinese (zh), Spanish (es), German (de), and English (en). The first four lan-
of it

---

2.2 Counterfactual Evaluation
3.1 Evaluation Benchmarks
Counterfactual evaluation is an approach that al- lows us to establish causal attribution: single input variable is modified a time, so that one can be sure that any changes in the output are due to that change (Pearl, 2009).
To evaluate social bias in our experiments, we use multiple different counterfactual benchmarks. Ta- ble I contains examples from all datasets. For English, we use the counterfactual corpus of Kir- itchenko and Mohammad (2018), which covers binary gender bias, and racial bias. Gender is rep- resented by common gender terms (he, she, sister, brother), and African American race is represented by African-Americat first names contrasted with European American ones, derived from Caliskan et al. (2017). For non-Engllish language bench- marks, we use the corpus of poldfarb-Tarran et al. (2023) which follows the methodology of Kir- itchenko and Mohammad (2018) 88 to create the same kind of benchmark in German, Spanish, Japanese, and Chinese, extended to respect linguistic and cul- tural specifics of those languages. In the Goldfarb- Tarrant et al. (2023) benchmark, all languages have test for gender bias, where gender is binary and is similarly represented by common gender terms (as above). The German resource covers anti- immigrant bias, using identity terms of race and nationality identified by governmental and NGO resources as immigrant categories that are targets of hate (Muigai, 2010; FADA) e.g. Turk, Arab, Muslim, Roma, Sinti. The Japanese resource covers bias against racial minorities, using identity terms of minoritised groups from sociology resources (Buckley, 2006; Weiner, 2009), egg Chinese, Ko- rean, Okinawan. The Spanish resource tests anti- immigrant bias via name proxies of immigrant first names, taken from Goldfarb-Tamme based on the social science research of Salamanca and Pereira (2013). The benchmark provides only gender bias tests for Chinese, so this work includes an analysis of gender bias only for Chinese. For reference, we have included the full set of racial and nationality groups covered in the benchmark in Appendix C.
Benchmarks for evaluating model faimess with this strategy are constructed so that model predic- tions should be invariant to changes in a demo- graphic or protected variable such as race or gender (Kusner et al., 2017).² For example, the sentiment scores of The conversation with that boy was ir- ritating and The conversation with that girl was irritating should be equal. If there is systematic difference in predicted sentiment scores between such pairs of sentences, we conclude that our model is biased. Biased models for sentiment analysis are likely to propagate representational harm (Craw- ford, 2017) by systematically associating minori- tised groups with more negative sentiment. They also propagate allocational harm by being less stable sentiment prediction in the presence of certain demographic information. Sentiment analy- sis often a component of another application, so the specific harm depends on the application.
3 Methodology
We treat sentiment polarity detection as a five-way classification problem: very negative (1), negative (2), neutral (3). positive (4), or very positive (5). In figures, we refer to these classes by using symbols and ++. This ordinal labeling scheme is
et al. (2021)
-, 0. commonly used when systems are trained user reviews with star rating (Poria al., 2020).
We train monolingual and cross-lingual mod- els, then evaluate them on counterfactual corpora and compare their differences in bias measures, We look at both average bias using aggregate met- rics and granular bias using a contingency table of counterfactuals. This enables us to build an overall picture of model comparability and also to differ- entiate between models with identical aggregate bias but different behaviour some models may make many small errors, and some may make few large errors, and this may matter for minimising real world harms.
In all datasets, counterfactual pairs are generated from template sentences (Table 1) that vary both the counterfactual and the sentiment polarity, by using placeholders for demographic words and emotion words, respectively. Demographic words are as de- scribed above, for emotion words, Kiritchenko and Mohammad (2018) use 40 English emotion words that fit into high level categories of fear, anger, joy, and sadness (this granularity allows testing more granular sentiment and emotion rather than simply polarity, if desired). Goldfarb- Colraanb-Tarrant et al. (2023)
"There are tasks where invariance to demographics docsn't make sense, such as hate specch classification. Our evaluation data are designed so that all examples should he invariant.

---

use emotion words from the same high level cate- gories and about 10 emotions per category as well, though sometimes this is many more than 10 ac- tual words to account for grammar in non-English languages (gender, case, etc). Datasets range from 3-5k pairs per language, which gives sufficient sta- tistical power for the differences we observe. We nonetheless include confidence intervals in all our analysis.
For more granular analysis we examine contin- geney tables of privileged vs. minoritised scores for each example. This enables us to distinguish be- tween many minor changes in sentiment or fewer large changes, which are otherwise obscured by aggregate metrics as described above."
Experimental Setup
Our goal is to simulate practical conditions as much as is possible with available resources and datasets, so we start with pre-trained models from hugging- face (Wolf et al., 2020) which are commonly used in sentiment benchmarks and previous work on our data. 4 We then fine-tune these models on su- pervised training for the polarity detection task and apply to the counterfactual evaluation set in the target language. Both monolingual and mul- tilingual models have as similar numbers of pa- rameters and fine-tuning procedures as is possible, to minimise confounds while being realistic (Ap- pendix A). Models fine fine-tuned until convergence using early stopping on the development set. All models (multilingual and monolingual) converge to equivalent performance as previous work (Keung et al., 2020), which is state of the art on this task. F1 scores and steps to convergence are included in Appendix B.
There exists an additional benchmark of the same construction covering Arabic also Câmara et al. (2022). was not yet available at the begin- ning of this work (and there is no equivalent Arabic sentiment data for us to use) so we did not use it in this work, but it may be helpful for future research to include an additional language family.
3.2 Metrics
We need an aggregate measure of overall bias and a way to look results in more detail. For our aggregate metric, we measure the difference in sen- timent score between each pair of counterfactual sentences, and then analyse the mean and variance over all pairs. Formally, each corpus consists of n sentences, S = {st...8n]. and a demographic variable A = (a,b) where is the privileged class (male or privileged unmarked race) and is the minoritised class (female or racial minority). The sentiment classifier produces score R for each sentence, and our aggregate measure of bias is:
Monolingual transfer (mono-T) models are -asse-u
based on pre-trained (Devlin et al., 2018) in the target language. We randomly initialise linear classification layer, then simultaneously train it and fine-tune the language model on monolingual supervision data. Our distilled monolingual model (distil- mono-T) is identical, except that it is based on sesttiibbet-base-ee 2019).
n
R(s: 1 a= R(s; A= b)
1=0
In this formulation, values greater than zero indi- cate bias against the minoritised group, values less than zero indicate bias against the privileged group, and zero indicates no bias. Scores are discrete integers ranging from 1 to 5, the range of pos- sible values is -4 to 4. For example, if a sentence received a score of with the male demographic term, and a score of 1 with the female demographic term, then the score gap for that example is
(Sanh et al.,
Multilingual models are based on pre-trained mbert cheert-baseuuccasee then fine-tuned on large volume of sentiment data in English only, the standard approach to zero-shot pross-lingual transfer (ZS-XLT). We also fine-tune a distilled ZSS- XLT model (distil-ZS-XLT), identical except that it is based on dist i Imbert base uncased.
To put our results in context, Kiritchenko and Mohammad (2018) found the average bias of a sys- tem to be < 3% of the output score range, which corresponds to gap of 0.12 our scale. In prac- tice, this is equivalent reducing the sentiment score by one for twelve out of every hundred re- views mentioning a minoritised group, or to flip- ping the score from maximally positive to maxi- mally negative for three out of every hundred.
*Readers familiar with Kiritchenko and Mohammad (2018) may recall that they provide aggregate measure in the form of a graph, as we do, and more granular measures of amount of bias per group (c.g. for male and female separately), in table. We forgo the table we use contingency tables our analysis, which contain superset of the same information (bias by group, as well as bias by label).
"htpps: / /apppraaaaaeeeee//e/ sent mennt - ana 1 lysis bbenchmarks
thoode. com task//

---

Temphie
Conneeriattal sntemnos
<n The co with sperson object> Mas <emotionna situation word>,
The corvereation with [himther', vad ieritating,
E022 senction word passivass
( 2022 4 8
sparson> Biiiiaf sonssional aituation waxd>,
[faye IfiRiaf $
de Dan Gespracl nit sperson dat. db jecz? wad senotional situation words, Dau Geapzäch nis Vihe) was ieclLierend.
La converseción operaon> Fue Senot ional situation vord Ferr eo. La sonversacion
[6liells] FuR ieritance.
Table I: Example sentence templates for each language and their counterfactual words that, when filled in, create a contrastive pair, in this case, for gender bias. For illustration, all five examples translations of the same sentence.
Since it not trained on target language data, we apply the same ZS-XLT model to each tar- get language. As an ablation, we also train mono-XLT models (one per language) based on pre-training data and fine-tuned on target language supervision. Al- though this setup atypical, it enables to deter- mine whether changes in behaviour between the mono-T and ZS-XLT models are attributable to multilingual pre-training data, English supervision data, or both.
We train each model five times with different ran- dom seeds and then ensemble by taking their major- ity vote, standard procedure to reduce variance. In our initial experiments, we observed that bias var- ied substantially across different random initialisa- tions our out-of-doman counterfactual corpora, despite stable performance on our in-domain train- ing/eval/test data. Previous work has also found dif- ferent seeds with identical in-domain performance to have wildly variable -f-mmann results (Mc- Coy et al., 2020) and bias (Sellam et al., 2022) and theorised that different local minima may have dif- fering generalisation performance. To combat this generalisation problem, we use classifier dropout in all of our neural models, which is theoretically equivalent to a classifier ensembling approach (Gal and Ghahramani, 2016; Baldi and Sadowski, 2013).
mbert-base-uncase
Fine-tuning data. Each mono-T and mono-XLT model is fine-tuned the target language subset of the Multilingual Amazon Reviews Corpus (MARC; Keung et al., 2020), which contains 200-word re- views in English, Japanese, German, French, Chi- nese and Spanish, with discrete polarity labels rang- ing from 1-5, balanced across labels. We use the provided train/dev/test splits of 200k, 5k, 5k ex- amples in each language). The ZS- ZS-XLT model is tune- on the US segment of the Amazon Cus- tomer reviews corpus. This dataset is not balanced across labels, so we balance it by downsampling overrepresented labels to match the maxinum num- ber the least frequent label, in order to make the label distribution identical to that of the mono-T and mono-XLL XLT fine-tuning data. After balancing we have dataset of 2 million reviews (ten times more than monolingual training data), which then concatenate with the English subset of MARC. We fix the random seed for the data shuffle to be the same across all fine-tuning runs. Since our pre-training data is from Wikipedia and Common- Crawl, Paracrawl, or the target language equiva- lent, there is domain shift between pre-training and fine-tuning data, and between fine-tuning and evaluation data, which are more similar to the pre- training: domain mismatches are common in SA.7
5 Results
We examine whether system bias is affected by a decision to use zero-shot cross-lingual transfer (ZS-XLT) instead of monolingual transfer. There are two potential sources of bias in ZS-XLT: from the multilingual pre-training, or from the English supervision. Bias from pre-training is of most con- cern, since it could influence many other types of multilingual models. To tease them apart, we look at the mono-XLT,, system: if has higher bias than the mono-T model, then we can conclude that bias is imported from the multilingual pre- vre-training data. If the ZS-XLT model is more biased than the moTo-XLT model, then we can conclude that bias is imported from the cross-lingual supervision.
5.1
RQ1: How does bias compare between monolingual models and ZS-XLT models? Are observed changes from pre-training or from supervision?
"https://st amazonaws com/ amazon- reviews- pds/ readmee .html
Figure 2 shows comparison between mono-T, mono-XLI and ZS-XLT models.
"As is common in user-generated review data, the distri- bution is skewed lowards extreme labels, and in the original review data I and are 73% of data.
not across languages, making it more difficult to make cross- linguistic comparisons, which is why we make very few and are predominantly interested in the effect of cross-linguistic transfer within one language.
'Note that pretraining data is fixed within one language, allowing comparison between models within language, but

---

JePanne Vial
Gemmeen
Seanshiess)
Chinse
German
Sparish (est
English
mone-XL1
032
ZS XLT
0.0
4.00
435
400
gender
Figure 2: Aggregate bias metries (RQ1): Comparison of mono-T (blue), and mono-XLT (orange), ZS DSL (green), Mean and 95% confidence interval of differences in the sentiment label under each counterfactual pair, one graph per language and type of bias tested. Higher numbers indicate greater bias against the minoritized group. The dashed line at zero indicates no bias, the shaded region corresponds to 3% of total range (see 3.2).
Which transfer learning strategy introduces more bias? Our results show that ZS-XLT mod- els have equal greater bias than monolingual models; bias often worsens, sometimes dramati- cally.
difference in bias, but not always. For Japanese, German, Spanish, and English gender bias, the multilingual model causes the most change, but for Chinese, the English data causes For German racial bias the multilingual model causes huge jump in bias, but for Spanish, the English data does. Overall, the multilingual pre-training causes a large increase in bias, rather than the supervision data, This is on the one hand not very surprising, as there is great deal of discriminatory content in mul- tilingual pre-training data (Luccioni and Viviano, 2021), likely much more than in sentiment analysis supervision data. However, is novel finding, since it means that either negative social biases can transfer between languages, or that some artifact of multilingual training increases bias.
In Figure 2 this is the comparison between the leftmost blue model and the rightmost green model (where the middle orange model is an experimental condition allowing us to isolate the contribution of data from that of model, used to answer the question in the next paragraph of the causal fac- tors behind this behaviour). Japanese. English, and Traditional Chinese have greater bias in ES-XLT models for gender, German and Spanish have changed bias a slighly lower mean bias, but with a much larger interval. race however, German and Spanish increase, and Japanese and English are equivalent. This adds understanding to the re- cent study showing that pre-trained models are less biased than models without pre-training (Goldfarb- Tarrant et al., 2023): our results show that cross- lingual zero-shot transfer exacerbates biases, even though these models are trained on much more data than monolingual transfer. Goldfarb-TH -arrant et al. (2023) looked at the effect of pre-training data within a monolingual setting (as compared to using only supervision data) and found that it lessened bias to add pre-training data, which they attributed to the increased stability of the models due to using much more data when pre-training. Our findings show that the relationship between biases transferred in pre-training significantly more complex in the case of cross-lingual transfer, as biases can worsen despite the use of more data.
What different behaviours are behind these changes? To examine model differences in more detail, we create contingency tables to find the pat- terns in bias behaviour. An unbiased model would have all values on the diagonal. We display subset of contingency tables in Figure illuminating both differences in bias patterns underlying similar bias levels; and the causes of extreme changes in aggre- gate bias, as we see with German. The complete set appears in Appendix D.
In the aggregate metric for Japanese gender bias, we can see that the model goes from nearly no bias in mono-T to significant anti-male bias in both mono-XI, and ZS-XLT models. Figure 3a shows three different patterns of behaviour for all three models. The leftmost matrix shows that the mono- T model displays equivalent bias in most areas and across most labels: there small total counterfac- tual errors, and what there are evenly distributed. The introduction of multilingual training with the mono-XIL model increases aggregate bias, but not uniformly - is largely accounted for by changes
Are biases imported from the multilingual pre- training data, or the English supervision data? The pattern is unfortunately not consistent. More frequently, the multilingual model causes a large

---

from neutral postive or negative sentiment; does not flip positive negative sentiment vice versa. The ZS-XLT model has less overall bias, but the source of different: the model overpredicts extremely positive sentiment for female examples (right vertical bar of matrix).
(a) Japanese (ja) Gender
Figure 3b shows the less frequent case of in- crease in bias from the supervision data rather than the multilingual pre- pre-training. The mono-T model has some bias, but in way that is driven by mi- nor changes, with the sentiment changing by only ordinal label (blue clustered around the diago- nal). The mono-XLT model, in the middle, is quite similar, but the failures are slightly more broadly distributed. The ZS- XLL model has extremely dif- ferent behaviour from the mono-T model. The aggregate bias in similar (though of flipped polar- ity and higher variance) but the failures under the counterfactual frequently flip between extremes. Even for similar levels of aggregate bias, the mono- T Chinese model is likely be better; the errors that makes are more reasonable than the ZS-XLT ones, which are more concerningly wrong.
(b) Chinese (zh) Gender
(c) German (dc) Race
Figure 3: Example confusion matrices for demographic counterfactual pairs for gender in Japanese and Chinese and race in German. From left right: mono-T models, mono-XLT models, and ZS -LLT models. ++ to are sentiment scores. Rows are predicted sentiment scores for the privileged group, columns predicted scores for the minoritised group. Higher colour saturation in the lower triangle bias against the minoritised group. the upper triangle is bias against the privileged group. Colour saturations are different scales for different mod- els. Not visualised here: actual (ground-truie sentiment scores.
Figure 3c presents analysis of the unusual be- haviour of the German cross-lingual models when evaluated for racial biases. We can see that the mono-XLT model inaccuratcly predicts maximally negative sentiment for racially minoritised groups (bottom row of matrix), and this underlies the huge increase in racial bias between the mono-T and mono-XLT models that we see in Figure 2. The ZS- XLT model ameliorates this behaviour, and brings the pattern closer to that of the mono-T model, but remains more biased overall than mono-T, since many of the errors are extreme flips from maxi- mally positive to negative (lower left corner cell of matrix), As well as having less aggregate bias, again we see that the mono-T T model is the only one that shows reasonable behaviour under the counter- factual.
all languages women are the minoritised group, whereas the minoritised racial group differs. We also expected this because some languages have stronger syntactic gender signal than others. Pre- vious work measuring gender bias in embedding spaces (McCurdy and Serbetci, 2017; Gonen et al.. 2019b; Zhao al., 2020) has shown that grammat- ical gender information has stronger effect on bias behaviour than content, due to dominating the contexts that words appear in. This previous work predicts that we would see a change in bias pre- dominantly from grammatical gender differences, despite changes in cultural baseline level of concep- tual gender bias. We hypothesised that this might manifest in changes in gender bias when introduc- ing multilingual model. Based on this previous work, we expected increased gender bias when ing cross-lingual transfer for languages with less gender agreement (Chinese, Japanese, English), and decreased gender bias when using transfer for
The Case of Gender The difference between mono-T and mono-XLT is generally small for race and large for gender (Figure 2) (except in German, which is clear outlier in mono-XLT for reasons we could discover). This demonstrates that bias from a language included pre-training can appear in a model targeted different target language.
The larger effect on gender than on race is as we expected; gender biases are less culturally spe- cific than racial biases, which makes them seem intuitively easier to amplify cross cross-linguse lingulling in

---

languages with more gender agreement (German, Spanish) (irrespective of cultural attitudes toward women, which are very variable). For all languages, our hypothesis holds, the first time this effect has been shown on a downstream task rather than inter- nally in a language model. For English, Chinese, and Japanese, monolingual models have less gen- der bias than their multilingual counterparts, while for Spanish and German, monolingual models have more gender bias.
biases, For gender bias, distillation always tend dampen bias when applied monolingual models, but frequently worsens bias when applied to cross- lingual models. German, Spanish, and Chinese all have significantly more bias for gender with distil-ZS-XLa XLT than with ZS-XLT models.
Perhaps this indicates that the sources of gender bias in Japanese and in English are different than in German, Spanish, and Chinese, or that there are more language-speciftr characteristics that inter- act differently with distillation. This mirrors the answer to RQ1 in this one way: that the effects of cross-lingual transfer on gender bias (even with distilled models) vary greatly across different lan- guages, whereas the effects for racial bias are clearer trend. We leave this investigation for fu- ture work, but consider these results be at least promising that model distillation may be an effec- tive approach to mitigate or at least avoid exacer- bating racial biases in cases where cross-linguu transfer must be used.
The Case of Race For racial bias, the source of the bias is less systematic: Sometimes the ZS-XLT model bias is unchanged-as with Japanese and English-and and sometimes it increases, as with Ger- man and Spanish. The presence of cross-lingual racial bias is surprising. Racial bias tends to be cul- turally specific, so we did not expect it to transfer across language data the way gender bias might; we expected ZS- XLL to have either equivalent or less racial bias than mono-T. A possible factor in this may be whether the languages that share infor- mation have overlapping racial biases. For instance, racial bias categories in Japanese, like Okinawan or Korean, are unlikely to be effected by pre-training on English. Whereas racial bias categories in Ger- man, though German-specific, may be shared by other high resource Western languages, such as Arab. Future work could investigate whether dif- ferences in cross-lingual transfer for racial bias are related to level of shared cultural context. It could also investigate whether language-specific imple- mentation details like monolingual vs. multilingual tokenisation (Rust et al., 2021) could be driving any of these effects, since that would be more likely to affect morphologically rich languages like German. There is, importantly, one factor in race that is very systematic, which is that aggregate bias is never against the privileged group (values are at or above the x-axis of zero). So while sentiment models may vary across languages and models in whether they inaccurately associate negative or positive sen- timent to male vs. female terms, they universally associate negative sentiment to racial terms, just to varying degrees.
7 Recommendations and Conclusions
This broad set of experiments has shown that bias can change drastically result of any of the stan- dard engineering choices for making SA system in a lower resourced language. In light of these results, we make the following recommendations: Do not assume that more data will improve biases Assess bias of all new model and data choices. Use granular bias by sentiment label, well as aggregate bias, to make decisions that best suit the intended application.
Don't rely solely aggregate measures. Our results highlight how summary statisties can make different underlying distributions appear identical, a point made by Matejka and Fitzmaurice (2017) in general, and by Zhao and Chang (2020) specifically for bias, but still frequently overlooked in most bias research. Though both are problematic, a model that consistently associates slightly more negative sentiment to minoritised group is qualitatively different from model that sometimes flips very positive sentiment to very negative sentiment. Beware of bias introduced cross-lingually. Bias can transfer across languages from pre-training from supervision data, which means that cross- lingual transfer has the opportunity to introduce non-local biases. These can be unexpected and hard to detect, and represent machine learning cul- tural imperialism that is best avoided.
6 RQ2: Do distilled models show the same trends?
Figure shows a comparison of standard and dis- tilled models for mono-T and ZS-XLT models. The patterns are still not consistent, but are striking. For cioss-lingual transfer, distillation dampens racial

---

Jehanne Vial
Gemmeen
Spanshress)
Chinse
Gemman
Sparish (est
English
cistil-mono-1
039
25 25-KCT
distii-2S-XIT
600
4.00
430
400
gender
Figure 4: Aggregate bias metries (RQ2): Comparison of mono-T (blue), and distil- distil-mono-T (orange), ZS-X XLL (green), and distil-ZS-XLT (red). Mean and 95% confidence interval of differences in the sentiment label under each counterfactual pair, one graph per language and type bias tested. mono-T and ZS-XLT models are repeated from Fig 2 to enable easier visual comparison to distilled models. Higher numbers indicate greater bias against the minoritized group. The dashed line at zero indicates no bias, the shaded region corresponds to 3% of total range (see 3.2). There are only Spanish models due lack of monolingual distilled pretrained Spanish model the time of thiw work.
Be particularly aware of racial biases. Racial biases were both more pervasive and generally of higher magnitude than gender biases, across many languages and models. Racial biases frequently overlooked in research (Field al., 2021), and our results show that this can be quite dangerous.
gender biases behave so differently from racial bi- ases? An analysis of how the model Iearns the bias behaviour over the course of training could also help understand the mechanisms better. Alterna- tively causal analysis, or saliency and attribution methods, could enable us to understand, and per- haps control, when cross-lingual transfer makes biases better and when it makes biases worse. We release our code, all models, and all intermediate checkpoints, to help expedite further analysis an- swering these and other questions.
Consider compressing models. Distilled models had lower bias across most languages and demo- graphics, with a few exceptions. This came a very low penalty for performance of one Fl point on average. Previous work had contradictory con- clusions regarding model compression, with some vision models showing worse bias in compressed models (Hooker et al., 2020) and some NLP gener- ation models showing less bias under compression (Vig al., 2020). Our results support the latter, suggesting that it may be worth using compressed models even when not computationally required. This also highlights the need for more work on the effect of model size on social bias, as models con- tinue to scale far beyond the sizes studied in this work.
Limitations
There are of course limitations to our study. We consider a range of models that achieve state-of- the-art results on sentiment analysis tasks, but it is not feasible to test all models currently in use. Also, resources exist across domains, we cannot isolate the effect of domain shift. In addition, with- out specific downstream application in mind, we can only measure the presence of bias but not esti- mate which specific harms (Blodgett et al., 2020) are likely to arise as a result.
We have done the first study of the impact of cross-lingual transfer on social biases in sentiment analysis. We have also raised many open questions. What are the key mechanisms of cross-lingual trans- fer causing these changes? Monolingual transfer was found to lessen biases due to increased stability and performance of the model Sholdfarb-Tarrann et al., 2023), is the lack of this effect in cross- lingual transfer due the curse of muitilingual- ity (Pfeiffer al., 2022), some other reason? Have negative stereotypes been imported across languages and cultures, or is the increase bias due to some other artifact of the transfer? Why do
The bias tests we use in this paper are only available in five languages. While this is sig- nificant step forward compared only testing for bias in English, it represents only fraction of the world's languages. A study involving more lan- guages would also allow testing the interactions between languages. For example, it is plausible that biases are more likely to be shared between languages that share the same alphabet.
Finally, this paper contributes to understanding how cross-lingual transfer affects the presence bias, but this is only one of the sources of bias.

---

Moreover, measuring bias is only the first step, and our approach only allows us make limited causal statements about why the biases are present. More research is needed for more detailed recommenda- tions for how to reduce it.
Arabic. In Proceedings of the Second Workshop on Language Technology for Equality, Diversity and In- clusion, pages 90-106, Dublin, Ireland. Association for Computational Linguistics.
Kate Crawford. 2017. The trouble with bias. In Con- ference on Neural Information Processing Systems, invited speaker.
9 Ethics Statement
Our work is direct response to the risks posed by biased AI. We hope that our work will help to re- duce the risk of bias (in this case, gender and racial bias) affecting sentiment classifiation decisions. In doing so, we are releasing models that we know to be biased. These models could, in theory, be used by others for dubious purposes. However, since we are aware that the models are biased and which racial and gender biases they have, it is unlikely that someone else will use them unintentionally. After weighing up the risks and benefits, we there- fore release them in the interest of reproducibility and of people who wish to build on our work.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language under- standing. CoRR, a10/0800.
Julian Eisenschlos, Sebastian Ruder, Piotr Czapla, Marcin Kadras, Sylvain Gugger, and Jeremy Howard. 2019. MultiFiT: Efficient multi-lingua, language model fine-tuning. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP- IJCNLP), pages 5702-5707, Hong Kong, China. As- sociation for Computational Linguistics.
The Federal Anti- Discrimination Agency (FADA). 2020. Equal rights, equal opportunities: Annual report of the federal anti-diserimination agency.
The dataset we use, which ultimately derives from the templates collected by Kiritchenko and Mohammad (2018), does not contain any informa- tion that names or uniquely identifies individual people or offensive content. Our use of this dataset is consistent with its intended use, to measure gen- der and racial bias in sentiment analysis systems.
Anjalie Field, Su Lin Blodgett, Zeerak Waseem, and Yulia Tsvetkov. 2021. A survey of race, racism, and anti-racism in NLP. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1905-1925, Online, Association for Computational Linguistics.
References
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33nd International Conference Machine Learning, 1CML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMI. Workshop and Conference Proceedings, pages 1050-1059. JML..org.
Pierre Baldi and Peter J. Sadowski, 2013. Understand- ing dropout. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Pro- ceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 2814-2822
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 5454- 5476, Online. Association for Computational Lin- guistics.
Seraphina Goldfarb-Tarrrnt Adam Lopez, Roi Blanco, and Diego Marcheggiani. 2023. Bias beyond English: Counterfactual tests for bias in sentiment analysis in four languages. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4458- 4468, Toronto, Canada. Association for Computa- tional Linguistics.
Sandra Buckley. 2006. Encyclopedia of contemporary Japanese culture. Routledge.
Seraphina Goldfarb- Tarraant,, Rebecca Marchant, Ri- cardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. 2021. Intrinsic bias metrics do not correlate with application bias. In Proceedings of the 59th Au- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1926-1940, Online. Association for Computational Linguistics.
Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from lan- guage corpora contain human-like biases. Science, 356:183-186.
António Câmara, Nina Taneja, Tamjeed Azad, Emily Allaway, and Richard Zemel. 2022. Mapping the multilingual margins: Intersectional biases of sen- timent analysis systems in English, Spanish, and
Hila Gonen, Yova Kementchedjhieva, and Yoav Gold- berg. 2019a. How does grammatical gender affect

---

noun representations in marddrg languages? In Proceedings of the 23rd Conference on Computa- tional Natural Language Learning (CoNIL), pages 463-471, Hong Kong, China. Association for Com- putational Linguistics.
Third BlackboxNLP Workshop on Analyzing and In- terpreting Neural Networks for NLP. pages 217-227, Online. Association for Computational Linguistics.
K. McCurdy and Oguz Serbetci. 2017. Grammat- ical gender associations outweigh topical gender bias in crosslinguistic word embeddings. ArXiv, abs/2005.0888 0.086.
Hila Gonen, Yova Kementchedjhieva, and Yoav Gold- berg. 2019b. How does grammatical gender affect noun representations in mender- languages? ArXiv, 1bs/1910.14161.
Saif M. Mohammad, Mohammad Salameh, and Svet- lana Kiritchenko. 2016. How translation alters senti- ment, J. Artif. Intell. Res., 55:95-130.
Suchin Gururangan. Ana Marasović. Swabha Swayamdipta, Kyle Lo, Iz Beltagy. Doug Downey, and Noah A. Smith. 2020. Don't t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342 8360, Online. Association for Computational Linguistics.
Githu Muigai. 2010. Report of the special rapporteur on contemporary forms of racism, racial discrimination, xenophobia and related intolerance, githu muigai, his mission to germany (22 june - 1 july 2009).
Hadas Orgad, Seraphina Goldfarb- Tarrant, and Yonatan Belinkov. 2022. How gender debiasing affects in- ternal model representations, and why it matters. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 2602 2622,, Seattle, United States. Association for Computational Linguistics.
Sara Hooker, Nyalleng Moorosi, G. Clark, S. Bengio, and Emily L. Denton. 2020. Characterising bias in compressed models. ArXiv, abs/2010.03058.
Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. 2020. The multilingual Amazon reviews cor-
pus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5563-4568, Online. Association for Computational Linguistics.
Judea Pearl. 2009. Causal inference in statistics: An overview, Statistics Surveys, 3:96-146.
Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross. Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3479-3495, Seattle, United States. Association for Computational Lin- guistics.
Svetlana Kiritchenko and Saif Mohammad. 2018. Ex- amining gender and race bias in two hundred senti- ment analysis systems. In Proceedings of the Sev- enth Joint Conference on Lexical and Computational Semantics, pages 43-53, New Orleans, Louisiana. Association for Computational Linguistics.
Matt J. Kusner, Joshua R. Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual fairness. In Ad- vances in Neural Information Processing Systems 30: Annual Conference on Neural Information Process- ing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4066-4076.
Telmo Pires, Eva Schlinger, and Dan Garrette, 2019. How multilingual is multilingual BERT? In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1996-5001, Flo- rence, Italy. Association for Computational Linguis- tics.
Alexandra Luccioni and Joseph Viviano. 2021. What's ss in the box? analysis of undesirable content in the Common Crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1th International Joint Confer- ence on Natural I Language Processing (Volume 2: Short Papers). pages 189, Online. Association for Computational Linguistics.
Soujanya Poria, Devamanyu Hazarika, Navonil Ma- jumder, and Rada Mihalcca. 2020. Beneath the tip of the iceberg: Current challenges and new directions in sentiment analysis research. CoRR, abs/2005.00357.
Tharindu Ranasinghe and Marcos Zampieri. 2020. Mul- tilingual offensive language identification with cross- lingual embeddings. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 5838-5844, Online. As- sociation for Computational Linguisties.
Justin Matejka and George Fitzmaurice. 2017. In Proceedings of the 2017 CHI Conference Hu- man Factors in Computing Systems, CHI '17, page 1290-1294, New York, NY, USA. Association for Computing Machinery, [link].
R. Thomas McCoy, Junghyun Min, and Tal Linzen. 2020. BERTs of feather do not generalize together: Large variability in generalization across models with similar test set performance. In Proceedings of the
Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65:569- 631.

---

Phillip Rust, Jonas Pfeiffer, Ivan Vulié, Sebastian Ruder. and Iryna Gurevych. 2021. How good is your tok- enizer? the monolingual performance of multilin- gual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Ith International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 31 111--135, Online. Association for Computational Linguistics.
Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Seao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas:
Gastä Salamanca and Lidia Pereira. 2013. PRESTI- GIO Y ESTIGMATYZACIÁ DE 60 NOMBRES PROPIOS EN 40 SUJTOS DE NIVEL EDUCA- CIONAL SUPERIOR. Universum (Talca), 28:35 57.
The surprising cross- moss-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-JCNILP), pages 833-844, Hong Kong, China. Association for Computational Linguis- tics.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, 10s/1910.0110S.
Jieyu Zhao and Kai-Wei Chang. 2020. LOGAN: Lo- cal group bias detection by clustering. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1968-1977 1977, Online. Association for Computational Linguistics.
Maarten Sap, D. Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias hate speech detection. In ACL.
Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D' Amour, Tal Linzen, Jasmijn Bastings, Iulia Raluca Ture, Jacob Eisenstein, Dipanjan Das, and Ellie Pavlick. 2022. The multiB- ERTs: BERT reproductions for robustness analysis. In Intermational Conference on Learning Representa- tions.
Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini, Kai- Wei Chang, and Ahmed Hassan Awadallah. 2020. Gender bias in multilingual embeddings and cross- lingual transfer. ArXiv, abs/2005.0066
Chris Sweeney and Maryam Najafian. 2020. Reducing sentiment polarity for demographic attributes word embeddings using adversarial learning. In Proceed- ings of the 2020 Conference on Fairness, Account- ability, and Transparency, FAT '20, page 359-368. New York, NY, USA. Association for Computing Machinery.
Mike Thelwall. 2018. Gender bias in sentiment analysis. Online Information Review.
Enrica Troiano, Roman Klinger, and Sebastian Padó. 2020. Lost in tack-translation: Emotion preserva- tion in neural machine translation. In Proceedings the 28th International Conference on Computational Linguistics, pages
Barcelona, Spain (On- line). International Committee Computational Lin- guistics.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stu- art Shieber. 2020. Investigating gender bias in lan- guage models using causal mediation analysis. In Advances in Neural Information Processing Systems, volume 33, pages 12388-12401. Curran Associates, Inc.
Michael Weiner. 2009. Japan's 's minorities: the illusion of homogeneity, volume 38. Taylor & Francis.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistae, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen,

---

A Model Implementation Details
C Demographics Included in Benchmark Datasets
Monolingual transformer models have 1 10 million parameters (# million) and vocabularies of 30- 32k with 768D embeddings. Multilingual models have 179 million parameters, vocabulary of 120K, with 768D embeddings. We train the monolingual models with the same training settings as preferred in Keung et al. (2020), and allow the pre-trained weights to fine-tune along with the newly initialised classification layer. The multilingual models are trained identically, save that they have 100x larger learning rate, and learning rate annealing.
Racial Minoritised Groups included in the bench- mark dataset of Goldfarb-Tarrant et al. (2023) are as below:
For German, this includes Jewish, Roma, Sinti, Arab and Muslim from the UN report, Sorbs an officially recognised minority, and Polish, Roma- nian, Turks, Kazakh, Kurds, Russian, Syria, Iraq. A Afghanistan, Vietnamese as official large immi- grant groups.
For Japanese, this is Chinese, Korean, Okinawan, and generic "Foreign".
All models were trained for 5 seeds, models trained on monolingual data (mono-T, mono-XLT, and distil-mono-T) were checkpointed 15 times. ZS-XLT models were checkpointed 6 times. In total we train 1525 models: 3 monolingual (non- baseline) model types with 5 seeds across 5 lan- guages and 15 checkpoints (1,225 models) and 2 multilingual model types (ZS-XLT, distil-XLT) with 5 seeds and 5 languages and checkpoints (300) models.
For Spanish there is list of proper names collected from sociology study that immigrant names (Salamanca and Pereira, 2013).
For English this a replication of Kiritchenko and Mohammad (2018) so it African American proper names.
D Full set of contingency tables comparing baseline and monolingual models.
This study was done on only the converged mod- cls, but all models are released for further study.
The contingency tables for all languages can be shown in Figure 5. A subset of these are included in the main body of the paper. It is worth noting that saturations not malised across all languages and models; this is not proxy for aggregate comparative bias, shows the pattern across sentiment scores. The contin- gency tables also do not show actual (ground-truin truth) sentiment scores. We include baseline models (left- column) not used in this work for maximum visual comparability to previous work on these bench- marks.
Computational Resources. Each model was trained 4 NVVIDIA Tesla V100 GPUs with 16GB memory. mono-T and mono-XLT models took 6-8 hours to converge, 7S-XLT and distil-ZS-XII took hours. This a total 620 total hours, or 2,480 GPU hours on our resource.
B Model Performance
Standard
Distilled
F1
Steps
Reference
Fl
Steps
ja
0.62
44370
0.57
0.61
60436
zh
0.56
35190
0.55
0.53
43750
de
0.63
36720
0.62
0.63
52621
es 0.61
41310
0.59
0.65
27050
0.63
0.65
44285
ZS-XLT
0.69
75000
0.68
33336
Table 2: F1 convergence and steps convergence for standard size and distilled models. Monolingual model performance is measured on the MARC data, ZS-XLT model performance the US reviews data. Refereence performance taken from Keung et al. (2020), classifi- cation accuracy. They don't 't train monolingual models, so the reference performance is mBERT classification accuracy.

---

baselne
manoo
detilmoro-T
mono-ALT
Z5
(a) Japanese (ja)
dieii-ZSaLL
(b) German (de)
baseline
distil-ZS-XLT
ZS-XLT
gender
race
(c) Spanish (es)
(d) English (en)
(c) Chinese (zh_en)
Figure 5: All confusion matrices for experiments in this paper. ++ to are sentiment scores. Rows are predicted sentiment scores for the privileged group, columns predicted scores for the minoritised group. Higher colour saturation in the lower triangle is therefore bias against the minoritised group. in the upper triangle is bias against the privileged group.


Reviews:
[{"review_id": "x7E8qcvKNI", "reviewer": "Reviewer_F94e", "summary": "", "questions": "What is the difference between this study and Goldfarb-Tarrant, et al 2023 which leads to contrasting results in terms of which models pre-trained ones or without pre-training (lines 388-394)? Do you have some assumptions regarding this? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the very helpful review!\n\nTo address the five listed improvements/reasons to reject:\n1. The demographic and emotion words were from the resource created by Goldfarb-Tarrant, et al 2023. We will remedy this and make this clearer in the text, but we will also use the new space to excerpt them and include examples of them in the text, and the entire list in the appendix.\n2. We will include the results of Keung et al 2020 in Appendix B, apologies for the oversight. These are their results, with ours in parentheses for comparison: ja 0.57 (0.62), zh 0.55 (0.56), de 0.62 (0.63), es 0.59 (0.61) en 0.63 (0.65). We exceed slightly in all cases but are very close.\n3. On reviewing the Case of Gender section, we realise some of what we wrote is unclear. We consider this to be primarily interesting because it is in line with previous work, our citation from McCurdy and Serbetci, 2017 actually has the same finding on word embeddings, that grammatical gender has a larger effect on bias than content, so we find it interesting that our results agree. This is also in line with Zhao et al 2020 in contextual embeddings. Their findings both show that the effect of grammatical gender is stronger than content, which implies that we’d see a change despite the change in culture. However, we agree with you that it is impossible to control for things like culture without creating synthetic data so we will revise this section to clarify that this is a hypothesis that comes from previous work, and that our work strengthens their findings.\n4. The caption is actually incorrect on Figure 2 from a previous version, our apologies, it still reads “mean and variance” but we had updated it so that the error bars were 95% confidence intervals, so what you see in this review version is in fact confidence intervals. We hope that this satisfies your question. An additional note is that the test sets we use from Goldfarb-Tarrant, et al 2023 are also quite large (3-5k per test) should ensure sufficient power. We will also include the test size stats excerpted from their paper in the final version to make that clear also. \n5. For figure 2, the comparison of interest for practical bias levels in use is the blue vs. green models (far left to far right) as the orange is essentially an experimental condition allowing us to isolate the contribution of data from that of model. So in comparing those: ja, en, and zh_cn have greater bias for gender, de and es have maybe a bit less but with a larger interval for the ZS-XLT model so we consider them equivalent. For race, only de and es increase, whereas ja and en are equivalent. We will write this out in detail with the additional space. We could also include an additional breakout graph with the orange experiment removed for ease of quick reading.\n\nRegarding the question as to differences between this work and Goldfarb-Tarrant, et al 2023, lines 388-394, their experiments only looked at the effect of pretraining data within a monolingual only setting (no cross lingual transfer). They found that it improved bias to add pretraining data. However, they attributed this finding to the increased stability of the models, since they were comparing to models with no pretraining at all (and thus very little data). \nOur findings on cross-lingual transfer don’t really contradict them, though they do augment and find that the situation is much more complex for cross-lingual transfer and sometimes bias can worsen. We will add this discussion.\n\nWe have made all the presentational improvements you suggest, thank you they were very helpful!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very well-written, with a clear motivation, experiential setup, presentation of the results, and discussion of the results. \nThis paper combines two relevant applications, sentiment analysis systems, and cross-lingual training, and test both gender and racial biases for a diverse set of languages.\nExperimental setup is sound.   \n", "reasons_to_reject": "Authors do not provide the list of demographic and emotion words used to create the counterfactual pairs. It is unclear if they were retrieved by the authors of the paper or base on a prior work’s resource.\nIn lines 287-291, authors mention that the performance of their models converges to be equivalent to previous work (Keung et al, 2020). The results of the prior work should be put in Appendix B next to their results to evaluate this claim.\nAuthors claim that the larger effect on gender in their experiments comes from the fact that some languages have stronger syntactic gender signal than others. While this statement could be true, it has not been tested in a wider setup in this paper and is rather an assumption given such a small number of languages (Japanese differs not only in gender-marking from German but also culturally which can also have an effect here).\nThis paper would benefit from stronger statistical testing. Results in Fig 2 could be tested on difference in means. The authors claim that the ZS-XLT models have equal or greater bias but this is not easily spotted for all the languages.\n", "questions_to_authors": "What is the difference between this study and Goldfarb-Tarrant, et al 2023 which leads to contrasting results in terms of which models pre-trained ones or without pre-training (lines 388-394)? Do you have some assumptions regarding this? ", "paper_topic_and_main_contributions": "This paper analyzes the performance of sentiment analysis systems in multilingual setups. The authors compare both mono- and cross-lingual transfer scenarios using counterfactual templates. They run their experiments in five languages. They find that cross-lingual transfer can lead to a more biased model, finding strong especially effects for racial biases. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "\n"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The abbreviation SA is introduced already in the abstract, no need to reintroduce it at the beginning of the introduction\nThe transition between the first and second sentences of the abstract is a bit rough (no connection between sentiment analysis and multilinguality).\nCaption Figure 1: The first sentence has a couple of mistakes\n4th finding could be made sharper. If compressing models via distillation does not always reduce the bias, you could explicitly give an example when that is the case \nFootnote 1 is not there "}}, {"review_id": "QUqDHMYA2Y", "reviewer": "Reviewer_S7SY", "summary": "", "questions": "A: L494-505 -- this is very confusing. Weak and strong gender agreement are not standard terms so it is not completely clear what is meant here. This seems to suggest that English (which has very little gender agreement) has strong gender agreement, which does not make sense. Can you please clarify?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review! We’ve made all the presentational improvements, and changed the shapes and label sizes on the graphs. \nWe also apologise about the missing Spanish monolingual distilled model – there wasn’t an available pretrained one so we had to leave it out of experiments, and somehow this explanation fell off the caption in one of the edits for length. We have added it back. \nThank you also for the additional reference that we had missed, we have read and added it also.\n\nRegarding your question on weak/strong gender agreement:  we re-read and see why it is confusing, and will revise. We will define the terms to be more explicit about the extent of grammatical gender marking rather than using ‘weak’ and ‘strong.’ \nWe intended to say that Chinese and Japanese have very little, German and Spanish have more, and English is in the middle of these two sets, such that when added to Chinese and Japanese gender information is increased and when added to German or Spanish it is decreased. We actually based this theory based on the previous cited work that did this in word embeddings but that isn’t very clear (McCurdy and Serbetci, 2017, and Zhao et al 2020) so we will clarify that also.\nIf you have additional suggestions on clarity for this section beyond that, please let us know!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Thorough analysis of gender and racial bias in cross-lingual transfer for sentiment in 5 languages. The authors control different sources of bias when possible and examine both aggregate and granular measures of bias. These results are interesting and likely important for researchers working in bias or sentiment.", "reasons_to_reject": "None", "questions_to_authors": "A: L494-505 -- this is very confusing. Weak and strong gender agreement are not standard terms so it is not completely clear what is meant here. This seems to suggest that English (which has very little gender agreement) has strong gender agreement, which does not make sense. Can you please clarify?", "paper_topic_and_main_contributions": "The authors analyze gender and racial biases in sentiment systems for 5 languages In particular, they are focused on bias differences between monolingual models (models trained directly on the target language, including pretraining data) and cross-lingual models (models trained with multilingual pretraining data and possibly also non-target language sentiment data). They evaluate on existing datasets which consist of pairs of sentences that differ in a single biased term (ex: a gendered pronoun, or a racial term). The authors then measure the score difference between the prediction on the sentence with the privileged group term (ex: male for gender) and the prediction on the sentence with the minority term (ex: female for gender). The authors observe that cross-lingual transfer can worsen bias in sentiment models, and they analyze whether the bias comes from the multilingual pretraining data or from English task data for each target language. The also find that distillation can help to mitigate bias in many but not all cases.\n\nMain contribution: Computationally-aided analysis of bias in multilingual NLP.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- https://arxiv.org/pdf/2204.03558.pdf also evaluate gender and racial bias in multilingual settings and creates new datasets for this purpose. Should at least mention this work, even if not evaluating on their data.", "Typos_Grammar_Style_and_Presentation_Improvements": "- L181: \"classes to using\" --> \"classes by using\"\n- L189: unnecessary semi-colon after \"metrics\"\n- L304: \"data in English only\" refers to sentiment data right? Should specify this directly. \n- Figures 2 and 4 are really difficult to follow in black and white. Please use different symbols in addition to colors to differentiate the models. \n- Figure 3: the font labeling each contingency table is way too small to be legible in printed form.\n- Figure 4: the spanish gender bias graph only has 3 models, one appears to be missing.\n"}}, {"review_id": "XcyDvxRGoJ", "reviewer": "Reviewer_hiFd", "summary": "", "questions": "Why is ZS-XLT finetuned on both MARC and Amazon Customer reviews corpus?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review!\n\nAs to your questions and reasons to reject:\n1. All demographics and emotion words were from the resource created by Goldfarb-Tarrant, et al 2023. We will make this clearer in the text, but we will also use the new space to excerpt them and include examples of them in the text, and the entire list in the appendix. We have also included them below for you.\n2. ZS-XLT is intentionally trained on additional data, because that is why someone would use a cross-lingual model (to leverage cross-lingual fine tuning data). We also ablate this to control for the effect of the model vs. the effect of the data, which is why we also train mono-XLT models (which are cross-lingual models without additional data). In section 4 lines 302-319 we discuss these choices and why its important to do all of them. This is one of our contributions.\n3. Re line 340 (domain shift of pretraining data): the pretraining data is consistent within one language, so it doesn’t affect comparisons within one language (which is what we do), it only would across languages, which we don’t focus on. We will clarify this in text. It would be better to have it fully consistent we agree, but that is nigh impossible to control for, since even different versions of commoncrawl can have vastly different data, so we set up our experiments to not require this consistency.\n4. We will use some of the additional space to break figure 4 into two graphs so they’re easier to read, and to discuss the compression results in more detail with more exact numbers to improve the consistency. They are not fully consistent within themselves as far as trends, but we decided to include them anyway since it’s a valuable research question. Distillation improves bias cross-lingually for gender for English and Japanese, but worsens for German, Spanish, and Chinese, whereas for race it always improve it (save with English where it makes no difference). We weren’t able to work out why this is, but we will revise the text and discuss it further. The important difference is the green to red dot for a practitioner who might want to use cross-lingual transfer to decide whether to distil.\n\nThank you for the presentational improvement also, we have increased the size of the labels in Figure 3!\n\nThe Racial Minoritised Groups:\nFor German, this includes Jewish, Roma, Sinti, Arab and Muslim from the UN report, Sorbs as an officially recognised minority, and Polish, Romanian, Turks, Kazakh, Kurds, Russian,  Syria, Iraq, Afghanistan, Vietnamese as official large immigrant groups.\nFor Japanese, this is Chinese, Korean, Okinawan, and generic “Foreign”.\nFor Spanish there is a list of proper names collected from a sociology study that are immigrant names (Gastã Salamanca and Lidia Pereira. 2013)\nFor English this is a replication of Kiritchenko and Mohammad so it is African American proper names.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The study of monolingual versus multilingual bias is broken down into pretraining and finetuning data, providing a more comprehensive analysis. In addition, the topic of cross-lingual bias transfer is important and timely, given the increased usage of models in non-English languages.\n\nThe authors study languages from a variety of language families.\n\nThe conclusion provides a nice overview of the study, while also highlighting specific areas of future research.", "reasons_to_reject": "There are no examples of racial/immigrant bias subjects in other languages. This makes it hard to determine the specific groups being studied in this paper.\n\nI have some issues with the training procedure. The first is that ZS-XLT is finetuned on additional data compared to other models, which can influence results. The second is that you mention in line 340 that pre-training data can stem from a variety of sources. If this indeed varies across models, this domain shift (e.g. Wikipedia data versus CommonCrawl) can introduce additional uncertainties when comparing models that are supposedly controlled on the pretraining data.\n\nThe results reported in Section 6 do not seem to be conclusive and consistent when compared to those shown in Figure 4.", "questions_to_authors": "Why is ZS-XLT finetuned on both MARC and Amazon Customer reviews corpus?", "paper_topic_and_main_contributions": "This paper studies how social biases can transfer across languages during cross-lingual transfer learning for the downstream sentiment analysis task. This analysis is broken down into investigations of bias transfer in the pretraining and finetuning data for five languages. Results show that multilingual pretraining has a larger impact on biases in comparison to pretraining data. In addition, both gender and racial biases are susceptible to transfer.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The labels in Figure 3 are extremely small"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper "Cross-lingual Transfer Can Worsen Bias in Sentiment Analysis" presents work on the effect of transfer learning on sentiment analysis in non-English languages and the import of biases from one languages to another. 

The arguments against accepting the paper refer to details of the experimental setup. 
The arguments for accepting the paper refer to the task as a whole, the depth and extent of the experiments and the results presented.
Decision: Accept-Main