Do Multilingual Large Language Models Mitigate Stereotype Bias? (2024)

Abstract: While preliminary findings indicate that multilingual LLMs exhibit reduced bias compared to monolingual ones, a comprehensive understanding of the effect of multilingual training on bias mitigation, is lacking. This study addresses this gap by systematically training six LLMs of identical size (2.6B parameters) and architecture: five monolingual models (English, German, French, Italian, and Spanish) and one multilingual model trained on an equal distribution of data across these languages, all using publicly available data. To ensure robust evaluation, standard bias benchmarks were automatically translated into the five target languages and verified for both translation quality and bias preservation by human annotators. Our results consistently demonstrate that multilingual training effectively mitigates bias. Moreover, we observe that multilingual models achieve not only lower bias but also superior prediction accuracy when compared to monolingual models with the same amount of training data, model architecture, and size.

heading: 1 Introduction
text: With an increasing rate of adopting Large Language Models (LLMs) in real-world applications such as healthcare, finance, and law (Yang et al., 2023, 2024), it is crucial to enhance their safe usage in terms of bias and fairness to avoid causing representational harm. Many studies have investigated the biases encoded in LLMs, which vary across models and languages (Kaneko et al., 2022; Zhou et al., 2019; Lalor et al., 2022; Kotek et al., 2023). Figure 1 shows an example of bias using our English and multilingual models. When asked about an uncertain situation, the monolingual model opts for an answer that is considered biased while the multilingual model chooses the unbiased option. Prior research has demonstrated the effectiveness of multilingual models in reducing language-specific ethnic bias (Ahn and Oh, 2021).

heading: I don't know
text: Furthermore, Levy et al. (2023) provided evidence that multilingual pre-training typically produces models with decreased bias. Building upon these findings, which primarily focused on BERT-like architectures, our study extends this investigation to larger, decoder-based Language Models (LLMs). We aim to examine the impact of monolingual versus multilingual training on model bias in these more advanced architectures. For this purpose, we train six novel 2.6B LLMs, one for each of Spanish, German, French, Italian, and English, as well as one multilingual model trained on all five languages but using the same number of tokens. We perform a human-validated automated translation of the CrowS-Pairs (Nangia et al., 2020) and BBQ (Parrish et al., 2022) bias evaluation benchmarks. In this controlled setting we find that the multilingual models are less biased and often outperform bigger LLMs with larger, less diverse training sets.

heading: 2 Related Work
text: Much research has been done to analyze bias in the NLP community, a trend that has increased as the focus has moved toward deep and large models (Garrido-Muñoz et al., 2021; Navigli et al., 2023a).
65
The evaluation of bias in LLMs mostly focuses on models and benchmarks in the English language and culture (Gallegos et al., 2023; Navigli et al., 2023b; Joshi et al., 2020). A survey of 146 papers found that in most studies there is no reasoning for why bias is harmful and to whom, which can lead to a mismatch between the objective and proposed methods (Blodgett et al., 2020). In this work, we use the definition from Crawford (2017), also mentioned in Parrish et al. (2022), that stereotype bias in our experiments relates to representational harm that “occurs when systems reinforce the subordination of some groups along the lines of identity.”
Metrics. There exists a broad range of metrics to quantify bias (Czarnowska et al., 2021), and mitigation approaches to reduce it (Gallegos et al., 2023). While some metrics are explicitly constructed to measure and reduce bias in datasets, the majority focuses on the evaluation of model bias. Gallegos et al. (2023) differentiate between embedding-based bias metrics (Caliskan et al., 2017), probability-based bias metrics (Webster et al., 2021), and generated text-based bias metrics (Bordia and Bowman, 2019). To evaluate the models in our setting, we focus on probability-based bias metrics.
Datasets. Recently, multiple benchmark datasets such as CrowS-Pairs (Nangia et al., 2020), BBQ (Parrish et al., 2022), StereoSet (Nadeem et al., 2021), and WinoGender (Rudinger et al., 2018) have been introduced that are applicable for specific NLP tasks or selected bias types. These datasets provide sentences that reflect stereotypes. As they cover a wider range of social groups, they are broadly used to benchmark NLP models. While some shortcoming of e.g. CrowS-Pairs and StereoSet could be mitigated, as suggested by Blodgett et al. (2021), the work of Liu (2024) demonstrates the value of the stereotype pairs to assess differences between disadvantaged and advantaged groups.
Multilingual bias. Addressing the lack of bias evaluation in different languages, there exist several studies examining bias in monolingual models including the evaluation of bias specifically related to a given culture. For instance, Malik et al. (2022) and Vashishtha et al. (2023) focus on the evaluation of bias in Indian culture and Indic languages. Zmigrod et al. (2019) and Zhou et al. (2019) focus on the mitigation of stereotypes in
gender-inflected languages. Besides a monolingual evaluation, Zhou et al. (2019) also evaluate bias in bilingual embeddings.
Multilinguality as bias mitigation. Similar to our work, Levy et al. (2023) compares biases and the impact of multilingual training across multiple languages by assessing bias in a downstream sentiment analysis task using templates adapted from Czarnowska et al. (2021). For five languages (Italian, Chinese, English, Hebrew, and Spanish), they reveal differences in the expression of bias and consistently show that models (mBERT, XLM-R) favor groups that are dominant within the culture of each language. Comparing the effects of multilingual pre-training and multilingual fine-tuning, they find a stronger effect on bias amplification using multilingual fine-tuning.
Notably, Ahn and Oh (2021) evaluate bias in monolingual models for six languages - English, German, Spanish, Korean, Turkish, and Chinese - and propose the use of multilingual models as a bias mitigation technique. Introducing the categorical bias score, they find for resource-rich languages a reduction of bias by using pre-trained or fine-tuned multilingual models.
While both of the above-mentioned studies examine bias in multilingual models, in our work we select Germanic and Romance languages and experiment with models of larger scale and transparent data origin. We translate commonly applied bias benchmarks to these languages and focus on the effect of pre-training by training our mono- and multilingual models.

heading: 3 Approach
text: To compare the encoded bias in mono- and multilingual models, first we use automatic translation to translate BBQ (Parrish et al., 2022) and CrowSPairs (Nangia et al., 2020) datasets and evaluate the translation quality with manual annotation. Then, we train six LLMs from scratch (one for each language plus one multilingual) and evaluate them on these benchmarks.

heading: 3.1 Datasets
text: While we discussed related bias datasets in §2, there are two datasets we chose for our experiments based on the wide array of stereotypes they covered. Coverage of different types of bias is particularly important when comparing monolingual and multilingual models, to identify how the usage of single
or multiple languages and associated cultural understanding increase or decrease model bias towards different protected attributes.
CrowS-Pairs. The Crowdsourced Stereotype Pairs benchmark (CrowS-Pairs) aims to measure nine types of social bias in language models, including race, gender, sexual orientation, religion, age, nationality, disability status, physical appearance, and socioeconomic status (Nangia et al., 2020). The dataset contains 1,508 examples distributed in two categories: the stereotypes (demonstrating a stereotype against a socially disadvantaged group) and the anti-stereotypes (violating a stereotype against a socially disadvantaged group). Each example consists of a pair, where a sentence about a disadvantaged group is paired with a minimally distant sentence about a contrasting advantaged group. The sentences are obtained via crowdsourcing with Amazon Mechanical Turk. Compared to other datasets which focus on only a few types of bias (commonly gender), CrowS-pairs covers a wide range of different bias categories (Gallegos et al., 2023).
BBQ. The Bias Benchmark for QA (BBQ) (Parrish et al., 2022) is a question-answering prompting dataset aiming to measure a model’s reliance on stereotypes depending on the given context and question formulation. It measures the same nine types of social biases like CrowS-Pairs, mostly derived from the protected demographic categories by the U.S. Equal Employment Opportunities Commission. The dataset contains 58,492 examples, generated from filling in a set of templates with a set of attributes, which have been handwritten by the authors and evaluated by experts and crowdworkers. For each type of bias, there are 1,000 test samples. Each sample consists of an underinformative (ambiguous) and adequately informative (disambiguous) context option, as well as two question types; a negative judgment question (e.g. Who never steals things?) and a non-negative judgment question (e.g. Who steals things?). For each example, the correct answer is given as multiplechoice with three options. (e.g. A. Person_1 steals things. B. Person_2 steals things. C. It’s not clear.)

heading: 3.2 Translation and Annotation
text: To compare monolingual and multilingual models for multiple languages, we need a consistent translation of both the CrowS-Pairs and BBQ datasets. While both datasets are originally available only in
English (en) and have been translated and curated into some other languages (Névéol et al., 2022; Jin et al., 2023), the full translation into our chosen language set is lacking. Existing works use both human translation (Nozza et al., 2021) and machine translation (Bernagozzi et al., 2021; Vanmassenhove et al., 2021). Since we want to make our approach extendable, we use machine translation with human quality control to adapt each example of CrowS-Pairs and BBQ into German (de), French (fr), Italian (it), and Spanish (es).
We evaluated two automatic translators DeepL1
and MetaTranslator2. For each language, at least one fluent speaker reviewed a sample of at least 60 sentences from the CrowS-Pairs dataset translated with both services. Each sample was assessed concerning the general translation quality and bias preservation. Annotators were asked to decide between wrong translation (incorrect sense), bumpy translation (grammatically incorrect or wrong expression), and correct translation, and the degree of the stereotype preservation (more bias, less bias, same bias, no bias, not reasonable bias). The latter two (no bias and not reasonable bias) were introduced because of the general missing quality of some bias swap templates of CrowS-pairs (Blodgett et al., 2021), which suffer from conceptual pitfalls. Only two instances (“Vince was rich, so he had very little money.” and “White people can’t drive because they have slanted eyes”) were rated as “not reasonable”, and thus removed from comparison of all language splits. The annotators were asked to provide free-text feedback to clarify their rating regarding the translation of the bias.
Table 1 shows the translation quality of the two online translators, and Table 2 shows the evaluation of the bias enhancement after the translation. For the languages with two annotators, we also evaluated the inter-annotator agreement by Cohen’s Kappa (Cohen, 1968) as shown in Table 3.
On average, the translation quality of DeepL was rated better, with a higher margin for French and German.In terms of Cohen’s κ, we see for MetaTranslator a moderate agreement and for DeepL a fair agreement.
The bias was rated by the annotators in a translation sample as equal to the English original in most cases. In a few instances, the annotators found no bias in either the CrowS-pairs sample or the trans-
1https://www.deepl.com/de/translator 2https://ai.meta.com/blog/seamless-m4t/
lation. This highlights a potential weakness of the CrowS-pairs dataset. A challenge within this evaluation is the different perception of bias, which gets, in particular, clear by the multi-annotation of two annotators in the same language that do not have a consistent agreement (compare A1 & A2, A3 & A4). Cases, where the annotators found an increase or decrease in bias due to the translation, were comparably infrequent in the translation of both automatic translators. We therefore decided on the use of DeepL due to the better translation quality. This evaluation using the CrowS-Pairs dataset informed our decision to use DeepL to also translate the BBQ benchmark.

heading: 4 Experiments
text: We train monolingual and multilingual variants of our causal language models and evaluate them using a zero-shot setup on both the CrowS-Pairs and BBQ benchmarks and compare them with several recently developed LLMs.

heading: 4.1 Task Formulation
text: For the CrowS-Pairs benchmark, we are given two sentences to compare. Each sentence can be given to a language model to compute an overall likelihood. These are compared with the intuition that the more similar the likelihood, the less biased the model is. Our evaluation follows the original setup from Nangia et al. (2020). For the BBQ dataset, however, our approach differs from that of the original paper, where BERT-based (Devlin et al., 2019) models were utilized. To evaluate bias, they finetuned their models on the RACE benchmark for reading comprehension (Lai et al., 2017a). The questions were collected from the English versions of middle-school and high-school student exams and contained multiple-choice answers. This step is not necessary to evaluate the bias of our models, where the likelihood of different options can be computed to determine an answer in a similar way to the CrowS-Pairs evaluation.
Since our models are not trained in a chat setting, prompt-based question answering is not effective. Instead, we first construct the initial model input by concatenating the context C and the question Q, denoted as X = concat(C,Q). For each answer option, Oi, i ∈ {0, 1, 2} we compute the loglikelihood li in an auto-regressive manner. Specifically, the likelihood of each word Oi,j in option Oi is calculated given the current state of input
X , which is iteratively updated by appending Oi,j . The formula for the log-likelihood calculation is as follows:
li =
|Oi|∑
j=0
log(p(Oi,j |Xj))
where Xj is updated by Xj = concat(Xj−1, Oi,j) after every iteration.
Ultimately, the option with the highest accumulated log-likelihood is selected as the model’s choice.

heading: 4.2 Our Models
text: To measure the effect of the language on the bias of the LLM, we trained one model for each language and one multilingual model, combining data from all five languages. Specifically, we trained a 2.6 billion parameter transformer-based decoder-only model for each of our five studied languages on 52 billion tokens following the scaling law proposed by Hoffmann et al. (2022). All models were trained based on the causal language modeling training objective. Further hyperparameters are shown in the Table 6 in the Appendix.
The models were primarily trained on web documents, more precisely, Common Crawl dumps processed with the Ungoliant pipeline (Abadji et al., 2022) and filtered based on the Ungoliant quality criteria and subsequently deduplicated. In addition, some curated datasets (cf. Appendix Table 5) such as Wikipedia and selected subsets of the The Pile (Gao et al., 2020) and RedPajama (Computer, 2023) were used. After compiling the five monolingual text corpora, 52 billion tokens were extracted from the corpora for the training of the models. The multilingual training corpus was created by sampling and combining 20% of each monolingual training corpus and therefore was trained on a comparable number of tokens.
For tokenization, we choose the sentence piece library (Kudo and Richardson, 2018) with a vocabulary size of 32,768 (monolingual) and 100,352 (multilingual, therefore 200 million more parameters) as recommended in Ali et al. (2023). Due to the difference in vocabulary size, the multilingual model has 2.8 billion parameters.
The training losses of all six mono- and multilingual models are shown in Figure 7 in the Appendix. Furthermore, we show in Figure 8 on a holdout validation set that all model trainings decrease to a perplexity of around 10± 2.5 depending on the
language. All of our models show a consistent improvement in training loss and validation perplexity during training.

heading: 4.3 Open-source Models
text: In this paper, we selected three well-known opensource large language models—Mistral, Falcon, and Llama2—for benchmarking. Since the parameter size of both our monolingual and multilingual models is 2.7b, we chose the smaller 7B versions of these open-source models for comparison. Additionally, we selected the base versions of these models and did not choose the fine-tuned versions, to maintain consistency with our model.
Falcon-7b (Almazrouei et al., 2023) Falcon is a causal decoder-only model that has been trained on 1.5 trillion tokens. Over 80% of its training data comes from RefinedWeb—a new web dataset based on CommonCrawl (Penedo et al., 2023). Additionally, Falcon-7b supports English, German, French, Spanish, and limited Italian, so we also conducted experiments with this model across all our target languages.
LLama2-7b (Touvron et al., 2023) LLama2 is also a decoder-only model. Compared to LLama1, it doubles the context length and uses Grouped Query Attention (GQA) to enhance the inference scalability. Additionally, it has been pre-trained on 2 trillion tokens of curated web data, with the majority of the pre-training corpus being in English (89.7%) and a small portion in programming code (8.38%). Hence, we only tested LLama2 on English datasets.
Mistral-7b (Jiang et al., 2023) Mistral is also a decoder-only model, which outperforms Llama2 on many benchmarking datasets. It also utilizes grouped query attention to enhance inference and employs sliding window attention to handle longer sequences with less memory. Unfortunately, the authors did not provide details about the training data, so we only tested it on English in this paper.

heading: 4.4 Evaluation
text: For the two datasets utilized, we applied the metrics mentioned in their respective papers. The following will provide individual introductions for each.

heading: 4.4.1 BBQ Metrics
text: Firstly, the accuracy of disambiguated and ambiguous samples for each bias category is calculated separately, and then the overall accuracy is also calculated. Subsequently, the benchmark authors
have defined methods for calculating the bias score in both ambiguous and disambiguated contexts, as these two different scenarios can reflect the model’s behavior from different perspectives. The bias scores range from −100% to 100% with 0% indicating the model is not biased, while 100% indicates that all answers align with the targeted bias, and −100% indicates that all answers go against the bias.
The definition of a biased answer is as follows: 1) The model outputs a biased target in response to a negative judgment question (e.g., answering "the girl" for "who is bad at math"). 2) The model outputs a non-biased target in response to a nonnegative judgment question (e.g., answering "the boy" for "who is good at math").
The calculation for the bias score in disambiguated contexts is as follows:
SDIS = 2( nbias_ans
nnon−UNKNOWN_outputs )− 1
With nbias_ans representing the number of samples that fall into the definition of biased answer, nnon−UNKNOWN_outputs represents the number of non-targeted outputs (i.e. answering "I don’t know").
For the bias score in ambiguous contexts, we also scale the score by the overall accuracy according to the original paper. This approach can to some extent reflect that biased answers, if frequently occurring, would be more harmful. The calculation for the bias score in ambiguous contexts is as follows:
SAMB = (1− accuracy)sDIS
Additionally, we contend that the bias score for ambiguous contexts should be interpreted from additional dimensions, rather than being entirely dependent on the bias score in disambiguated contexts. To be more specific, in the sample of ambiguous contexts from the BBQ dataset, the context lacks complete information or solid evidence, hence the ground truth is always a non-targeted answer. Therefore, we believe that accuracy in ambiguous contexts, specifically the probability of the model responding with "I don’t know", more accurately reflects the model’s tendencies when information is insufficient.

heading: 4.4.2 Metrics on CrowS-Pairs
text: In the CrowS-Pairs dataset, each sample consists of two sentences: one is biased, and the other is not. Each sentence comprises two parts: the modified part and the unmodified part. For example, in the pair "John ran into his old football friend" versus "Shaniqua ran into her old football friend" the modified tokens are {John, his} for the first sentence and {Shaniqua, her} for the second sentence. The unmodified tokens for both sentences are {ran, into, old, football, friend}. For a sentence S, let U = {u0, ..., ul} be the unmodified tokens, and M = {m0, ...,mn} be the modified tokens (S = U ∪ M ). Based on these definitions, the pseudo-log-likelihood (Wang and Cho, 2019) of the unmodified tokens conditioned on the modified tokens, p(U |M, θ) is calculated. This approach differs from p(M |U, θ) from (Nadeem et al., 2021), primarily because the authors of this dataset believe it can help with avoiding bias caused by the frequent appearance of common names in the training data. The calculation of the score definition is as follows:
score(S) =
|C|∑
i=0
logP (ui ∈ U |U\ui ,M, θ)
The pseudo-log-likelihood of all unmodified tokens is calculated iteratively and then summed up as the final score of sentence S.
Based on the score of each sentence, we measured 1) the average score difference across all samples and 2) the percentage of examples where the model assigns a higher pseudo-log-likelihood to the stereotyping sentence. These are applied to every bias category.

heading: 5 Results and Discussion
text: Results for the CrowS-Pairs benchmark are shown in the heatmap in Figure 2. Numbers shown are the percentage stereotype, we subtracted 50 from all the values, meaning that values greater than 0 indicate a tendency towards the stereotype sentence, while values less than 0 indicate a tendency towards the non-stereotype sentence. The perfect score is 0, where neither sentence is preferred over the other. We find that the multilingual model has scores that are closer to 0 in all languages compared to its monolingual counterpart and also opensource LLMs.
Results for the BBQ benchmark are shown in the heatmap in Figure 3 On the BBQ dataset, Our
multilingual model also has better overall accuracy compared to their monolingual counterparts, and also better than most of the open-source LLMs across languages. Falcon outperforms the other open-source models and, in the case of German, outperforms our models. The high performance of the model, in particular for gender identity and the German language is difficult to determine, but may be attributed to the filtering done to construct the RefinedWeb corpus on which it was trained (Penedo et al., 2023).
Breaking down the accuracy on the BBQ dataset in Figure 4, we can also compare the accuracy of ambiguous and disambiguated contexts. we can observe that on the accuracy of ambiguous con-
text, the multilingual model does much better than the monolingual models, while on the accuracy of disambiguated contexts, performance drops. The mixture of languages in the training data for the multilingual model seems to make it more conservative, hence the model is more likely to respond with “I don’t know” when the information is insufficient, but this nature also causes loss of accuracy when dealing with the disambiguated samples (where the answer is always known).
However, after balancing the two sides, the final outcome is favorable for our multilingual model. In Parrish et al. (2022), their UnifiedQA model reached the average ambiguous accuracy of 60.8% and average disambiguated accuracy of 91.4%. The
large difference in performance is due to first finetuning their model on the RACE dataset (Lai et al., 2017b), which is also a text-based multiple-choice dataset. The fine-tuning made their model familiar with the QA format, and hence, they got a high score. For a fairer comparison, we do not fine-tune any models on the QA task and the results from open-source models are on par with our results.
Additionally, some papers also evaluated models in a zero-shot setting on the BBQ dataset. Shaikh et al. (2022) with GPT3 got 55.73% accuracy overall. Si et al. (2022) with an instruction fine-tuned version of GPT3, Text-Davinci-001 got 60.5% and 43.2% for ambiguous and disambiguated context, respectively. One notable comparison is the parameter size difference between our models and GPT3. While GPT3 has 150b parameters, ours only have 2.7b. Our models achieve lower accuracy at 20.83% lower than GPT3, yet surpassing Falcon7b by 1.54% across all 5 languages, LLama2-7b by 6.9% on English, Mistral-7b by 3.8% on English, on average. Due to limited computational resources, we cannot perform this comparison at 150b parameters and leave a controlled exploration of the relationship between bias and parameter size to future work.

heading: 6 Validity of The Models
text: To validate our model’s capabilities beyond bias evaluation, we additionally conducted tests on the Belebele benchmark (Bandarkar et al., 2023), a common sense-based multiple-choice question-
answering dataset designed to test the model’s understanding capabilities in different language contexts. To fit our model, we also reformulated this dataset into QA format.
The model’s results are shown in Table 4. All the data in the table including those from other papers, were obtained under the zero-shot setting. Additionally, the inference method is consistent with the BBQ method described in Section 4.1.
From the Belebele results, the monolingual models generally perform better than the multilingual model. This may be due to the fact that during the training of the multilingual model, the data for each language is only 20% of that for the corresponding monolingual model, leading to insufficient commonsense knowledge. However, given that our data-controlled models have less than half the parameters compared to other open-source models, our LLM benchmark results are satisfactory.

heading: 7 Conclusion
text: In this work, we systematically explored the relationship between the language of data a large language model is trained on and the stereotype bias that is encoded in the model. We trained six models with around 2.7b parameters from scratch using a causal language modeling objective and evaluated them on the CrowS-Pairs and BBQ benchmarks for English, French, German, Italian, and Spanish. To ensure that our approach can be extended to other languages and benchmarks, the datasets were automatically translated. For quality
assurance, a sample of the translations was evaluated by humans, who generally found that the translation quality was high and biases were preserved. We found that multilingual models trained on the same number of tokens as monolingual models were less biased for all languages and both benchmarks than the monolingual models. We also found that our models were generally less biased than selected open-source LLMs which had 7b parameters, though they fall short of zero-shot prompt-based approaches with GPT3. Publicly released material to our experiments can be found under http://lamarr-institute.org/research/ natural-language-processing/.
Limitations
In our work, we use machine translation to evaluate monolingual and multilingual models across multiple languages. Using machine translation might affect the quality and the expression of bias of the translated datasets. By evaluating the translation process with human evaluators as described in §3.2, we aim to reduce these effects. Nevertheless, we are aware that the small number of annotators might decrease the significance of our results as in particular the evaluation of the bias in the translation is
influenced by the perception of the annotator. In future work, we aim to extend this evaluation to all the studied languages and to more native annotators and methods that can ensure the quality of the automated translations.
The biases that exist in the benchmarks we used may be specific to English speaking regions. When translating the benchmark, bias may decrease because the biases that manifest in the translated language are specific to the regions that speak that language, which might not be the same as English speaking regions. Future work should consider creating new bias benchmarks for each language that represent the biases of the populations that speak those languages. Without this, we cannot be sure that the translated benchmarks cover the biases that are likely to occur in a given language. The significance of our results might be limited by CrowSpairs quality as shown in Blodgett et al. (2021). (Blodgett et al., 2021) finds that 97% of the dataset are not admissible. Generating a french version of CrowS-pairs, also Névéol et al. (2022) scrutinizes and even improves the original CrowS-pairs dataset. They present the statistics of the different adaptation types (compare Table 2 in (Névéol et al., 2022). In addition to the sentences modified to suit
to the French culture, 150 samples in total (10% of the dataset) were adapted due to the identified limitations within the original CrowS-pairs dataset (non-minimal pairs (22), double switches (64) or bias-type mismatches (64)). Even if the findings of (Blodgett et al., 2021) show severe shortcomings, we decided on using CrowS-Pairs due to its broad usage in the literature and its coverage of many different bias categories and social groups. The findings of (Liu, 2024) proof at least significant differences between the stereotype and antistereotype sentence pairs. Within our own sampled evaluation also only a small rate of sentences needed to be excluded in general. To validate our findings despite of the ambiguities, we used BBQ as a second benchmarking dataset. In future work, we plan to extend the experiments to other datasets, such as the published revised version of CrowSpairs (Névéol et al., 2022) or the HONEST dataset (Nozza et al., 2021). Moreover, since the languages involved in this paper are all European languages, their high similarity may lead to certain stereotypical knowledge being shared, making it easier for stereotypes to transfer between languages.

heading: Acknowledgments
text: This work has been supported by the German Federal Ministry of Education and Research (BMBF) as a part of the AI Safety project (project No. 05D2022), the Federal Ministry of Education and Research of Germany and the state of NorthRhine Westphalia as part of the Lamarr-Institute for Machine Learning and Artificial Intelligence, LAMARR22B as well as by the German Federal Ministry for Economic Affairs and Climate Action (BMWK) through the project OpenGPT-X (project No. 68GX21007D) and by the European Union’s Horizon 2020 research and innovation program under grant agreement No. 101135671 (TrustLLM) and 952215 (TAILOR). The authors gratefully acknowledge the Gauss Centre for Supercomputing e.V. for funding this project by providing computing time on the GCS Supercomputer JUWELS at Jülich Supercomputing Centre (JSC). We acknowledge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer Leonardo, hosted by CINECA (Italy) and the Leonardo consortium through a EuroHPC Benchmark Access call.

heading: A BBQ Bias Scores
text: Here we present the bias scores for the BBQ dataset covering the nine demographic attributes. Figure 5 we show the bias scores for the monolingual and multilingual LLMs that we trained and in Figure 6 we show the scores for the open-source models.
B Intrinsic Evaluation of the LLMs
The training losses of all six mono- and multilingual models are depicted in Figure 7 in the Appendix. Additionally, in Figure 8, we illustrate that during training, all models consistently decrease to a perplexity of approximately 10 ± 2.5 on a holdout validation set, with slight variations observed depending on the language. As all models use different tokenizers, the training loss and the validation perplexity are not directly comparable to each other. Also, the curated corpora, and therefore the training- and validation sets differ slightly depending on the language. Nonetheless, all models show a consistent improvement during training.

heading: C Datasets
text: Our web documents in the corpora consist of Oscars3 (Abadji et al., 2021), that were generated by the ungoliant pipeline4 based on 20 Common Crawl WET Archives (2014-42, 2015-14, 2015- 48, 2016-22, 2016-43, 2017-13, 2017-47, 2018- 30, 2018-47, 2019-22, 2020-24, 2020-45, 2021-31, 2021-49, 2022-27, 2022-40, 2022-49, 2023-06, and 2023-14).
The curated datasets consist of The Pile (Gao et al., 2020), RedPajama (Computer, 2023), and single datasets that do not belong to a collection. From the Pile subcorpora, we selected: Phil Archive, PMC Abstracts, PMC Extracts, OpenWebText, NIH Exporter, and Free Law Opinions V2. From RedPajama we use Books and StackExchange.
The remaining datasets are:
1. The Wikimedia dump of 2023-09-015
2. All the News V2.06 is a corpus of newspaper articles crawled from over 26 different publications from January 2016 to April 1, 2020.
3https://oscar-project.org/ 4https://github.com/oscar-project/ungoliant 5https://dumps.wikimedia.org/backup-index.
html 6https://metatext.io/datasets/all-the-news-2. 0
3. CoStEP7 is a cleaned-up and corrected version of the EuroParl corpus(Graën et al., 2014)(Koehn, 2005)
4. DCEP8 is a companion corpus to CoStEP, containing documents published by the European Parliament. (Hajlaoui et al., 2014)
5. Dissertations9 is a collection of dissertations from the Deutsche Nationalbibliothek.
6. MAREC/IREC10: The MAtrixware REsearch Collection / The Information retrieval facility Research Collection is a patent corpus of over 19 million documents from the EP, WO, US, and JP patent offices.
7. Medi-Notice11 is part of the Zurich Parallel Corpus Collection. It is a multilingual corpus compiled from information leaflets for medications and pharmaceutical products published by the Swiss Agency for Therapeutic Products.(Graën et al., 2019)
8. Swiss Policy12 contains documents of the Swiss Legislation Corpus (Höfler and Piotrowski, 2011)
9. OpenSubtitles 201813,14 is a collection of translated movie subtitles.
10. The peS2o (Soldaini and Lo, 2023) dataset is a collection of 40M creative open-access academic papers, cleaned, filtered, and formatted for pre-training of language models (Lison and Tiedemann, 2016)
11. The EUR-Lex dataset15 is a multilingual collection of case laws, decisions, directives, recommendations, regulations, and proposals of the European Union.
7https://pub.cl.uzh.ch/wiki/public/costep/ start
8https://joint-research-centre.ec. europa.eu/language-technology-resources/ dcep-digital-corpus-european-parliament_en
9https://www.dnb.de/DE/Professionell/ Services/Dissonline/dissonline_node.html
10https://researchdata.tuwien.ac.at/records/ 2zx6e-5pr64
11https://pub.cl.uzh.ch/wiki/public/pacoco/ medi-notice
12https://pub.cl.uzh.ch/wiki/public/pacoco/ swiss_legislation_corpus
13https://opus.nlpl.eu/OpenSubtitles-v2018.php 14https://www.opensubtitles.org/de/index.cgi 15https://huggingface.co/datasets/joelniklaus/
eurlex_resources
12. Bundestag - Plenarprotokolle16 comprises transcripts of sessions of the German Bundestag.
13. Bundestag - Drucksachen17 contains all bills that are negotiated in the Bundestag.
14. Bundesgerichtshof - Entscheidungen18 is a collection of decisions of the German Federal Court.
15. German legal cases contain German court decisions and the corresponding citation network(Ostendorff et al., 2020).
16https://www.bundestag.de/dokumente/ protokolle/plenarprotokolle
17https://www.bundestag.de/drucksachen 18https://www.bundesgerichtshof.de/DE/
Entscheidungen/entscheidungen_node.html

heading: Future Work
text: In future work, we aim to extend this evaluation to all the studied languages and to more native annotators and methods that can ensure the quality of the automated translations. The biases that exist in the benchmarks we used may be specific to English speaking regions. When translating the benchmark, bias may decrease because the biases that manifest in the translated language are specific to the regions that speak that language, which might not be the same as English speaking regions. Future work should consider creating new bias benchmarks for each language that represent the biases of the populations that speak those languages. Without this, we cannot be sure that the translated benchmarks cover the biases that are likely to occur in a given language. The significance of our results might be limited by CrowSpairs quality as shown in Blodgett et al. (2021). (Blodgett et al. , 2021) finds that 97% of the dataset are not admissible. Generating a french version of CrowS-pairs, also Névéol et al. (2022) scrutinizes and even improves the original CrowS-pairs dataset. They present the statistics of the different adaptation types (compare Table 2 in (Névéol et al. , 2022). In addition to the sentences modified to suit
to the French culture, 150 samples in total (10% of the dataset) were adapted due to the identified limitations within the original CrowS-pairs dataset (non-minimal pairs (22), double switches (64) or bias-type mismatches (64)). Even if the findings of (Blodgett et al. , 2021) show severe shortcomings, we decided on using CrowS-Pairs due to its broad usage in the literature and its coverage of many different bias categories and social groups. The findings of (Liu, 2024) proof at least significant differences between the stereotype and antistereotype sentence pairs. Within our own sampled evaluation also only a small rate of sentences needed to be excluded in general. To validate our findings despite of the ambiguities, we used BBQ as a second benchmarking dataset. In future work, we plan to extend the experiments to other datasets, such as the published revised version of CrowSpairs (Névéol et al. , 2022) or the HONEST dataset (Nozza et al. , 2021). Moreover, since the languages involved in this paper are all European languages, their high similarity may lead to certain stereotypical knowledge being shared, making it easier for stereotypes to transfer between languages