# Introducing MascuLead: the First Gender Bias Leaderboard

Fanny Ducel, Jeffrey André, Aurélie Névéol, Karën Fort

► To cite this version:

Fanny Ducel, Jeffrey André, Aurélie Névéol, Karën Fort. Introducing MascuLead: the First Gender Bias Leaderboard. EALM 2025 - Ethic and Alignment of (Large) Language Models, Jun 2025, Marseille, France. pp.12-19. hal-05282981

HAL Id: hal-05282981

https://inria.hal.science/hal-05282981v1

Submitted on 25 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Introducing MascuLead:
the First Gender Bias Leaderboard

Fanny Ducel* Jeffrey André† Aurélie Névéol* Karën Fort†

* Université Paris-Saclay, CNRS, LISN, 91400, Orsay, France

† Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France

*prenom.nom@lisn.fr, †prenom.nom@loria.fr

## ABSTRACT

We present MascuLead, the first leaderboard centered on a gender bias detection task for inflected languages. We apply an existing framework on several Large Language Models and compare their performance. We also highlight the importance of including bias benchmarks in leaderboards, and question the very notion of leaderboards.

**MOTS-CLÉS** : biais, stéréotype, modèle de langue, genre, français.

**KEYWORDS**: bias, stereotype, large language model, gender, French.

# 1 On the Urge of Including Bias Benchmarks in Leaderboards

The field of Natural Language Processing (NLP) is experiencing exponential growth, leading to exponential development of Large Language Models (LLMs). The number of LLMs calls for robust evaluation methods, in order to identify state-of-the art models, but also for users to be able to pick the best-suited model for their needs. Leaderboards are a popular form of evaluation which consists in the publication of ranked LLMs, based on their performance on one or several tasks. For example, the *Open LLM Leaderboard* (Fourrier et al., 2024) compares over 4,500 LLMs on 6 benchmarks, and has been liked by over 13,000 HuggingFace users<fnref n="1" />. A French version of this leaderboard is also available, covering about 30 LLMs on the same 6 translated benchmarks (Mohamad Alhajar, 2024).

Most leaderboards only include tasks such as question-answering, language understanding, or algorithmic reasoning. However, ethical aspects of LLMs are rarely taken into account, even though they can have a crucial impact on outputs and users. In this paper, we will focus on introducing stereotypical biases in NLP leaderboards. Numerous studies propose evaluation metrics for different types of stereotypical biases regarding gender (Choenni et al., 2021), race (Hofmann et al., 2024), socio-economic status (Cercas Curry et al., 2024), etc. Even if this subfield of research focuses on English and US-centric stereotypes, efforts are carried out to address more diverse languages and socio-cultural contexts (Malik et al., 2022; Fort et al., 2024).

We believe that stereotypical biases should carry weight in leaderboards. Unbiased LLMs should be favored in comparison to biased and potentially harmful models. Further, including bias metrics in popular leaderboards would give more visibility to bias research. It could also "encourage researchers and engineers to pay more attention and direct more resources towards developing more exhaustive bias mitigation techniques and tackling more sources of biases" (Ducel et al., 2024a).

Therefore, we introduce MascuLead, the first leaderboard evaluating gender biases for French LLMs.

<figure><img src="image_2.png" /></figure>

<footnote n="1">¹As of April 2025.</footnote>
We use and extend the framework presented in *Ducel et al. (2024b)* to build our leaderboard. We also develop an online demonstrator to perform gender detection and bias evaluation, as well as update leaderboards<fnref n="2" />. The central task is cover letter generation, and gender biases are evaluated with two metrics: GenderGap and GenderShift (see Section 2). This framework presents advantages that make it a fit candidate to constitute a leaderboard task: it can be easily adapted to inflected languages and it does not rely on a corpus, which prevents overfitting³. We focus on French only and apply this framework on more LLMs than the original publication, generating 74,490 new cover letters with *Mistral-7B-v0.3*, *Mistral-7B-Instruct-v0.3*, *CroissantLLMBase*, *CroissantLLMChat*, *gemma-2-2b*, *gemma-2-2b-it*, *Llama-3.2-3B*, and *Llama-3.2-3B-Instruct*.

The additional models were chosen based on their popularity (they are among the most downloaded LLMs on HuggingFace as of April 2025) and the availability of both a base and Instruct version. The same two combinations of hyperparameters as the original article were selected⁴. For common models, we obtain the same results as the original paper, showing the reproducibility of the research.

## 2 Comparing LLMs with MascuLead

The resulting leaderboards are presented in Tables 1 to 3. The highest ranked LLMs are the ones that performed the best with the framework, and are supposedly the least biased according to the GenderGap (GG) and GenderShift (GS) metrics.

According to *Ducel et al. (2024b)*, GG is the difference in proportions of masculine vs. feminine generated texts. Hence, the ideal GG is 0, i.e. there are as many masculine as feminine generated texts. A positive GG indicates a bias towards masculine markers, whereas a negative GG indicates a bias towards feminine markers. GS is the likelihood that texts are not consistent with the gender of the prompt (e.g., a model prompted with feminine markers generates a masculine cover letter). As we use either neutral or gendered prompts, we present separate leaderboards for each type of prompt.

Table 1 is an attempt at aggregating all three separate leaderboards. The ranking is based on the average of GG and GS. However, as GG uses a scale from -100 to 100, we convert the scores to absolute values, and separate them depending on the direction of the bias and of the type of prompts. Hence, the ideal values become 0, offering better compatibility with GS.

We still display the separate leaderboards (Table 2, 3), as the ranking changes and they illustrate various issues. Table 2 (left) unveils the models that are the more/less likely to favor a gender over the other, when the prompts do not contain gender information. Table 2 (right) highlights a similar phenomenon, but when using gendered prompts, which supposes that some texts do not respect the prompted gender. These two tables, based on GG, are related to representational harm as they imply stereotypical associations and the preference of one gender. Finally, Table 3 illustrates the likelihood that the gender of the prompt is overridden. This can be related to allocational harm, as users can be misgendered, and their request (a cover letter consistent with their gender) is not met.

All leaderboards highlight the important biases generated by the LLAMA models (including vigogne, a LLAMA model fine-tuned for French). *Mistral-7b-v0.3*, *xglm-2*, and *croissantbase* remain in the top 5 across all leaderboards. Table 1 allows for better generalization and readability of common points.

<footnote n="2">²The demonstrator is currently being deployed but the URL will be shared during the oral presentation.

³StereoSet (Nadeem et al., 2021) and CrowsPairs (Nangia et al., 2020) are widely used bias benchmarks. However, since they rely on corpora, models can be trained or fine-tuned on the material, an increasingly common phenomenon known as "data contamination" (Balloccu et al., 2024; Deng et al., 2024). In consequence, the bias metrics results can be misleading.

⁴top_p: 0.75 and top_k: 100 / top_p: 0.95 and top_k: 10 – except for Mistral, for which the first combination was replaced with temperature: 0.5 and top_p: 0.7 due to poor quality results with the initial combination</footnote>
We can thus establish that all Instruct models appear more biased than their base counterparts. Further, the most biased models seem to be the most popular, but also best performing models according to traditional benchmarks (see Table 4 for a comparison with the *OpenLLMLeaderBoard* ranking).

<table><caption>Table 1: Global MascuLead. Italic: models used in the original paper. *: Instruct models.</caption><thead><tr><th>Rank</th><th>Model</th><th>Avg (↓)</th><th>GG-masc-N</th><th>GG-fem-N</th><th>GG-masc-G</th><th>GG-fem-G</th><th>GS</th></tr></thead><tbody><tr><td>1</td><td>xglm-2</td><td>13.64</td><td>1.08</td><td>/</td><td>7.05</td><td>/</td><td>32.79</td></tr><tr><td>2</td><td>mistral-7b-v0.3</td><td>17.87</td><td>0.71</td><td>/</td><td>/</td><td>7.73</td><td>45.18</td></tr><tr><td>3</td><td>croissantbase</td><td>24.98</td><td>/</td><td>8.15</td><td>9.07</td><td>/</td><td>57.71</td></tr><tr><td>4</td><td>bloom-560m</td><td>27.35</td><td>15.82</td><td>/</td><td>1.15</td><td>/</td><td>65.09</td></tr><tr><td>5</td><td>llama-3.2-3b</td><td>27.88</td><td>33.05</td><td>/</td><td>10.05</td><td>/</td><td>40.54</td></tr><tr><td>6</td><td>gemma-2-2b</td><td>30.27</td><td>23.7</td><td>/</td><td>10.39</td><td>/</td><td>56.71</td></tr><tr><td>7</td><td>gpt2-fr</td><td>31.66</td><td>12.81</td><td>/</td><td>21.81</td><td>/</td><td>60.35</td></tr><tr><td>8</td><td>bloom-7b</td><td>32.25</td><td>11.04</td><td>/</td><td>19.93</td><td>/</td><td>65.78</td></tr><tr><td>9</td><td>croissant-chat*</td><td>33.88</td><td>23.89</td><td>/</td><td>11.44</td><td>/</td><td>66.32</td></tr><tr><td>10</td><td>bloom-3b</td><td>36</td><td>18.95</td><td>/</td><td>17.23</td><td>/</td><td>71.82</td></tr><tr><td>11</td><td>mistral-7b-instruct-v0.3*</td><td>38.52</td><td>47.67</td><td>/</td><td>/</td><td>0.35</td><td>67.53</td></tr><tr><td>12</td><td>gemma-2-2b-it*</td><td>44.22</td><td>57.18</td><td>/</td><td>28.88</td><td>/</td><td>46.59</td></tr><tr><td>13</td><td>vigogne-2-7b</td><td>50.77</td><td>69.23</td><td>/</td><td>18.4</td><td>/</td><td>64.69</td></tr><tr><td>14</td><td>llama-3.2-3b-it*</td><td>58.14</td><td>65.57</td><td>/</td><td>25.47</td><td>/</td><td>83.37</td></tr></tbody></table>

## 3 Discussion – Leaderboards Flaws

MascuLead allows for a first evaluation and comparison of LLMs in terms of gender biases in French. However, this approach has several flaws, some of which are inherent to the notion of leaderboards. Leaderboards mostly focus on overall performance, which masks important biases, e.g. in relation to gender stereotypes. We argue that these grading systems should incorporate bias metrics in order to make biases, and the impact they have on performance and generations, more visible. By proposing an ethical reconfiguration of benchmarks, this contribution calls for the evolution of evaluation standards towards more responsible and equitable practices.

A closer look at the generated cover letters shows that data quality plays an important part in ranking models. Some models, such as *mistral-7b-v0.3*, appear to be among the least biased across all leaderboards, but exhibit major quality issues, e.g. irrelevant texts, gibberish<fnref n="5" />, or the production of only a few words in French before switching to English<fnref n="6" />. Further analysis of the generations' quality could be conducted, in order to determine whether or not GS correlates with poor text consistency. If so, GS could be a relevant metrics to assess both gender bias and the general quality of generations.

These types of issues can question the very notion of leaderboards, since they only present aggregated scores without examples or error analyses. Moreover, aggregated scores often mix very different tasks, which can question the relevance and meaning of the scores. LLMs that have medium performance on all tasks may be ranked higher than some models that excel in specific tasks, fields or languages (including the task/field/language of interest for a specific user). Similarly, there does not seem to be a notion of weighting of scores, whereas some tasks may be more important than others. As shown by Raji et al. (2021) and Ethayarajh & Jurafsky (2020), the matter of general evaluation of LLMs and other NLP systems is at stake and should be further investigated. Minimally, stereotypical bias evaluation, as well as other types of ethical evaluations, should be included and carry more weight. We hope that MascuLead constitutes a first step towards bias inclusion in leaderboards, and that it will be expanded with other bias tasks, bias types, and more linguistic and cultural contexts.

<footnote n="5">⁵See an example in Appendix 3.</footnote>

<footnote n="6">⁶A first naive approach based on language detection with GlotLID (Kargaran et al., 2023) reveals that 48% of gendered texts and 40% of neutral texts generated with Mistral are labeled as English. See Tables 5 and 6 in Appendix for full results.</footnote>
## References

BALLOCCU S., SCHMIDTOVÁ P., LANGO M. & DUSEK O. (2024). Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs. In Y. GRAHAM & M. PURVER, Éds., *Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)*, p. 67–93, St. Julian’s, Malta: Association for Computational Linguistics.

CERCAS CURRY A., ATTANASIO G., TALAT Z. & HOVY D. (2024). Classist tools: Social class correlates with performance in NLP. In L.-W. KU, A. MARTINS & V. SRIKUMAR, Éds., *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, p. 12643–12655, Bangkok, Thailand: Association for Computational Linguistics. DOI: 10.18653/v1/2024.acl-long.682.

CHOENNI R., SHUTOVA E. & VAN ROOIJ R. (2021). Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you? In M.-F. MOENS, X. HUANG, L. SPECIA & S. W.-T. YIH, Éds., *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, p. 1477–1491, Online and Punta Cana, Dominican Republic: Association for Computational Linguistics. DOI: 10.18653/v1/2021.emnlp-main.111.

DENG C., ZHAO Y., TANG X., GERSTEIN M. & COHAN A. (2024). Investigating data contamination in modern benchmarks for large language models. In K. DUH, H. GOMEZ & S. BETHARD, Éds., *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, p. 8706–8719, Mexico City, Mexico: Association for Computational Linguistics. DOI: 10.18653/v1/2024.nacl-long.482.

DUCEL F., NÉVÉOL A. & FORT K. (2024a). Desiderata for Actionable Bias Research. In *New Perspectives on Bias and Discrimination in Language Technology*, Amsterdam, Netherlands. HAL: hal-04755691.

DUCEL F., NÉVÉOL A. & FORT K. (2024b). "You'll be a nurse, my son!" Automatically Assessing Gender Biases in Autoregressive Language Models in French and Italian. *Language Resources and Evaluation*. DOI: 10.1007/s10579-024-09780-6, HAL: hal-04803403.

ETHAYARAJH K. & JURAFSKY D. (2020). Utility is in the eye of the user: A critique of NLP leaderboards. In B. WEBBER, T. COHN, Y. HE & Y. LIU, Éds., *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, p. 4846–4853, Online: Association for Computational Linguistics. DOI: 10.18653/v1/2020.emnlp-main.393.

FORT K., ALONSO ALEMANY L., BENOTTI L., BEZANÇON J., BORG C., BORG M., CHEN Y., DUCEL F., DUPONT Y., IVETTA G., LI Z., MIESKES M., NAGUIB M., QIAN Y., RADAELLI M., SCHMEISSER-NIETO W. S., RAIMUNDO SCHULZ E., SACI T., SAIDI S., TORROBA MARCHANTE J., XIE S., ZANOTTO S. E. & NÉVÉOL A. (2024). Your stereotypical mileage may vary: Practical challenges of evaluating biases in multiple languages and cultural contexts. In N. CALZOLARI, M.-Y. KAN, V. HOSTE, A. LENCI, S. SAKTI & N. XUE, Éds., *Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)*, p. 17764–17769, Torino, Italia: ELRA and ICCL.

FOURRIER C., HABIB N., LOZOVSKAYA A., SZAFER K. & WOLF T. (2024). Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard.
HOFMANN V., KALLURI P. R., JURAFSKY D. & KING S. (2024). Ai generates covertly racist decisions about people based on their dialect. *Nature*, **633**(8028), 147–154.

KARGARAN A. H., IMANI A., YVON F. & SCHUETZE H. (2023). GlotLID: Language identification for low-resource languages. In H. BOUAMOR, J. PINO & K. BALI, Éds., *Findings of the Association for Computational Linguistics: EMNLP 2023*, p. 6155–6218, Singapore: Association for Computational Linguistics. DOI: 10.18653/v1/2023.findings-emnlp.410.

MALIK V., DEV S., NISHI A., PENG N. & CHANG K.-W. (2022). Socially aware bias measurements for Hindi language representations. In M. CARPUAT, M.-C. DE MARNEFFE & I. V. MEZA RUIZ, Éds., *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, p. 1041–1052, Seattle, United States: Association for Computational Linguistics. DOI: 10.18653/v1/2022.nacl-main.76.

MOHAMAD ALHAJAR A. L. (2024). Open llm french leaderboard v0.2. https://huggingface.co/spaces/le-leadboard/OpenLLMFrenchLeaderboard.

NADEEM M., BETHKE A. & REDDY S. (2021). StereoSet: Measuring stereotypical bias in pretrained language models. In C. ZONG, F. XIA, W. LI & R. NAVIGLI, Éds., *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, p. 5356–5371, Online: Association for Computational Linguistics. DOI: 10.18653/v1/2021.acl-long.416.

NANGIA N., VANIA C., BHALERAO R. & BOWMAN S. R. (2020). CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In B. WEBBER, T. COHN, Y. HE & Y. LIU, Éds., *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, p. 1953–1967, Online: Association for Computational Linguistics. DOI: 10.18653/v1/2020.emnlp-main.154.

RAJI D., DENTON E., BENDER E. M., HANNA A. & PAULLADA A. (2021). Ai and the everything in the whole wide world benchmark. In J. VANSCHOREN & S. YEUNG, Éds., *Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks*, volume 1.
# Appendix

## Example of generated, gibberish text

"ça me semble être ce que je cherche. Je suis un homme de 30 ans, très sérieux et très motivé. Je suis à votre disposition pour plus de détails.

\# 2017 Ford F-150

\## 2017 Ford F-150 Interior Review

Note: This interior review was created when the 2017 Ford F-150 was new.

\###### Scorecard

The 2017 Ford F-150 has a well-built cabin that's comfortable and spacious. The F-150's front seats are supportive and offer plenty of room for taller drivers. The rear seats are also comfortable and roomy, and the truck's cabin is quiet on the highway.

- "The F-150's cabin is a pleasant place to spend time, with comfortable"

<table><thead><tr><td>Rank</td><td>Model</td><td>GenderGap (|x| = 0)</td></tr></thead><tbody><tr><td>1</td><td>mistral-7b-v0.3</td><td>0.71</td></tr><tr><td>2</td><td>xglm-2</td><td>1.08</td></tr><tr><td>3</td><td>croissantbase</td><td>-8.15</td></tr><tr><td>4</td><td>bloom-7b</td><td>11.04</td></tr><tr><td>5</td><td>gpt2-fr</td><td>12.81</td></tr><tr><td>6</td><td>bloom-560m</td><td>15.82</td></tr><tr><td>7</td><td>bloom-3b</td><td>18.95</td></tr><tr><td>8</td><td>gemma-2-2b</td><td>23.70</td></tr><tr><td>9</td><td>croissant-chat*</td><td>23.89</td></tr><tr><td>10</td><td>llama-3.2-3b</td><td>33.05</td></tr><tr><td>11</td><td>mistral-7b-instruct-v0.3*</td><td>47.67</td></tr><tr><td>12</td><td>gemma-2-2b-it*</td><td>57.18</td></tr><tr><td>13</td><td>llama-3.2-3b-it*</td><td>65.57</td></tr><tr><td>14</td><td>vigogne-2-7b</td><td>69.23</td></tr></tbody></table>

<table><caption>Table 2: MascuLead, on neutral (left) / gendered (right) prompts.</caption><thead><tr><td>Rank</td><td>Model</td><td>GenderGap (|x| = 0)</td></tr></thead><tbody><tr><td>1</td><td>mistral-7b-instruct-v0.3*</td><td>-0.35</td></tr><tr><td>2</td><td>bloom-560m</td><td>1.15</td></tr><tr><td>3</td><td>xglm-2</td><td>7.05</td></tr><tr><td>4</td><td>mistral-7b-v0.3</td><td>-7.73</td></tr><tr><td>5</td><td>croissantbase</td><td>9.07</td></tr><tr><td>6</td><td>llama-3.2-3b</td><td>10.05</td></tr><tr><td>7</td><td>gemma-2-2b</td><td>10.39</td></tr><tr><td>8</td><td>croissant-chat*</td><td>11.44</td></tr><tr><td>9</td><td>bloom-3b</td><td>17.23</td></tr><tr><td>10</td><td>vigogne-2-7b</td><td>18.40</td></tr><tr><td>11</td><td>bloom-7b</td><td>19.93</td></tr><tr><td>12</td><td>gpt2-fr</td><td>21.81</td></tr><tr><td>13</td><td>llama-3.2-3b-it*</td><td>25.47</td></tr><tr><td>14</td><td>gemma-2-2b-it*</td><td>28.88</td></tr></tbody></table>
<table><caption>Table 3: MascuLead, on gendered prompts, with GS as the key metrics.</caption><thead><tr><th>Rank</th><th>Model</th><th>GenderShift (↓)</th></tr></thead><tbody><tr><td>1</td><td>xglm-2</td><td>32.79</td></tr><tr><td>2</td><td>llama-3.2-3b</td><td>40.54</td></tr><tr><td>3</td><td>mistral-7b-v0.3</td><td>45.18</td></tr><tr><td>4</td><td>gemma-2-2b-it*</td><td>46.59</td></tr><tr><td>5</td><td>gemma-2-2b</td><td>56.71</td></tr><tr><td>6</td><td>croissantbase</td><td>57.71</td></tr><tr><td>7</td><td>gpt2-fr</td><td>60.35</td></tr><tr><td>8</td><td>vigogne-2-7b</td><td>64.69</td></tr><tr><td>9</td><td>bloom-560m</td><td>65.09</td></tr><tr><td>10</td><td>bloom-7b</td><td>65.78</td></tr><tr><td>11</td><td>croissant-chat*</td><td>66.32</td></tr><tr><td>12</td><td>mistral-7b-instruct-v0.3*</td><td>67.53</td></tr><tr><td>13</td><td>bloom-3b</td><td>71.82</td></tr><tr><td>14</td><td>llama-3.2-3b-it*</td><td>83.37</td></tr></tbody></table>

<table><caption>Table 4: Scores of LLMs in the OpenLLMLeaderboard (English version), as of 04/29/2025. Global Rank is out of 4576 LLMs. Number of downloads is global ("all time"), from URLs such as this. Note: We use the English version of *OpenLLMLeaderboard* as the models of our experiment are not in the French version. Moreover, we hypothesize that most users would refer to the more popular, English version, even if to work on other languages</caption><thead><tr><th>MascuLead</th><th>Rank</th><th>Model</th><th>Avg (%)</th><th>Global Rank</th><th>Nb. downloads</th></tr></thead><tbody><tr><td>14</td><td>1</td><td>Llama-3.2-3B-Instruct</td><td>24.2</td><td>1,768</td><td>11,592,453</td></tr><tr><td>11</td><td>2</td><td>Mistral-7B-Instruct-v0.3</td><td>19.23</td><td>2,799</td><td>11,556,197</td></tr><tr><td>12</td><td>3</td><td>gemma-2-2b-it</td><td>17.05</td><td>3,062</td><td>3,972,389</td></tr><tr><td>2</td><td>4</td><td>Mistral-7B-v0.3</td><td>14.58</td><td>3,390</td><td>6,797,298</td></tr><tr><td>6</td><td>5</td><td>gemma-2-2b</td><td>10.36</td><td>3,709</td><td>21,185,617</td></tr><tr><td>5</td><td>6</td><td>Llama-3.2-3B</td><td>8.7</td><td>3,809</td><td>3,740,053</td></tr><tr><td>10</td><td>7</td><td>bloom-3b</td><td>4.39</td><td>4,384</td><td>656,781</td></tr><tr><td>4</td><td>8</td><td>bloom-560m</td><td>3.51</td><td>4,525</td><td>21,939,835</td></tr></tbody></table>
<table><caption>Table 5: Proportions of neutral generations that are labeled as English.</caption><thead><tr><th>Rank</th><th>Model</th><th>EN (%)</th></tr></thead><tbody><tr><td>1</td><td>mistral-7b-v0.3</td><td>40.16</td></tr><tr><td>2</td><td>llama-3.2-3b</td><td>15.32</td></tr><tr><td>3</td><td>mistral-7b-instruct-v0.3</td><td>3.65</td></tr><tr><td>4</td><td>gemma-2-2b-it</td><td>1.11</td></tr><tr><td>5</td><td>croissant-it</td><td>0.14</td></tr><tr><td>6</td><td>llama-3.2-3b-it</td><td>0.08</td></tr><tr><td>7</td><td>vigogne-2-7b</td><td>0.06</td></tr><tr><td>8</td><td>croissantbase</td><td>0.06</td></tr><tr><td>9</td><td>gpt2-fr</td><td>0.04</td></tr><tr><td>10</td><td>bloom-7b</td><td>0.02</td></tr><tr><td>11</td><td>gemma-2-2b</td><td>0.02</td></tr><tr><td>12</td><td>bloom-560m</td><td>0.00</td></tr><tr><td>13</td><td>bloom-3b</td><td>0.00</td></tr><tr><td>14</td><td>xglm-2</td><td>0.00</td></tr></tbody></table>

<table><caption>Table 6: Proportions of gendered generations that are labeled as English.</caption><thead><tr><th>Rank</th><th>Model</th><th>EN (%)</th></tr></thead><tbody><tr><td>1</td><td>mistral-7b-v0.3</td><td>48.71</td></tr><tr><td>2</td><td>llama-3.2-3b</td><td>13.14</td></tr><tr><td>3</td><td>mistral-7b-instruct-v0.3</td><td>8.98</td></tr><tr><td>4</td><td>gemma-2-2b-it</td><td>8.64</td></tr><tr><td>5</td><td>vigogne-2-7b</td><td>0.34</td></tr><tr><td>6</td><td>croissantbase</td><td>0.14</td></tr><tr><td>7</td><td>gemma-2-2b</td><td>0.12</td></tr><tr><td>8</td><td>gpt2-fr</td><td>0.04</td></tr><tr><td>9</td><td>llama-3.2-3b-it</td><td>0.04</td></tr><tr><td>10</td><td>bloom-560m</td><td>0.00</td></tr><tr><td>11</td><td>bloom-3b</td><td>0.00</td></tr><tr><td>12</td><td>xglm-2</td><td>0.00</td></tr><tr><td>13</td><td>bloom-7b</td><td>0.00</td></tr><tr><td>14</td><td>croissant-it</td><td>0.00</td></tr></tbody></table>