<figure><img src="image_1.png" /></figure>

HAL
open science

# CroissantLLM: A Truly Bilingual French-English Language Model

Manuel Faysse, Patrick Fernandes, Nuno Guerreiro, Antonio Loison, Duarte Alves, Caio Corro, Nicolas Boizard, Jaoe Alves, Ricardo Rei, Pedro Raphaël Martins, et al.

<figure><img src="image_2.png" /></figure>

## To cite this version:

Manuel Faysse, Patrick Fernandes, Nuno Guerreiro, Antonio Loison, Duarte Alves, et al., CroissantLLM: A Truly Bilingual French-English Language Model. 2024. [hal-04574908](https://hal.science/hal-04574908)

HAL Id: hal-04574908

https://hal.science/hal-04574908

Preprint submitted on 14 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
# CroissantLLM: A Truly Bilingual French-English Language Model

Manuel Faysse, Patrick Fernandes, Nuno Guerreiro, Antonio Loison, Duarte Alves, Caio Corro, Nicolas Boizard, Jaoe Alves, Ricardo Rei, Pedro Raphaël Martins, et al.

► To cite this version:

Manuel Faysse, Patrick Fernandes, Nuno Guerreiro, Antonio Loison, Duarte Alves, et al. Crois-santLLM: A Truly Bilingual French-English Language Model. Transactions of Machine Learning Research, 2025. hal-04574908

HAL Id: hal-04574908

https://hal.science/hal-04574908v1

Submitted on 14 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

# CroissantLLM: A Truly Bilingual French-English Language Model

Manuel Faysse¹,⁵

Patrick Fernandes⁶,⁸,¹¹

Nuno M. Guerreiro²,⁵,⁶,⁸

António Loison¹

Duarte M. Alves⁶,⁸

Caio Corro⁹

Nicolas Boizard⁴,⁵

João Alves²

Ricardo Rei²,⁷,⁸

Pedro H. Martins²

Antoni Bigata Casademunt¹⁰

François Yvon⁹

André F.T. Martins²,⁶,⁸

Gautier Viaud¹

Céline Hudelot⁵

Pierre Colombo³,⁵

¹Illuin Technology

²Unbel

³Equall

⁴Diabolocom

⁵MICS, CentraleSupélec, Université Paris-Saclay

⁶Instituto de Telecomunicações, Lisboa

⁷INESC-ID, Lisboa

⁸Instituto Superior Técnico, Universidade de Lisboa (Lisbon ELLIS Unit)

⁹Sorbonne Université, CNRS, ISIR, Paris

¹⁰Imperial College London

¹¹Language Technologies Institute, Carnegie Mellon University,

manuel.faysse@centralesupelec.fr

## Abstract

We introduce CroissantLLM, a 1.3B language model pre-trained on a set of 3T English and French tokens, to bring to the research and industrial community a high-performance, fully open-sourced bilingual model that runs swiftly on consumer-grade local hardware. To that end, we pioneer the approach of training an intrinsically bilingual model with a 1:1 English-to-French pretraining data ratio, a custom tokenizer, and bilingual finetuning datasets. We release the training dataset, notably containing a French split with manually curated, high-quality, and varied data sources. To assess performance outside of English, we craft a novel benchmark, FrenchBench, consisting of an array of classification and generation tasks, covering various orthogonal aspects of model performance in the French Language. Additionally, rooted in transparency and to foster further Large Language Model research, we release codebases, and dozens of checkpoints across various model sizes, training data distributions, and training steps, as well as fine-tuned Chat models, and strong translation models. We evaluate our model through the FMTI framework (Bommasani et al., 2023) and validate 81% of the transparency criteria, far beyond the scores of even most open initiatives. This work enriches the NLP landscape, breaking away from previous English-centric work to strengthen our understanding of multilingualism in language models.
<figure><img src="image_2.png" /></figure>

# CroissantLLM: A Truly Bilingual French-English Language Model

Manuel Faysse¹,⁵

Patrick Fernandes⁶,⁸,¹¹

Nuno M. Guerreiro²,⁵,⁶,⁸

António Loison¹

Duarte M. Alves⁶,⁸

Caio Corro⁹

João Alves²

Ricardo Rei²,⁷,⁸

Pedro H. Martins²

Antoni Bigata Casademunt¹⁰

François Yvon⁹

André F.T. Martins²,⁶,⁸

Gautier Viaud¹

Céline Hudelot⁵

Pierre Colombo³,⁵

¹Illuin Technology

²Unbelab

³Equall

⁴Diabolocom

⁵MICS, CentraleSupélec, Université Paris-Saclay

⁶Instituto de Telecomunicações, Lisboa

⁷INESC-ID, Lisboa

⁸Instituto Superior Técnico, Universidade de Lisboa (Lisbon ELLIS Unit)

⁹Sorbonne Université, CNRS, ISIR, Paris

¹⁰Imperial College London

¹¹Language Technologies Institute, Carnegie Mellon University,

manuel.faysse@centralesupelec.fr

## Abstract

We introduce CroissantLLM, a 1.3B language model pre-trained on a set of 3T English and French tokens, to bring to the research and industrial community a high-performance, fully open-sourced bilingual model that runs swiftly on consumer-grade local hardware. To that end, we pioneer the approach of training an intrinsically bilingual model with a 1:1 English-to-French pretraining data ratio, a custom tokenizer, and bilingual finetuning datasets. We release the training dataset, notably containing a French split with manually curated, high-quality, and varied data sources. To assess performance outside of English, we craft a novel benchmark, FrenchBench, consisting of an array of classification and generation tasks, covering various orthogonal aspects of model performance in the French Language. Additionally, rooted in transparency and to foster further Large Language Model research, we release codebases, and dozens of checkpoints across various model sizes, training data distributions, and training steps, as well as fine-tuned Chat models, and strong translation models. We evaluate our model through the FMTI framework (Bommasani et al., 2023) and validate 81% of the transparency criteria, far beyond the scores of even most open initiatives. This work enriches the NLP landscape, breaking away from previous English-centric work to strengthen our understanding of multilingualism in language models.
# 1 Introduction

Large Language Models (LLM) have taken over the Natural Language Processing (NLP) landscape in the past years. Although a few proprietary models are still considered to run ahead of the pack (OpenAI et al., 2023), open weights models such as Llama (Touvron et al., 2023a,b), Qwen (Bai et al., 2023a) or Mistral (Jiang et al., 2023, 2024) are rapidly bridging the performance gap. However, widespread industrial and research adoption of such technology remains challenging for several reasons, including the lack of transparency in the data collection and training processes, the scarcity of existing resources outside of English, and the large-scale and costly nature of existing high-performing models.

**Lack of transparency.** State-of-the-art models, both proprietary and open-weights are often designed and trained by heavily investor-backed companies, that aim to retain a moat by keeping their training data mix and strategy secret, hindering the rest of the field's ability to fully study and understand these models. This lack of transparency, ranging from training set composition to lack of evaluation or unclear usage policies, has been characterized by previous works, such as those by Bommasani et al. (2023) and Casper et al. (2024), pushing for full transparency as a key component for safe model development and use. The dangers of closed, non-auditable datasets have been exemplified by recent findings showcasing the potential dangers of dataset contamination, whether intentional (Hubinger et al., 2024) or not.¹ Furthermore, legal questions arise surrounding data ownership in LLM training corpora (NewYorkTimes, 2023; Samuelson, 2023) and recent developments in the political landscape, regarding AI (EU AI Act, US senate hearings)² have further emphasized the importance of transparent approaches, both from a legal perspective and to build user trust.

**Bias towards English.** Although the exact training mix of most well-performing LLMs is not publicly available information, most large models are trained on very English-centric corpora (Touvron et al., 2023a). This is the consequence of the important amount of English resources compared to other languages, both in terms of data availability and benchmarks. As could be expected, all publicly available LLMs display a large performance disparity between English and non-English languages when evaluated on downstream tasks (Bildarkar et al., 2023). Moreover, cultural knowledge and biases are mechanically integrated through the training data, leading to models with greater knowledge of American events or biases (Bender et al., 2021; Santurkar et al., 2023). This puts non-English users at a disadvantage when it comes to language model usage and adoption. While the non-English NLP community has produced multilingual datasets (Soboleva et al., 2023; Laurençon et al., 2023) and models (Scao et al., 2022; Shliazhko et al., 2022) in the last few years, the available resources still largely lag behind English ones, hindering industrial adoption in non-English settings.

**Challenging to use at scale.** Although benefits of scaling models to enormous sizes have been amply demonstrated in terms of performance (Hoffmann et al., 2022; Wei et al., 2022b; Chowdhery et al., 2022), scale comes at a large cost in terms of hardware requirements and inference speed. Download statistics on HuggingFace show the smallest Llama model (Touvron et al., 2023a) to be the most adopted by the community, demonstrating the interest in small but capable models. LLM scaling laws demonstrate the diminishing returns of training a model of a given size past a certain amount of tokens. By continuing pre-training way past the compute-optimal threshold, performance has been shown not to saturate, enabling the training of "inference-optimal" models of high interest to the industrial and

¹https://purl.stanford.edu/kh752sm9123
²https://www.commerce.senate.gov/2023/9/the-need-for-transparency-in-artificial-intelligence
# 1 Introduction

Large Language Models (LLM) have taken over the Natural Language Processing (NLP) landscape in the past years. Although a few proprietary models are still considered to run ahead of the pack (OpenAI et al., 2023), open weights models such as Llama (Touvron et al., 2023a,b), Qwen (Bai et al., 2023a) or Mistral (Jiang et al., 2023, 2024) are rapidly bridging the performance gap. However, widespread industrial and research adoption of such technology remains challenging for several reasons, including the lack of transparency in the data collection and training processes, the scarcity of existing resources outside of English, and the large-scale and costly nature of existing high-performing models.

**Lack of transparency.** State-of-the-art models, both proprietary and open-weights are often designed and trained by heavily investor-backed companies, that aim to retain a moat by keeping their training data mix and strategy secret, hindering the rest of the field's ability to fully study and understand these models. This lack of transparency, ranging from training set composition to lack of evaluation or unclear usage policies, has been characterized by previous works, such as those by Bommasani et al. (2023) and Casper et al. (2024), pushing for full transparency as a key component for safe model development and use. The dangers of closed, non-auditable datasets have been exemplified by recent findings showcasing the potential dangers of dataset contamination, whether intentional (Hubinger et al., 2024) or not.<fnref n="1" /> Furthermore, legal questions arise surrounding data ownership in LLM training corpora (NewYorkTimes, 2023; Samuelson, 2023) and recent developments in the political landscape, regarding AI (EU AI Act, US senate hearings)² have further emphasized the importance of transparent approaches, both from a legal perspective and to build user trust.

**Bias towards English.** Although the exact training mix of most well-performing LLMs is not publicly available information, most large models are trained on very English-centric corpora (Touvron et al., 2023a). This is the consequence of the important amount of English resources compared to other languages, both in terms of data availability and benchmarks. As could be expected, all publicly available LLMs display a large performance disparity between English and non-English languages when evaluated on downstream tasks (Bildarkar et al., 2023). Moreover, cultural knowledge and biases are mechanically integrated through the training data, leading to models with greater knowledge of American events or biases (Bender et al., 2021; Santurkar et al., 2023). This puts non-English users at a disadvantage when it comes to language model usage and adoption. While the non-English NLP community has produced multilingual datasets (Soboleva et al., 2023; Laurençon et al., 2023) and models (Scao et al., 2022; Shliazhko et al., 2022) in the last few years, the available resources still largely lag behind English ones, hindering industrial adoption in non-English settings.

**Challenging to use at scale.** Although benefits of scaling models to enormous sizes have been amply demonstrated in terms of performance (Hoffmann et al., 2022; Wei et al., 2022b; Chowdhery et al., 2022), scale comes at a large cost in terms of hardware requirements and inference speed. Download statistics on HuggingFace show the smallest Llama model (Touvron et al., 2023a) to be the most adopted by the community, demonstrating the interest in small but capable models. LLM scaling laws demonstrate the diminishing returns of training a model of a given size past a certain amount of tokens. By continuing pre-training way past the compute-optimal threshold, performance has been shown not to saturate, enabling the training of "inference-optimal" models of high interest to the industrial and

<footnote n="1">¹https://purl.stanford.edu/kh752sm9123
²https://www.commerce.senate.gov/2023/9/the-need-for-transparency-in-artificial-intelligence</footnote>
<figure><img src="image_4.png" /><figcaption>Figure 1: Conversation example with CroissantLLMChat</figcaption></figure>

research communities (Sardana and Frankle, 2023). It is still not fully understood how model performance continues to improve at these later stages of training.

## 1.1 Contributions

In this work, we attempt to bridge the aforementioned limitations and gaps. Our main contributions can be summarized as follows.

**Contribution 1: Introduction of a highly-curated, diverse corpus in French.** We collect and openly release a 303B token corpus spanning internet data, but also literary work, speech transcripts, legal and administrative documents, scientific articles, business-related documents, etc. The corpus is distributed under permissive licenses, allowing commercial use with no restriction, and is heavily filtered, curated, and deduplicated. To our knowledge, it is the largest multi-source French language corpus released to date of sufficient quality for language modeling purposes<fnref n="3" />.

**Contribution 2: Training CroissantLLM, a truly bilingual language model.** Nowadays, most models display some multilingual capabilities. For example, Bloom has been trained to be massively multilingual (Scao et al., 2022), Llama contains a minor proportion of non-English data in its training set (Touvron et al., 2023a) and Qwen included a significative portion of Chinese data (Bai et al., 2023a).⁴ However, to our knowledge, outside of Chinese with a different alphabet (Zeng et al., 2022), no work has studied or attempted to train a multilingual model of significant scale in which English is not the dominant training language.

In this work, we train a model on a 1:1 ratio of English to French with a tokenizer optimized for bilingualism. Our end goal is to have a model less skewed towards English performance or cultural biases. We motivate this ratio by conducting careful experimentation on smaller model scales to uncover the trade-offs behind language distribution ratios. We opt for a strategy enabling a “best of both worlds” performance in these two languages, as empirically validated by scaling law experiments. Experimentally, we show the importance of integrating data from another cultural source in the acquisition of cultural knowledge, underlining the importance of the effort.

**Contribution 3: FrenchBench: A novel LLM benchmark for the French Language.** To evaluate models in French, we introduce a benchmark encompassing various tasks to assess

<footnote n="3">³By this, we imply of sufficient quality to train a language model (little OCR errors, high-quality text) and not exclusively composed of Internet data.

⁴Qwen-VL (Bai et al., 2023b) reports 77.3% & 22.7% Chinese, but no information is given for the Qwen base model.</footnote>
<figure><img src="image_3.png" /><figcaption>Figure 1: Conversation example with CroissantLLMChat</figcaption></figure>

research communities (Sardana and Frankle, 2023). It is still not fully understood how model performance continues to improve at these later stages of training.

## 1.1 Contributions

In this work, we attempt to bridge the aforementioned limitations and gaps. Our main contributions can be summarized as follows.

**Contribution 1: Introduction of a highly-curated, diverse corpus in French.** We collect and openly release a 303B token corpus spanning internet data, but also literary work, speech transcripts, legal and administrative documents, scientific articles, business-related documents, etc. The corpus is distributed under permissive licenses, allowing commercial use with no restriction, and is heavily filtered, curated, and deduplicated. To our knowledge, it is the largest multi-source French language corpus released to date of sufficient quality for language modeling purposes<fnref n="3" />.

**Contribution 2: Training CroissantLLM, a truly bilingual language model.** Nowadays, most models display some multilingual capabilities. For example, Bloom has been trained to be massively multilingual (Scao et al., 2022), Llama contains a minor proportion of non-English data in its training set (Touvron et al., 2023a) and Qwen included a significative portion of Chinese data (Bai et al., 2023a).⁴ However, to our knowledge, outside of Chinese with a different alphabet (Zeng et al., 2022), no work has studied or attempted to train a multilingual model of significant scale in which English is not the dominant training language.

In this work, we train a model on a 1:1 ratio of English to French with a tokenizer optimized for bilingualism. Our end goal is to have a model less skewed towards English performance or cultural biases. We motivate this ratio by conducting careful experimentation on smaller model scales to uncover the trade-offs behind language distribution ratios. We opt for a strategy enabling a “best of both worlds” performance in these two languages, as empirically validated by scaling law experiments. Experimentally, we show the importance of integrating data from another cultural source in the acquisition of cultural knowledge, underlining the importance of the effort.

**Contribution 3: FrenchBench: A novel LLM benchmark for the French Language.** To evaluate models in French, we introduce a benchmark encompassing various tasks to assess

<footnote n="3">³By this, we imply of sufficient quality to train a language model (little OCR errors, high-quality text) and not exclusively composed of Internet data.

⁴Qwen-VL (Bai et al., 2023b) reports 77.3% & 22.7% Chinese, but no information is given for the Qwen base model.</footnote>
factual knowledge, generative capabilities, language understanding, etc. This benchmark
is constructed both from openly available datasets, as well as newly released manually
annotated data. We evaluate and report results for our models as well as other models with
French-speaking capabilities.⁵⁶

**Contribution 4: Releasing high-performing, inference-optimal models for the industrial community, together with a wide range of resources for the research community.** The models we train are all released under open licenses. Our largest model is trained on a 3000:1 token to parameter ratio (150 times longer than a Chinchilla Optimal 1.3B model) leading to very strong performance in its size category. We show that model performance on downstream tasks continues to dramatically improve with lengthened pre-training runs, although model perplexity does not significantly improve. We release all checkpoints for all model sizes, as well as the exact training data seen at every step for research purposes.<fnref n="7" /> These models are extremely efficient to run, leading to low-latency, energy-efficient on-edge inference, even on low-resource devices such as phones or personal computers. These releases<fnref n="8" /> are motivated by a commitment to transparency to allow research and reassuring users for industrial deployment: our models comply with 81% of criteria listed by the Foundation Model Transparency Index (Bommasani et al., 2023) (see Section 6).

## 2 Data

The dominance of the English language in the training data of most current models is undeniable. While multilingual models like Llama leverage some non-English data (Touvron et al., 2023a), it corresponds to only a minor part of the corpus, leading to a significant drop in performance across non-English data, with noticeable "American" bias (Santurkar et al., 2023; Navigli et al., 2023). This work aims to offset this trend by using a more balanced bilingual corpus comprising English and French, as well as additional code data. Although both languages belong to the Indo-European language family, they exhibit different morpho-syntactic structures<fnref n="9" /> and French has a richer morphology.<fnref n="10" /> We study whether this corpus helps in reducing biases, enabling more varied knowledge sets, and unlocking non-English performance.

A variety of sources are integrated into our corpus, including carefully filtered internet data and high-quality data from a range of sources, all devoid of restrictive licenses ensuring

<footnote n="5">⁵Code for evaluation is currently available at https://github.com/EleutherAI/lm-evaluation-harness</footnote>

<footnote n="6">⁶Another complementary initiative has been led for French model evaluation and released concurrently in Bawden et al. (2024)</footnote>

<footnote n="7">⁷Training data indexing order will be released in a second stage.</footnote>

<footnote n="8">⁸Code for dataset collection and filtering is available at https://github.com/ManuelFay/llm-data-hub. Code for model training is hosted at https://github.com/CoderPat/croissant-llm-training. Datasets and model checkpoints are available at https://huggingface.co/CroissantLLM.</footnote>

<footnote n="9">⁹For example, pronominal objects are placed before (resp. after) the verb in French (resp. English), both languages have different noun phrase constructions ("la commission européenne" vs. "the European commission"), etc.</footnote>

<footnote n="10">¹⁰For example, English has 5 verb forms whereas French has 48, French has explicit inflections for grammatical genders, etc. However, note that only English adjectives have morphological constructions for expressing comparison (e.g. easy, easier, easiest). We refer to WALS for more details, e.g. https://wals.info/feature/21B#2/26.7/152.4</footnote>
factual knowledge, generative capabilities, language understanding, etc. This benchmark
is constructed both from openly available datasets, as well as newly released manually
annotated data. We evaluate and report results for our models as well as other models with
French-speaking capabilities.⁵⁶

**Contribution 4: Releasing high-performing, inference-optimal models for the industrial community, together with a wide range of resources for the research community.** The models we train are all released under open licenses. Our largest model is trained on a 3000:1 token to parameter ratio (150 times longer than a Chinchilla Optimal 1.3B model) leading to very strong performance in its size category. We show that model performance on downstream tasks continues to dramatically improve with lengthened pre-training runs, although model perplexity does not significantly improve. We release all checkpoints for all model sizes, as well as the exact training data seen at every step for research purposes.<fnref n="7" /> These models are extremely efficient to run, leading to low-latency, energy-efficient on-edge inference, even on low-resource devices such as phones or personal computers. These releases<fnref n="8" /> are motivated by a commitment to transparency to allow research and reassuring users for industrial deployment: our models comply with 81% of criteria listed by the Foundation Model Transparency Index (Bommasani et al., 2023) (see Section 6).

## 2 Data

The dominance of the English language in the training data of most current models is undeniable. While multilingual models like Llama leverage some non-English data (Touvron et al., 2023a), it corresponds to only a minor part of the corpus, leading to a significant drop in performance across non-English data, with noticeable "American" bias (Santurkar et al., 2023; Navigli et al., 2023). This work aims to offset this trend by using a more balanced bilingual corpus comprising English and French, as well as additional code data. Although both languages belong to the Indo-European language family, they exhibit different morpho-syntactic structures<fnref n="9" /> and French has a richer morphology.<fnref n="10" /> We study whether this corpus helps in reducing biases, enabling more varied knowledge sets, and unlocking non-English performance.

A variety of sources are integrated into our corpus, including carefully filtered internet data and high-quality data from a range of sources, all devoid of restrictive licenses ensuring

<footnote n="5">⁵Code for evaluation is currently available at https://github.com/EleutherAI/lm-evaluation-harness</footnote>

<footnote n="6">⁶Another complementary initiative has been led for French model evaluation and released concurrently in Bawden et al. (2024)</footnote>

<footnote n="7">⁷Training data indexing order will be released in a second stage.</footnote>

<footnote n="8">⁸Code for dataset collection and filtering is available at https://github.com/ManuelFay/llm-data-hub. Code for model training is hosted at https://github.com/CoderPat/croissant-llm-training. Datasets and model checkpoints are available at https://huggingface.co/CroissantLLM.</footnote>

<footnote n="9">⁹For example, pronominal objects are placed before (resp. after) the verb in French (resp. English), both languages have different noun phrase constructions ("la commission européenne" vs. "the European commission"), etc.</footnote>

<footnote n="10">¹⁰For example, English has 5 verb forms whereas French has 48, French has explicit inflections for grammatical genders, etc. However, note that only English adjectives have morphological constructions for expressing comparison (e.g. easy, easier, easiest). We refer to WALS for more details, e.g. https://wals.info/feature/21B#2/26.7/152.4</footnote>
<table><caption>Table 1: Final Data mix for CroissantLLM training</caption><thead><tr><th></th><th>Size (GB)</th><th>Docs. (M)</th><th>Tokens (B)</th><th>Token/Doc</th><th>Sampling Ratio</th><th># tokens (B)</th></tr></thead><tbody><tr><td>French</td><td>1258.70</td><td>376.27</td><td>303.51</td><td>806.63</td><td>4.09</td><td>1240.08</td></tr><tr><td>English</td><td>2351.13</td><td>591.23</td><td>655.64</td><td>1108.94</td><td>1.89</td><td>1240.09</td></tr><tr><td>Code</td><td>366.87</td><td>81.90</td><td>141.43</td><td>1726.76</td><td>2.04</td><td>288.92</td></tr><tr><td>Parallel</td><td>113.91</td><td>408.03</td><td>35.78</td><td>87.68</td><td>6.13</td><td>219.26</td></tr><tr><td><strong>Total</strong></td><td>4090.61</td><td>1457.43</td><td>1136.35</td><td>779.70</td><td>14.15</td><td>2988.35</td></tr></tbody></table>

complete openness of the data and the trained model. Data statistics are available in Table 1.<fnref n="11" />

The scrapping and processing code are available in our code base.<fnref n="12" /> The license information of all datasets used is given, all allowing for permissive commercial use.

## 2.1 French Data

Table 10 lists the source and some information regarding the French corpus. Details about the data sources are expanded further below.

**Web Data.** We collect internet data from various web scraps (Oscar (Abadji et al., 2022), mC4 (Xue et al., 2021)), leveraging the CulturaX corpus (Nguyen et al., 2023) for heuristic and perplexity filtering, as well as exact and fuzzy deduplication. In total, this represents over 363 million webpages and more than 292 billion tokens, that we split using our custom tokenizer fitted on equal amounts of French and English data.<fnref n="13" />

We ensure data is of good quality and correctly tagged in French through sampled manual inspection, and confirm French-speaking countries are well represented within the dataset. Notably, we include several news sources scrapped from Belgium, Switzerland, Canada, and Lebanon, as well as multiple African countries (Senegal, Morocco, Algeria, Cameroon, etc.)

**Legal and Administrative Data.** We introduce 5.3B tokens of data from the French government's open data initiative, ranging from legal tasks to parliamentary discussions and financial transcripts (e.g. legal and administrative jargon). These texts originate from 13 different datasets (the OpenData corpus) and were collected from the French government's open data platform.<fnref n="14" /> To ensure other French-speaking countries are represented, we add 68M tokens of data from Swiss legislation retrieved from government sources. We perform steps to process, filter, and run exact deduplication on these documents.

<footnote n="11">¹¹As further detailed in 3.4, our data corpus contains different amounts of unique English, French, and Code tokens. We obtain our balanced training corpus by upsampling French, Code, and English data with different sampling ratios, such that no performance loss is to be expected (Muennighoff et al., 2023).</footnote>

<footnote n="12">¹²https://github.com/ManuelFay/llm-data-hub</footnote>

<footnote n="13">¹³The mC4 corpus https://huggingface.co/datasets/allenai/c4 is released under the ODC-BY licence https://opendatacommons.org/licenses/by/1.0/ whereas Oscar (Abadji et al., 2022) does not claim data ownership, provides an opt-out strategy for data inclusion, and filtering metadata is released under the Creative Commons CC0 license. https://creativecommons.org/publicdomain/zero/1.0/</footnote>

<footnote n="14">¹⁴Data is released at https://echanges.dila.gouv.fr/OPENDATA/ with the ETALAB open license https://www.etalab.gouv.fr/wp-content/uploads/2017/04/ETALAB-Licence-Ouverte-v2.0.pdf</footnote>
<table><caption>Table 1: Final Data mix for CroissantLLM training</caption><thead><tr><th></th><th>Size (GB)</th><th>Docs. (M)</th><th>Tokens (B)</th><th>Token/Doc</th><th>Sampling Ratio</th><th># tokens (B)</th></tr></thead><tbody><tr><td>French</td><td>1258.70</td><td>376.27</td><td>303.51</td><td>806.63</td><td>4.09</td><td>1240.08</td></tr><tr><td>English</td><td>2351.13</td><td>591.23</td><td>655.64</td><td>1108.94</td><td>1.89</td><td>1240.09</td></tr><tr><td>Code</td><td>366.87</td><td>81.90</td><td>141.43</td><td>1726.76</td><td>2.04</td><td>288.92</td></tr><tr><td>Parallel</td><td>113.91</td><td>408.03</td><td>35.78</td><td>87.68</td><td>6.13</td><td>219.26</td></tr><tr><td><strong>Total</strong></td><td>4090.61</td><td>1457.43</td><td>1136.35</td><td>779.70</td><td>14.15</td><td>2988.35</td></tr></tbody></table>

complete openness of the data and the trained model. Data statistics are available in Table 1.<fnref n="11" />

The scrapping and processing code are available in our code base.<fnref n="12" /> The license information of all datasets used is given, all allowing for permissive commercial use.

## 2.1 French Data

Table 10 lists the source and some information regarding the French corpus. Details about the data sources are expanded further below.

**Web Data.** We collect internet data from various web scraps (Oscar (Abadji et al., 2022), mC4 (Xue et al., 2021)), leveraging the CulturaX corpus (Nguyen et al., 2023) for heuristic and perplexity filtering, as well as exact and fuzzy deduplication. In total, this represents over 363 million webpages and more than 292 billion tokens, that we split using our custom tokenizer fitted on equal amounts of French and English data.<fnref n="13" />

We ensure data is of good quality and correctly tagged in French through sampled manual inspection, and confirm French-speaking countries are well represented within the dataset. Notably, we include several news sources scrapped from Belgium, Switzerland, Canada, and Lebanon, as well as multiple African countries (Senegal, Morocco, Algeria, Cameroon, etc.)

**Legal and Administrative Data.** We introduce 5.3B tokens of data from the French government's open data initiative, ranging from legal tasks to parliamentary discussions and financial transcripts (e.g. legal and administrative jargon). These texts originate from 13 different datasets (the OpenData corpus) and were collected from the French government's open data platform.<fnref n="14" /> To ensure other French-speaking countries are represented, we add 68M tokens of data from Swiss legislation retrieved from government sources. We perform steps to process, filter, and run exact deduplication on these documents.

<footnote n="11">¹¹As further detailed in 3.4, our data corpus contains different amounts of unique English, French, and Code tokens. We obtain our balanced training corpus by upsampling French, Code, and English data with different sampling ratios, such that no performance loss is to be expected (Muennighoff et al., 2023).</footnote>

<footnote n="12">¹²https://github.com/ManuelFay/llm-data-hub</footnote>

<footnote n="13">¹³The mC4 corpus https://huggingface.co/datasets/allenai/c4 is released under the ODC-BY licence https://opendatacommons.org/licenses/by/1.0/ whereas Oscar (Abadji et al., 2022) does not claim data ownership, provides an opt-out strategy for data inclusion, and filtering metadata is released under the Creative Commons CC0 license. https://creativecommons.org/publicdomain/zero/1.0/</footnote>

<footnote n="14">¹⁴Data is released at https://echanges.dila.gouv.fr/OPENDATA/ with the ETALAB open license https://www.etalab.gouv.fr/wp-content/uploads/2017/04/ETALAB-Licence-Ouverte-v2.0.pdf</footnote>
**Cultural Data.** We introduce cultural data from various sources. Notably, we retrieve all Project Gutenberg (Hart, 1971) books in the French language as of October 2023, corresponding to books released in the public domain (302 million tokens). We also download and aggressively filter manuscripts and documents from the French National Library (Bibliothèque Nationale de France), and filter for documents that belong to the public domain, have undergone an OCR process, and are of high quality.<fnref n="15" /> To filter out low-quality OCR documents, we implement custom heuristics which we release within our code base. We run all documents through perplexity filters using KenLM 5-grams¹⁶ fitted on the French Wikipedia split, and discard documents with perplexity values that are too high (noisy) or too low (repetitive patterns). Thresholds are set through a manual verification process. We deliberately choose to be aggressive in our filtering to ensure only high-quality data is kept and discard the largest portion of the original corpus, keeping about 27M tokens. We choose not to keep any data from the newspaper archives, as the OCR transcription is often too noisy. Additionally, we introduce famous public domain French poems custom scrapped from a French poetry website,¹⁷ and run a set of podcasts through a high-quality speech-to-text model to obtain a textual transcription. This process is hard to scale and data splits from these sources are limited in quantity. Data from the OpenSubtitles¹⁸ initiative is integrated, corresponding to 41.8 million tokens originating from movie subtitles.¹⁹ Finally, we add the French data from Wikisource collected as part of the BigScience initiative (Scao et al., 2022) and obtain 2.7 billion tokens from the process.²⁰

**Encyclopedia Data.** To introduce high-quality factual data to the training corpus, we integrate the French Wikipedia split from November 2023.²¹ This corresponds to the latest knowledge cutoff in the training data. In total, more than 2.5 million articles are used, spanning more than 2 billion tokens.

**Industrial Data.** We scrap high-quality and publicly available data from industrial PDFs via a manually crafted list of websites, from large French and French Canadian (Quebec) companies to government agencies. This business-focused data boosts performance on a series of downstream applications related to industrial NLP. We collect over 36000 PDF multi-page documents and filter them through carefully crafted heuristics, followed by aggressive perplexity filtering.²² In total, we obtain over 290000 documents and 191 million tokens.

## 2.2 English Data

Our English data is primarily drawn from the SlimPajama corpus (Soboleva et al., 2023), excluding copyrighted documents. Splits per data source are detailed in Table 11.

**Internet Data.** Similarly to the French dataset, we rely on carefully filtered content from an assortment of internet sources, including miscellaneous web pages and blogs. The filtering process includes heuristics and perplexity filtering, as well as large-scale deduplication

<footnote n="15">¹⁵Metadata is licensed under Open Etalab license https://gallica.bnf.fr/edit/und/conditions-dutilisation-des-contenus-de-gallica

¹⁶https://github.com/kpu/kenlm

¹⁷https://www.poesie-française.fr/

¹⁸https://opus.nlpl.eu/OpenSubtitles2016.php

¹⁹https://www.opensubtitles.org

²⁰Licensed under CC BY-SA 4.0, https://en.wikisource.org/wiki/Wikisource:Copyright_policy

²¹https://huggingface.co/datasets/wikimedia/wikipedia with a CC-By-SA 3.0 license

²²Data is public internet data that we release under MIT license with an opt-out scheme https://huggingface.co/datasets/manu/illuin_layout_dataset_text_only</footnote>
**Cultural Data.** We introduce cultural data from various sources. Notably, we retrieve all Project Gutenberg (Hart, 1971) books in the French language as of October 2023, corresponding to books released in the public domain (302 million tokens). We also download and aggressively filter manuscripts and documents from the French National Library (Bibliothèque Nationale de France), and filter for documents that belong to the public domain, have undergone an OCR process, and are of high quality.<fnref n="15" /> To filter out low-quality OCR documents, we implement custom heuristics which we release within our code base. We run all documents through perplexity filters using KenLM 5-grams¹⁶ fitted on the French Wikipedia split, and discard documents with perplexity values that are too high (noisy) or too low (repetitive patterns). Thresholds are set through a manual verification process. We deliberately choose to be aggressive in our filtering to ensure only high-quality data is kept and discard the largest portion of the original corpus, keeping about 27M tokens. We choose not to keep any data from the newspaper archives, as the OCR transcription is often too noisy. Additionally, we introduce famous public domain French poems custom scrapped from a French poetry website,¹⁷ and run a set of podcasts through a high-quality speech-to-text model to obtain a textual transcription. This process is hard to scale and data splits from these sources are limited in quantity. Data from the OpenSubtitles¹⁸ initiative is integrated, corresponding to 41.8 million tokens originating from movie subtitles.¹⁹ Finally, we add the French data from Wikisource collected as part of the BigScience initiative (Scao et al., 2022) and obtain 2.7 billion tokens from the process.²⁰

**Encyclopedia Data.** To introduce high-quality factual data to the training corpus, we integrate the French Wikipedia split from November 2023.²¹ This corresponds to the latest knowledge cutoff in the training data. In total, more than 2.5 million articles are used, spanning more than 2 billion tokens.

**Industrial Data.** We scrap high-quality and publicly available data from industrial PDFs via a manually crafted list of websites, from large French and French Canadian (Quebec) companies to government agencies. This business-focused data boosts performance on a series of downstream applications related to industrial NLP. We collect over 36000 PDF multi-page documents and filter them through carefully crafted heuristics, followed by aggressive perplexity filtering.²² In total, we obtain over 290000 documents and 191 million tokens.

## 2.2 English Data

Our English data is primarily drawn from the SlimPajama corpus (Soboleva et al., 2023), excluding copyrighted documents. Splits per data source are detailed in Table 11.

**Internet Data.** Similarly to the French dataset, we rely on carefully filtered content from an assortment of internet sources, including miscellaneous web pages and blogs. The filtering process includes heuristics and perplexity filtering, as well as large-scale deduplication

<footnote n="15">¹⁵Metadata is licensed under Open Etalab license https://gallica.bnf.fr/edit/und/conditions-dutilisation-des-contenus-de-gallica

¹⁶https://github.com/kpu/kenlm

¹⁷https://www.poesie-française.fr/

¹⁸https://opus.nlpl.eu/OpenSubtitles2016.php

¹⁹https://www.opensubtitles.org

²⁰Licensed under CC BY-SA 4.0, https://en.wikisource.org/wiki/Wikisource:Copyright_policy

²¹https://huggingface.co/datasets/wikimedia/wikipedia with a CC-By-SA 3.0 license

²²Data is public internet data that we release under MIT license with an opt-out scheme https://huggingface.co/datasets/manu/illuin_layout_dataset_text_only</footnote>
(Soboleva et al., 2023). The SlimPajama corpus includes internet data from the Common-Crawl¹²³ and C4²⁴ web scraps, as well as data sourced from Github textual content<fnref n="25" /> and the StackExchange forums.<fnref n="26" />

**Miscellaneous.** Other non-internet-based data sources are included in the SlimPajama dataset, such as scientific articles from Arxiv<fnref n="27" /> and English documents from Wikipedia.<fnref n="28" /> The SlimPajama dataset is also comprised of the “Books” subcorpus, obtained by downloading all book documents from Bibliotik.<fnref n="29" /> Some of the documents within this last corpora have been flagged by their owner as proprietary data. We filter out all documents from this subcorpus, and replace them with data from the open-source Project Gutenberg (Hart, 1971) English books under public domains.<fnref n="30" />

**Gutenberg Canaries.** To assess model memorization to inform about the risks of including private or sensitive data within the training set, we stress test the model by including “canaries” (Carlini et al., 2019). These correspond to samples that have been intentionally modified and/or repeated and included within the model training set, and that will enable a posteriori evaluation of the model capacity to memorize data in a “worse than worst-case” situation.<fnref n="31" /> In total the canaries represent 555 million tokens, representing less than 0.04 % of the total tokens seen during training.

## 2.3 Code Data

In line with most recent models (Chowdhery et al., 2022; Scao et al., 2022; Touvron et al., 2023a), we integrate code data into our training corpus. Notably, previous work shows that code data benefits natural language tasks and can be particularly useful in data-constrained settings (Muennighoff et al., 2023). Therefore, we include 140B tokens of code data in several common programming languages. Splits and number of tokens are detailed in Table 12.

**The Stack & StarCoder.** We rely on the efforts of the StarCoder project (Li et al., 2023a), and use their high-quality filtered code data from The Stack corpus (Kocetkov et al., 2022).<fnref n="32" /> We keep only high-resource programming languages (Java, Javascript, Python, C, C++, SQL) and Jupyter<fnref n="33" /> notebooks, as well as a few samples of formatted data (JSON, Tex) and scripting languages (shell scripts, Dockerfiles).<fnref n="34" />

**Extra Python code.** We extend the corpus with several other sources of Python code due to the popularity of the language in the community. Firstly, we add Pypi packages from recent

<footnote n="23">²³Common Crawl license https://commoncrawl.org/terms-of-use</footnote>

<footnote n="24">²⁴ODC-BY license</footnote>

<footnote n="25">²⁵Only content under MIT, BSD, and Apache licenses are kept</footnote>

<footnote n="26">²⁶CC-By-SA 4.0 license https://archive.org/details/stackexchange</footnote>

<footnote n="27">²⁷https://arxiv.org/, with author opt-out options</footnote>

<footnote n="28">²⁸CC-By-SA 3.0 license</footnote>

<footnote n="29">²⁹https://huggingface.co/datasets/the_pile-books3</footnote>

<footnote n="30">³⁰From https://huggingface.co/datasets/pg19 with an Apache 2.0 license</footnote>

<footnote n="31">³¹This work is led in parallel to the CroissantLLM project and findings will be independently published.</footnote>

<footnote n="32">³²Data are drawn from various sources and falls under 193 different permissive licenses. We use the version 1.2 of the corpus, which has been filtered with respect to data owners opt-out option information.</footnote>

<footnote n="33">³³https://jupyter.org/</footnote>

<footnote n="34">³⁴https://docs.docker.com/engine/reference/builder/</footnote>
(Soboleva et al., 2023). The SlimPajama corpus includes internet data from the Common-Crawl¹²³ and C4²⁴ web scraps, as well as data sourced from Github textual content<fnref n="25" /> and the StackExchange forums.<fnref n="26" />

**Miscellaneous.** Other non-internet-based data sources are included in the SlimPajama dataset, such as scientific articles from Arxiv<fnref n="27" /> and English documents from Wikipedia.<fnref n="28" /> The SlimPajama dataset is also comprised of the “Books” subcorpus, obtained by downloading all book documents from Bibliotik.<fnref n="29" /> Some of the documents within this last corpora have been flagged by their owner as proprietary data. We filter out all documents from this subcorpus, and replace them with data from the open-source Project Gutenberg (Hart, 1971) English books under public domains.<fnref n="30" />

**Gutenberg Canaries.** To assess model memorization to inform about the risks of including private or sensitive data within the training set, we stress test the model by including “canaries” (Carlini et al., 2019). These correspond to samples that have been intentionally modified and/or repeated and included within the model training set, and that will enable a posteriori evaluation of the model capacity to memorize data in a “worse than worst-case” situation.<fnref n="31" /> In total the canaries represent 555 million tokens, representing less than 0.04 % of the total tokens seen during training.

## 2.3 Code Data

In line with most recent models (Chowdhery et al., 2022; Scao et al., 2022; Touvron et al., 2023a), we integrate code data into our training corpus. Notably, previous work shows that code data benefits natural language tasks and can be particularly useful in data-constrained settings (Muennighoff et al., 2023). Therefore, we include 140B tokens of code data in several common programming languages. Splits and number of tokens are detailed in Table 12.

**The Stack & StarCoder.** We rely on the efforts of the StarCoder project (Li et al., 2023a), and use their high-quality filtered code data from The Stack corpus (Kocetkov et al., 2022).<fnref n="32" /> We keep only high-resource programming languages (Java, Javascript, Python, C, C++, SQL) and Jupyter<fnref n="33" /> notebooks, as well as a few samples of formatted data (JSON, Tex) and scripting languages (shell scripts, Dockerfiles).<fnref n="34" />

**Extra Python code.** We extend the corpus with several other sources of Python code due to the popularity of the language in the community. Firstly, we add Pypi packages from recent

<footnote n="23">²³Common Crawl license https://commoncrawl.org/terms-of-use</footnote>

<footnote n="24">²⁴ODC-BY license</footnote>

<footnote n="25">²⁵Only content under MIT, BSD, and Apache licenses are kept</footnote>

<footnote n="26">²⁶CC-By-SA 4.0 license https://archive.org/details/stackexchange</footnote>

<footnote n="27">²⁷https://arxiv.org/, with author opt-out options</footnote>

<footnote n="28">²⁸CC-By-SA 3.0 license</footnote>

<footnote n="29">²⁹https://huggingface.co/datasets/the_pile-books3</footnote>

<footnote n="30">³⁰From https://huggingface.co/datasets/pg19 with an Apache 2.0 license</footnote>

<footnote n="31">³¹This work is led in parallel to the CroissantLLM project and findings will be independently published.</footnote>

<footnote n="32">³²Data are drawn from various sources and falls under 193 different permissive licenses. We use the version 1.2 of the corpus, which has been filtered with respect to data owners opt-out option information.</footnote>

<footnote n="33">³³https://jupyter.org/</footnote>

<footnote n="34">³⁴https://docs.docker.com/engine/reference/builder/</footnote>
code dumps,<fnref n="35" /> that are filtered to keep only Python and Jupyter files.<fnref n="36" /> Secondly, in order to promote high-quality problem-solving-centered code, we integrate 1.2B tokens of Python3 data from competitive coding contests (Li et al., 2022).<fnref n="37" /> Lastly, following the success of learning from textbooks (Li et al., 2023b), we add commented Python code constructed by combining code and text cells from Jupyter Notebooks through the CodeParrot initiative.<fnref n="38" />

## 2.4 Parallel Data

Following previous work (Anil et al., 2023), we incorporate vast quantities of parallel data, in our case high-quality English-French translation pairs, in order to improve the multilingual capabilities of the model (Briakou et al., 2023).

**Opus.** We extract subsets of sentence pairs spanning multiple domains from the OPUS corpus (Tiedemann, 2012).<fnref n="39" /> Statistics are described in Table 10. In total, we include 400 million parallel sentences and about 36 billion tokens. The data is filtered through a rigorous cleaning pipeline: (1) BiFixer (Ramírez-Sánchez et al., 2020)<fnref n="40" /> is first used to remove duplicate data through fuzzy matching techniques; (2) BiCleaner<fnref n="41" /> is then used to filter data using heuristic and perplexity filters; (3) finally, the state-of-the-art NMT quality estimator CometKiwi (Rei et al., 2022b) is used to keep only top quality translation pairs.

**Theses.** To incorporate versatile academic and scientific language, we augment our dataset with French theses abstracts along with their author-generated English translations. This corresponds to 95000 documents and more than 80 million high-quality tokens.<fnref n="42" />

**Song Lyrics.** Our dataset integrates song lyrics in both French and English, scrapped from a specialized community-driven lyrics translation website.<fnref n="43" /> As such, our model is trained with radically different linguistic styles (e.g. colloquialism), and the wide range of topics can help the model to capture cultural nuances. Lyrics have been translated by hand by the website community. With a total of 70k songs, we have built up a corpus of 53M tokens covering different periods (80s, 90s, 20s, etc.) and musical genres (rap, rock, jazz, etc.) To preserve colloquial expressions and cultural subtleties, we have not filtered song lyrics for explicit content. We validate the original language metadata of the songs through Google's language-detection algorithm.

## 3 Training

Our main goal was to train a highly performing, yet resource-friendly bilingual model while optimizing performances across both languages. To focus on the specific challenges of the bilingual paradigm, we rely on previous work to motivate many of our design and hyperparameter choices (Touvron et al., 2023b).

<footnote n="35">³⁵https://py-code.org/datasets from permissive licensed code</footnote>

<footnote n="36">³⁶https://huggingface.co/datasets/vikp/pypi_clean</footnote>

<footnote n="37">³⁷Under CC-By-4.0 license</footnote>

<footnote n="38">³⁸https://huggingface.co/datasets/codeparrot.github-jupyter-code-to-text Apache 2.0 license</footnote>

<footnote n="39">³⁹Free license https://opus.nlpl.eu/</footnote>

<footnote n="40">⁴⁰https://github.com/bitextor/bifixer</footnote>

<footnote n="41">⁴¹https://github.com/bitextor/bicleaner</footnote>

<footnote n="42">⁴²Data licensed under Etalab open license https://www.etalab.gouv.fr/wp-content/uploads/2017/04/ETALAB-Licence-Ouverte-v2.0.pdf</footnote>

<footnote n="43">⁴³https://www.lacoccinelle.net, with opt-out options for data owners and community-contributed translations under free-use license.</footnote>
code dumps,<fnref n="35" /> that are filtered to keep only Python and Jupyter files.<fnref n="36" /> Secondly, in order to promote high-quality problem-solving-centered code, we integrate 1.2B tokens of Python3 data from competitive coding contests (Li et al., 2022).<fnref n="37" /> Lastly, following the success of learning from textbooks (Li et al., 2023b), we add commented Python code constructed by combining code and text cells from Jupyter Notebooks through the CodeParrot initiative.<fnref n="38" />

## 2.4 Parallel Data

Following previous work (Anil et al., 2023), we incorporate vast quantities of parallel data, in our case high-quality English-French translation pairs, in order to improve the multilingual capabilities of the model (Briakou et al., 2023).

**Opus.** We extract subsets of sentence pairs spanning multiple domains from the OPUS corpus (Tiedemann, 2012).<fnref n="39" /> Statistics are described in Table 10. In total, we include 400 million parallel sentences and about 36 billion tokens. The data is filtered through a rigorous cleaning pipeline: (1) BiFixer (Ramírez-Sánchez et al., 2020)<fnref n="40" /> is first used to remove duplicate data through fuzzy matching techniques; (2) BiCleaner<fnref n="41" /> is then used to filter data using heuristic and perplexity filters; (3) finally, the state-of-the-art NMT quality estimator CometKiwi (Rei et al., 2022b) is used to keep only top quality translation pairs.

**Theses.** To incorporate versatile academic and scientific language, we augment our dataset with French theses abstracts along with their author-generated English translations. This corresponds to 95000 documents and more than 80 million high-quality tokens.<fnref n="42" />

**Song Lyrics.** Our dataset integrates song lyrics in both French and English, scrapped from a specialized community-driven lyrics translation website.<fnref n="43" /> As such, our model is trained with radically different linguistic styles (e.g. colloquialism), and the wide range of topics can help the model to capture cultural nuances. Lyrics have been translated by hand by the website community. With a total of 70k songs, we have built up a corpus of 53M tokens covering different periods (80s, 90s, 20s, etc.) and musical genres (rap, rock, jazz, etc.) To preserve colloquial expressions and cultural subtleties, we have not filtered song lyrics for explicit content. We validate the original language metadata of the songs through Google's language-detection algorithm.

## 3 Training

Our main goal was to train a highly performing, yet resource-friendly bilingual model while optimizing performances across both languages. To focus on the specific challenges of the bilingual paradigm, we rely on previous work to motivate many of our design and hyperparameter choices (Touvron et al., 2023b).

<footnote n="35">³⁵https://py-code.org/datasets from permissive licensed code</footnote>

<footnote n="36">³⁶https://huggingface.co/datasets/vikp/pypi_clean</footnote>

<footnote n="37">³⁷Under CC-By-4.0 license</footnote>

<footnote n="38">³⁸https://huggingface.co/datasets/codeparrot.github-jupyter-code-to-text Apache 2.0 license</footnote>

<footnote n="39">³⁹Free license https://opus.nlpl.eu/</footnote>

<footnote n="40">⁴⁰https://github.com/bitextor/bifixer</footnote>

<footnote n="41">⁴¹https://github.com/bitextor/bicleaner</footnote>

<footnote n="42">⁴²Data licensed under Etalab open license https://www.etalab.gouv.fr/wp-content/uploads/2017/04/ETALAB-Licence-Ouverte-v2.0.pdf</footnote>

<footnote n="43">⁴³https://www.lacoccinelle.net, with opt-out options for data owners and community-contributed translations under free-use license.</footnote>
<table><caption>Table 2: Model information for scaling laws. Parameter count excludes embedding and output parameters.</caption><thead><tr><th>Model</th><th>Params (M)</th><th>Layers</th><th>Hidden size</th><th>Inter. size</th><th>KV heads</th></tr></thead><tbody><tr><td>XXS</td><td>100.7</td><td>6</td><td>1024</td><td>4096</td><td>8</td></tr><tr><td>XS</td><td>202.5</td><td>12</td><td>1024</td><td>4128</td><td>8</td></tr><tr><td>S</td><td>341.5</td><td>12</td><td>1536</td><td>4128</td><td>12</td></tr><tr><td>Base</td><td>1214.3</td><td>24</td><td>2048</td><td>5504</td><td>16</td></tr></tbody></table>

## 3.1 Model Architecture

We use the Llama architecture (Touvron et al., 2023a), a decoder-based transformer, trained with rotary position encodings (Su et al., 2023) and a context length of 2048. We construct 4 different model sizes by jointly scaling the number of attention heads, hidden size, and hidden layers. Table 2 summarizes the sizes of each model in the family.

## 3.2 Tokenizer

Most LLM tokenizers are fitted on English-centric corpora with an information-theoretic optimization objective, for example, Byte-Pair encoding (Sennrich et al., 2016) or Unigram (Kudo, 2018), leading to good fertility values (low token per word ratio) on English text, but high fertility in other languages. These phenomena make processing in other languages slower and more costly (Rust et al., 2021). Furthermore, subword splits in non-English languages mechanically carry less semantical meaning, potentially being a factor in the degraded performance of models on non-English languages (Rust et al., 2021).

**gb Tokenizer training.** We fit our CroissantLLM tokenizer on a corpus of 100B high-quality tokens, with splits of English, French, and code data. We use SentencePiece<fnref n="44" /> to train a Byte-Pair Encoding tokenizer with a vocabulary size of 32000 tokens, 100 special placeholder tokens, whitespace separation, and byte fallback, inspired by Touvron et al. (2023a); Jiang et al. (2023). The data corpus used to fit the tokenizer is made available,<fnref n="45" /> and notably contains large amounts of French data to skew the vocabulary construction process towards optimizing for French as well.

**Improved fertility rates.** The focus on English, French, and Code enables the CroissantLLM tokenizer to display smaller fertility rates on French texts than the Mistral and Llama models with similar vocabulary sizes, all the while also displaying slightly smaller rates than both in English and Code (Figure 2). This is due to the multilingual support of both Llama and Mistral tokenizers which need to allocate some vocabulary tokens to frequent character patterns from other languages. Roughly, the Llama tokenizer is 17% less token efficient at encoding French internet data, and up to 40% less efficient on clean encyclopedia French texts, implying that the 303B unique French tokens in our data training set correspond to more than 360B tokens with the Llama tokenizer. This enables us to pack more data in fewer tokens, leading to improvements in training and inference efficiency.

<footnote n="44">⁴⁴https://github.com/google/sentencepiece</footnote>

<footnote n="45">⁴⁵https://huggingface.co/datasets/manu/tok-corpus-shuffled</footnote>
<table><caption>Table 2: Model information for scaling laws. Parameter count excludes embedding and output parameters.</caption><thead><tr><th>Model</th><th>Params (M)</th><th>Layers</th><th>Hidden size</th><th>Inter. size</th><th>KV heads</th></tr></thead><tbody><tr><td>XXS</td><td>100.7</td><td>6</td><td>1024</td><td>4096</td><td>8</td></tr><tr><td>XS</td><td>202.5</td><td>12</td><td>1024</td><td>4128</td><td>8</td></tr><tr><td>S</td><td>341.5</td><td>12</td><td>1536</td><td>4128</td><td>12</td></tr><tr><td>Base</td><td>1214.3</td><td>24</td><td>2048</td><td>5504</td><td>16</td></tr></tbody></table>

## 3.1 Model Architecture

We use the Llama architecture (Touvron et al., 2023a), a decoder-based transformer, trained with rotary position encodings (Su et al., 2023) and a context length of 2048. We construct 4 different model sizes by jointly scaling the number of attention heads, hidden size, and hidden layers. Table 2 summarizes the sizes of each model in the family.

## 3.2 Tokenizer

Most LLM tokenizers are fitted on English-centric corpora with an information-theoretic optimization objective, for example, Byte-Pair encoding (Sennrich et al., 2016) or Unigram (Kudo, 2018), leading to good fertility values (low token per word ratio) on English text, but high fertility in other languages. These phenomena make processing in other languages slower and more costly (Rust et al., 2021). Furthermore, subword splits in non-English languages mechanically carry less semantical meaning, potentially being a factor in the degraded performance of models on non-English languages (Rust et al., 2021).

**gb Tokenizer training.** We fit our CroissantLLM tokenizer on a corpus of 100B high-quality tokens, with splits of English, French, and code data. We use SentencePiece<fnref n="44" /> to train a Byte-Pair Encoding tokenizer with a vocabulary size of 32000 tokens, 100 special placeholder tokens, whitespace separation, and byte fallback, inspired by Touvron et al. (2023a); Jiang et al. (2023). The data corpus used to fit the tokenizer is made available,<fnref n="45" /> and notably contains large amounts of French data to skew the vocabulary construction process towards optimizing for French as well.

**Improved fertility rates.** The focus on English, French, and Code enables the CroissantLLM tokenizer to display smaller fertility rates on French texts than the Mistral and Llama models with similar vocabulary sizes, all the while also displaying slightly smaller rates than both in English and Code (Figure 2). This is due to the multilingual support of both Llama and Mistral tokenizers which need to allocate some vocabulary tokens to frequent character patterns from other languages. Roughly, the Llama tokenizer is 17% less token efficient at encoding French internet data, and up to 40% less efficient on clean encyclopedia French texts, implying that the 303B unique French tokens in our data training set correspond to more than 360B tokens with the Llama tokenizer. This enables us to pack more data in fewer tokens, leading to improvements in training and inference efficiency.

<footnote n="44">⁴⁴https://github.com/google/sentencepiece</footnote>

<footnote n="45">⁴⁵https://huggingface.co/datasets/manu/tok-corpus-shuffled</footnote>
<figure><img src="image_5.png" /><figcaption>Figure 2: Fertility on unseen test sets using various tokenizers. Lower is better.</figcaption></figure>

3.3 Selecting an optimal language ratio

A crucial question when training a bilingual model is how to effectively *weight* data from the
two languages to achieve a good trade-off between performance in both. While, intuitively,
training on an equal mixture of English and French data may seem to be the obvious
solution, differences in data quality available for each language coupled with transfer
learning dynamics between both languages could imply that a balanced mix might be
sub-optimal. However, training multiple models with different data mixes for comparison
is prohibitively expensive.

To offset this computational cost, we leverage recent findings on scaling laws (Kaplan et al., 2020a) that show that we can predict the performance of our model by training smaller models on the same dataset. In particular, Fernandes et al. (2023) found that, for *multilingual* models, by training smaller models with varying weights for each language in the data mix, one can fit a *multilingual, joint* scaling law that can predict the language performance trade-off of larger models, even for novel language weightings not encountered during the fitting of the scaling law.

As such, we fit a joint scaling law as described by Fernandes et al. (2023) for each language, by training 3 smaller model sizes on 3 different data mixes with varied ratios of English and French data (keeping the amount of Code data fixed). The corpus for these scaling law experiments is a subsampled variant of the larger corpus and is detailed in Appendix B.5. We define 3 data mixes by varying the language sampling ratio: (1) **equal** containing 40% English data, 40% French data and 20% Code data; (2) **frplus** containing 20% English data, 60% French data and 20% Code data; and (3) **enplus** containing 60% English data, 20% French data and 20% Code data. We then trained a 1.3B model on these subsets of the data for one of the data mixes to validate their predictive power.

Figure 3 shows the performance predicted by jointly-fitted scaling laws as we scale the model and vary the language weightings on the Wiki data validation split. First, we see that the fitted scaling law is able to predict the performance of the larger model almost perfectly. Secondly, changing the weight of each language in training has a non-symmetrical impact on language performance: by increasing the (relative) weight of French from 50% to 75%, we get a marginal performance increase in French, while performance in English drops
<figure><img src="image_4.png" /><figcaption>Figure 2: Fertility on unseen test sets using various tokenizers. Lower is better.</figcaption></figure>

3.3 Selecting an optimal language ratio

A crucial question when training a bilingual model is how to effectively *weight* data from the
two languages to achieve a good trade-off between performance in both. While, intuitively,
training on an equal mixture of English and French data may seem to be the obvious
solution, differences in data quality available for each language coupled with transfer
learning dynamics between both languages could imply that a balanced mix might be
sub-optimal. However, training multiple models with different data mixes for comparison
is prohibitively expensive.

To offset this computational cost, we leverage recent findings on scaling laws (Kaplan et al., 2020a) that show that we can predict the performance of our model by training smaller models on the same dataset. In particular, Fernandes et al. (2023) found that, for *multilingual* models, by training smaller models with varying weights for each language in the data mix, one can fit a *multilingual, joint* scaling law that can predict the language performance trade-off of larger models, even for novel language weightings not encountered during the fitting of the scaling law.

As such, we fit a joint scaling law as described by Fernandes et al. (2023) for each language,
by training 3 smaller model sizes on 3 different data mixes with varied ratios of English
and French data (keeping the amount of Code data fixed). The corpus for these scaling law
experiments is a subsampled variant of the larger corpus and is detailed in Appendix B.5.
We define 3 data mixes by varying the language sampling ratio: (1) **equal** containing 40 %
English data, 40 % French data and 20 % Code data; (2) **frplus** containing 20 % English data,
60 % French data and 20 % Code data; and (3) **enplus** containing 60 % English data, 20 %
French data and 20 % Code data. We then trained a 1.3B model on these subsets of the data
for one of the data mixes to validate their predictive power.

Figure 3 shows the performance predicted by jointly-fitted scaling laws as we scale the
model and vary the language weightings on the Wiki data validation split. First, we see that
the fitted scaling law is able to predict the performance of the larger model almost perfectly.
Secondly, changing the weight of each language in training has a non-symmetrical impact
on language performance: by increasing the (relative) weight of French from 50 % to 75 %,
we get a marginal performance increase in French, while performance in English drops
<figure><img src="image_6.png" /><figcaption>Figure 3: Evolution of test cross-entropy loss with model size in English (left) and French (right), for the *wiki* domain, as well as the fitted joint scaling laws,</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 4: *Effective capacity ratio* (as predicted by our fitted joint scaling law) for English and French as we change the weight of each language.</figcaption></figure>

significantly. This fact is made clear by plotting the *effective capacity ratio*<fnref n="46" /> of each language as we change the language weight (Figure 4): the “gains” in parameters from increasing weight of French data are minimal past the 50% mark.

These findings showcase that **multilinguality comes at a price**, and training a bilingual model implies accepting a performance loss on a target language compared to an equivalent model trained on a monolingual corpus.

We find equal ratios of English and French data lead to minimized performance hits across both languages (Figure 3) and opt to train our base model in this data configuration.

## 3.4 Final data distribution

Our final dataset is composed of 1.1T unique tokens that originate from sources of various languages, qualities, and quantities. To craft a training set with a language and data distribution that suits our objectives, we upsample some of the sources, notably to balance out French and English data and increase the share of parallel data in our training run. Following work by Muennighoff et al. (2023) and Luukkonen et al. (2023) on data-constrained

<footnote n="46">⁴⁶The number of parameters that it would take a monolingual model to match the performance of the multilingual, divided by the size of the multilingual model. See Fernandes et al. (2023) for more details</footnote>
<figure><img src="image_5.png" /><figcaption>Figure 3: Evolution of test cross-entropy loss with model size in English (left) and French (right), for the *wiki* domain, as well as the fitted joint scaling laws,</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 4: *Effective capacity ratio* (as predicted by our fitted joint scaling law) for English and French as we change the weight of each language.</figcaption></figure>

significantly. This fact is made clear by plotting the *effective capacity ratio*<fnref n="46" /> of each language as we change the language weight (Figure 4): the “gains” in parameters from increasing weight of French data are minimal past the 50% mark.

These findings showcase that **multilinguality comes at a price**, and training a bilingual model implies accepting a performance loss on a target language compared to an equivalent model trained on a monolingual corpus.

We find equal ratios of English and French data lead to minimized performance hits across both languages (Figure 3) and opt to train our base model in this data configuration.

## 3.4 Final data distribution

Our final dataset is composed of 1.1T unique tokens that originate from sources of various languages, qualities, and quantities. To craft a training set with a language and data distribution that suits our objectives, we upsample some of the sources, notably to balance out French and English data and increase the share of parallel data in our training run. Following work by Muennighoff et al. (2023) and Luukkonen et al. (2023) on data-constrained

<footnote n="46">⁴⁶The number of parameters that it would take a monolingual model to match the performance of the multilingual, divided by the size of the multilingual model. See Fernandes et al. (2023) for more details</footnote>
language modeling scaling laws, we upsample French text by a factor of two, and parallel
data by a factor of 3. For a 3T token training run, this enables the model to see French data
at most 4 times, and English and code data twice, which should have negligible impacts on
the performance (Muennighoff et al., 2022). The final data distribution is shown in Table 1.

All data is provided from the above-listed sources and no synthetic or augmented data
is used. Data licenses and copyright information are given for every split to the best of
our ability. The data collection and filtering process to construct our final mix from the
above-listed sources is entirely done by the authors of this paper, who are employed by the
universities or private companies described through their affiliations, under their countries'
data protection laws, and compensated at market rates or following the academic salary
grid of their institution.

3.5 Training framework

We train our models on a modified version of Megatron-Deepspeed,<fnref n="47" /> a training framework built on top of PyTorch. Training is done on a dedicated Nvidia A100 SXM4 80 Gb partition of the Jean Zay supercomputer owned by the French government, on 30 octo-GPU nodes. Compute is obtained through a Jean Zay grant (2023-AD011014668R1). We rely on the HuggingFace Transformers and Datasets library for model and data manipulation.

To maximize efficiency, we set the micro-batch size per device to 8 sequences of length 2048, and use 4 gradient accumulation steps, resulting in a total batch size of 8 × 4 × 30 × 8 = 7680 samples, or 7680 × 2048 = 15,728,640 tokens. We achieve a mean efficiency of around 120 TFLOP<fnref n="48" /> per second with activation checkpointing, leading to a total compute estimate of 4.30e22 FLOPS. Standard Cross-Entropy losses are used on a Causal Language Modeling objective.

3.6 Training losses

Training lasts 17 days for a total of 99648 GPU hours, and we chose not to manually
intervene, letting the model recover on its own after occasional loss spikes. We train with a
max learning rate of 3e-4, 1000 warmup steps, and a cosine learning rate with a minimum
value of 1e-5. Curves suggest the model still has not reached a performance plateau after
3T tokens (Figure 5). Checkpoints are stored every 5k steps and released with the rest of the
project artifacts.

3.7 Environmental impact

The model was exclusively trained on Jean Zay, a French government-owned supercomputer, operating on low-carbon nuclear electricity. Between experimental runs, scaling laws, and the final training, 123k A100 hours were used. The Thermal Design Power of the NVIDIA A100 SXM4 80Gb used is 400W corresponding to a total power consumption of 49.2 MWH and considering a grid carbon intensity of 57 gCO2eq/kWh, we estimate a carbon footprint of 2.80 tons of CO2 (Luccioni et al., 2022) emitted during training.<fnref n="49" />

Interestingly, the model we trained is not “compute-optimal” according to Chinchilla laws
(Hoffmann et al., 2022), meaning that less computing could have been used to train a larger

<footnote n="47">⁴⁷https://github.com/deep-spin/Megatron-DeepSpeed</footnote>

<footnote n="48">⁴⁸160 TFLOP per second for our scaling law experiments with one GPU node only</footnote>

<footnote n="49">⁴⁹Factoring in an intentionally large data center Power Usage Effectiveness of 1.2 (Luccioni et al., 2022), we estimate an emission of 3.36 tons of CO2.</footnote>
language modeling scaling laws, we upsample French text by a factor of two, and parallel
data by a factor of 3. For a 3T token training run, this enables the model to see French data
at most 4 times, and English and code data twice, which should have negligible impacts on
the performance (Muennighoff et al., 2022). The final data distribution is shown in Table 1.

All data is provided from the above-listed sources and no synthetic or augmented data
is used. Data licenses and copyright information are given for every split to the best of
our ability. The data collection and filtering process to construct our final mix from the
above-listed sources is entirely done by the authors of this paper, who are employed by the
universities or private companies described through their affiliations, under their countries'
data protection laws, and compensated at market rates or following the academic salary
grid of their institution.

3.5 Training framework

We train our models on a modified version of Megatron-Deepspeed,<fnref n="47" /> a training framework built on top of PyTorch. Training is done on a dedicated Nvidia A100 SXM4 80 Gb partition of the Jean Zay supercomputer owned by the French government, on 30 octo-GPU nodes. Compute is obtained through a Jean Zay grant (2023-AD011014668R1). We rely on the HuggingFace Transformers and Datasets library for model and data manipulation.

To maximize efficiency, we set the micro-batch size per device to 8 sequences of length 2048, and use 4 gradient accumulation steps, resulting in a total batch size of 8 × 4 × 30 × 8 = 7680 samples, or 7680 × 2048 = 15,728,640 tokens. We achieve a mean efficiency of around 120 TFLOP<fnref n="48" /> per second with activation checkpointing, leading to a total compute estimate of 4.30e22 FLOPS. Standard Cross-Entropy losses are used on a Causal Language Modeling objective.

3.6 Training losses

Training lasts 17 days for a total of 99648 GPU hours, and we chose not to manually
intervene, letting the model recover on its own after occasional loss spikes. We train with a
max learning rate of 3e-4, 1000 warmup steps, and a cosine learning rate with a minimum
value of 1e-5. Curves suggest the model still has not reached a performance plateau after
3T tokens (Figure 5). Checkpoints are stored every 5k steps and released with the rest of the
project artifacts.

3.7 Environmental impact

The model was exclusively trained on Jean Zay, a French government-owned supercomputer, operating on low-carbon nuclear electricity. Between experimental runs, scaling laws, and the final training, 123k A100 hours were used. The Thermal Design Power of the NVIDIA A100 SXM4 80Gb used is 400W corresponding to a total power consumption of 49.2 MWH and considering a grid carbon intensity of 57 gCO2eq/kWh, we estimate a carbon footprint of 2.80 tons of CO2 (Luccioni et al., 2022) emitted during training.<fnref n="49" />

Interestingly, the model we trained is not “compute-optimal” according to Chinchilla laws
(Hoffmann et al., 2022), meaning that less computing could have been used to train a larger

<footnote n="47">⁴⁷https://github.com/deep-spin/Megatron-DeepSpeed</footnote>

<footnote n="48">⁴⁸160 TFLOP per second for our scaling law experiments with one GPU node only</footnote>

<footnote n="49">⁴⁹Factoring in an intentionally large data center Power Usage Effectiveness of 1.2 (Luccioni et al., 2022), we estimate an emission of 3.36 tons of CO2.</footnote>
<figure><img src="image_8.png" /><figcaption>Figure 5: (Left) Training loss with respect to the number of seen tokens. (Right) Validation perplexity (Averaged Log Likelihood) on Wikitext (English), computed with a rolling stride</figcaption></figure>

model with the same performance. However, our model aims to be used for inference purposes at industrial scales. Our training paradigm is thus to absorb the downstream inference costs, by training a smaller model on a lot more tokens to obtain an inference-optimized model equivalent in performance to a bigger compute-optimal model (Sardana and Frankle, 2023). Each inference of the final model is thus vastly more energy-efficient than a Chinchilla optimal model of equivalent performance (> 3B parameters), and can even run on CPU or mobile devices. Relying on estimates of Kaplan et al. (2020b), at inference, CroissantLLM represents roughly 2.6 GFLOPS per token.

# 4 Evaluation Benchmarks

We hope to extend base model evaluation past English benchmarking alone and assess model capabilities in French, aiming for broad coverage across orthogonal capabilities to observe the effect of truly bilingual pre-training. Our evaluation efforts are rooted in transparency, and all results reported in the main technical report are reproducible through code that is open-sourced and public data.<fnref n="50" />

## 4.1 English

In English, we evaluate on standard LLM evaluation benchmarks.

**HellaSwag.** HellaSwag (Zellers et al., 2019) is a dataset specifically crafted to challenge common-sense reasoning abilities of models by requiring them to predict the endings of sentences in a way that relies on information not present in the preceding context. It focuses on capturing a nuanced and context-dependent understanding of language.

**PiQA.** PIQA is a dataset for common-sense reasoning and was created to investigate the physical knowledge of existing NLP models (Bisk et al., 2019).

**SciQ.** The SciQ dataset contains 13,679 crowdsourced science exam questions about Physics, Chemistry, and Biology, among others. The questions are in multiple-choice format with 4 answer options each (Welbl et al., 2017).

<footnote n="50">⁵⁰Evaluation is ran through the LM Evaluation harness (Gao et al., 2021) at https://github.com/EleutherAI/lm-evaluation-harness. Datasets used are grouped in the following collection: https://huggingface.co/collections/manu/frenchbench-evaluation-datasets-65bbcbf3987c4be8a5e9a2ef</footnote>
<figure><img src="image_7.png" /><figcaption>Figure 5: (Left) Training loss with respect to the number of seen tokens. (Right) Validation perplexity (Averaged Log Likelihood) on Wikitext (English), computed with a rolling stride</figcaption></figure>

model with the same performance. However, our model aims to be used for inference purposes at industrial scales. Our training paradigm is thus to absorb the downstream inference costs, by training a smaller model on a lot more tokens to obtain an inference-optimized model equivalent in performance to a bigger compute-optimal model (Sardana and Frankle, 2023). Each inference of the final model is thus vastly more energy-efficient than a Chinchilla optimal model of equivalent performance (> 3B parameters), and can even run on CPU or mobile devices. Relying on estimates of Kaplan et al. (2020b), at inference, CroissantLLM represents roughly 2.6 GFLOPS per token.

# 4 Evaluation Benchmarks

We hope to extend base model evaluation past English benchmarking alone and assess model capabilities in French, aiming for broad coverage across orthogonal capabilities to observe the effect of truly bilingual pre-training. Our evaluation efforts are rooted in transparency, and all results reported in the main technical report are reproducible through code that is open-sourced and public data.<fnref n="50" />

## 4.1 English

In English, we evaluate on standard LLM evaluation benchmarks.

**HellaSwag.** HellaSwag (Zellers et al., 2019) is a dataset specifically crafted to challenge common-sense reasoning abilities of models by requiring them to predict the endings of sentences in a way that relies on information not present in the preceding context. It focuses on capturing a nuanced and context-dependent understanding of language.

**PiQA.** PIQA is a dataset for common-sense reasoning and was created to investigate the physical knowledge of existing NLP models (Bisk et al., 2019).

**SciQ.** The SciQ dataset contains 13,679 crowdsourced science exam questions about Physics, Chemistry, and Biology, among others. The questions are in multiple-choice format with 4 answer options each (Welbl et al., 2017).

<footnote n="50">⁵⁰Evaluation is ran through the LM Evaluation harness (Gao et al., 2021) at https://github.com/EleutherAI/lm-evaluation-harness. Datasets used are grouped in the following collection: https://huggingface.co/collections/manu/frenchbench-evaluation-datasets-65bbcbf3987c4be8a5e9a2ef</footnote>
**Arc-C.** The AI2 reasoning challenge dataset (Clark et al., 2018) consists of 7,787 authentic grade-school level, multiple-choice science questions, designed to stimulate research in advanced question-answering. The dataset is divided into a Challenge Set and an Easy Set, with the Challenge Set comprising questions that were answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm. Additionally, the dataset includes a corpus of over 14 million science sentences relevant to the task and provides three neural baseline models.

**MT-Bench.** MT-Bench (Zheng et al., 2023) contains a set of prompts designed to evaluate models on their multi-turn conversation and instruction-following abilities, covering various core model abilities; writing, roleplay, extraction, reasoning, math, coding, knowledge I (STEM), and knowledge II (humanities/social science). MT-Bench performance has been shown to best correlate with human-rated appreciation of a model through the LM-Sys model arena.

## 4.2 French

We aim to evaluate models on their capabilities in French, along several axes including vocabulary, grammar, reading comprehension, factual knowledge, biases, and generative capacities, etc. To this end, we introduce *FrenchBench*, a novel LLM evaluation benchmark for the French language, testing a large array of model skills in various settings.

FrenchBench comprises several tasks, some included from previous benchmark datasets, others newly released with this work.

### 4.2.1 FrenchBench Gen

FrenchBench assesses the generative capabilities of LLMs in a few-shot setting. Tasks include title generation, summarization, question generation, and question answering. We detail the tasks and the evaluation metrics used below.

**FQuaD.** FQuaD (d'Hoffschmidt et al., 2020) is a French Question Answering dataset, containing manually annotated sets of Wikipedia passages, questions, and extractive answer spans in the Squad format. This high-quality dataset is one of the rare human-annotated French datasets and we rely on its public evaluation split for 4 of the FrenchBench tasks.

*FQuADGenQ* is a question generation task in which passages and answers are given to the model in a few-shot manner, and we compute the ROUGE1 score (Lin, 2004) with the gold questions.

*FquadGenAns* is the classic question-answering task, but models generate the answer span themselves, and the ROUGE1 score is computed with the gold extractive answer span.

*MultiFQuAD*<sup>51</sup> is a FQuAD variant, with a publicly released evaluation set, in which answers can consist of multiple disjoint spans. We evaluate performance on the concatenation of these gold extractive spans using the ROUGE1 score.

**French Trivia.** The French Trivia dataset is built from online trivia questions pertaining to French culture. Answers are short and meant to assess latent model knowledge and the impact of pre-training data and cultural references. Intentionally, questions are formulated in English for comparison with monolingual English models.<sup>52</sup>

<footnote n="51"><sup>51</sup>Previously unreleased dataset, evaluation set is released under CC-By-NC SA 4.0 license with this work

<sup>52</sup>This is a previously unreleased dataset, released under MIT license with this work.</footnote>
**Arc-C.** The AI2 reasoning challenge dataset (Clark et al., 2018) consists of 7,787 authentic grade-school level, multiple-choice science questions, designed to stimulate research in advanced question-answering. The dataset is divided into a Challenge Set and an Easy Set, with the Challenge Set comprising questions that were answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm. Additionally, the dataset includes a corpus of over 14 million science sentences relevant to the task and provides three neural baseline models.

**MT-Bench.** MT-Bench (Zheng et al., 2023) contains a set of prompts designed to evaluate models on their multi-turn conversation and instruction-following abilities, covering various core model abilities; writing, roleplay, extraction, reasoning, math, coding, knowledge I (STEM), and knowledge II (humanities/social science). MT-Bench performance has been shown to best correlate with human-rated appreciation of a model through the LM-Sys model arena.

## 4.2 French

We aim to evaluate models on their capabilities in French, along several axes including vocabulary, grammar, reading comprehension, factual knowledge, biases, and generative capacities, etc. To this end, we introduce *FrenchBench*, a novel LLM evaluation benchmark for the French language, testing a large array of model skills in various settings.

FrenchBench comprises several tasks, some included from previous benchmark datasets, others newly released with this work.

### 4.2.1 FrenchBench Gen

FrenchBench assesses the generative capabilities of LLMs in a few-shot setting. Tasks include title generation, summarization, question generation, and question answering. We detail the tasks and the evaluation metrics used below.

**FQuaD.** FQuaD (d'Hoffschmidt et al., 2020) is a French Question Answering dataset, containing manually annotated sets of Wikipedia passages, questions, and extractive answer spans in the Squad format. This high-quality dataset is one of the rare human-annotated French datasets and we rely on its public evaluation split for 4 of the FrenchBench tasks.

*FQuADGenQ* is a question generation task in which passages and answers are given to the model in a few-shot manner, and we compute the ROUGE1 score (Lin, 2004) with the gold questions.

*FquadGenAns* is the classic question-answering task, but models generate the answer span themselves, and the ROUGE1 score is computed with the gold extractive answer span.

*MultiFQuAD*<sup>51</sup> is a FQuAD variant, with a publicly released evaluation set, in which answers can consist of multiple disjoint spans. We evaluate performance on the concatenation of these gold extractive spans using the ROUGE1 score.

**French Trivia.** The French Trivia dataset is built from online trivia questions pertaining to French culture. Answers are short and meant to assess latent model knowledge and the impact of pre-training data and cultural references. Intentionally, questions are formulated in English for comparison with monolingual English models.<sup>52</sup>

<footnote n="51"><sup>51</sup>Previously unreleased dataset, evaluation set is released under CC-By-NC SA 4.0 license with this work

<sup>52</sup>This is a previously unreleased dataset, released under MIT license with this work.</footnote>
### 4.2.2 FrenchBench Multiple Choice

We also assess reasoning, factual knowledge, linguistic capabilities, and model biases through a series of few-shot classification tasks, on which models are given multiple completions (multiple choice), and the answer with the highest likelihood is selected. We experimented with multiple question templates. In the MMLU format, the multiple potential answers are given after the question prefixed by a letter (A, B, C, D) and the model must guess the correct answer by predicting the correct answer's letter. In HellaSwag formatting, the model must complete the sentence and the model chooses the most likely continuation sequence, without prior knowledge of all other options. We find HellaSwag formatting is less abstract, and enables smaller size models to perform better.

**French Language Test.** The French Language Test is a dataset crafted to assess the grammar and vocabulary capabilities of models through language tests. It provides a structured evaluation of a model's linguistic proficiency, aiming to measure its competency in understanding and generating coherent and grammatically accurate sentences in the French language. It is composed of a *fr-grammar* and *fr-vocabulary* multiple choice test.

**French Hellaswag and Arc-C.** These datasets correspond to machine translations made by GPT3.5 of HellaSwag and Arc-C to French.<fnref n="53" /> Manual verification of the translation quality indicates the translations to be far from perfect but sufficient for these datasets to act as a correct performance proxy.

**OrangeSum.** OrangeSum⁵⁴ (Eddine et al., 2020) is a summarization dataset constructed from online News articles. Two standard French summarization tasks span from this dataset: *OSum(T)* in which the model is tasked with generating the title from the article body, and *OSum(A)* in which the model must generate the first paragraph of the article aimed to be an abstract of the article. We select the abstract generation task, and measure performance with the ROUGE1 score.

## 4.3 Other Tasks

**MT-Bench French.** Mt-Bench French⁵⁵ is a translated and adapted version of MT-Bench in French with all questions having undergone rigorous human review and adaption to guarantee authentic wording, and coherence, and to account for cultural discrepancies.

**Translation.** Translation capabilities are evaluated through the test set of the 2014 WMT French-English and English-French tasks (Alves et al., 2023). We measure performance using BLEU score (sacreBLEU, Papineni et al., 2002; Post, 2018), and COMET (Rei et al., 2022a). We also report FLORES (Team et al., 2022) and TICO (Anastasopoulos et al., 2020) scores.

**Belebele.** Belebele is a challenging reading comprehension dataset, with multiple choices, released across 122 languages in parallel format (Bandarkar et al., 2023). We leverage the English and French splits.

<footnote n="53">⁵³https://github.com/laiviet/lm-evaluation-harness/tree/main/datasets
⁵⁴https://huggingface.co/datasets/orange_sum
⁵⁵https://huggingface.co/datasets/bofenghuang/mt-bench-french</footnote>
### 4.2.2 FrenchBench Multiple Choice

We also assess reasoning, factual knowledge, linguistic capabilities, and model biases through a series of few-shot classification tasks, on which models are given multiple completions (multiple choice), and the answer with the highest likelihood is selected. We experimented with multiple question templates. In the MMLU format, the multiple potential answers are given after the question prefixed by a letter (A, B, C, D) and the model must guess the correct answer by predicting the correct answer's letter. In HellaSwag formatting, the model must complete the sentence and the model chooses the most likely continuation sequence, without prior knowledge of all other options. We find HellaSwag formatting is less abstract, and enables smaller size models to perform better.

**French Language Test.** The French Language Test is a dataset crafted to assess the grammar and vocabulary capabilities of models through language tests. It provides a structured evaluation of a model's linguistic proficiency, aiming to measure its competency in understanding and generating coherent and grammatically accurate sentences in the French language. It is composed of a *fr-grammar* and *fr-vocabulary* multiple choice test.

**French Hellaswag and Arc-C.** These datasets correspond to machine translations made by GPT3.5 of HellaSwag and Arc-C to French.<fnref n="53" /> Manual verification of the translation quality indicates the translations to be far from perfect but sufficient for these datasets to act as a correct performance proxy.

**OrangeSum.** OrangeSum⁵⁴ (Eddine et al., 2020) is a summarization dataset constructed from online News articles. Two standard French summarization tasks span from this dataset; *OSum(T)* in which the model is tasked with generating the title from the article body, and *OSum(A)* in which the model must generate the first paragraph of the article aimed to be an abstract of the article. We select the abstract generation task, and measure performance with the ROUGE1 score.

## 4.3 Other Tasks

**MT-Bench French.** Mt-Bench French⁵⁵ is a translated and adapted version of MT-Bench in French with all questions having undergone rigorous human review and adaption to guarantee authentic wording, and coherence, and to account for cultural discrepancies.

**Translation.** Translation capabilities are evaluated through the test set of the 2014 WMT French-English and English-French tasks (Alves et al., 2023). We measure performance using BLEU score (sacreBLEU, Papineni et al., 2002; Post, 2018), and COMET (Rei et al., 2022a). We also report FLORES (Team et al., 2022) and TICO (Anastasopoulos et al., 2020) scores.

**Belebele.** Belebele is a challenging reading comprehension dataset, with multiple choices, released across 122 languages in parallel format (Bandarkar et al., 2023). We leverage the English and French splits.

<footnote n="53">⁵³https://github.com/laiviet/lm-evaluation-harness/tree/main/datasets
⁵⁴https://huggingface.co/datasets/orange_sum
⁵⁵https://huggingface.co/datasets/bofenghuang/mt-bench-french</footnote>
# 5 Benchmark results

**Baseline models.** To evaluate CroissantLLM, we compare with an array of various models, varying in parameter size, pre-training language distribution, training corpus size, etc.

For “monolingual” English models, we evaluate Pythia-1.4B (Biderman et al., 2023) trained on 300B tokens, OPT-1.3B (Zhang et al., 2022) trained on 180B tokens, and TinyLlama(1.1B) (Zhang et al., 2024). TinyLlama is a very strong English baseline, as it holds many similarities to CroissantLLM. It is a 1.1B model trained on 3 trillion tokens with the same English corpus as the Croissant base. Although it contains some amount of high-quality non-English data, it is only a minor share of the training corpus, the main data sources being English and code data. As such, it trains on much more English tokens than CroissantLLM. All models are trained way past Chinchilla optimality (~26B tokens for a 1.3B model).

For monolingual French models, we use GPT-fr (Simoulin and Crabbé, 2021), a 1B model trained on 16.3B tokens, as well as the PagnolXL(1.5B) model (Launay et al., 2021), both in their author submitted HuggingFace implementations.

We also compare CroissantLLM with multilingual models, notably Llama2(7B) (Touvron et al., 2023b) trained on 2T tokens, Mistral7B (Jiang et al., 2023), and Bloom (Scao et al., 2022) models (from 1.1B to 3B), trained on 350B tokens each. We note that although the largest Bloom model is undertrained according to Chinchilla optimality (Hoffmann et al., 2022), smaller models are trained on the same number of tokens, making them largely more inference optimal and thus strong contenders. Finally, in the same size category, we evaluate mGPT (Shliazhko et al., 2022) a 1.3B model trained on 440B tokens.

Finally, to assess the impact of including instruction-like data within the pretraining dataset of models (as done in Bloom), we continue CroissantBase pretraining with a short cooldown phase on an instruction dataset without any formatting, and call the resulting model CroissantCool.

## 5.1 Base model

CroissantLLM obtains strong performances in its model size category, achieving on-par performance with the best monolingual English models on English benchmarks and largely outperforming existing mono and multilingual models on French benchmarks.

**English.** On English benchmarks (Table 3), CroissantLLM displays performances almost equivalent to those of TinyLlama, which has trained on much more English data. We see training on such a large quantity of English tokens enables our model to edge out similarly sized monolingual models trained on fewer tokens (OPT, Pythia), and larger multilingual models (Bloom 3B) demonstrating the interest of pursuing training past Chinchilla optimality, especially when splitting model capacity across languages.

**French.** On French classification benchmarks, CroissantLLM largely outperforms models of similar sizes trained on mostly monolingual English or French data, and multilingual models (Table 4). Performance is on par with the Bloom(3B) model, which is about 3 times as large. An interesting phenomenon can be noticed, especially on generative benchmarks assessed in few-shot settings: “base” models trained with instruction-like data perform a lot better. This is noticeable with the Bloom(3B) model which outperforms the otherwise vastly superior Llama2(7B) model on several tasks, or through the performance gains of CroissantCool with respect to CroissantBase.
# 5 Benchmark results

**Baseline models.** To evaluate CroissantLLM, we compare with an array of various models, varying in parameter size, pre-training language distribution, training corpus size, etc.

For “monolingual” English models, we evaluate Pythia-1.4B (Biderman et al., 2023) trained on 300B tokens, OPT-1.3B (Zhang et al., 2022) trained on 180B tokens, and TinyLlama(1.1B) (Zhang et al., 2024). TinyLlama is a very strong English baseline, as it holds many similarities to CroissantLLM. It is a 1.1B model trained on 3 trillion tokens with the same English corpus as the Croissant base. Although it contains some amount of high-quality non-English data, it is only a minor share of the training corpus, the main data sources being English and code data. As such, it trains on much more English tokens than CroissantLLM. All models are trained way past Chinchilla optimality (~26B tokens for a 1.3B model).

For monolingual French models, we use GPT-fr (Simoulin and Crabbé, 2021), a 1B model trained on 16.3B tokens, as well as the PagnolXL(1.5B) model (Launay et al., 2021), both in their author submitted HuggingFace implementations.

We also compare CroissantLLM with multilingual models, notably Llama2(7B) (Touvron et al., 2023b) trained on 2T tokens, Mistral7B (Jiang et al., 2023), and Bloom (Scao et al., 2022) models (from 1.1B to 3B), trained on 350B tokens each. We note that although the largest Bloom model is undertrained according to Chinchilla optimality (Hoffmann et al., 2022), smaller models are trained on the same number of tokens, making them largely more inference optimal and thus strong contenders. Finally, in the same size category, we evaluate mGPT (Shliazhko et al., 2022) a 1.3B model trained on 440B tokens.

Finally, to assess the impact of including instruction-like data within the pretraining dataset of models (as done in Bloom), we continue CroissantBase pretraining with a short cooldown phase on an instruction dataset without any formatting, and call the resulting model CroissantCool.

## 5.1 Base model

CroissantLLM obtains strong performances in its model size category, achieving on-par performance with the best monolingual English models on English benchmarks and largely outperforming existing mono and multilingual models on French benchmarks.

**English.** On English benchmarks (Table 3), CroissantLLM displays performances almost equivalent to those of TinyLlama, which has trained on much more English data. We see training on such a large quantity of English tokens enables our model to edge out similarly sized monolingual models trained on fewer tokens (OPT, Pythia), and larger multilingual models (Bloom 3B) demonstrating the interest of pursuing training past Chinchilla optimality, especially when splitting model capacity across languages.

**French.** On French classification benchmarks, CroissantLLM largely outperforms models of similar sizes trained on mostly monolingual English or French data, and multilingual models (Table 4). Performance is on par with the Bloom(3B) model, which is about 3 times as large. An interesting phenomenon can be noticed, especially on generative benchmarks assessed in few-shot settings: “base” models trained with instruction-like data perform a lot better. This is noticeable with the Bloom(3B) model which outperforms the otherwise vastly superior Llama2(7B) model on several tasks, or through the performance gains of CroissantCool with respect to CroissantBase.
<table><caption>Table 3: English Benchmarks (5-shot results)</caption><thead><tr><th>Task</th><th>Arc-e</th><th>Belebele (eng)</th><th>Hellaswag</th><th>PiQA</th><th>SciQ</th><th>Avg</th></tr></thead><tbody><tr><td>GPT-fr(1B)</td><td>0.27</td><td>0.28</td><td>0.29</td><td>0.54</td><td>0.68</td><td>0.41</td></tr><tr><td>Pagnol-XL(1.5B)</td><td>0.34</td><td>0.25</td><td>0.31</td><td>0.56</td><td>0.76</td><td>0.44</td></tr><tr><td>mGPT(1.3B)</td><td>0.48</td><td>0.23</td><td>0.35</td><td>0.66</td><td>0.62</td><td>0.47</td></tr><tr><td>Bloom(1.1B)</td><td>0.55</td><td>0.24</td><td>0.36</td><td>0.68</td><td>0.89</td><td>0.54</td></tr><tr><td>OPT(1.3B)</td><td>0.61</td><td>0.23</td><td>0.42</td><td>0.72</td><td>0.92</td><td>0.58</td></tr><tr><td>Pythia(1.4b)</td><td>0.63</td><td>0.25</td><td>0.42</td><td>0.71</td><td>0.92</td><td>0.59</td></tr><tr><td>Bloom(3B)</td><td>0.64</td><td>0.24</td><td>0.42</td><td>0.71</td><td>0.93</td><td>0.59</td></tr><tr><td><strong>CroissantLLM</strong></td><td>0.62</td><td>0.28</td><td>0.42</td><td>0.72</td><td>0.92</td><td>0.59</td></tr><tr><td><strong>CroissantCool</strong></td><td>0.62</td><td>0.26</td><td>0.43</td><td>0.73</td><td>0.92</td><td>0.59</td></tr><tr><td>TinyLlama(1.1B)</td><td>0.65</td><td>0.26</td><td>0.45</td><td>0.73</td><td>0.94</td><td>0.61</td></tr><tr><td>Llama2(7B)</td><td>0.79</td><td>0.46</td><td>0.56</td><td>0.79</td><td>0.97</td><td>0.72</td></tr><tr><td>Mistral(7B)</td><td>0.83</td><td>0.85</td><td>0.60</td><td>0.82</td><td>0.98</td><td>0.81</td></tr></tbody></table>

<table><caption>Table 4: FrenchBench MC (5-shot results)</caption><thead><tr><th>Task</th><th>Hellaswag(fr)</th><th>Arc-c(fr)</th><th>fr-vocab</th><th>fr-grammar</th><th>Belebele(fr)</th><th>Avg</th></tr></thead><tbody><tr><td>OPT(1.3B)</td><td>0.28</td><td>0.19</td><td>0.50</td><td>0.61</td><td>0.28</td><td>0.37</td></tr><tr><td>Pythia(1.4B)</td><td>0.30</td><td>0.20</td><td>0.61</td><td>0.76</td><td>0.23</td><td>0.42</td></tr><tr><td>TinyLlama(1.1B)</td><td>0.33</td><td>0.23</td><td>0.64</td><td>0.67</td><td>0.25</td><td>0.42</td></tr><tr><td>mGPT(1.3B)</td><td>0.27</td><td>0.20</td><td>0.71</td><td>0.73</td><td>0.23</td><td>0.43</td></tr><tr><td>GPT-fr(1B)</td><td>0.30</td><td>0.19</td><td>0.70</td><td>0.79</td><td>0.24</td><td>0.44</td></tr><tr><td>Bloom(1.1B)</td><td>0.34</td><td>0.22</td><td>0.76</td><td>0.79</td><td>0.24</td><td>0.47</td></tr><tr><td>Pagnol-XL(1.5B)</td><td>0.33</td><td>0.21</td><td>0.77</td><td>0.82</td><td>0.27</td><td>0.48</td></tr><tr><td><strong>CroissantCool</strong></td><td>0.40</td><td>0.26</td><td>0.77</td><td>0.78</td><td>0.23</td><td>0.49</td></tr><tr><td><strong>CroissantLLM</strong></td><td>0.40</td><td>0.26</td><td>0.75</td><td>0.80</td><td>0.27</td><td>0.50</td></tr><tr><td>Bloom(3B)</td><td>0.40</td><td>0.27</td><td>0.78</td><td>0.81</td><td>0.23</td><td>0.50</td></tr><tr><td>Llama2(7B)</td><td>0.44</td><td>0.38</td><td>0.76</td><td>0.77</td><td>0.43</td><td>0.56</td></tr><tr><td>Mistral(7B)</td><td>0.49</td><td>0.47</td><td>0.78</td><td>0.78</td><td>0.78</td><td>0.66</td></tr></tbody></table>

**Improvements throughout training.** The model performance continues to improve on downstream tasks during the entirety of training. We report WMT14 translation performance in Figure 6, and observe similar trends across all tasks. The benefits of training past Chinchilla optimality are clear, and although there are diminishing returns past a certain number of steps, training does not seem to saturate. In low training step settings, performance appears to emerge suddenly, reflecting emergent performance experiments in the literature most often obtained through model scaling (Wei et al., 2022b).

**Overall.** The 1.3B CroissantLLM displays top-of-its-class performance across both languages and all benchmarks, even edging out larger models such as Bloom(3B) on most tasks. All models remain far off from the performance of the strong 7B Llama and Mistral models.
<table><caption>Table 3: English Benchmarks (5-shot results)</caption><thead><tr><th>Task</th><th>Arc-e</th><th>Belebele (eng)</th><th>Hellaswag</th><th>PiQA</th><th>SciQ</th><th>Avg</th></tr></thead><tbody><tr><td>GPT-fr(1B)</td><td>0.27</td><td>0.28</td><td>0.29</td><td>0.54</td><td>0.68</td><td>0.41</td></tr><tr><td>Pagnol-XL(1.5B)</td><td>0.34</td><td>0.25</td><td>0.31</td><td>0.56</td><td>0.76</td><td>0.44</td></tr><tr><td>mGPT(1.3B)</td><td>0.48</td><td>0.23</td><td>0.35</td><td>0.66</td><td>0.62</td><td>0.47</td></tr><tr><td>Bloom(1.1B)</td><td>0.55</td><td>0.24</td><td>0.36</td><td>0.68</td><td>0.89</td><td>0.54</td></tr><tr><td>OPT(1.3B)</td><td>0.61</td><td>0.23</td><td>0.42</td><td>0.72</td><td>0.92</td><td>0.58</td></tr><tr><td>Pythia(1.4b)</td><td>0.63</td><td>0.25</td><td>0.42</td><td>0.71</td><td>0.92</td><td>0.59</td></tr><tr><td>Bloom(3B)</td><td>0.64</td><td>0.24</td><td>0.42</td><td>0.71</td><td>0.93</td><td>0.59</td></tr><tr><td><strong>CroissantLLM</strong></td><td>0.62</td><td>0.28</td><td>0.42</td><td>0.72</td><td>0.92</td><td>0.59</td></tr><tr><td><strong>CroissantCool</strong></td><td>0.62</td><td>0.26</td><td>0.43</td><td>0.73</td><td>0.92</td><td>0.59</td></tr><tr><td>TinyLlama(1.1B)</td><td>0.65</td><td>0.26</td><td>0.45</td><td>0.73</td><td>0.94</td><td>0.61</td></tr><tr><td>Llama2(7B)</td><td>0.79</td><td>0.46</td><td>0.56</td><td>0.79</td><td>0.97</td><td>0.72</td></tr><tr><td>Mistral(7B)</td><td>0.83</td><td>0.85</td><td>0.60</td><td>0.82</td><td>0.98</td><td>0.81</td></tr></tbody></table>

<table><caption>Table 4: FrenchBench MC (5-shot results)</caption><thead><tr><th>Task</th><th>Hellaswag(fr)</th><th>Arc-c(fr)</th><th>fr-vocab</th><th>fr-grammar</th><th>Belebele(fr)</th><th>Avg</th></tr></thead><tbody><tr><td>OPT(1.3B)</td><td>0.28</td><td>0.19</td><td>0.50</td><td>0.61</td><td>0.28</td><td>0.37</td></tr><tr><td>Pythia(1.4B)</td><td>0.30</td><td>0.20</td><td>0.61</td><td>0.76</td><td>0.23</td><td>0.42</td></tr><tr><td>TinyLlama(1.1B)</td><td>0.33</td><td>0.23</td><td>0.64</td><td>0.67</td><td>0.25</td><td>0.42</td></tr><tr><td>mGPT(1.3B)</td><td>0.27</td><td>0.20</td><td>0.71</td><td>0.73</td><td>0.23</td><td>0.43</td></tr><tr><td>GPT-fr(1B)</td><td>0.30</td><td>0.19</td><td>0.70</td><td>0.79</td><td>0.24</td><td>0.44</td></tr><tr><td>Bloom(1.1B)</td><td>0.34</td><td>0.22</td><td>0.76</td><td>0.79</td><td>0.24</td><td>0.47</td></tr><tr><td>Pagnol-XL(1.5B)</td><td>0.33</td><td>0.21</td><td>0.77</td><td>0.82</td><td>0.27</td><td>0.48</td></tr><tr><td><strong>CroissantCool</strong></td><td>0.40</td><td>0.26</td><td>0.77</td><td>0.78</td><td>0.23</td><td>0.49</td></tr><tr><td><strong>CroissantLLM</strong></td><td>0.40</td><td>0.26</td><td>0.75</td><td>0.80</td><td>0.27</td><td>0.50</td></tr><tr><td>Bloom(3B)</td><td>0.40</td><td>0.27</td><td>0.78</td><td>0.81</td><td>0.23</td><td>0.50</td></tr><tr><td>Llama2(7B)</td><td>0.44</td><td>0.38</td><td>0.76</td><td>0.77</td><td>0.43</td><td>0.56</td></tr><tr><td>Mistral(7B)</td><td>0.49</td><td>0.47</td><td>0.78</td><td>0.78</td><td>0.78</td><td>0.66</td></tr></tbody></table>

**Improvements throughout training.** The model performance continues to improve on downstream tasks during the entirety of training. We report WMT14 translation performance in Figure 6, and observe similar trends across all tasks. The benefits of training past Chinchilla optimality are clear, and although there are diminishing returns past a certain number of steps, training does not seem to saturate. In low training step settings, performance appears to emerge suddenly, reflecting emergent performance experiments in the literature most often obtained through model scaling (Wei et al., 2022b).

**Overall.** The 1.3B CroissantLLM displays top-of-its-class performance across both languages and all benchmarks, even edging out larger models such as Bloom(3B) on most tasks. All models remain far off from the performance of the strong 7B Llama and Mistral models.
<table><caption>Table 5: FrenchBench Gen (5-shot ROUGE1 results). Bloom models seem to have strong performance on QA tasks (Fquad), likely due to the inclusion of Question Answering datasets in its pretraining corpus (Laurençon et al., 2023). Pagnol-XL and GPT-fr are trained exclusively on French text and as such cannot be fairly evaluated on the French Trivia test.</caption><thead><tr><th>Task</th><th>FGenQ</th><th>FGenAns</th><th>MultiFQuAD</th><th>OSum(A)</th><th>FTrivia</th><th>Avg</th></tr></thead><tbody><tr><td>Pagnol-XL(1.5B)</td><td>0.06</td><td>0.04</td><td>0.03</td><td>0.03</td><td>-</td><td>*0.04</td></tr><tr><td>GPT-fr(1B)</td><td>0.04</td><td>0.02</td><td>0.05</td><td>0.11</td><td>-</td><td>*0.06</td></tr><tr><td>mGPT(1.3B)</td><td>0.01</td><td>0.00</td><td>0.02</td><td>0.03</td><td>0.33</td><td>0.08</td></tr><tr><td>OPT(1.3B)</td><td>0.09</td><td>0.18</td><td>0.21</td><td>0.17</td><td>0.39</td><td>0.21</td></tr><tr><td>Bloom(1.1B)</td><td>0.17</td><td>0.28</td><td>0.26</td><td>0.10</td><td>0.31</td><td>0.23</td></tr><tr><td>Pythia(1.4B)</td><td>0.15</td><td>0.34</td><td>0.27</td><td>0.21</td><td>0.44</td><td>0.28</td></tr><tr><td><strong>CroissantLLM</strong></td><td>0.19</td><td>0.40</td><td>0.33</td><td>0.10</td><td>0.52</td><td>0.31</td></tr><tr><td>Bloom(3B)</td><td>0.21</td><td>0.47</td><td>0.37</td><td>0.18</td><td>0.47</td><td>0.34</td></tr><tr><td>TinyLlama(1.1B)</td><td>0.18</td><td>0.46</td><td>0.41</td><td>0.23</td><td>0.45</td><td>0.35</td></tr><tr><td><strong>CroissantCool</strong></td><td>0.20</td><td>0.45</td><td>0.36</td><td>0.27</td><td>0.53</td><td>0.36</td></tr><tr><td>Llama2(7B)</td><td>0.25</td><td>0.68</td><td>0.60</td><td>0.30</td><td>0.70</td><td>0.50</td></tr><tr><td>Mistral(7B)</td><td>0.33</td><td>0.78</td><td>0.64</td><td>0.31</td><td>0.74</td><td>0.56</td></tr></tbody></table>

<figure><img src="image_9.png" /><figcaption>Figure 6: Performance evolution on the WMT Translation task (5-shot)</figcaption></figure>
<table><caption>Table 5: FrenchBench Gen (5-shot ROUGE1 results). Bloom models seem to have strong performance on QA tasks (Fquad), likely due to the inclusion of Question Answering datasets in its pretraining corpus (Laurençon et al., 2023). Pagnol-XL and GPT-fr are trained exclusively on French text and as such cannot be fairly evaluated on the French Trivia test.</caption><thead><tr><th>Task</th><th>FGenQ</th><th>FGenAns</th><th>MultiFQuAD</th><th>OSum(A)</th><th>FTrivia</th><th>Avg</th></tr></thead><tbody><tr><td>Pagnol-XL(1.5B)</td><td>0.06</td><td>0.04</td><td>0.03</td><td>0.03</td><td>-</td><td>*0.04</td></tr><tr><td>GPT-fr(1B)</td><td>0.04</td><td>0.02</td><td>0.05</td><td>0.11</td><td>-</td><td>*0.06</td></tr><tr><td>mGPT(1.3B)</td><td>0.01</td><td>0.00</td><td>0.02</td><td>0.03</td><td>0.33</td><td>0.08</td></tr><tr><td>OPT(1.3B)</td><td>0.09</td><td>0.18</td><td>0.21</td><td>0.17</td><td>0.39</td><td>0.21</td></tr><tr><td>Bloom(1.1B)</td><td>0.17</td><td>0.28</td><td>0.26</td><td>0.10</td><td>0.31</td><td>0.23</td></tr><tr><td>Pythia(1.4B)</td><td>0.15</td><td>0.34</td><td>0.27</td><td>0.21</td><td>0.44</td><td>0.28</td></tr><tr><td><strong>CroissantLLM</strong></td><td>0.19</td><td>0.40</td><td>0.33</td><td>0.10</td><td>0.52</td><td>0.31</td></tr><tr><td>Bloom(3B)</td><td>0.21</td><td>0.47</td><td>0.37</td><td>0.18</td><td>0.47</td><td>0.34</td></tr><tr><td>TinyLlama(1.1B)</td><td>0.18</td><td>0.46</td><td>0.41</td><td>0.23</td><td>0.45</td><td>0.35</td></tr><tr><td><strong>CroissantCool</strong></td><td>0.20</td><td>0.45</td><td>0.36</td><td>0.27</td><td>0.53</td><td>0.36</td></tr><tr><td>Llama2(7B)</td><td>0.25</td><td>0.68</td><td>0.60</td><td>0.30</td><td>0.70</td><td>0.50</td></tr><tr><td>Mistral(7B)</td><td>0.33</td><td>0.78</td><td>0.64</td><td>0.31</td><td>0.74</td><td>0.56</td></tr></tbody></table>

<figure><img src="image_8.png" /><figcaption>Figure 6: Performance evolution on the WMT Translation task (5-shot)</figcaption></figure>
## 5.2 Finetuning

Beyond base model performance, we evaluate CroissantLLM downstream performance once finetuned on generalist chat and instruction data, or on specific target tasks (translation, summarization).

### 5.2.1 Chat Model

It has been shown that supervised fine-tuning on instruction or chat datasets enables leveraging model capabilities to their fullest (Wei et al., 2022a).

**Training.** We finetune the base model on public Chat datasets Ultrachat (Ding et al., 2023) and Wildchat (Zhao et al., 2024) containing ChatGPT interactions in English and French. We also incorporate 12k samples of translation data (4% of the SFT dataset). We run finetuning on CroissantLLM, as well as the Bloom-1b7 and TinyLlama models for comparison. The obtained models are further suffixed with "Chat".

**MT-Bench.** We evaluate models on the MT-Bench benchmarks, both in English and French. Although a large difference in performance can be noted between the Bloom model and Croissant in favor of the latter, performance differences with TinyLlama are not as significant, neither in English nor in French. CroissantLLMChat performs strongly in open-ended writing categories (writing, roleplay, humanities) but struggles with reasoning and extractive tasks. Turn 2 performance (reformulation under constraints) is largely lower than Turn 1 performance as can be seen in Figures 11 and 12. Our CroissantLLMChat model also vastly outperforms the BloomZ 3B model trained by CMArkea on a large chat finetuning corpus (Delestre, 2023).

This hints at the fact that quasi-monolingual models with only a minor share of another language in their pretraining corpus can be adapted to a reasonable extent, through subsequent finetuning or continued pretraining, although large pre-training corpora are necessary to incorporate sufficient knowledge and reasoning abilities within the base models. We notice large correlations between generation temperature and performance and find CroissantLLMChat works a lot better with higher temperatures (≥ 0.4). For fair comparisons, we only report results obtained with low temperature settings in line with other model evaluations.

**French Trivia.** One main question this work attempts to tackle is whether training on bilingual data goes beyond augmenting the language understanding and writing capabilities of a model in another language, but also equips the models with novel knowledge and different cultural biases. We evaluate French cultural knowledge on a Trivia task, consisting of questions about France-related topics, asked in English (Table 6). As heuristic-based metrics are insufficient to capture the diversity of possible answers (Faysse et al., 2023), we rely on GPT-4 to score the results. Similar results to the MT-Bench experiments can be noted: although CroissantLLM is the best performing, no huge margins exist with respects to the best English model, TinyLlama, finetuned on the same bilingual SFT set. A potential bias might be that questions originate from American websites and are asked in English, which might slightly favor TinyLlama. Both CroissantLLM and TinyLlama largely outperform Bloom pretrained on a smaller corpora but finetuned on the same data. This further underlines the importance of pretraining data quantity, and the large knowledge overlap between various sources and languages.
## 5.2 Finetuning

Beyond base model performance, we evaluate CroissantLLM downstream performance once finetuned on generalist chat and instruction data, or on specific target tasks (translation, summarization).

### 5.2.1 Chat Model

It has been shown that supervised fine-tuning on instruction or chat datasets enables leveraging model capabilities to their fullest (Wei et al., 2022a).

**Training.** We finetune the base model on public Chat datasets Ultrachat (Ding et al., 2023) and Wildchat (Zhao et al., 2024) containing ChatGPT interactions in English and French. We also incorporate 12k samples of translation data (4% of the SFT dataset). We run finetuning on CroissantLLM, as well as the Bloom-1b7 and TinyLlama models for comparison. The obtained models are further suffixed with "Chat".

**MT-Bench.** We evaluate models on the MT-Bench benchmarks, both in English and French. Although a large difference in performance can be noted between the Bloom model and Croissant in favor of the latter, performance differences with TinyLlama are not as significant, neither in English nor in French. CroissantLLMChat performs strongly in open-ended writing categories (writing, roleplay, humanities) but struggles with reasoning and extractive tasks. Turn 2 performance (reformulation under constraints) is largely lower than Turn 1 performance as can be seen in Figures 11 and 12. Our CroissantLLMChat model also vastly outperforms the BloomZ 3B model trained by CMArkea on a large chat finetuning corpus (Delestre, 2023).

This hints at the fact that quasi-monolingual models with only a minor share of another language in their pretraining corpus can be adapted to a reasonable extent, through subsequent finetuning or continued pretraining, although large pre-training corpora are necessary to incorporate sufficient knowledge and reasoning abilities within the base models. We notice large correlations between generation temperature and performance and find CroissantLLMChat works a lot better with higher temperatures (≥ 0.4). For fair comparisons, we only report results obtained with low temperature settings in line with other model evaluations.

**French Trivia.** One main question this work attempts to tackle is whether training on bilingual data goes beyond augmenting the language understanding and writing capabilities of a model in another language, but also equips the models with novel knowledge and different cultural biases. We evaluate French cultural knowledge on a Trivia task, consisting of questions about France-related topics, asked in English (Table 6). As heuristic-based metrics are insufficient to capture the diversity of possible answers (Faysse et al., 2023), we rely on GPT-4 to score the results. Similar results to the MT-Bench experiments can be noted: although CroissantLLM is the best performing, no huge margins exist with respects to the best English model, TinyLlama, finetuned on the same bilingual SFT set. A potential bias might be that questions originate from American websites and are asked in English, which might slightly favor TinyLlama. Both CroissantLLM and TinyLlama largely outperform Bloom pretrained on a smaller corpora but finetuned on the same data. This further underlines the importance of pretraining data quantity, and the large knowledge overlap between various sources and languages.
<figure><img src="image_10.png" /><figcaption>Figure 7: MT Bench Results (Both Turns)</figcaption></figure>

<table><caption>Table 6: French Trivia performance, evaluated by the gold answer and GPT4. Questions are asked in English and accuracy is measured by GPT-4. To remove bias due to Supervised Finetuning, all base models are finetuned on the same dataset.</caption><thead><tr><td>Models</td><td>Accuracy</td></tr></thead><tbody><tr><td>Bloom 1b7 Chat</td><td>24.47%</td></tr><tr><td>TinyLlamaChat</td><td>44.47%</td></tr><tr><td>CroissantLLMChat</td><td>47.11%</td></tr></tbody></table>

**Translation.** We run translation evaluations on the Chat models<fnref n="56" /> and report results in Table 7. CroissantLLMChat displays extremely strong performances, in line with the strong few-shot performance of the CroissantLLM base model, outperforming models like Mistral7B or Llama13B in few-shot settings, and even matching the open source state-of-the-art specialized translation model for the size category, the NLLB 1.3B (Team et al., 2022), trained on vastly superior amounts of parallel data.

### 5.2.2 Dialog Summarization finetuning

To assess performance on specific downstream applications, we finetune base models on a custom dialog summarization dataset.<fnref n="57" /> Models are finetuned for three epochs on 6000 samples and results are computed through ROUGE and GPT-4 judgment metrics (Table 8).

CroissantLLM and Bloom(1.7B) models appear to yield strong, yet very similar results, trailing behind the larger Mistral7B model. This hints at the fact that base model performance is not always directly correlated to downstream performance post-finetuning, notably on tasks requiring few to no prior knowledge (here, keypoint extraction and reformulation).

<footnote n="56">⁵⁶TinyLLaMAChat corresponds to the TinyLlama model finetuned with the same SFT set as CroissantLLMChat.</footnote>

<footnote n="57">⁵⁷Proprietary dataset belonging to Illuin Technology, corresponds to organic customer interactions with a counselor.</footnote>
<figure><img src="image_9.png" /><figcaption>Figure 7: MT Bench Results (Both Turns)</figcaption></figure>

<table><caption>Table 6: French Trivia performance, evaluated by the gold answer and GPT4. Questions are asked in English and accuracy is measured by GPT-4. To remove bias due to Supervised Finetuning, all base models are finetuned on the same dataset.</caption><thead><tr><td>Models</td><td>Accuracy</td></tr></thead><tbody><tr><td>Bloom 1b7 Chat</td><td>24.47%</td></tr><tr><td>TinyLlamaChat</td><td>44.47%</td></tr><tr><td>CroissantLLMChat</td><td>47.11%</td></tr></tbody></table>

**Translation.** We run translation evaluations on the Chat models<fnref n="56" /> and report results in Table 7. CroissantLLMChat displays extremely strong performances, in line with the strong few-shot performance of the CroissantLLM base model, outperforming models like Mistral7B or Llama13B in few-shot settings, and even matching the open source state-of-the-art specialized translation model for the size category, the NLLB 1.3B (Team et al., 2022), trained on vastly superior amounts of parallel data.

### 5.2.2 Dialog Summarization finetuning

To assess performance on specific downstream applications, we finetune base models on a custom dialog summarization dataset.<fnref n="57" /> Models are finetuned for three epochs on 6000 samples and results are computed through ROUGE and GPT-4 judgment metrics (Table 8).

CroissantLLM and Bloom(1.7B) models appear to yield strong, yet very similar results, trailing behind the larger Mistral7B model. This hints at the fact that base model performance is not always directly correlated to downstream performance post-finetuning, notably on tasks requiring few to no prior knowledge (here, keypoint extraction and reformulation).

<footnote n="56">⁵⁶TinyLLaMAChat corresponds to the TinyLlama model finetuned with the same SFT set as CroissantLLMChat.</footnote>

<footnote n="57">⁵⁷Proprietary dataset belonging to Illuin Technology, corresponds to organic customer interactions with a counselor.</footnote>
<table><caption>Table 7: Performance in machine translation, according to COMET-22 and BLEU, across three different benchmarks: WMT14, TICO and FLORES. All translation outputs, unless stated otherwise, were generated using greedy decoding. We omit results with our Chat models (–) on WMT14, since WMT14 was used during fine-tuning.</caption><thead><tr><th rowspan="2"></th><th colspan="4">WMT 14</th><th colspan="4">TICO</th><th colspan="4">FLORES</th></tr><tr><th colspan="2">en→fr</th><th colspan="2">fr→en</th><th colspan="2">en→fr</th><th colspan="2">en→fr</th><th colspan="2">fr→en</th><th colspan="2">fr→en</th></tr><tr><th></th><th>COMET</th><th>BLEU</th><th>COMET</th><th>BLEU</th><th>COMET</th><th>BLEU</th><th>COMET</th><th>BLEU</th><th>COMET</th><th>BLEU</th><th>COMET</th><th>BLEU</th></tr></thead><tbody><tr><td colspan="14"><strong>NMT models</strong></td></tr><tr><td>NLLB 1.3B<br>0-shot</td><td>86.82</td><td>41.59</td><td>84.55</td><td>36.47</td><td>81.15</td><td>40.22</td><td>87.10</td><td>47.49</td><td>87.21</td><td>40.47</td><td></td><td></td></tr><tr><td colspan="14"><strong>Pre-trained models</strong></td></tr><tr><td>LLaMA-2 7B<br>5-shot</td><td>84.37</td><td>32.98</td><td>86.66</td><td>38.57</td><td>78.05</td><td>33.75</td><td>85.03</td><td>38.59</td><td>88.75</td><td>41.83</td><td></td><td></td></tr><tr><td>LLaMA-2 13B<br>5-shot</td><td>85.94</td><td>36.76</td><td>87.02</td><td>39.93</td><td>80.04</td><td>38.21</td><td>86.67</td><td>43.49</td><td>89.03</td><td>42.71</td><td></td><td></td></tr><tr><td>Mistral-7B-v0.1<br>5-shot</td><td>84.99</td><td>34.82</td><td>87.01</td><td>39.55</td><td>79.34</td><td>37.82</td><td>86.07</td><td>41.31</td><td>88.36</td><td>42.56</td><td></td><td></td></tr><tr><td>TinyLLaMA<br>5-shot</td><td>73.03</td><td>18.13</td><td>82.99</td><td>29.85</td><td>69.20</td><td>20.55</td><td>74.40</td><td>21.17</td><td>85.86</td><td>33.10</td><td></td><td></td></tr><tr><td>CroissantLLM<br>5-shot</td><td>85.11</td><td>38.09</td><td>85.70</td><td>36.30</td><td>78.74</td><td>38.49</td><td>86.85</td><td>46.58</td><td>88.58</td><td>42.83</td><td></td><td></td></tr><tr><td colspan="14"><strong>SFT models</strong></td></tr><tr><td>TowerInstruct-7B-v0.1<br>0-shot</td><td>88.07</td><td>46.19</td><td>88.14</td><td>46.75</td><td>81.53</td><td>41.27</td><td>88.38</td><td>48.57</td><td>89.56</td><td>46.34</td><td></td><td></td></tr><tr><td>TinyLLaMAChat<br>0-shot</td><td>-</td><td>-</td><td>-</td><td>-</td><td>73.04</td><td>23.61</td><td>78.08</td><td>27.24</td><td>86.26</td><td>32.80</td><td></td><td></td></tr><tr><td>CroissantLLMChat<br>0-shot</td><td>-</td><td>-</td><td>-</td><td>-</td><td>80.27</td><td>36.99</td><td>86.82</td><td>44.79</td><td>88.38</td><td>41.54</td><td></td><td></td></tr><tr><td>CroissantLLMChat<br>0-shot (Beam Search)</td><td>-</td><td>-</td><td>-</td><td>-</td><td>80.72</td><td>38.34</td><td>87.68</td><td>47.11</td><td>88.71</td><td>42.90</td><td></td><td></td></tr></tbody></table>

<table><caption>Table 8: Dialog Summarization Results. Except for ROUGE1, scores are measured by GPT-4, out of a maximum of 5.</caption><thead><tr><th>Models</th><th>ROUGE1</th><th>Coherence</th><th>Consistence</th><th>Fluidity</th><th>Relevance</th></tr></thead><tbody><tr><td>CroissantLLM</td><td>0.550</td><td>4.56</td><td>3.93</td><td>4.73</td><td>4.09</td></tr><tr><td>Bloom(1.7B)</td><td>0.550</td><td>4.52</td><td>3.96</td><td><strong>4.76</strong></td><td>4.08</td></tr><tr><td>Mistral(7B)</td><td><strong>0.588</strong></td><td><strong>4.60</strong></td><td><strong>4.73</strong></td><td>4.72</td><td><strong>4.59</strong></td></tr></tbody></table>
<table><caption>Table 7: Performance in machine translation, according to COMET-22 and BLEU, across three different benchmarks: WMT14, TICO and FLORES. All translation outputs, unless stated otherwise, were generated using greedy decoding. We omit results with our Chat models (–) on WMT14, since WMT14 was used during fine-tuning.</caption><thead><tr><th rowspan="2"></th><th colspan="4">WMT 14</th><th colspan="4">TICO</th><th colspan="4">FLORES</th></tr><tr><th colspan="2">en→fr</th><th colspan="2">fr→en</th><th colspan="2">en→fr</th><th colspan="2">en→fr</th><th colspan="2">fr→en</th><th colspan="2">fr→en</th></tr><tr><th></th><th>COMET</th><th>BLEU</th><th>COMET</th><th>BLEU</th><th>COMET</th><th>BLEU</th><th>COMET</th><th>BLEU</th><th>COMET</th><th>BLEU</th><th>COMET</th><th>BLEU</th></tr></thead><tbody><tr><td colspan="14"><strong>NMT models</strong></td></tr><tr><td>NLLB 1.3B<br>0-shot</td><td>86.82</td><td>41.59</td><td>84.55</td><td>36.47</td><td>81.15</td><td>40.22</td><td>87.10</td><td>47.49</td><td>87.21</td><td>40.47</td><td></td><td></td></tr><tr><td colspan="14"><strong>Pre-trained models</strong></td></tr><tr><td>LLaMA-2 7B<br>5-shot</td><td>84.37</td><td>32.98</td><td>86.66</td><td>38.57</td><td>78.05</td><td>33.75</td><td>85.03</td><td>38.59</td><td>88.75</td><td>41.83</td><td></td><td></td></tr><tr><td>LLaMA-2 13B<br>5-shot</td><td>85.94</td><td>36.76</td><td>87.02</td><td>39.93</td><td>80.04</td><td>38.21</td><td>86.67</td><td>43.49</td><td>89.03</td><td>42.71</td><td></td><td></td></tr><tr><td>Mistral-7B-v0.1<br>5-shot</td><td>84.99</td><td>34.82</td><td>87.01</td><td>39.55</td><td>79.34</td><td>37.82</td><td>86.07</td><td>41.31</td><td>88.36</td><td>42.56</td><td></td><td></td></tr><tr><td>TinyLLaMA<br>5-shot</td><td>73.03</td><td>18.13</td><td>82.99</td><td>29.85</td><td>69.20</td><td>20.55</td><td>74.40</td><td>21.17</td><td>85.86</td><td>33.10</td><td></td><td></td></tr><tr><td>CroissantLLM<br>5-shot</td><td>85.11</td><td>38.09</td><td>85.70</td><td>36.30</td><td>78.74</td><td>38.49</td><td>86.85</td><td>46.58</td><td>88.58</td><td>42.83</td><td></td><td></td></tr><tr><td colspan="14"><strong>SFT models</strong></td></tr><tr><td>TowerInstruct-7B-v0.1<br>0-shot</td><td>88.07</td><td>46.19</td><td>88.14</td><td>46.75</td><td>81.53</td><td>41.27</td><td>88.38</td><td>48.57</td><td>89.56</td><td>46.34</td><td></td><td></td></tr><tr><td>TinyLLaMAChat<br>0-shot</td><td>-</td><td>-</td><td>-</td><td>-</td><td>73.04</td><td>23.61</td><td>78.08</td><td>27.24</td><td>86.26</td><td>32.80</td><td></td><td></td></tr><tr><td>CroissantLLMChat<br>0-shot</td><td>-</td><td>-</td><td>-</td><td>-</td><td>80.27</td><td>36.99</td><td>86.82</td><td>44.79</td><td>88.38</td><td>41.54</td><td></td><td></td></tr><tr><td>CroissantLLMChat<br>0-shot (Beam Search)</td><td>-</td><td>-</td><td>-</td><td>-</td><td>80.72</td><td>38.34</td><td>87.68</td><td>47.11</td><td>88.71</td><td>42.90</td><td></td><td></td></tr></tbody></table>

<table><caption>Table 8: Dialog Summarization Results. Except for ROUGE1, scores are measured by GPT-4, out of a maximum of 5.</caption><thead><tr><th>Models</th><th>ROUGE1</th><th>Coherence</th><th>Consistence</th><th>Fluidity</th><th>Relevance</th></tr></thead><tbody><tr><td>CroissantLLM</td><td>0.550</td><td>4.56</td><td>3.93</td><td>4.73</td><td>4.09</td></tr><tr><td>Bloom(1.7B)</td><td>0.550</td><td>4.52</td><td>3.96</td><td><strong>4.76</strong></td><td>4.08</td></tr><tr><td>Mistral(7B)</td><td><strong>0.588</strong></td><td><strong>4.60</strong></td><td><strong>4.73</strong></td><td>4.72</td><td><strong>4.59</strong></td></tr></tbody></table>
<table><caption>Table 9: Inference Results in French and English on an A100 GPU with 40GB VRAM (average results over 100 tokens generations with 100 tokens input based on 100 Wikipedia text samples, vLLM backend and batch size 1)</caption><thead><tr><th>Model</th><th>Parameters (B)</th><th>Tokens Per Second</th><th>Words Per Second</th></tr></thead><tbody><tr><td colspan="4"><strong>French</strong></td></tr><tr><td>Llama 2</td><td>13</td><td>38.56</td><td>22.18</td></tr><tr><td>Llama 2</td><td>7</td><td>64.05</td><td>37.12</td></tr><tr><td>Croissant</td><td>1.3</td><td>145.40</td><td><strong>101.12</strong></td></tr><tr><td>TinyLlama</td><td>1.1</td><td><strong>152.60</strong></td><td>90.08</td></tr><tr><td colspan="4"><strong>English</strong></td></tr><tr><td>Llama2</td><td>13</td><td>38.17</td><td>28.16</td></tr><tr><td>Llama2</td><td>7</td><td>62.60</td><td>46.49</td></tr><tr><td>Croissant</td><td>1.3</td><td>139.64</td><td>111.41</td></tr><tr><td>TinyLlama</td><td>1.1</td><td><strong>150.16</strong></td><td><strong>112.36</strong></td></tr></tbody></table>

## 5.3 Optimized Inference

Our largest model, CroissantLLM, with 1.3B parameters is dimensioned to be extremely lightweight when compared to the main proprietary models and the smallest versions of the Llama and Mistral model family. This is motivated by the fact that widespread model adoption is bounded by inference compute resources, and most high-performing LLMs require expensive specialized infrastructures to run, which leads to high inference costs and model deployment difficulties. The most downloaded Llama model on the HuggingFace model hub is the 7B variant, reflecting the interest in small, yet effective, models.

At a 1.3B scale, CroissantLLM runs easily on local hardware (personal computers, low-end smartphones) and is easy to deploy on inexpensive CPU servers or low-end GPU servers, unlocking new applications with widespread usage. On higher-end GPUs (Table 9), CroissantLLM is both faster (latency) and less memory intensive enabling it to fit bigger batch sizes (throughput). Performance benchmarks are given in Table 9.

The decoder nature of CroissantLLM enables to benefit from the rich inference optimization ecosystem that has boomed recently. CroissantLLM is compatible with all main model serving libraries and platforms and can easily be quantized or optimized to run on personal devices. We performed 4bit quantization in the GGUF<fnref n="58" /> format and were able to run the model on lower-end smartphones at a speed of more than 5 tokens per second.

## 5.4 Model limitations

Evaluation results indicate the model is strong in its size category, and offers decent performances on writing-based tasks and internal knowledge, and very strong performance on translation tasks. The small size of the CroissantLLM model however hinders its capacity to perform more complex reasoning-based tasks, at least in a zero or few-shot manner in its generalist base or chat-model versions. This is aligned with other models of size and underlines the importance of scale for more abstract tasks (Wei et al., 2022b).

<footnote n="58">⁵⁸https://github.com/ggerganov/ggml/blob/master/docs/gguf.md</footnote>
<table><caption>Table 9: Inference Results in French and English on an A100 GPU with 40GB VRAM (average results over 100 tokens generations with 100 tokens input based on 100 Wikipedia text samples, vLLM backend and batch size 1)</caption><thead><tr><th>Model</th><th>Parameters (B)</th><th>Tokens Per Second</th><th>Words Per Second</th></tr></thead><tbody><tr><td colspan="4"><strong>French</strong></td></tr><tr><td>Llama 2</td><td>13</td><td>38.56</td><td>22.18</td></tr><tr><td>Llama 2</td><td>7</td><td>64.05</td><td>37.12</td></tr><tr><td>Croissant</td><td>1.3</td><td>145.40</td><td><strong>101.12</strong></td></tr><tr><td>TinyLlama</td><td>1.1</td><td><strong>152.60</strong></td><td>90.08</td></tr><tr><td colspan="4"><strong>English</strong></td></tr><tr><td>Llama2</td><td>13</td><td>38.17</td><td>28.16</td></tr><tr><td>Llama2</td><td>7</td><td>62.60</td><td>46.49</td></tr><tr><td>Croissant</td><td>1.3</td><td>139.64</td><td>111.41</td></tr><tr><td>TinyLlama</td><td>1.1</td><td><strong>150.16</strong></td><td><strong>112.36</strong></td></tr></tbody></table>

## 5.3 Optimized Inference

Our largest model, CroissantLLM, with 1.3B parameters is dimensioned to be extremely lightweight when compared to the main proprietary models and the smallest versions of the Llama and Mistral model family. This is motivated by the fact that widespread model adoption is bounded by inference compute resources, and most high-performing LLMs require expensive specialized infrastructures to run, which leads to high inference costs and model deployment difficulties. The most downloaded Llama model on the HuggingFace model hub is the 7B variant, reflecting the interest in small, yet effective, models.

At a 1.3B scale, CroissantLLM runs easily on local hardware (personal computers, low-end smartphones) and is easy to deploy on inexpensive CPU servers or low-end GPU servers, unlocking new applications with widespread usage. On higher-end GPUs (Table 9), CroissantLLM is both faster (latency) and less memory intensive enabling it to fit bigger batch sizes (throughput). Performance benchmarks are given in Table 9.

The decoder nature of CroissantLLM enables to benefit from the rich inference optimization ecosystem that has boomed recently. CroissantLLM is compatible with all main model serving libraries and platforms and can easily be quantized or optimized to run on personal devices. We performed 4bit quantization in the GGUF<fnref n="58" /> format and were able to run the model on lower-end smartphones at a speed of more than 5 tokens per second.

## 5.4 Model limitations

Evaluation results indicate the model is strong in its size category, and offers decent performances on writing-based tasks and internal knowledge, and very strong performance on translation tasks. The small size of the CroissantLLM model however hinders its capacity to perform more complex reasoning-based tasks, at least in a zero or few-shot manner in its generalist base or chat-model versions. This is aligned with other models of size and underlines the importance of scale for more abstract tasks (Wei et al., 2022b).

<footnote n="58">⁵⁸https://github.com/ggerganov/ggml/blob/master/docs/gguf.md</footnote>
**Knowledge Cutoff.** The model training dataset has a data cutoff date corresponding to the November 2023 Wikipedia dump. This is the de facto knowledge cutoff date for our base model, although a lot of information dates back further.<fnref n="59" /> Updated versions can be trained through continued pre-training or subsequent fine-tuning.

**Multilingual performance.** CroissantLLM is mostly a French and English model. Code performance is relatively limited, and although some amount of data from other languages is included within the SlimPajama training set, out-of-the-box performance in other languages is not to be expected, although some European languages do work quite well.

**Hallucinations.** CroissantLLM can hallucinate (Ji et al., 2023; Guerreiro et al., 2023) and output factually incorrect data,<fnref n="60" /> especially regarding complex topics. This is to be expected given the small model size, and hallucination rates seem inferior to most models of the same size category although no quantitative assessments have been conducted outside of MT-Bench experiments.

# 6 Foundation Model Transparency Index

To assess the transparency of our work, we evaluate our model through the Stanford Transparency Index (Bommasani et al., 2023) and obtain a total score of 81%, far ahead of proprietary models, as well as most staple open-weights models and large-scale open-source efforts (Figure 8).<fnref n="61" />

**Upstream.** The upstream categories include data, compute, and methods dimensions. The fully open-source nature and extensive disclosure of training information enable CroissantLLM to score 88% of the points. The difficulties in identifying personal information and in guaranteeing the exact license, and creators of all data included in internet scale corpora prohibit our work from obtaining the full points, although strong efforts have been made in only using data under free-use or open licenses and with no copyright issues, notably by excluding copyright flagged content from our English language corpus.

**Model.** The model categories include model information, as well as characterizations and mitigations of risks, limitations, trustworthiness, and mitigation. CroissantLLM obtains an average of 73% on this domain due to the wide array of reproducible evaluation results reported, but hindered by the lack of third-party external evaluation at the moment, and an evaluation of potential harms that is not as extensive as required.

**Downstream.** Downstream categories refer to usage policies, user statistics, distribution, documentation, and model impact assessment. The fully open-access nature of our model and distribution channel avoids most of the transparency pitfalls linked to restricted usage policies and user information processing, but the impact of our work remains difficult to assess until the model is released. The aggregated score for this category is 80%.

<footnote n="59">⁵⁹Prompted with "Who is the current French prime minister ?", it responds: "The current French prime minister is Jean Castex." which is outdated by more than 18 months at the time of the writing.</footnote>

<footnote n="60">⁶⁰As an example, prompted with "Which French club won the UEFA Champions League ?", it answers "The Paris Saint-Germain (PSG) club won the UEFA Champions League in 2020-2021."</footnote>

<footnote n="61">⁶¹Methodology is described in the appendix, and outline the fact our work relies on the index to guide its efforts in transparency, thus putting it at an advantage with respects to prior work such as Bloom (Scao et al., 2022).</footnote>
**Knowledge Cutoff.** The model training dataset has a data cutoff date corresponding to the November 2023 Wikipedia dump. This is the de facto knowledge cutoff date for our base model, although a lot of information dates back further.<fnref n="59" /> Updated versions can be trained through continued pre-training or subsequent fine-tuning.

**Multilingual performance.** CroissantLLM is mostly a French and English model. Code performance is relatively limited, and although some amount of data from other languages is included within the SlimPajama training set, out-of-the-box performance in other languages is not to be expected, although some European languages do work quite well.

**Hallucinations.** CroissantLLM can hallucinate (Ji et al., 2023; Guerreiro et al., 2023) and output factually incorrect data,<fnref n="60" /> especially regarding complex topics. This is to be expected given the small model size, and hallucination rates seem inferior to most models of the same size category although no quantitative assessments have been conducted outside of MT-Bench experiments.

# 6 Foundation Model Transparency Index

To assess the transparency of our work, we evaluate our model through the Stanford Transparency Index (Bommasani et al., 2023) and obtain a total score of 81%, far ahead of proprietary models, as well as most staple open-weights models and large-scale open-source efforts (Figure 8).<fnref n="61" />

**Upstream.** The upstream categories include data, compute, and methods dimensions. The fully open-source nature and extensive disclosure of training information enable CroissantLLM to score 88% of the points. The difficulties in identifying personal information and in guaranteeing the exact license, and creators of all data included in internet scale corpora prohibit our work from obtaining the full points, although strong efforts have been made in only using data under free-use or open licenses and with no copyright issues, notably by excluding copyright flagged content from our English language corpus.

**Model.** The model categories include model information, as well as characterizations and mitigations of risks, limitations, trustworthiness, and mitigation. CroissantLLM obtains an average of 73% on this domain due to the wide array of reproducible evaluation results reported, but hindered by the lack of third-party external evaluation at the moment, and an evaluation of potential harms that is not as extensive as required.

**Downstream.** Downstream categories refer to usage policies, user statistics, distribution, documentation, and model impact assessment. The fully open-access nature of our model and distribution channel avoids most of the transparency pitfalls linked to restricted usage policies and user information processing, but the impact of our work remains difficult to assess until the model is released. The aggregated score for this category is 80%.

<footnote n="59">⁵⁹Prompted with "Who is the current French prime minister ?", it responds: "The current French prime minister is Jean Castex." which is outdated by more than 18 months at the time of the writing.</footnote>

<footnote n="60">⁶⁰As an example, prompted with "Which French club won the UEFA Champions League ?", it answers "The Paris Saint-Germain (PSG) club won the UEFA Champions League in 2020-2021."</footnote>

<footnote n="61">⁶¹Methodology is described in the appendix, and outline the fact our work relies on the index to guide its efforts in transparency, thus putting it at an advantage with respects to prior work such as Bloom (Scao et al., 2022).</footnote>
<figure><img src="image_11.png" /><figcaption>Figure 8: Aggregated FMTI scores by major dimension of transparency. CroissantLLM scores are calculated by the authors, the rest by (Bommasani et al., 2023).</figcaption></figure>

## Ethics Statement

This work aims to offset recent English-centric work by enabling the study of the impact of language distribution within the pre-training dataset. The objective is to offer valuable resources to strengthen the community's understanding of induced model behavior and biases in that multilingual setup and inform future model and dataset development to be more inclusive.

**Model and Resource Release.** The models and all related artifacts are released openly on the CroissantLLM HuggingFace organization<fnref n="62" /> under an MIT license. No usage restrictions are imposed on users whatsoever. We indicate that users are responsible for the content they generate through the use of CroissantLLM and no redress mechanisms exist for harmful content disclosure. The model is offered to users openly, and downstream developers are accountable for using the model responsibly, although usage examples are provided.

Users are free to download and use the model and the associated resources at their will, and no monitoring information is kept by the CroissantLLM team regarding individual model usage or download information. The distribution platform, HuggingFace, does not share any non-public data with the CroissantLLM authors. Any modifications to the models or ulterior versions of the resources will be released under different version numbers, and

<footnote n="62">⁶²https://huggingface.co/croissantllm</footnote>
<figure><img src="image_10.png" /><figcaption>Figure 8: Aggregated FMTI scores by major dimension of transparency. CroissantLLM scores are calculated by the authors, the rest by (Bommasani et al., 2023).</figcaption></figure>

## Ethics Statement

This work aims to offset recent English-centric work by enabling the study of the impact of language distribution within the pre-training dataset. The objective is to offer valuable resources to strengthen the community's understanding of induced model behavior and biases in that multilingual setup and inform future model and dataset development to be more inclusive.

**Model and Resource Release.** The models and all related artifacts are released openly on the CroissantLLM HuggingFace organization<fnref n="62" /> under an MIT license. No usage restrictions are imposed on users whatsoever. We indicate that users are responsible for the content they generate through the use of CroissantLLM and no redress mechanisms exist for harmful content disclosure. The model is offered to users openly, and downstream developers are accountable for using the model responsibly, although usage examples are provided.

Users are free to download and use the model and the associated resources at their will, and no monitoring information is kept by the CroissantLLM team regarding individual model usage or download information. The distribution platform, HuggingFace, does not share any non-public data with the CroissantLLM authors. Any modifications to the models or ulterior versions of the resources will be released under different version numbers, and

<footnote n="62">⁶²https://huggingface.co/croissantllm</footnote>
original resources will not be deleted. We encourage discussions and feedback, either through the HuggingFace model page in the discussion tab, or the issues section of the associated GitHub repository.

**Risk mitigation.** We intend for our training process to be fully transparent and as such release all artifacts related to training. As such, our base model is released as is, without any risk mitigation methods beyond the extensive data curation that has gone into creating the pre-training data to remove toxic content as much as possible. In our Chat variant of the model, chat instructions have been explicitly sampled to include alignment instructions that train the model not to respond to certain prompts.<fnref n="63" />

**Data Leakage.** Through the inclusion of canaries in the training set, experiments were conducted on model memorization.<fnref n="64" /> These experiments confirm only artificially extreme cases of data repetition lead to in-weight information of inclusion within the training set. This enables us to confidently release the model without fear of potentially private data leakage that data filtering methods were unable to detect.

**Risk Assessment.** Our extensive evaluation process and the small scale of the CroissantLLM models allow us to confidently release all artifacts in our efforts of transparency without fear of potential misuse beyond what existing models of larger size already enabled. We staged our release by first giving model access to a dozen individuals and enabling them to experiment with them, whether through finetuning experiments, chat interactions, etc. Their feedback was aligned with the authors' observations in terms of the model capabilities and limitations and no warning flag was raised in terms of toxic content generation or otherwise harmful model behavior. We are confident the release will enable in-depth studying of large language models and outweigh the potential risks. To further strengthen compliance with FMTI guidelines, we will inform of any government inquiries regarding our model. We also indicate that users are responsible for the content they generate through the use of CroissantLLM and no redress mechanisms exist for harmful content disclosure.

<footnote n="63">⁶³For example when prompted "Give me medical advice about how to solve my back issues.", CroissantLLM responds I don't have a physical body or medical knowledge. However, I can provide you with some general tips on how to solve back issues: 1. Stay active: exercises such as yoga and stretching can help alleviate back pain. Regular physical activity can help improve flexibility, strengthen the muscles, and reduce the risk of future back problems. 2. Maintain a healthy diet: a balanced diet rich in fruits, vegetables, and lean protein can help reduce inflammation and improve digestion. Avoid high-fat and high-sugar foods, which can aggravate back pain. 3. Manage stress: chronic stress can worsen back pain. Try relaxation techniques such as meditation, deep breathing, or progressive muscle relaxation to help manage stress levels. 4. Get regular back checkups: regular checkups with a healthcare provider can help detect any underlying back issues early on. This can help prevent future problems. 5. Seek professional help: if you have persistent back pain, it may be necessary to seek the help of a healthcare professional. They can help diagnose and treat any underlying back issues that may be contributing to your pain.</footnote>

<footnote n="64">⁶⁴This research is conducted by an independent research team and results will be published after the Technical Report release.</footnote>
original resources will not be deleted. We encourage discussions and feedback, either through the HuggingFace model page in the discussion tab, or the issues section of the associated GitHub repository.

**Risk mitigation.** We intend for our training process to be fully transparent and as such release all artifacts related to training. As such, our base model is released as is, without any risk mitigation methods beyond the extensive data curation that has gone into creating the pre-training data to remove toxic content as much as possible. In our Chat variant of the model, chat instructions have been explicitly sampled to include alignment instructions that train the model not to respond to certain prompts.<fnref n="63" />

**Data Leakage.** Through the inclusion of canaries in the training set, experiments were conducted on model memorization.<fnref n="64" /> These experiments confirm only artificially extreme cases of data repetition lead to in-weight information of inclusion within the training set. This enables us to confidently release the model without fear of potentially private data leakage that data filtering methods were unable to detect.

**Risk Assessment.** Our extensive evaluation process and the small scale of the CroissantLLM models allow us to confidently release all artifacts in our efforts of transparency without fear of potential misuse beyond what existing models of larger size already enabled. We staged our release by first giving model access to a dozen individuals and enabling them to experiment with them, whether through finetuning experiments, chat interactions, etc. Their feedback was aligned with the authors' observations in terms of the model capabilities and limitations and no warning flag was raised in terms of toxic content generation or otherwise harmful model behavior. We are confident the release will enable in-depth studying of large language models and outweigh the potential risks. To further strengthen compliance with FMTI guidelines, we will inform of any government inquiries regarding our model. We also indicate that users are responsible for the content they generate through the use of CroissantLLM and no redress mechanisms exist for harmful content disclosure.

<footnote n="63">⁶³For example when prompted "Give me medical advice about how to solve my back issues.", CroissantLLM responds I don't have a physical body or medical knowledge. However, I can provide you with some general tips on how to solve back issues: 1. Stay active: exercises such as yoga and stretching can help alleviate back pain. Regular physical activity can help improve flexibility, strengthen the muscles, and reduce the risk of future back problems. 2. Maintain a healthy diet: a balanced diet rich in fruits, vegetables, and lean protein can help reduce inflammation and improve digestion. Avoid high-fat and high-sugar foods, which can aggravate back pain. 3. Manage stress: chronic stress can worsen back pain. Try relaxation techniques such as meditation, deep breathing, or progressive muscle relaxation to help manage stress levels. 4. Get regular back checkups: regular checkups with a healthcare provider can help detect any underlying back issues early on. This can help prevent future problems. 5. Seek professional help: if you have persistent back pain, it may be necessary to seek the help of a healthcare professional. They can help diagnose and treat any underlying back issues that may be contributing to your pain.</footnote>

<footnote n="64">⁶⁴This research is conducted by an independent research team and results will be published after the Technical Report release.</footnote>
## Contributors and Acknowledgements

In our efforts of transparency, we provide a summary of contributions for all authors and persons associated to the paper. Manuel, Patrick, Nuno, and Pierre belong to the core team and have participated throughout all steps of the project, from decision-making to report writing. Manuel coordinated the project and led the data collection, design decisions, and model evaluation efforts, and strongly participated in the model training. Pierre is the senior author of the project and was instrumental through constant feedback, project coordination, securing the compute grant, and design decisions. Patrick led the scaling law efforts and spearheaded the model training on distributed compute clusters. Nuno led the Chat and translation finetuning efforts, including constructing model finetuning pipelines and datasets, and gave constant feedback throughout the project. Pedro provided help on the development of the pre-training codebase and gave feedback on the pre-training stream of the work. João and Ricardo constructed the parallel data used for pre-training, which included efforts in both large-scale data collection and filtering. Duarte assisted with the fine-tuning efforts. António worked on base model finetuning on specific tasks and was in charge of the Chat model evaluation and the inference speed benchmark. Caio assisted with data collection efforts and provided high-quality, extensive feedback and notes on the report. Nicolas assisted with data collection efforts and data scrapping. Antoni adapted the model to swiftly run on mobile devices. Gautier, Céline, François, André are senior researchers who provided valuable feedback and important guidance throughout the project and were instrumental in obtaining compute grants.

This work is a collaboration of academic and industrial partners. On the academic side, core authors are affiliated with CentraleSupélec (Université Paris Saclay) and Instituto Superior Técnico de Lisboa, and other contributors are linked to Sorbonne Université and Imperial College London. On the industrial side, core authors receive funding from respectively Illuin Technology (Paris), Unbelabel (Lisboa), Equall (New York, Lisboa, Paris). Training compute is obtained on the Jean Zay supercomputer operated by GENCI IDRIS through compute grant 2023-AD011014668R1 as well as on Adastra through compute grant AD010614770. Part of the work was also supported by EU's Horizon Europe Research and Innovation Actions (UTTER, contract 101070631), DECOLLAGE (ERC-2022-CoG 101088763), Center for Responsible AI (2022-C05i0102-02), and by Fundação para a Ciência e Tecnologia through contract UIDB/50008/2020. Evaluation and finetuning experiments are done on the Ruche platform (Université Paris Saclay), as well as on privately owned compute centers.

Many other contributors, not listed as authors of the paper, lent a very welcome helping hand to the project, either with data collection efforts, feedback, interesting discussions, grant obtention, etc. In no particular order, we extend our thanks to Hélène Rousset, Bruno Hays, Bilel Omrani, Sacha Muller, Elena Hinnekens and Robert Vesoul (Illuin Technology), Paul-Henry Cournède, Renaud Monnet, Géraud Faye (CentraleSupélec), Pierre-Etienne Devineau (DINUM), Louise-Anne Charles (BnF Datalab), the Unbelabel TowerLLM team. Finally, we would like to warmly thank Stéphane Requena for allowing us access to Jeanzay, as well as Rémi Lacroix, and Etienne Malaboeuf for the debugging and technical support.
## Contributors and Acknowledgements

In our efforts of transparency, we provide a summary of contributions for all authors and persons associated to the paper. Manuel, Patrick, Nuno, and Pierre belong to the core team and have participated throughout all steps of the project, from decision-making to report writing. Manuel coordinated the project and led the data collection, design decisions, and model evaluation efforts, and strongly participated in the model training. Pierre is the senior author of the project and was instrumental through constant feedback, project coordination, securing the compute grant, and design decisions. Patrick led the scaling law efforts and spearheaded the model training on distributed compute clusters. Nuno led the Chat and translation finetuning efforts, including constructing model finetuning pipelines and datasets, and gave constant feedback throughout the project. Pedro provided help on the development of the pre-training codebase and gave feedback on the pre-training stream of the work. João and Ricardo constructed the parallel data used for pre-training, which included efforts in both large-scale data collection and filtering. Duarte assisted with the fine-tuning efforts. António worked on base model finetuning on specific tasks and was in charge of the Chat model evaluation and the inference speed benchmark. Caio assisted with data collection efforts and provided high-quality, extensive feedback and notes on the report. Nicolas assisted with data collection efforts and data scrapping. Antoni adapted the model to swiftly run on mobile devices. Gautier, Céline, François, André are senior researchers who provided valuable feedback and important guidance throughout the project and were instrumental in obtaining compute grants.

This work is a collaboration of academic and industrial partners. On the academic side, core authors are affiliated with CentraleSupélec (Université Paris Saclay) and Instituto Superior Técnico de Lisboa, and other contributors are linked to Sorbonne Université and Imperial College London. On the industrial side, core authors receive funding from respectively Illuin Technology (Paris), Unbelabel (Lisboa), Equall (New York, Lisboa, Paris). Training compute is obtained on the Jean Zay supercomputer operated by GENCI IDRIS through compute grant 2023-AD011014668R1 as well as on Adastra through compute grant AD010614770. Part of the work was also supported by EU's Horizon Europe Research and Innovation Actions (UTTER, contract 101070631), DECOLLAGE (ERC-2022-CoG 101088763), Center for Responsible AI (2022-C05i0102-02), and by Fundação para a Ciência e Tecnologia through contract UIDB/50008/2020. Evaluation and finetuning experiments are done on the Ruche platform (Université Paris Saclay), as well as on privately owned compute centers.

Many other contributors, not listed as authors of the paper, lent a very welcome helping hand to the project, either with data collection efforts, feedback, interesting discussions, grant obtention, etc. In no particular order, we extend our thanks to Hélène Rousset, Bruno Hays, Bilel Omrani, Sacha Muller, Elena Hinnekens and Robert Vesoul (Illuin Technology), Paul-Henry Cournède, Renaud Monnet, Géraud Faye (CentraleSupélec), Pierre-Etienne Devineau (DINUM), Louise-Anne Charles (BnF Datalab), the Unbelabel TowerLLM team. Finally, we would like to warmly thank Stéphane Requena for allowing us access to Jeanzay, as well as Rémi Lacroix, and Etienne Malaboeuf for the debugging and technical support.
## References

Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022. Towards a cleaner document-oriented multilingual crawled corpus. In *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pages 4344–4355, Marseille, France. European Language Resources Association.

Duarte M Alves, Nuno M Guerreiro, João Alves, José Pombal, Ricardo Rei, José GC de Souza, Pierre Colombo, and André FT Martins. 2023. Steering large language models for machine translation with finetuning and in-context learning. *arXiv preprint arXiv:2310.13448*.

Antonios Anastasopoulos, Alessandro Cattelan, Zi-Yi Dou, Marcello Federico, Christian Federman, Dmitriy Genzel, Francisco Guzmán, Junjie Hu, Macduff Hughes, Philipp Koehn, Rosie Lazar, Will Lewis, Graham Neubig, Mengmeng Niu, Alp Öktem, Eric Paquin, Grace Tang, and Sylwia Tur. 2020. Tico-19: the translation initiative for covid-19.

Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. 2023. Palm 2 technical report.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023a. Qwen technical report.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023b. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. *arXiv preprint arXiv:2308.12966*.
## References

Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022. Towards a cleaner document-oriented multilingual crawled corpus. In *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pages 4344–4355, Marseille, France. European Language Resources Association.

Duarte M Alves, Nuno M Guerreiro, João Alves, José Pombal, Ricardo Rei, José GC de Souza, Pierre Colombo, and André FT Martins. 2023. Steering large language models for machine translation with finetuning and in-context learning. *arXiv preprint arXiv:2310.13448*.

Antonios Anastasopoulos, Alessandro Cattelan, Zi-Yi Dou, Marcello Federico, Christian Federman, Dmitriy Genzel, Francisco Guzmán, Junjie Hu, Macduff Hughes, Philipp Koehn, Rosie Lazar, Will Lewis, Graham Neubig, Mengmeng Niu, Alp Öktem, Eric Paquin, Grace Tang, and Sylwia Tur. 2020. Tico-19: the translation initiative for covid-19.

Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. 2023. Palm 2 technical report.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023a. Qwen technical report.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023b. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. *arXiv preprint arXiv:2308.12966*.
Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023. The Belebele benchmark: a parallel reading comprehension dataset in 122 language variants.

Rachel Bawden, Hatim Bourfoune, Bertrand Cabot, Nathan Cassereau, Pierre Cornette, Marco Naguib, Aurélie Névéol, and François Yvon. 2024. Les modèles Bloom pour le traitement automatique de la langue française. Working paper or preprint.

Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In *Proceedings of the 2021 ACM conference on fairness, accountability, and transparency*, pages 610–623.

Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling.

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. PIQA: Reasoning about physical commonsense in natural language.

Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, and Percy Liang. 2023. The foundation model transparency index.

Eleftheria Briakou, Colin Cherry, and George Foster. 2023. Searching for needles in a haystack: On the role of incidental bilingualism in PaLM's translation capability. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 9432–9452, Toronto, Canada. Association for Computational Linguistics.

Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In *28th USENIX Security Symposium (USENIX Security 19)*, pages 267–284, Santa Clara, CA. USENIX Association.

Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, Lee Sharkey, Satyapriya Krishna, Marvin Von Hagen, Silas Alberti, Alan Chan, Qinyi Sun, Michael Gerovitch, David Bau, Max Tegmark, David Krueger, and Dylan Hadfield-Menell. 2024. Black-box access is insufficient for rigorous AI audits.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta,
Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023. The Belebele benchmark: a parallel reading comprehension dataset in 122 language variants.

Rachel Bawden, Hatim Bourfoune, Bertrand Cabot, Nathan Cassereau, Pierre Cornette, Marco Naguib, Aurélie Névéol, and François Yvon. 2024. Les modèles Bloom pour le traitement automatique de la langue française. Working paper or preprint.

Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In *Proceedings of the 2021 ACM conference on fairness, accountability, and transparency*, pages 610–623.

Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling.

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. PIQA: Reasoning about physical commonsense in natural language.

Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, and Percy Liang. 2023. The foundation model transparency index.

Eleftheria Briakou, Colin Cherry, and George Foster. 2023. Searching for needles in a haystack: On the role of incidental bilingualism in PaLM's translation capability. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 9432–9452, Toronto, Canada. Association for Computational Linguistics.

Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In *28th USENIX Security Symposium (USENIX Security 19)*, pages 267–284, Santa Clara, CA. USENIX Association.

Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, Lee Sharkey, Satyapriya Krishna, Marvin Von Hagen, Silas Alberti, Alan Chan, Qinyi Sun, Michael Gerovitch, David Bau, Max Tegmark, David Krueger, and Dylan Hadfield-Menell. 2024. Black-box access is insufficient for rigorous AI audits.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta,
Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaLM: scaling language modeling with pathways.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge.

Cyrile Delestre. 2023. [link].

Martin d'Hoffschmidt, Wacim Belblidia, Tom Brendlé, Quentin Heinrich, and Maxime Vidal. 2020. Fquad: French question answering dataset.

Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. *arXiv preprint arXiv:2305.14233*.

Moussa Kamal Eddine, Antoine J-P Tixier, and Michalis Vazirgiannis. 2020. Barthez: a skilled pretrained french sequence-to-sequence model.

Manuel Faysse, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2023. Revisiting instruction fine-tuned model evaluation to guide industrial applications. In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*. Association for Computational Linguistics.

Patrick Fernandes, Behrooz Ghorbani, Xavier Garcia, Markus Freitag, and Orhan Firat. 2023. Scaling laws for multilingual neural machine translation. In *Proceedings of the 40th International Conference on Machine Learning*, volume 202 of *Proceedings of Machine Learning Research*, pages 10053–10071. PMLR.

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation.

Nuno M Guerreiro, Duarte M Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André FT Martins. 2023. Hallucinations in large multilingual translation models. *Transactions of the Association for Computational Linguistics*, 11:1500–1517.

Michael Hart. 1971. Project gutenberg.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models.

Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, Kshitij Sachan, Michael Sellitto, Mrinank Sharma, Nova DasSarma, Roger Grosse, Shauna Kravec, Yuntao Bai, Zachary Witten, Marina
Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaLM: scaling language modeling with pathways.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge.

Cyrile Delestre. 2023. [link].

Martin d'Hoffschmidt, Wacim Belblidia, Tom Brendlé, Quentin Heinrich, and Maxime Vidal. 2020. Fquad: French question answering dataset.

Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. *arXiv preprint arXiv:2305.14233*.

Moussa Kamal Eddine, Antoine J-P Tixier, and Michalis Vazirgiannis. 2020. Barthez: a skilled pretrained french sequence-to-sequence model.

Manuel Faysse, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2023. Revisiting instruction fine-tuned model evaluation to guide industrial applications. In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*. Association for Computational Linguistics.

Patrick Fernandes, Behrooz Ghorbani, Xavier Garcia, Markus Freitag, and Orhan Firat. 2023. Scaling laws for multilingual neural machine translation. In *Proceedings of the 40th International Conference on Machine Learning*, volume 202 of *Proceedings of Machine Learning Research*, pages 10053–10071. PMLR.

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation.

Nuno M Guerreiro, Duarte M Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André FT Martins. 2023. Hallucinations in large multilingual translation models. *Transactions of the Association for Computational Linguistics*, 11:1500–1517.

Michael Hart. 1971. Project gutenberg.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models.

Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, Kshitij Sachan, Michael Sellitto, Mrinank Sharma, Nova DasSarma, Roger Grosse, Shauna Kravec, Yuntao Bai, Zachary Witten, Marina
Favaro, Jan Brauner, Holden Karnofsky, Paul Christiano, Samuel R. Bowman, Logan Graham, Jared Kaplan, Sören Mindermann, Ryan Greenblatt, Buck Shlegeris, Nicholas Schiefer, and Ethan Perez. 2024. Sleeper agents: Training deceptive llms that persist through safety training.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. *ACM Computing Surveys*, 55(12):1–38.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mixtral of experts.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020a. Scaling laws for neural language models. *CoRR*, abs/2001.08361.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020b. Scaling laws for neural language models.

Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Fer-
randis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau,
Leandro von Werra, and Harm de Vries. 2022. The stack: 3 tb of permissively licensed
source code.

Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates.

Julien Launay, Elena Tommasone, Baptiste Pannier, François Boniface, Amélie Chatelain, Alessandro Cappelli, Iacopo Poli, and Djamé Seddah. 2021. Pagnol: An extra-large french generative model.

Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jernite. 2023. The bigscience roots corpus: A 1.6tb composite multilingual dataset.
Favaro, Jan Brauner, Holden Karnofsky, Paul Christiano, Samuel R. Bowman, Logan Graham, Jared Kaplan, Sören Mindermann, Ryan Greenblatt, Buck Shlegeris, Nicholas Schiefer, and Ethan Perez. 2024. Sleeper agents: Training deceptive llms that persist through safety training.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. *ACM Computing Surveys*, 55(12):1–38.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mixtral of experts.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020a. Scaling laws for neural language models. *CoRR*, abs/2001.08361.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020b. Scaling laws for neural language models.

Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Fer-
randis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau,
Leandro von Werra, and Harm de Vries. 2022. The stack: 3 tb of permissively licensed
source code.

Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates.

Julien Launay, Elena Tommasone, Baptiste Pannier, François Boniface, Amélie Chatelain, Alessandro Cappelli, Iacopo Poli, and Djamé Seddah. 2021. Pagnol: An extra-large french generative model.

Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jernite. 2023. The bigscience roots corpus: A 1.6tb composite multilingual dataset.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Cheng-hao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Ab-ulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zh-danov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023a. Starcoder: may the source be with you!

Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023b. Textbooks are all you need ii: phi-1.5 technical report.

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Suther-land Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022. Competition-level code generation with alphacode. *Science*, 378(6624):1092-1097.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In *Text Summarization Branches Out*, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. 2022. Estimating the carbon footprint of bloom, a 176b parameter language model.

Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, and Sampo Pyysalo. 2023. FinGPT: Large generative models for a small language. In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 2710–2726, Singapore. Association for Computational Linguistics.

Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2023. Scaling data-constrained language models.

Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. Crosslingual generalization through multitask finetuning. *arXiv preprint arXiv:2211.01786*.

Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. Crows-pairs: A challenge dataset for measuring social biases in masked language models.

Roberto Navigli, Simone Conia, and Björn Ross. 2023. Biases in large language models: Origins, inventory, and discussion. *J. Data and Information Quality*, 15(2).
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Cheng-hao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Ab-ulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zh-danov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023a. Starcoder: may the source be with you!

Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023b. Textbooks are all you need ii: phi-1.5 technical report.

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Suther-land Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022. Competition-level code generation with alphacode. *Science*, 378(6624):1092–1097.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In *Text Summarization Branches Out*, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. 2022. Estimating the carbon footprint of bloom, a 176b parameter language model.

Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, and Sampo Pyysalo. 2023. FinGPT: Large generative models for a small language. In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 2710–2726, Singapore. Association for Computational Linguistics.

Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2023. Scaling data-constrained language models.

Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. Crosslingual generalization through multitask finetuning. *arXiv preprint arXiv:2211.01786*.

Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. Crows-pairs: A challenge dataset for measuring social biases in masked language models.

Roberto Navigli, Simone Conia, and Björn Ross. 2023. Biases in large language models: Origins, inventory, and discussion. *J. Data and Information Quality*, 15(2).