CroissantLLM: A Truly Bilingual French-English Language Model
Authors: Manuel Faysse
Patrick Fernandes
Nuno M Guerreiro
António Loison
Duarte Miguel Alves
Caio Corro
Nicolas Boizard
João Alves
Ricardo Rei
Pedro Henrique Martins
Antoni Bigata Casademunt
François Yvon
Andre Martins
Gautier Viaud
CELINE HUDELOT
Pierre Colombo
Venue: Accepted by TMLR ()

Abstract: We introduce CroissantLLM, a 1.3B language model pretrained on a set of 3T English and French tokens, to bring to the research and industrial community a high-performance, fully open-sourced bilingual model that runs swiftly on consumer-grade local hardware. To that end, we pioneer the approach of training an intrinsically bilingual model with a 1:1 English-to-French pretraining data ratio, a custom tokenizer, and bilingual finetuning datasets. We release the training dataset, notably containing a French split with manually curated, high-quality, and varied data sources. To assess performance outside of English, we craft a novel benchmark, FrenchBench, consisting of an array of classification and generation tasks, covering various orthogonal aspects of model performance in the French Language. Additionally, rooted in transparency and to foster further Large Language Model research, we release codebases, and dozens of checkpoints across various model sizes, training data distributions, and training steps, as well as fine-tuned Chat models, and strong translation models. We evaluate our model through the FMTI framework, and validate 81 % of the transparency criteria, far beyond the scores of even most open initiatives. This work enriches the NLP landscape, breaking away from previous English-centric work in order to strengthen our understanding of multilinguality in language models.

Published in Transactions on Machine Learning Research (01/2025)
CroissantLLM: A Truly Bilingual French-English Language Model
Manuel Faysse 1,5
Patrick Fernandes 6.8,11 Nuno M. Guerreiro 2,56,
António Loison'
Duarte M. A
Caio Corro
Nicolas Boizard João Alves?
Ricardo Ref277
Pedro H. Martins²
Antoni Bigata Casademunt François Yvon" André F.T. Martins
Gautier Viaud
Céline Hudclot Pierre Colombo
IIlluin Technology Unbabel
'Equall
*Diabolocom
'MICS, CentraleSupélec
"Instituto de Telecomunicagōes, Lisboa
"INESC-ID, Lisboa "Instituto Superior Téénico, Universidade de Lisboa
"Sorbonne Université, CNRS, ISIR, Paris
10 Imperial College London
11 "Language Technologies Institute, Carnegie Mellon University
Reviewed OpenReview:
Abstract
We introduce CroissantLLM, a 1.3B language model pre-trained on a set of 3T English and French tokens, bring to the research and industrial community a high-performance fully open-sourced bilingual model that runs swiftly on consumer- grade local hardware. To that end, we pioneer the approach of training intrinsically bilingual model with 1:1 English-to-Freet French pretraining data ratio, custom tokenizer, and bilingual finetuning datasets. We release the training dataset, notably containing French split with manually curated, high-quality, and varied data sources. To assess performance outside of English, we craft novel benchmark, FrenchBench, consisting of an array of classification and generation tasks, covering various orthogonal aspects of model performance in the French Language. Additionally, rooted in transparency and foster further Large Language Model research, release codebases, and dozens of checkpoints across various model sizes, training data distributions, and training steps, well finc-tuned Chat models, and strong translation models. We evaluate our model through the FMTI framework (Bommasani el al. (023)) and validate 81 % of the transparency criteria, far beyond the scores of even most open initiatives. This work enriches the NLP landscape, breaking away from previous English-centric work to strengthen our understanding of multilingualism in language models.
1 Introduction
Large Language Models ( have taken the Natural Language Processing (NLP) landscape in the past years. Although few proprietary models are still considered to run ahead of the pack (OpenAT et al. 2023). open weights models such as Llama [Touvron et al. 2023a b), Qwen (Bai et al. 2023a) or Mistral (Jiang et al. 2023 2024 are rapidly bridging the performance gap. However, widespread industrial and research adoption
such technology remains challenging for several reasons, including the lack of transparency in the data collection and training processes, the scarcity of existing resources outside of English, and the large-scale and costly nature of existing high-performing models.
Lack of transparency. State-of-the-art models, both proprictary and wein-wiights often designed and trained by heavily investor-backed companies, that aim to retain moat by keeping their training data mix and strategy secret, hindering the rest of the field's ability to fully study and understand these models. This lack of transparency, ranging from training set composition to lack evaluation or unclear usage policies,
See Appendix F definition.

---

Published in Transactions on Machine Learning Research (01/202)
Explain the French expression "poser un lapin"?
"Poser lapin" an expression that means to delay avoid something that expected to do, usually by not showing up all. often used in jest or sarcastically, but can also have more serious meaning. In French, the expression often used to describe someone who is unreliable or unreasonable, and who is not willing to follow through on their commitments. It can also be used to describe someone who is rude dismissive of others' needs.
doing it at
Figure Conversation example with Croissant
has been characterized by previous works, such as those by Bommasani et al. (2023) and Casper et al. (2024), pushing for full transparency as a key component. for sale model development and use. The dangers of closed, non-auditable datasets have been exemplified by recent findings showcasing the potential dangers of dataset contamination, whether intentional (Hubinger et al. 2024) or not Furthermore, legal questions arise surrounding data ownership in LLM training corpora (New YorkTTimes.
2023 SSmmeeson 2023 and
recent developments in the political landscape, regarding AT (EU AT Act, US senate hearings
$ have further emphasized the importance of transparent approaches, both from legal perspective and to build user trust.
Bias towards English. Although the exact training mix of most well-performing LLMs is not publicly available information, most large models are trained on very English-centric corpora (Touvron et al. 2023a This the consoquence of the important amount of English resources compared to other languages, both in terms data availability and benchmarks. As could be expected, publicly available LLMs display large performance disparity between English and non-English languages when evaluated on downstream tasks (Bandarkar et al. 2023). Morcover, cultural knowledge and biases are mechanically integrated through the training data, leading to models with greater knowledge of American events biases (Bender et al. 2021 Santurkar et al.] 2023 This puts non-English users at a disad when it comes to language model usage and adoption. While the non-English NLP community has produced multilingual datasets (Soboleva et al. 2023 Laurençon et al.) 2023) and models (Scao et al., 2022 Shliazhko et al.) 2022) in the last. lew years, the available resources still Jargely lag behind English ones, hindering industrial adoption
non-English settings.
Challenging to use at scale. Although benefits of scaling models to enormous sizes have been amply demonstrated in terms of performance (Hoffmann et al., 2022 Wei et al. 2022b Chowdhery et al. 2022 scale comes at large cost in terms of hardware requirements and inference speed. Download statistics on HuggingFace show the smallest Llama model (Touvron et al. 2023a) to be the most adopted by the community, demonstrating the interest in small but capable models. LLM scaling laws demonstrate the diminishing returns of training a model of a given size past a certain amount of tokens. By continuing pre- training way past the compute-optimal threshold, performance has been shown not to saturate, enabling the training of "inference-optimal"" models of high interest to the industrial and research communities (Sardanal & Frankle 2023). It is still not fully understood how model performance continues to improve at these later stages of training.
1.1 Contributions
In this work, attempt to bridge the aforcmentioned limitations and gaps. Our main contributions can be summarized as follows.
http:://purl.staaffrrt act.
https://www.ommeecc.. commerce. senate. ttthe-need-foeee hhee-eee---------rrrrrre-----iie http:///.......mppppp..aimtttttd
2

---

Published in Transactions on Machine Learning Research (01/2025)
Contribution 1: Introduction of highly-curated, diverse corpus in French. We collect and openly release 303B token corpus spanning internet data, but also literary work, speech transcripts, legal and administrative documents, scientific articles, business-related documents, etc. The corpus is distributed under permissive licenses, allowing commercial use with no restriction, and is heavily filtered, curated, and deduplicated. To our knowledge, it is the largest multi-source French language corpus released to date of sufficient quality for language modeling purpose:
Contribution 2: Traininn LLoissantLLM, a truly bilingual language model. Nowadays, most mod- els display some multilingual capabilities. For example, Bloom has been trained to be massively multilingual (Scao ct al. 2022), Llama contains a minor proportion of non-English data in its training set (Touvron ett al. 2023a and Qwen included a significative portion of Chinese data (Bai et al. 2023a 5 However, to our knowledge, outside of Chinese with a different alphabet (Zeng et al. 2022), no work has studied or attempted to train a multilingual model of significant scale in which English is not the dominant. training language.
In this work, we train model on 1:1 ratio of English to French with tokenizer optimized for bilingualism. Our end goal is to have model less skewed towards English performance cultural biases. We motivate this ratio by conducting careful experimentation on smaller model scales to uncover the trade-offs behind language distribution ratios. We opt for a strategy enabling "best of both worlds" performance in these two languages, as empirically validated by scaling law experiments. Experimentally, we show the importance of integrating data from another cultural source in the acquisition of cultural knowledge, underlining the importance of the effort.
Contribution 3: FrenchBench: A novel LLM benchmark for the French Language. To evaluate models in French, we introduce benchmark encompassing various tasks to assess factual knowledge, gener- alive capabilities, language understanding, etc. This benchmark is constructed both from openly available datasets, as well as newly released manually annotated data. We evaluate and report results for our models as well as other models with French-speaking capabilities
Contribution 4: Releasing high-performing, inferencee-ptinmal models for the industrial com- munity, together with a wide range of resources for the research community. The models we train are all released under open licenses. Our largest model is trained on 2300:1 token to parameter ratio (115 times longer than Chinchilla Optimal 1.3B model) leading to very strong performance in its size category. We show that model performance on downstream tasks continues to dramatically improve with lengthened pre-training runs, although model perplexity does not significantly improve. We release all checkpoints for all model sizes, as well as the exact training data seen at every step for rescarch purposes R These models are remely efficient to run, leading to low-latency, energy-efficient on-edge inference, even on low-resource devices such as phones or personal computers. These relcasos are motivated by a commitment to trans- parency to allow research and reassuring users for industrial deployment: our models comply with 81 % of criteria listed by the Foundation Model Transparency Index (Bommasani et al., 2023) (see Section 6].
The CroissantLLM initiative aims to adress the aforementionned limitations of current models through the release of an inference-oplimizze, small but capable model that performs well outside of English settings, and that is designed to be as open and transparent as possible. Beyond facilitating industrial LLM adoption and unlocking new generative model use cases, this project is also a proven platform for researching LLMs and
kickstarting future pretraining efforts (Martins al., 2024 Moeus et al. 2024a Shilov et al., 2024 et al. 2024b).
Mocus
"By this, we imply of sufficient quality to train a language model (litcle OCR errors, high-quality text)) and not exclusively composed of Internet data,
" "Qwen-VL VL (Bai et al. 2023bb reports 77.3% & 22.7 % Chinese, but no information is given for the Qwen base model. "Code for evaluation is available at hattps://githut com/ Ellerberrtl/L---vallaaiooor harnesss
7 Another complementary initiative has been led for French model evaluation and released concurrently in Bawden et al. (024)
" "Training data indexing order will be released second stage.
9 "ooe for dataset collection and filtering is available hattps://githu/ com Maa////laaa///laaaaaaaaaaaaa Code for model
training is hosted at htttps://github. cww./ooderPtt/rriieeaaaa 11m training Datasets and model checkpoints are available
at httppa://uugginggeeee wtc.aacce...eeeeeeeeeeeeeeeeea ca/Cro Crtoossiian
3

---

Published in Transactions on Machine Learning Research (01/2025)
Size (GB)
Docs. (M)
Tokens (B)
Token/ /Doe Sampling Ratio
tokens (B)
Freuch
1258.70
376.27
303.51
806.63
4.09
1240.08 08
English
2351.13
591.23
655.64
1108.94
1.89
1240.09
Code
366.87
81.90
141.43
1726.76
2.04
288.92
Parallel
113.91
408.03
35.78
87.68
6.13
219.26
Total
4090.61
1457.43
1136.35
779.70
14.15
2988.35
Table 1: Final Data mix for CroissantLLM training
2 Data
The dominance of the Euglish language in the training data of most current models is undeniable. While multilingual models like Llama leverage some non- non-English data (Touvron et al.) (202a)), it corresponds only minor part of the corpus, leading to significant drop in performance across non-English data, with noticeable "American" bias (Santurkar et al. 2023 Navigli et al. 2023). This work aims to offset this trend by using more balanced bilingual corpus comprising English and French, as well as additional code data. Although both languages belong to the Indo Europennn language family, they exhibit different morpho- syntactic structures and French has a richer morphology We study whether this corpus helps in reducing biases, enabling more varied knowledge sets, and unlocking non-English performance.
A variety of sources are integrated into our corpus, including carefully filtered internet data and high-quality data from range of sources, all devoid of restrictive licenses ensuring complete openness of the data and the trained model. Data statistics are available in Table 12
The scrapping and processing code are available in our code base used is given, all allowing for permissive commercial use.
13
The license information of all datasets
2.1 French Data
Table D lists the source and some information regarding the French corpus. Details about the data sources are expanded further below.
Web Data. We collect internet data from various web scraps (Oscar (Abadji et al. 2022), mC4 (ue et al. 2021)), leveraging the CulturaX corpus (Nguyen et al.) (223) for heuristic and perplexity filtering, as well exact and fuzzy deduplication. In total, this represents over 363 million webpages and more than 292 billion tokens, that we split using our custom tokenizer fitted on equal amounts of French and English data
We ensure data is of good quality and correctly tagged in French through sampled manual inspection and confirm French-speaking countries are well represented within the dataset. Notably, we include several news sources scrapped from Belgium, Switzerland, Canada, and Lebanon, as well as multiple African countries (Senegal, Morocco, Algeria, Cameroon, etc.)
10For example, pronominal objects are placed before (resp. after) the verb in French (resp. English), both languages have different noun phrase constructions (*Ia commission européenne" vs. "the European commission") "), etc.
For example, English has verb forms whereas French has 48, French has explicit inflections for grammatical genders, etc. However, note that only English adjectives have morphological constructions for expressing comparison (e.g. easy, easier, easiest). We refer to WALS for more details, e.g. atttp://vall into//eeaurre//1eeee
7/152.4
As further detailed in 3.4 our data corpus contains different amounts of unique English, French, and Code tokens. We obtain our balanced training corpus by upsampling French, Code, and English data with different sampling ratios, such that performance loss is to be expected (Mucnnighoff et all 2023).
"/ttpps//gittuu wwt..b..coommee//e/eea Waaaaneeaaa/lnnddaaaaaaa
14The mC4 corpus https://huggingteent wttpaaaccccoccaaeeeeeeeeee co//ataasts//llennt/cc. is released under the ODC-BY licence https://
whereas Oscar (Abadji et al. 2022 does not claim data ownership, provides upl-out stralegy for data inclusion, and filtering metadata is released under the Creative Commons CCO license/https:
@pendatacommons.orgee wttp.r//.e..eeeeee/////o0000
//creativecommons.oo wwtp:///wwrppppp.o.oonraoaaampee wwwaa///mmomam/oooooo//ooo
Contrary ocrtain languages such Arabic Portuguese which the language differs greatly depending the country, the written French language these countries are largely similar with only minor share of regionally introduced syntax and expressions
4

---

Published in Transactions on Machine Learning Research (01/2025)
Legal and Administrative Data. We introduce 5.3B tokens of data from the French government's open data initiative, ranging from legal tasks to parliamentary discussions and financial transcripis (e.g. legal and administrative jargon). These texts originate from 13 different datasets (the OpenData corpus) and were collected from the French government's open data platform T6 To ensure other French-speaking countries are represented, we add 68M tokens of data from Swiss legislation retrieved from government sources. We perform steps process, filter, and run exact deduplication on these documents.
Cultural Data. We introduce cultural data from various sources. Notably, retrieve all Project Guten- berg (Hart 1971 books in the French language of October 2023, corresponding to books released in the public domain (302 million tokens). We also download and aggressively filter manuscripts and documents from the French National Library (Bibliothèque Nationale de France), and filter for documents that belong to the public domain, have undergone an OCR process, and are of high quality T7 To filter out low-quality OCR documents, we implement custom heuristics which we release within our code basc. We run all docu- ments through perplexity filters using KenLM 5-grams fitted on the French Wikipedia split, and discard documents with perplexity values that are too high (noisy) too low (repetitive patterns). Thresholds are set through manual verification process. We deliberately choose to be aggressive in our filtering to ensure only high-quality data is kept and discard the largest portion of the original corpus, keeping about 27M tokens. We choose not to keep any data from the newspaper archives, as the OCR transcription is often too noisy. Additionally, we introduce famous public domain French poems custom scrapped from a French poetry website and run a set of podcasts through a high-quality speech-to-text model to obtain a textual transcription. This process is hard to scale and data splits from these sources are limited in quantity. Data from the OpenSubtitles? 200 initiative integrated, corresponding to 41.8 million tokens originating from movie subtitles Finally, we add the French data from Wikisource collected part of the BigScience initiative Scao et al. 2022) and obtain 2.7 billion tokens from the process
Encyclopedia Data. To introduce high-quality factual data to the training corpus, we integrate the French Wikipedia split from November 2023 This corresponds to the latest knowledge cutoff in the training data. In lotal, more than 2.5 million articles are used, spanning more than billion tokens.
Industrial Data. We scrap high-quality and publicly available data from industrial PDFs via a manually crafted list of websites, from large French and French Canadian (Quebec) companics to government agencies. This musiness-focuser data boosts performance on series of downstream applications related to industrial NLP. We collect over 36000 PDF multi-page documents and filter them through carefully crafted heuristics, followed by aggressive perplexity filtering 24 In total, we obtain over 290000 documents and 191 million tokens.
2.2 English Data
Our English data is primarily drawn from the SlimPajama corpus (Soboleva et al. 2023 excluding copy- righted documents. Splits per data source are detailed in Table I0]
Internet Data. Similarly to the French dataset, rely carefully filtered content from assortment of internet sources, including miscellaneous web pages and blogs. The filtering process includes heuristics
and perplexity filtering, well as large-scale deduplication (Soboleva et al. 2023). The SlimPajama corpus
Data is released at attp:/ wwwp:/eeeessddll.....eeeeeeeaaes www../a....t..ttt////AAAAAAAAA with the ETALAB licnne/tttps://wweetaaabb.
v2f0.
"etadata
18
licensed
under
Open
Etalab
license
attps://gallicaabbrt buf wtt.ttt///t//////dddda/aaa
https://github
http:///wwww.osssetrrreeeeeeee fr/
https://www.penanubeete
Licensed under CC BY-SA 4.0, https:://n. vikiooure. http://wwwwr..reerre////iiiiaaaa ott.://..............oo.o..o..o.
25 htttp::/huggiigffcc. htt..//.aattttttiiaaaa/aaaaaaaaa
with CC-By-SA 3.0 license
24 Data public internet data that release under MIT license with an opt-ont scheme http:://hgggii///scccc///eeee
dataset_ wtw.aatttteet__ooorre textt_oniyy
5

---

Published in Transactions on Machine Learning Research (01/2025)
includes internet data from the CommonCraw 20 and C4 web scraps, well as data sourced from Github
textual conten 27 and the StackExchange forums 25
Miscellaneous. Other non-internet-based data sources are included in the SlimPajama dataset, such as scientific articles from Arxiv and English documents from Wikipedia, 30 The SlimPajama dataset is also comprised of the "Books" subcorpus, obtained by downloading all book documents from Bibliotik 31 Some of the documents within this last corpora have been flagged by their owner as proprietary data. We filter out all documents from this subcorpus, and replace them with data from the open-source Project Gutenberg Hart 1971h English books under public domains
Gutenberg Canaries. To assess model memorization to inform about the risks of including private or sensitive data within the training sel, we stress Lest the model by including "canaries" (Carlini el. al.] 2019). These correspond to samples that have been intentionally modified and/or repeated and included within the model training set, and that will enable posteriori evaluation of the model capacity to memorize data "worse than worst-case" situation In total the canaries represent 555 million tokens, representing less than 0.04 % of the total tokens seen during training.
2.3
Code Data
In line with most recent models (Chowdhery ct al. 2022 Scao et al. 2022 Touvron et al. 2023a we integrate code data into our training corpus. Notably, previous work shows that code data benefits natural language tasks and can be particularly useful in data-constrained settings (Muennighoff al. 2023). Therefore, we include 140B tokens of code data in several common programming languages. Splits and number of tokens are detailed in Table 11
The Stack & StarCoder. We rely on the efforts of the StarCoder project Li et al. high-quality filtered code data from The Stack corpus (Kocetkov et al. 2022 programming languages (Java, Javascript, Python, C, C++, SQL) and Jupyte notebooks, as well as a few samples of formatted data (JSON, Tex) and scripting languages (shell scripts, Dockerfiles)
2023a and use their We keep only high-resource
34
35
36
Extra Python code. We extend the corpus with several other sources of Python code due to the popularity of the language in the community. Firstly, we add Pypi packages from recent code dumps that are filtered to keep only Python and Jupyter files Secondly, in order to promote high-quality problem-solvin solving-
33
centered code, we integrate 1.2B tokens of Python3 data from competitive coding contests (Li ct al. 2022
39
Lastly, following the success of learning from textbooks (Li et al. 2023b), we add commented Python code constructed by combining code and text cells from Jupyter Notebooks through the CodeParrot initiative
2.4 Parallel Data
Following previous work (Anil et al.) 2023), we incorporate vast quantities of parallel data, in case high-quality English-French translation pairs, in order to improve the multilingual capabilities of the model Briakou et al. 2023).
25 Commoo Crawl license httppp////mmmmara...//oo/oo////a 20DDC-B license
27 Only content under MIT, BSD, and Apache licenses are kept
28 CBCC-By-S 4.0 license /tttps://rrchiee... http::://rcciie..rgg/eeeas///// atg///essse/sttceeeceeent
/tttps://arxiv. org// with author opt-out options
30 CCCBBB-AA 3.0 license
/pttpe://uuggiggacceee http:/awww...ccattttttttttaetaat pile books3
32From httpp:: hacippa://uuggae hhtgg//uggggggceeeeeeeeeea httpg/aaaeeeccc.aeeeeeeeeeeeeeee wtt.////ott////eeeeo pgig with an Apache 2.0 license
T9This work is led in parallel to the Croissant LLM project and findings will be independently published.
34Dat are drawn from various sources and falls under 193 different permissive licenses. We use the version 1.2 of the corpus, which has been filtered with respect to data owners opt-out option information.
35 httppe///upyterr.oo org/
http:://pr-codeeorr//aaaasass from permissive licensed code
Under CC-By-4.1 license
40
juw.-uub---upytt--oooodeeooooeee -od-ttttooo-----tttttt Apache license

---

Published in Transactions on Machine Learning Research (01/2025)
Model
Params (M)
Layers
Hidden size
Inter. size
KV heads
XXS
100.7
1024
4096
XS
202.5
12
1024
4128
S
341.5
12
1536
4128
12
Base
1214.3
24
2048
5504
Table 2: Model information for scaling laws. Parameter count excludes embedding and output parameters.
Opus.
We extract subsets of sentence pairs spanning multiple domains from the OPUS corpus Tiedemann
2012 Statistics are described in Table 10 In total, we include 400 million parallel sentences and about
36 billion tokens. The data is filtered through a rigorous cleaning pipeline: (1) BiFixer (Ramirez-Sánchez
ct al. 2020 is first used to remove duplicate data through fuzzy matching techniques; (2) BiClcaner is
then used to filter data using heuristic and perplexity filters; (3) finally, the state-of-the-ar NMT quality isator CimetKiwi (Rei et al. 2022b) is used to keep only top quality translation pairs.
Theses. To incorporate versatile academic and scientific language, we augment our dataset with French the- ses abstracts along with their author-generee generated English translations. This corresponds to 95000 documents and more than 80 million high-quality tokens
Song Lyrics. Our dataset integrates song lyrics in both French and English, scrapped from a specialized As such, our model is trained with radically different linguistic styles (e.g. colloquialism), and the wide range of topics can help the model to capture cultural nuances. Lyrics have been translated by hand by the website community. With a total of 70k songs, we have built up a corpus of 53M tokens covering different periods (80s, 90s, 20s, etc.) and musical genres (rap, rock, jazz, etc.) To preserve colloquial expressions and cultural subtleties, we have not filtered song lyrics for explicit content. We validate the original language metadata of the songs through Googlc's language-detcction algorithm.
community-driven lyries translation website 45
3 Training
Our main goal was to train a high-performing, yet resource-friendly bilingual model while optimizing per- formances across both languages. To focus on the specific challenges of the bilingual paradigm, we rely on previous work to motivate many of our design and hyperparameter choices (Touvron et al. 2023b).
3.1 Model Architecture
We use the Llama architecture Touvron et al. 2023a decoder-based transformer, trained with rotary position encodings (Su et al. 2023 and context length of 2048. We construct different model sizes by jointly scaling the number of attention heads, hidden size, and hidden layers. Table summarizes the sizes of each model the family.
3.2 Tokenizer
Most LLM tokenizers are fitted on English-centric corpora with an information-theored optimization ob- jective, for example, Byte-Pair encoding (Snnnich et al.) (0166) or Unigram (Kudo 2018). leading to good
fortility 40 values (low token per word ratio) on English text, but high fertility in other languages. These
41 Free license https::/opus alpl. ou/
"tttps///itthuu httpchubb..oom//beeett/////t///t
hittps://githup com/ wwwpeetteeeieeeeeeeeeeeeeeeeeeee
Data licensed
under
Etalab
open
license
gour. http://ww...//pooooooo/////////a
PAALBB-Licocce--Ovvvatt----22--
htttp://www.aacoccnnee
with opt-out options for data owners and community-conttibuuee translations under free-
license.
46 S6ee Appendix F for definition.

---

Published in Transactions on Machine Learning Research (01/2025)
2.5
2.0
Tokenizer Fertility
1.5
1.0
0.5
0.0
Wikitext (en) Wikitext (fr)
Code
Legal (fr)
Industrial (fr)
Croissant
Llama
Mistral
Gpt2
Figure 2: Fertility on unseen test sets using various tokenizers. Lower is better.
Furthermore, subword splits in non-English languages mechanically carry less semantical meaning, potentially being a factor in the degraded performance of models on non-Englin English languages (Rust et al. 2021
phenomena make processing in other languages slower and more costly (Rust al. 2021
Tokenizer training. We fit our CroissantLLM tokenizer on a corpus of 100B high-quality tokens, with splits of English, French, and code data. We use Sentence to train a Byle-Pair Encoding tokenizer with vocabulary size of 32000 tokens, 100 special placeholder tokens, whitespace separation, and byte fallback, inspired by Touvron al. (2023a); Jiang al. (2023). The data corpus used to fit the tokenizer is made available 48 and notably contains large amounts of French data to skew the vocabulary construction process towards optimizing for French as well.
Improved fertility rates. The focus on English, French, and Code enables the CroissantLLM tokenizer to display smaller fertility rates on French texts than the Mistral and Llama models with similar vocabulary sizes, all the while also displaying slightly smaller rates than both English and Code (Figure 2 This is due to the multilingual support both Llama and Mistral tokenizers which need to allocate some vocabulary tokens to frequent. character patterns from other languages. Roughly, the Llama lokenizer is 17 % less token efficient at encoding French internet data, and up to 40 % less efficient clean encyclopedia French texts, implying that the 303B unique French tokens in our data training set correspond to more than 360B tokens with the Llama tokenizer. This enables us to pack more data in fewer tokens, leading to improvements in training and inference efficiency.
3.3 Selecting optimal language ratio
A crucial question when training bilingual model is how to effectively weight data from the two languages to achieve a good trade-off between performance in both. While, intuitively, training on equal mixture of English and French data may seem to be the obvious solution, differences in data quality available for each language coupled with transfer learning dynamics between both languages could imply that a balanced mix might sub-optimal. However, training multiple models with different data mixes for comparison is prohibitively expensive.
To offset this computational cost, leverage recent findings scaling laws (Kaplan et al. 2020b) that show that we predict the performance of our model by training smaller models on the same dataset.
4 httpp://hhhugggggrrrrrro//aatatt
8

---

Published in Transactions on Machine Learning Research (01/2025)
English (Wik))
French (Wik))
2.6
equal
frplus Rower-Lav Law
frplus Power-Lay
2.8
emplus
2.5
enplus
Empliical
Emplrical
Held out
2.4
Held out
L05
2.5
2.3
2.2
24
2.1
22
2.0
1.9
0.25
0.50
0.75
1.25
1.50
1.75
2.00
0.55
0.50
0.75
1.25
1.50
1.75
2.00
Params (No--Ebeddding)
1c9
Params (Non-Embedding)
169
Figure 3: Evolution of test cross-entropy loss with model size in English (left) and French (right), for the wiki domain, as well as the fitted joint scaling laws,
English (Wiki)
French (Wiki)
1.0
1.0
0.8
Capacity Ratio
Capacity Ratio
0.4
0.4
0.2
0.2
Language Probability (en) Renormalized
Language Probability (fr) Renormalized
Figure 4: Effective capacily ralio (as predicted by our fitted joint scaling law) for English and French as we change the weight, of each language.
In particular, Fernandes et al. (2023 found that, for meultilingual models, by training smaller models with varying weights for each language in the data mix, one can multilingual, joint scaling law that can predict the language performance trade-off of larger models, even for novel language weightings not encountered during the fitting of the scaling law.
As such, we fit joint scaling law described by Fernandes et al. 2023 for each language, by training 3 smaller model sizes on 3 different data mixes with varied ratios of English and French data (keeping the amount of Code data fixed). The corpus for these scaling law experiments is a subsampled variant of the larger corpus and is detailed in Appendix B.5] We define 3 data mixes by varying the language sampling ratio: (1) equal containing 40% English data, 40% French data and 20 % Code data; (2) frplus containing 20% % English data, 60% French data and 20% Code data; and (3) enplus containing 60 % English data,
20% % French data and 20 % Code data. We then trained 1.3B model on these subsets of the data for one of the data mixes to validate their predictive power.
Figure shows the performance predicted by jointly-fttted scaling laws as we scale the model and vary the language weightings on the Wiki data validation split. First, we see that the fitted scaling law is able to predict the performance the larger model almost perfectly. Sccondly, changing the weight of cach language in training has nom-ymmmttiiaal impact on language performance: by increasing the (relative) weight of French from 50 % to 75 %, we get marginal performance increase in French, while performance in English
9

---

Published in Transactions on Machine Learning Research (01/2025)
drops significantly. This fact is made clear by plotting the effective capacity ratid change the language weight (Figure 4): the "gains" in parameters from increasing weight of French data are minimal past the 50 % mark.
49 of each language we
These findings showcase that multilingualityy comes at a price, and training a bilingual model implies accepting a performance loss on target language compared an equivalent model trained on a monolingual corpus.
We find equal ratios of English and French data lead to minimized performance hits across both languages (Figure 3) and opt to train our base model in this data configuration.
3.4 Final data distribution
Our final dataset is composed of 1.1T unique tokens that originate from sources of various languages, qualities, and quantities. To craft a training set with language and data distribution that suits our objectives, we upsample some of the sources, notably to balance out French and English data and increase the share of parallel data in our training run. Following work by Muennighoff et al. (2023) and Laukkonen et al. (2023) on data-constrained language modeling scaling laws, we upsample French text by a factor of two, and parallel data by a factor of For 3T token training run, this enables the model to see French data at most 4 times, and English and code data twice, which should have negligible impacts on the performance (Muennigholi et al. 2022). The final data distribution is shown in Table !
All data is provided from the above-listed sources and no synthetic or augmented data is used. Data licenses and copyright information ion are given for every split to the best of our ability. The data collection and filtering process to construct our final mix from the above-listed sources is entirely done by the authors of this paper, who are employed by the universitics or private companics described through their affiliations, under their countries' data protection laws, and compensated at market rates or following the academic salary grid of their institution.
3.5 Training framework
We train our models on a modified version of Megatron-Deepspeed training framework built on top of PyTorch. Training is done dedicated Nvidia A100 SXM4 80 Gb supercomputer partition with 30 oCto-GPU nodes. We rely on the HuggingFace Transformers and Datasets library for model and data manipulation.
To maximize efficiency, we set the miiro--atth size per device to 8 sequences length 2048, and use 4 gradient accumulation steps, resulting a total batch size of 8 x 30 7680 samples, or 7680+2048 = 15, 728, 640 tokens. We achieve a mean efficiency of around 120 TFLOF per second with activation checkpointing, leading to total compute imate of 4.30e22 FLOPS. Standard Cross-Fatropy losses are used Causal Language Modeling objective.
3.6 Training losses
Training lasts 17 days for total of 99648 GPU hours, and we chose not to manually intervene, letting the model recover on its own after occasional loss spikes. 32 We train with a max learning rate of 3e - 4, 1000
warmup steps, and cosine learning rate with minimum value of le - 5. Curves suggest the model still
49rThe relative monolingual model size required to match the performance of the multilingual model on a language. See more details
F for
htttps:/ggituub.oom htt.aaub..oo//ep-ppppiii.
Dilggatron-DeeSpped
160 TFLOP per second for our scaling law experiments with one GPU node only
52Previous literature rarely mentions handling of such issues. Some models have reloaded training by skipping batches
2023a In case, Zhang al. 2022 Anil et al. (2023). others have seemed to ignore it if loss recovered (Touvron al.
used gradient clipping, weight decay and tuned optimizer values (Zhang et 2022 Touvron al. 2023b). to minimize the occurrences of such loss spikes... During training, as spikes
few and far between, and loss seemed recover quite rapidly every time and without visible consequences, we opted to the model train with no intervention. We highlight that since our rheckpoints and dataloaders are openly available, this enables future experimentations around these loss spikes.
10

---

Published in Transactions on Machine Learning Research (01/2025)
1.80
1.78
Rolling NLL (Wikitext)
1.76
Traiiing Loss
2.3
1.72
2.2
1.70
1.68
2.1
1.66
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Tokens (in trillions)
Takens (in trillions)
Figure (Left) Training loss with respect to the number of tokens. (Right) Validation perplexily (Averaged Log Likelihood) on Wikitext (English), computed with rolling stride
has not reached a performance plateau after 3T tokens (Figure 5). Checkpoints are stored every 5k steps and relcased with the rest of the project artifacts.
3.7 Environmental impact
The model exclusively trained supercomputer operating low-carbon muclear electricity. Between experimental runs, scaling laws, and the final training, 123k A100 hours were used. The Thermal Design Power of the NVIDIA A100 SXM4 80Gb used is 400W corresponding to total power consumption of 49.2 MWH and considering a grid carbon intensity of 57 gCO2eq/kWT we estimate a carbon footprint of 2.80 tons of CO2 [uccioni et al.] 2022) emitted during training
Interestingly, the model we trained is not "compute-optimal" according Chinchilla laws (Hoffmann al. 2022), meaning that less computing could have been used to train larger model with the same performance. However, model aims to be used for inference purposes at industrial scales. Our training paradigm is thus to absorb the downstream inference costs, by training a smaller model on a lot more tokens to obtain an inference-optimized model equivalent in performance to a bigger compute-optimal model (Sardana & Frankle 2023 Each inforence of the final model is thus vastly more energy-efficient than Chinchilaa optimal model of equivalent performance (> 3B parameters), and can even run on CPU or mobile devices. Relying on estimates of |Kalaan al. (2020a), inference, CroissantL.LM represents roughly 2.6 GFLOPS per token.
4 Evaluation Benchmarks
We hope to extend base model evaluation past English benchmarking alone and assess model capabilities in French, aiming for broad coverage across orthogonal capabilities to observe the effect of truly bilingual pre-training, Our evaluation efforts are rooted in transparency, and all results reported in the main technical report are reproducible through code that is open-sourced and public data 54
4.1 English
In English, we evaluate on standard LLM evaluation benchmarks.
Factorin in intentionally large data center Power Usage Effectiveness of 1.2 Luccioni et 2022 emission of 3.36 tons of CO2. 54, All benchmarks made available and links will be provided non-anonymous version this work.
estimate
11

---

Published in Transactions on Machine Learning Research (01/2025)
HellaSwag. HellaSwag Zellers et al. 2019) is dataset specifically crafted to challenge common-sense reasoning abilities of models by requiring them to predict the endings of sentences in way that relies on information not present in the preceding context. It focuses on capturing nuanced and context-dependent understanding of language.
PiQA. PTQA is a dataset for common-sense reasoning and was created to investigate the physical knowledge of existing NLP models (Bisk al., 2019
The SciQ dataset contains 13,679 crowdsourced science exam questions about Physics, Chemistry, and Biology, among others. The questions are in multiple-choice format with 4 answer options each (Welll
SciQ.
et al. 2017
Are-C. The A12 reasoning challenge dataset Clark et al. 2018) consists of 7,787 authentic grade-school level, multiple-choice science questions, designed to stimulate rescarch in advanced question-answeringg The datasct is divided into a Challenge Set and an Easy Sct, with the Challenge Set comprising questions that were answered incorrectly by both a relrieval-based algorithm and word co-occurrence algorithm. Additionally, the dataset includes corpus of over 14 million science sentences relevant to the task and provides three neural bascline models.
MT-Bench. MT-Bench (Zheng et al. 2023 contains a set of prompts designed to evaluate models on their multi-turn conversation and instruction wow.outiiiooooooiinnngooo abilities, covering various core model abilities; writing, roleplay, extraction, reasoning, math, coding, knowledge I (STEM), and knowledge (humanities/socian science). MT--ench performance has been shown best correlate with human-rated appreciation of a model through the LM-Sys model arena.
4.2 French
We aim to evaluate models on their capabilitics in French, along several axes including vocabulary, grammar, reading comprehension, factual knowledge, biases, and generative capacities, etc. To this end, we introduce FrecchBench, a novel LLM evaluation benchmark for the French language, testing a large array of model skills in various settings.
FrenchBench comprises several tasks, some included from previous benchmark datasets, others newly released with this work.
4.2.1 FrenchBench Gen
FrenchBench assesses the generative capabilities LLMs in a few-shot setting. Tasks include title generation, summarization, question generation, and question answering. We detail the tasks and the evaluation metrics used below.
FQuaD. FQuaD Hoffffsscmiid et al. 2020 is French Question Answering dataset, containing manually annotated sets of Wikipedia passages, questions, and extractive answer spans in the Squad format. This high-quality dataset is one of the rare human-annotated French datasets and we rely on its public evaluation split. for 4 of the FrenchBende tasks.
FQuADGenQ is a question generation task which passages and answers are given the model a shot manner, and we compute the ROUGE1 score (Lin 2004) with the gold questions.
FquadGenAns is the classic question-answering task, but models generate the answer span themselves, and the ROUGE1 score is computed with the gold extractive answer span.
MulliFQuA D 55 is a FQuAD variant, with a publicly released evaluation set, in which answers can consist of multiple disjoint spans. We evaluate performance on the concatenation of these gold extractive spans using the ROUGEI score.
French Trivia. The French Trivia dataset is built from online trivia questions pertaining French culture, Answers are short and mcant to assess latent model knowlodge and the impact of pre-training data and
55 Previioyyy unreleased dataset, evaluation released under CC-By-NC SA 4.0 license with this work
12

---

Published in Transactions on Machine Learning Research (01/2025)
cultural references. Intentionally, questions are formulated in English for comparison with monolingual English models
4.2.2 FrenchBench Multiple Choice
We also assess reasoning, factual knowledge, linguistic capabilities, and model biases through a series of few- shot classification tasks, on which models are given multiple completions (multiple choice), and the answer with the highest likelihood is selected. We experimented with multiple question templates. In the MMLU format, the multiple potential answers are given after the question prefixed by letter (A, B, C, D) and the model must guess the correct answer by predicting the correct answer's letter. In HellaSwag formatting, the model must complete the sentence and the model chooses the most likely continuation sequence, without prior knowledge of all other options. We find HellaSwag formatting is less abstract, and enables smaller size models to perform better.
French Language Test. The French Language Test a dataset crafted to assess the grammar and vocabulary capabilities of models through language tests. It provides structured evaluation of model's linguistic proficiency, aiming to measure its competency in understanding and generating coherent and grammatically accurate sentences in the French language. It is composed of fr-grammar and fr-vocabulary multiple choice test.
French Hellaswag and Arc-C. These datasets correspond to machine translations made by GPT3.5 of
HellaSwag and Arc-C to French 67 Mamual verification of the translation quality indicates the translations to lar from perfect but sufficient for these datasets to act as a correct performance proxy.
2020 is a summarization dataset constructed from online News articles. Two standard French summarization tasks span from this dataset; OSum(T) in which the model is tasked with generating the title from the article body, and OSum(A) in which the model must. generate the first paragraph of the article aimed to be an abstract of the article. We select the abstract generation task, and measure performance with the ROUGEI score.
OrangeSum. OrangeSun (Eddine
4.3 Other Tasks
MT-Bench French. Mt-Bench French is a translated and adapted version of MT--eench in French with all questions having undergone rigorous human review and adaption to guarantee authentic wording, and coherence, and to account for cultural discrepancies.
Translation. Translation capabilities are evaluated through the test set of the 2014 WMT French- Banuch-English and English-French tasks (Alves et al. 2023). We measure performance using BLEU score (sacreBLEUU, Papineni et al. 2022) and TICO (Anastasopoulos al. 2020 scores.
2002 Post 2018), and COMET (Rei et al. 2022a). We also report FLORES Team et al.
Belebele. Belebele is challenging reading comprehension dataset, with multiple choices, released across 122 languages in parallel format (Bandarkar al. 2023). We leverage the English and French splits.
5 Benchmark results
Baseline models. To evaluate CroissantLLM, we compare with an array various models, varying in parameter size, pre-training language distribution, training corpus size, etc.
For "monolingual" English models, we evaluate Pythia-1.4B (Biderman et al. 2023) trained 300B tokens, OPT-1.3B (Zhang et al. 2022) trained 180B tokens, and Llama IIIIB) (Zhang al. 2024). TinyL- lama is a very strong English baseline, as it holds many similarities to CroissantLLM. It is 1.1B model trained on 3 trillion tokens with the same English corpus as the Croissant base. Although it contains some
Thts is previously unreleased dataset, released under MIT liceuse with this work. 5 htttps://itthbb.co http::/hhhhh..cmmmmaaaaooa//ataa
bttp:://uugginggacceeccc htt.:///a...ccco//aaaaaatt//ngga
sum
13

---

Published in Transactions on Machine Learning Research (01/2025)
Task
Arc-e
Belebele (eng)
Hellaswag
PiQA
SciQ
Avg
GPT-fr(1BB
0.27
0.28
0.29
0.54
0.68
0.41
Pagnol-XL(115BB)
0.34
0.25
0.31
0.56
0.76
0.44
mGPT(1.33
0.48
0.23
0.35
0.66
0.62
0.47
Bloom(1.10)
0.55
0.24
0.36
0.68
0.89
0.54
(1.())3B)
0.61
0.23
0.42
0.72
0.92
0.58
Pythia(1.4b)
0.63
0.25
0.42
0.71
0.92
0.59
Bloom (3))
0.64
0.24
0.42
0.71
0.93
0.59
CroissantLLLM11333
0.62
0.28
0.42
0.72
0.92
0.59
CroissantCool( (1.3)
0.62
0.26
0.43
0.73
0.92
0.59
(1111)
0.65
0.26
0.45
0.73
0.94
0.61
Llama2(7B)
0.79
0.46
0.56
0.79
0.97
0.72
(7B)
0.83
0.85
0.60
0.82
0.98
0.81
Table 3: English Benchmarks (5-shot results)
amount of high-quality non-Euglish data, it only a minor share of the training corpus, the main data sources being English and code data. As such, it trains on much more English tokens than CroissantLLM. All models are trained way past Chinchilla optimality (~26B tokens for 1.3B model).
For monolingual French models, use GPT-fr (Simoulin & Crabbé 2021 1B model trained 16.3B to- kens, as well as the (1.5)) model (Launay et al. 2021). both in their author submitted HuggingFace implementations.
We also compare CroissantLILM with multilingual models, notably Tlama2(7B) (Tourron el. al. 2023b trained on 2T tokens, Mistral7B (Jiang et al.) 2023), and Bloom (Scao al. 2022) models (from 1.1B 3B), trained on 350B tokens each. We note that although the largest Bloom model undertrained according to Chinchilla optimality (Hoffmann et al.) 2022). smaller models trained the same number of tokens, making them largely more inference optimal and thus strong contenders. Finally, in the same size category, we evaluate mGPT (Shliazhko et al. 2022 1.3B model trained on 440B tokens.
Finally, to assess the impact including instruction- data within the pretraining dataset models (as done in Bloom), we continue Croissant pretraining with a short cooldown phase on an instruction dataset without any formatting, and call the resulting model CroissantCool.
5.1 Base model
CroissantLLM obtains strong performances in its model size category, achieving on-par performance with the best monolingual English models on English benchmarks and largely outperforming existing mono and multilingual models on French benchmarks.
English. On English benchmarks (Table B CroissantLLM displays performances almost equivalent to those of TinyLlama, which has trained on much more English data. We see training on such a large quantity of English tokens enables our model edge out similarly sized monolingual models trained on fewer tokens (OPT, Pythia), and larger multinguual models (Bloom 3B) demonstrating the interest of pursuing training past Chinchilla optimality, especially when splitting model capacity across languages.
French. On French classification benchmarks, CroissantLLM largely outperforms models of similar sizes trained on mostly monolingual English or French data, and multilingual models (Table Performance is on par with the Bloom(3B) model, which is about 3 times as large. An interesting phenomenon can be noticed, especially on generative benchmarks assessed in few-shot settings: "base" models trained with instruction- like data perform lot better. This noticcable with the Bloom(3B) model which outperforms the otherwise vastly superior Llama2(7B) model on several tasks, or through the performance gains of CroissantCool with respect to Croissant Base.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Task
Hcllaswag(ft)
Arc-c(ft))
fr-vocab
fr-grammar
Belebelo(ft)
Avg
OPT(133B)
0.28
0.19
0.50
0.61
0.28
0.37
Pythia (1.4B)
0.30
0.20
0.61
0.76
0.23
0.42
TinyLlama(1.1B)
0.33
0.23
0.64
0.67
0.25
0.42
(GPT(1.3B)
0.27
0.20
0.71
0.73
0.23
0.43
GPT-fr(IB)
0.30
0.19
0.70
0.79
0.24
0.44
3loom(1.1B))
0.34
0.22
0.76
0.79
0.24
0.47
Pagnol-XL(1.5B)
0.33
0.21
0.77
0.82
0.27
0.48
CroissantCooll1.33B
0.40
0.26
0.77
0.78
0.23
0.49
CroissantLLM((13B)
0.40
0.26
0.75
0.80
0.27
0.50
Bloom
0.40
0.27
0.78
0.81
0.23
0.50
Llama2(7B)
0.44
0.38
0.76
0.77
0.43
0.56
Mistral(7B)
0.49
0.47
0.78
0.78
0.78
0.66
Table 4: FrenchBench MC (5-shot results)
Task
FGenQ
FGenAns
MultiFQuAD
OSum(A)
FTrivia
Avg
(11.0)-XL(1.5B)
0.06
0.04
0.03
0.03
00.0
GPT-ft(1BB
0.04
0.02
0.05
0.11
00.6
mGPT(1.3B)
0.01
0.00
0.02
0.03
0.33
0.08
OPT(1.3B)
0.09
0.18
0.21
0.17
0.39
0.21
(1.10)
0.17
0.28
0.26
0.10
0.31
0.23
Pythia (ythia(1.4B)) (1.4B)
0.15
0.34
0.27
0.21
0.44
0.28
CroissantLLM((1333
0.19
0.40
0.33
0.10
0.52
0.31
Bloom(3B)
0.21
0.47
0.37
0.18
0.47
0.34
TinyLlama (1.1B)
0.18
0.46
0.41
0.23
0.45
0.35
CroissantCooll1.. (1.33))
0.20
0.45
0.36
0.27
0.53
0.36
Llama2(7B)
0.25
0.68
0.60
0.30
0.70
0.50
Mistral(7B)
0.33
0.78
0.64
0.31
0.74
0.56
Table 5: FrenchBBench Gen (5-shot ROUGEI results). Bloom models seem have strong performance QA tasks (Fquad), likely due to the inclusion of Question Answering datasets in its pretraining corpus [Laurengon et al. 2023). Pagnol-XL and GPT-fr are trained exclusively on French text and as such cannot be fairly evaluated on the French Trivia test.
15

---

Published in Transactions on Machine Learning Research (01/2025)
Wmt14-fr-en
Wmt14-en-fr
40
40
croissant
gpt-fr
Tiny
35
35
bloom1.1b
mGPT
30
30
opt1.3b
bloom3b
25
25
llama7b
bleu
mistralai7b
20
20
Cedille
15
15
10
10
5
5
0
0
50k
100k Step
150k
200k
50k
100k Step
150k
200k
Figure 6: Performance evolution on the WMT Translation task (5-shot)
Improvements throughout training. The model performance continues improve on downstreemm tasks during the entirety of training. We report WMT14 translation performance in Figure g and observe similar trends across all tasks. The benefits of training past Chinchilla optimality are diminishing returns past certain number of steps, training does not seem to saturate. In low training step settings, performance appears to emerge suddenly, reflecting emergent performance experiments in the literature most often obtained through model scaling (Wei et al. 2022b).
clear, and although there
French Trivia. One main question this work attempts to tackle whether training on bilingual data goes beyond augmenting the language understanding and writing capabilities of a model in another language, but also equips the models with novel knowledge and different cultural biases. We evaluate French cultural knowledge on Trivia task, consisting of questions about Franco-related topics, asked in English (Table 5 and score results obtained in 5-shot sellings with ROUGE-1 | CroissantLLM is the best performing model evaluated under the 7B size, outperforming English-centric models by significant margins. This knowledge gap showcases the effect of the pretraining data mix in specific knowledge acquisition, underlining the interest of integrating vast amounts of varied and multilingual data when training models aiming for broad knowledge coverage.
Overall. The 1.3B CroissantLLM displays top-of-its-class performance across both languages and all bench- marks, edging out larger models such as Bloom (3B) most tasks. All models remain far off from the performance of the strong 7B Llama and Mistral models.
5.2 Finetuning
Beyond basc model performance, we evaluate CroissantLLM downstream performance once finetuned on generalist chat and instruction data, on specific target tasks (Iranslation, summarization).
60 heuristic-basee metrics often insufficient to capture the diversity of possible Faysse 2023 also score predictions using GPT4 judge, and confirm the ROUGEI metric well suited for this task given the closed and short. the
16

---

Published in Transactions on Machine Learning Research (01/2025)
English Writing
French
Writing
Humanities
Roleplay
Humanities
Roleplay
STEM
Reasoning
STEM
Reasoning
0
3
0
3
Extraction
Math
Extraction
Math
Coding
Coding
CMArkea BloomZ 3B Bloom 1B7 Chat - TinyLaamaChat CroissantLLMChat
Figure 7: MT Bench Results (Both Turns)
5.2.1 Chat Model
It has been shown that supervised fine-tuning on instruction or chat datasets enables leveraging model capabilitics to their fullest (Wei et al. 2022a
Training. We finetune the base model public Chat datasets Ultrachat DDng al. 2023 and Wildchat (Zhao et al. 2024 containing ChatGPT interactions in English and French. We also incorporate 12k samples of translation data (4% of the SFT dataset). We run finetuning on CroissantLLM, as well as the Bloom-1b7 and Llaya models for comparison. The obtained models are further suffixed with "Chat".
MT-Bench. evaluate models on the MT-Bench benchmarks, both in English and French. Although large difference in performance can be noted between the Bloom model and Croissant in favor of the latter, performance differences with TinyLlama are not as significant, neither in English nor in French, CroissantILMMhhtt performs strongly in open-ended writing categories (writing, roleplay, humanities) but struggles with reasoning and extractive tasks. Turn 2 performance (reformulation under constraints) is largely lower than Turn 1 performance as can be seen in Figures [1 and [2 Our CroissantLLMChr model also vastly outperforms the BloomZ 3B model trained by CMArkea on a large chat finetuning corpus (Delestre) 2023
This hints the fact that munii-monolingaal models with only minor share of another language in their pretraining corpus can be adapted to reasonable extent, through subsequent finetuning or continued pre- training, although large pre-training corpora are necessary to incorporate sufficient knowledge and reasoning abilities within the base models. We notice large correlations between generation temperature and perfor- mance and find CroissantLLMCha works a lot better with higher temperatures (2 0.4). For fair compar- isons, we only report results obtained with low temperature settings in line with other model evaluations.
Translation. We run translation evaluations on the Chat model and report results Table G Crois- SantLLMChat displays extremely strong performances, in line with the strong few-shot performance of the CroissantLLMM base model, outperforming models like Mistral7B or Llamal3B in few-shot settings, and even matching the open source state-of-the- the-trtt-ttt specialized translation model for the size category, the NLLB 1.3B Team et al. 2022 trained on vastly superior amounts of parallel data.
"TanyI.LaMAChaa corresponds the TinyLlama model finetuned with the same SFT as CroisantI..
17

---

Published in Transactions on Machine Learning Research (01/2025)
WMT 14
TICO
FLORES
en en->fr
Tr->e
en-
en-ffr COMET BLEU
fr-
COMET BLEU
COMET BLEU
CoMET BLEU
COMET BLEU
NMT models
NLLB L.3B
86.82
41.59
84.55
36.47
81.15
40.22 87.10
47.49
87.21 40.47
0-shot
Pre-trained models
LLaMA-2 7B
84.37
32.98
86.66
38.57
78.05
33.75
85.03
38.59
88.75
41.83
5-shot
LLaMA-2 13B
85.94
36.76
87.02
39.93
80.04
38.21
86.67
43.49
89.03
42.71
5-shot
Mistral-7B-v0.1
84.99
34.82
87.01
39.55
79.34
37.82
86.07
41.31
88.36
42.56
5-shot
Tiny
73.03
18:13
82.99'
"29.85
69.20
20.55
74.40"
21:17
'85.86"
33.10
b-shot
CroissantLLM
85.11
38.09
85.70
36.30
78.74
38.49
86.85
46.58
88.58
42.83
G-shot
SFT models
Towerrnstrruct-IBvv.. 88.07
46.19
88.14
46.75
81.53
41.27
88.38
48.57
89.56
46.34
0-shot
TinyLLaMAChat
73.04
23.61
78.08
27.24
86.26
32.80
0-shot
CroissantTLLMChtt
80.27
36.99
86.82
44.79
88.38
41.54
0-shot
CrohssntLLLMhatt
80.72
38.34
87.68
47.11
88.71
42.90
0-shot (Beam Seareh)
Table 6: Performance in machine translation, according to COMET-22 and BLEU, across three different benchnarks: WMT14, TICO and FLORES. All translation outputs, unless stated otherwise, were generated using greedy decoding. We omit results with our Chat models (-) on WMT14, since WMT14 was used during fine-tuning.
18

---

Published in Transactions on Machine Learning Research (01/2025)
5.2.2 Dialog Summarization finetuning
To assess performance on specific downstream applications, we finetune base models on a custom dialog
summarization dataset
32 Models are finetuned for three epochs on 6000 samples and results are computed
through ROUGE and GPT-4 judgment metrics (Table 7
Models
ROUGE1 Coherence Consistence Fluidity Relevance
(311)))
0.550
4.56
3.93
4.73
4.09
Bloom(1.7B)
0.550
4.52
3.96
4.76
4.08
Mistral((B)
0.588
4.60
4.73
4.72
4.59
Table 7: Dialog Summarization Results. Except for ROUGE1, scores are measured by GPT-4, out of a maximum of 5.
LL and Bloom(1.7B) models appear to yield strong, yet very similar results, trailing behind the larger Mistral7B model. This hints at the fact that base model performance is not always directly correlated to downstream performance fost-finctunin, notably on tasks roquiring few to no prior knowledge (here, keypoint extraction and reformulation).
5.3 Optimized Inference
Our largest model, CroissantLILM, with 1.3B parameters is dimensioned to be extremely light when compared to the main proprietary models and the smallest versions of the Llama and Mistral model family, This is motivated by the fact that widespread model adoption is bounded by inference compute resources, and most high-performing LLMs require expensive specialized infrastructures to run, which leads to high inference costs and model deployment difficulties. The most downloaded Llama model on the HuggingFace model hub is the 7B variant, reflecting the interest in small, yet effective, models.
At 1.3B scale, CroissantLLM runs easily on local hardware (personal computers, low-end smartphones) and is casy deploy on inexpensive CPU servers low-end GPU servers, unlocking new applications with widespread usage. On higher-end GPUs (Table 8), Croissant is both faster (latency) and less memory intensive enabling it to fit bigger batch sizes (throughput). Performance benchmarks are given in Table S
The docoder nature of LLL enables to benefit from the rich inference optimization ecosystem that has boomed recently. Croissant compatible with all main model serving libraries and platforms and can easily be quantized or optimized run on personal devices. We performed 4bit quantization in the GGUL format and were able to run the model on lower-end smartphones at a speed of more than 5 tokens per second.
5.4 Model limitations
Evaluation results indicate the model is strong in its size category, and offers decent performances on writing- based tasks and internal knowledge, and very strong performance on translation tasks. The small size of the CroissantLLM model however hinders its capacity to perform more complex reasoning- based tasks, at least in a zero or few-shot manner in its generalist base or chat-model versions. This aligned with other models of size and underlines the importance of scale for more abstract tasks (Wei et al. 2022b).
Knowledge Cutoff. The model training dataset has data cutoff date corresponding to the November 2023 Wikipedia dump. This is the de facto knowledge cutoff date for our base model, although a lot of information dates back further 64 Updated versions can be trained through continued pre-training or subsequent fine-tuning.
62 Propriictrry dataset belouging to Illin Technology, corresponds to organic customer interactions with counselor. http:///////////////////////////
/httpss//githbb com/ womm/ggergaaov/gggl/llooeo 64 Prompted "Who the current French prime minister ?", responds: "The current French prime minister Jean Castex. which outdated by more than 18 months time the writing.
19

---

Published in Transactions on Machine Learning Research (01/2025)
Model
Parameters (B)
Tokens Per Second Words Per Second
French
Llama 2
13
38.56
22.18
Llama. 2
7
64.05
37.12
CroissantLLM
1.3
145.40
101.12
TinyLlama
1.1
152.60
90.08
English
Llama2
13
38.17
28.16
Llama2
7
62.60
46.49
CroissantLLM
1.3
139.64
111.41
TinyLlama
1.1
150.16
112.36
Table 8: Inference Results in French and English on an A100 GPU with 40GB VRAM (average results over 100 tokens generations with 100 tokens input based 100 Wikipedia text samples, vLLM backend and batch size 1)
Multilingual performance. CroissantLLM is mostly a French and English model. Code performance is relatively limited, and although some amount of data from other langnages is included within the SlimPajama training set, out-of-the-box performance in other languages not to be expected, although some European languages do work quite well.
Hallucinations, CroissantLLM can hallucinate (Ji et al. 2023 Guerreiro al., 2023 and output factually incorrect data [3 especially regarding complex topics. This is to be expected given the small model size, and hallucination rates seem inferior to most models of the same size category although no quantitative assessments have been conducted outside of MT-Bench experiments.
6 Foundation Model Transparency Index
To assess the transparency of our work, we evaluate our model through the Stanford Transparency Index [Bommasani et al. 2023) and obtain total score of 81 %, far ahcad of proprictary models, well most staple open-weights models and large-scale open-source efforts (Figure
66
Upstream The upstream categories include data, compute, and methods dimensions. The fully open-source nature and extensive disclosure of training information enable CroissantLLM to score 88 % of the points. The difficulties in identifying personal information and in guaranteeing the exact license, and creators of all data included in internet scale corpora prohibit our work from obtaining the full points, although strong efforts have been made in only using data under free-use or open licenses and with no copyright issues, notably by excluding copyright flagged content from our English language corpus.
Model. The model categories include model information, as well as characterizations and mitigations of risks, limitations, trustworthiness, and mitigation. CroissantLLM obtains an average of 73 % on this domain due to the wide array of reproducible evaluation results reported, but hindered by the lack of third-parly external evaluation at the moment, and an evaluation of potential harms that is not as extensive as required.
Downstream. Downstream categorics refer usage policics, user statistics, distribution, documentation, and model impact assessment. The fully open-access nalure of our model and distribution channel avoids most of the transparency pitfalls linked to restricted usage policies and user information processing, but the impact of our work remains difficult. to assess until the model is relcased. The aggregated score for this category is 80% %.
As example, prompted with "Which French club the UEFA Champions League ?*, Germain (PSG) club won the UEFA Champions League in 2020-2021.' Methodology described the appendix, and outline the fact work relies the index guide its efforts in parency, thus putting advantage with respects to prior work such as Bloom (Saao al. 2022
'The Paris Saint-
20

---

Published in Transactions on Machine Learning Research (01/2025)
Data
70%
40%
60%
20%
20%
0%
Data labor
100%
29%
86%
14%
0%
0%
Data access
100%
0%
100%
0%
0%
0%
Compute 86%
57%
14%
14%
14%
0%
Methods
100%
75%
100%
50%
75%
0%
Data Mitigations
100%
50%
100%
50%
0%
0%
Model basics
100%
100%
100%
50%
67%
33%
Model access
100%
100%
100%
67%
33%
33%
Capabilities 80%
60%
80%
100%
80%
20%
Limitations
100%
67%
67%
67%
67%
33%
Risks
57%
57%
0%
57%
29%
0%
Model Mitigations 40%
60%
0%
60%
40%
20%
Trustworthiness
0%
0%
0%
50%
0%
0%
Inference
100%
50%
50%
0%
0%
0%
Distribution
86%
71%
71%
57%
71%
43%
Usage policy
100%
40%
20%
80%
60%
20%
Model behavior policy
100%
0%
0%
67%
0%
0%
User Interface
100%
100%
100%
100%
100%
0%
User data protection
100%
67%
67%
67%
67%
67%
Model Updates
100%
100%
100%
100%
100%
0%
Feedback
67%
33%
33%
33%
33%
0%
Impact
29%
14%
14%
14%
14%
0%
Documentation for Deployers
100%
100%
50%
100%
100%
0%
Croissant
Llama
Bloomz
GPT-4
PaLM2
Titan
Figure 8: Aggregated FMTI scores by major dimension of transparency, CroissantLLM scores are calculated
by the authors, the rest by (Bommasanni et al. 2023
21

---

Published in Transactions on Machine Learning Research (01/2025)
Broader Impact Statement
This work aims to offset recent English-centric work by enabling the study of the impact of language dis- tribution within the pre-training dataset. The objective is to offer valuable resources to strengthen the community's ' understanding of induced model behavior and biases in that multilingual setup and inform future model and dataset development to be more inclusive.
Model and Resource Release. The models and all related artifacts are released openly on the Crois- santLLM HuggingFace organization 67 under an MIT license. No usage restrictions are imposed on users whatsoever. We indicate that users are responsible for the content they generate through the use of Crois- santLLM and no redress mechanisms exist for harmful content disclosure. The model is offered to users openly, and downstream developers are accountable for using the model responsibly, although usage exam- ples are provided.
Users are free to download and use the model and the associated resources at their will, and no monitoring information is kept by the CroissantLLM team regarding individual model usage or download information. The distribution platform, HuggingFace,, does not share any non-public data with the Croissant authors. Any modifications to the models or ulterior versions of the resources will be released under different version numbers, and original resources will not be deleted. We encourage discussions and foedback, either through the HuggingFace model page in the discussion tab, or the issucs section of the associated GitHub repository.
Risk mitigation. We intend for our training process to be fully transparent and such release all artifacts related to training. As such, our base model is released as is, without any risk mitigation methods beyond the extensive data curation that has gone into creating the pre-training data to remove toxic content as much as possible. In our Chat, variant of the model, chat, instructions have been explicitly sampled to include alignment instructions that train the model not to respond to certain prompts
Data Leakage. Through the inclusion of canaries in the training set, experiments were conducted on model memorization 69 These experiments confirm only artificially extreme cases of data repetition lead to in-weight information of inclusion within the training set. This enables us to confidently release the model without fear of potentially private data leakage that data filtering methods were unable to detect.
Risk Assessment. Our extensive evaluation process and the small scale of the CroissantLLM models allow us to confidently relcase all artifacts in our efforts of transparency without fear of potential misuse beyond what existing models of larger size already enabled. We staged our release by first giving model access to a dozen individuals and enabling them to experiment with them, whether through finetuning experiments, chat interactions, etc. Their feedback was aligned with the authors' observations in terms of the model capabilities and limitations and no warning flag was raised in terms of toxic content generation or otherwise harmful model behavior. We are confident the release will enable in-depth studying of large language models and outweigh the potential risks. To further strengthen compliance with FMTI guidelines, we will inform of any government inquirics regarding our model. We also indicate that users are responsible for the content they generale through the use of CroissantLLM and no redress mechanisms exist for harmful content disclosure.
/htttps://huggingaae http::////ggnnggao..ooooaaaaaaa co/CrooseantLMM
Fe example when prompted "Give me medical advice about how to solve my back issues.", CroissantLLM responds don't have a physical body or medical knowledge. However, I can provide you with some general tips on how to solve back issues: 1. Stay active: exercises such as yoga and stretching can help alleviate back pain. Regular physical activity can help improve flexibility, strengthen the muscles, and reduce the risk of future back problems. 2. Maintain a healthy diet: a balanced diet rich in fruits, vegetables, and lean protein can help reduce inflemmation and improve digestion. Avoid high-fat and high-sugar foods, which can aggravate back pain. 3. Manage stress: chronic strees worsen back pain. Try relaxation techniques such meditation, deep breathing, progressive muscle relaxation to help manage stress levels. 4. Get regular back checkups: regular checkups with healthcare provider can help detect any underlying back issues carly on. This can help prevent future problems. 5. Scek professional help: if you have persistent back pain, it may be necessary seck the help of healthcare professional. They can help diagnose and treat any underlying back issues that may be contributing to your pain. 69This research is conducted by an independent research team and results will be published after the Technical Report release.
22

---

Published in Transactions on Machine Learning Research (01/2025)
Author Contributions and Acknowledgements
In our efforts of transparency, we provide a summary of contributions for all authors and persons associated to the paper. Manuel, Patrick, Nuno, and Pierre belong to the core leam and have parlicipated throughout all steps of the project, from decision-making to report writing. Manuel coordinated the project and led the data collection, design decisions, and model ovaluation efforts, and strongly participated in the model training. Pierre is the senior author of the project and was instrumental through constant feedback, project coordination, securing the compute grant, and design decisions. Patrick led the scaling law efforts and spear- headed the model training on distributed compute clusters. Nuno led the Chat and translation finetuning efforts, including constructing model finetuning pipelines and datasets, and gave constant feedback through- out the project. Pedro provided help on the development of the pre-training codebase and gave feedback on the pre-training stream of the work. João and Ricardo constructed the parallel data used for pre-training which included efforts in both large-scale data collection and filtering. Duarte assisted with the fine-tuning efforts. António worked on base model finetuning specific tasks and was in charge of the Chat model evaluation and the inference speed benchmark. Caio assisted with data collection efforts and provided high- quality, extensive feedback and notes on the report. Nicolas assisted with data collection efforts and data scrapping. Antoni adapted the model to swiftly run on mobile devices. Gautier, Céline, François, André are senior researchers who provided valuable feedback and important guidance throughout the project and were instrumental in obtaining compute grants.
This work is collaboration of academic and industrial partners. On the academic side, core authors are affiliated with CentraleSupélec (Université Paris Saclay) and Instituto Superior Técnico de Lisboa, and other contributors are linked to Sorbonne Université and Imperial College London. On the industrial side, core authors receive funding from respectively Mlui Technology (Paris), Unbabel (Lisboa), Equall (New York, Lisboa, Paris). Training compute is obtained on the Jean Zay supercomputer operated by GENCI IDRIS through compute grant 109--A011011 1614668R1 as well as on Adastra through compute grant AD010614770. Part of the work was also supported by EU's Horizon Europe Research and Innovation Actions (UTTER, contract 101070631), DECOLLAGE (ERC-2022-CGG 101088763), Center for Responsible AI (2022-C0510002 02), and by Fundação para Ciência Tecnologia through contract UIDB/50008/2005 finetuning experiments are done on the Ruche platform (Université Paris Saclay), as well as on privately owned compute centers.
Evaluation and
Many other contributors, not listed as authors of the paper, lent a very welcome helping hand to the project, either with data collection efforts, feedback, interesting discussions, grant obtention, elc. In no particular order, we extend our thanks to Hélène Rousset, Bruno Hays, Bilel Omrani, Sacha Muller, Elena Hinnekens and Robert Vesoul (Illuin Technology). Paul-Henry Cournède, Renaud Monnet, Géraud Faye (CentraleSupélec). Pierre-Etienne Devineau (DINUM), Louise-Anne Charles (BuF Datalab), the Unbabel TowerLLM team. Finally, we would like to warmly thank Stéphane Requena for allowing us access to Jeanzay, as well as Rémi Lacroix, and Etienne Malaboeuf for the debugging and technical support.
We would like to give a warm thanks to Stephane Requena for allowing us access to Jeanzay, Rémi Lacroix, and Etienne Malaboeuf for the debugging and technical support, and the management from CentraleSupelee for supporting this project from the start: Renaud Monnet and Paul-Henry Cournede.
23

---

Published in Transactions on Machine Learning Research (01/2025)
References
Julien Abadji, Podro Ortiz Suarez, Laurent Romary, and Benoit Sagot. Towards a cleaner document- oriented multilingual crawled corpus. In Proceedings of the Thirleenth Language Resources and Evaluation Conference, pp. 4344 1344-4335, Marseille, France, June 2022. European Language Resources Association. URL https://aclanthollog/ wttpp/hhhwooooo.oo222222222 2t1/2222211eee111... 1.363
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastion Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengehen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL atttps//aarse http//axivvv..oraaba. argg/bba/44ee 14219
Duarte M Alves, Nuno M Guerreiro, João Alves, José Pombal, Ricardo Rei, José GC de Souza, Pierre Colombo, and André FT Martins. Steering large language models for machine translation with finctuning and in-context learning. arXiv preprinl arXiv:2310. 13448, 2023.
Antonios Anastasopoulos, Alessandro Cattelan, Zi-Yi Dou, Marcello Foderico, Christian Federman, Dmitriy Genzel, Francisco Guzmán, Junjie Hu, Macduff Hughes, Philipp Koehn, Rosie Lazar, Will Lewis, Graham Neubig, Mengmeng Niu, Alp Öktem, Eric Paquin, Grace Tang, and Sylwia Tur. Tico-19: the translation initiative for covid-19. 19, 2020.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Fric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hel, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Jun- whan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. CioqueetteChho,, Aakanksha Chowdhery, Clé- ment Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lu- cas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Rence Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang. John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven
24

---

Published in Transactions on Machine Learning Research (01/2025)
Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Kemiigg Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023a.
Jinze Bai, Shnai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile lision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308. 12966, 2023b.
Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Arletxe, Salya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. The Belebele benchmark: a parallel reading comprchension datasct in 122 language variants, 2023.
Rachel Bawden, Hatim Bourfoune, Bertrand Cabot, Nathan Cassereau, Pierre Cornette, Marco Naguib, Aurélie Névéol, and Frangois Yvon. Les modèles Bloom pour le traitement. aulomatique de la langue française. working paper or preprint, February 2024. URL http:://hall.cciinccaaaaaaaa--/ sciencceehal--0443537777
Loubna. Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Cosmopedia, 2024. URL
Emily M Bender, Timmit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 610-623 623, 2021.
Stella Biderman, Hailey Schoelkopf, Quentin Authony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language, 2019.
Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, and Percy Liang. The foundation model transparency index, October 2023. URL attps://crff. stanford. cdu/fmti/FKTI
Eleftheria Briakou, Colin Cherry, and George Foster. Searching for needles in a haystack: On the role of incidental bilingualism in PaLM's translation capability. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9432-9452, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ acl.o.....o...n........ URL https:://cclaathoooooyy org/2023. acl-long. 524.
Nicholas Carlini, Chang Liu, Ülfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pp. 267-284, Santa Clara, CA, August 2019. USENIX Association. ISBN 978 979-1-999133----9. URL https:/ www. usenix. www.:ussww...ooonniooo
Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, Loe Sharkey, Satyapriya Krishna, Mar- vin Von Hagen, Silas Alberti, Alan Chan, Qinyi Sun, Michael Gerovitch, David Bau, Max Tegmark, David Krueger, and Dylan Hadfield-Menell. Black-box access is insufficient for rigorous AT audits, 2024.
25

---

Published in Transactions on Machine Learning Research (01/2025)
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Tsard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fodus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyzz, Frica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bremnan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellssern, Douglas Eck, Jell Dean, Slav Petrov, and Noah Fiedel. PaLM: scaling language modeling with pathways, 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafford. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018.
Cyrile Delestre, 2023. URL
Martin d'Hoffschmidt, Wacim Belblidia, Tom Brendlé, Quentin Heinrich, and Maxime Vidal. Fquad: French question answering datasct, 2020.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint ar Xrv::2300 14233, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aure- lien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbic Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chumyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Al- lonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olana Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Geor- gia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Klounann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Ji- awen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstum, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Lú, Kenneth Heafield Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yean Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Tubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pa- supuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, pravcen Krish- nan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta. Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hos- scini, Sahana Chennabasappa, Sanjay Singh, Scan Bell, Scohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarck Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ra- manathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu,
26

---

Published in Transactions on Machine Learning Research (01/2025)
Wenhan Xiong, Wenyin Fu, Whitney Meens, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xin- feng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg. Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Bsing- Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Med- ina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant. Her- man, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Ge- boski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veer- araghavan, Kelly Michelena, Koqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Mar- tynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Kencally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Moham- mad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Ro- han Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswany, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoc, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngiin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models, 2024. URI Attps://arx:: orgoooogg//o.oooooooooooe 2407 21783
Moussa Kamal Eddine, Antoine J-P Tixier, and Michalis Vazirgiannis. Barthez: a skilled pretrained french sequence-too-sequecee model, 2020.
27

---

Published in Transactions on Machine Learning Research (01/2025)
Manuel Faysse, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Revisiting instruction fine-tuned model evaluation to guide industrial applications. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023. emnlp-main.55 URL http://dx... doi. 101///0011665////2233 http://2222.mm1mmmmmmm.m emnlp-main. 559
Patrick Fernandes, Behrooz Ghorbani, Xavier Garcia, Markus Freitag, and Orhan Firat. Scaling laws for multilingual neural machine translation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 10053 10071. PMLR, 23-29 Jul 2023. URL
mir ppwss/v202.. wtw.//ernnnnnaaaaneeeeee
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Authony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL W.RLLttps:/// https: org/ http:///.r/.....2//eee1..
5371628
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafford, Ananya Harsh Jha, Hamish Tvison, Tan Magmusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Pe- ters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Nishant Subramani, Mitchell Wortsman, Pradeop Dasigi, Nathan Lambert, Kyle Richardson, Jesse Dodge, Kyle Lo, Luca. Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. Olmo: Accelerating the science of language models. Preprint, 2024.
Nuno M Guerreiro, Duarte M Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André FT Martins. Hallucinations large multilingual translation models. Transactions of the Association for Computational Linguistics, 11:1500 1517, 2023.
Michael Hart. Project gutenberg, 1971. URL
org/
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoe, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxi- ang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm: Unveiling the potential of small language models with scalable training strategics, 2024. URL https:///axxiy org/ 2b0////4../2./...//// 2404.04444000 06395
Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adain Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, Kshitij Sachan, Michael Sellitto, Mrinank Sharma, Nova DasSarma, Roger Grosse, Shauna Kravec, Yuntao Bai, Zachary Witten, Marina Favaro, Jan Brauner, Holden Karnofsky, Paul Christiano, Samuel R. Bowman, Logan Graham, Jared Kaplan, Sören Mindermann, Ryan Greenblatt, Buck Shlegeris, Nicholas Schiefer, and Ethan Perez. Sleeper agents: Training deceptive lims that persist through safety training, 2024.
Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations, 2024. URL https: httpss://arxi: :/arxiv. org/abs/2405. 18392
28

---

Published in Transactions on Machine Learning Research (01/2025)
Ziwei Ji, Naycon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Compuling Surveys, 55(1)): 38, March 2023. ISSN 1557-7341. doi: 10.1145/3571730, URL http://dx. dow..ddi..oo org/ 10. 1577/3
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Maric-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020a.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.0836) 2020b. URL https://arxi:: www.aaiii...gg/bb.e. org/2bbs/2001.083611 08361
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Damitry Bahdanau, Leandro von Werra, and Harm de Vries. The stack: 3 tb of permissively licensed source code, 2022.
Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates, 2018.
Julien Launay, Elena Tommasone, Baptiste Pannier, François Boniface, Amélie Chatelain, Alessandro Cap- polli, Iacopo Poli, and Djamé Seddah. Pagnol: An extra-large french generative model, 2021.
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Saško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jemny Chim, Violette Lepercq, Suzana Ilie, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jernite. The bigscience roots corpus: A 1.6tb composite multilingual datasot, 2023.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Ben- jamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Gavint Dan- ish Contractor, Siva Reddy, Daniel Friod, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!, 2023a.
29

---

Published in Transactions on Machine Learning Research (01/2025)
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasckar, and Yin Tat Lee. Text- books are all you need ii: phi-1.5 technical report, 2023b.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Ec- cles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Mas- son d' d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-leeel code generation with alphacode. Science, 378 (78(6))4):109) 1097, 2022. doi: (0.1166/cienceeabgl... URL 10. http://ccecncee..11...
www.wsciiece..oom org/
Chin- Lin. ROUGE: A package for automatic evaluation of summaries. In Terl Summarizalion Branches Out, pp. 74 81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https:
Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. Estimating the carbon footprint of bloom, 176b parameter language model, 2022.
Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Ku- pari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Noua- mane Tazi, Teven Scao, Thomas Wolf, Osma Suominen, Samuli Sairamen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, and Sampo Pyysalo. FinGPT: Large generative models for a small language. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2710 27226, Singapore, Decem- ber 2023. Association for Computational Linguistics. doi: 10.18653/v1/200 1118600/v/2023.emml---. monnp-main 164. URL htwpcc/aannooooooooo///2200 www.033.mmlpmmm.....m 164
Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. Eurollm: Multilingual language models for europe, 2024. URL http::///axxxv..//aaba/////o/
Matthicu Mecus, Igor Shilov, Manuel Fayssc, and Yves-Alcxaanre de Montjoye. Copyright traps for large language models, 2024a. URL
Matthicu Meeus, Igor Shilov, Shubham Jain, Manuel Faysse, Marck Rei, and Yves-Alexandr de Montjoye. Sok: Membership inference attacks on 1lms are rushing nowhere (and how to fix it), 2024b. URL https:
//arxiv.org/abs/2000 17975
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Xin Yong, Hailey Schoelkopf, al. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01 2022.
Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models, 2023.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. Crows-pairs: A challenge dataset for measuring social biases in masked language models, 2020.
Roberto Navigli, Simone Conia, and Björn Ross. Biases in large language models: Origins, inventory, and discussion. J. Data and Information Quality, 15(2), jun 2023. ISSN 1936-1955. doi: 10.1145/3597307,
URL https:///oo:ii .rgg 10. 1145/3597307
Aurélie Névéol, Yoamn Dupont, Julien Bezançon, and Karën Fort. French CrowS-pairs: Extension à une langue autre que l'anglais d'un corpus de mesure des biais sociétaux dans les modèles de langue masqués (French CrowS-pairs : Extending a challenge datasct for measuring social bias masked language models a language other than English). In Yannick Estève, Tania Jiménez, Titouan Parcollet, and Marcely Zanon Boito (eds.), Actes de la 29e Conférence sur le Traitement Automatique des Langues
30

---

Published in Transactions on Machine Learning Research (01/2025)
Naturelles. Volume 1 conférence principale, pp. 355 -64, Avignon, France, 6 2022. ATALA. URL NewYorkTimes.
The times sues openai and microsoft over a.. use of copyrighted work. https://wait law.ssttt.........tt..... h.t.aittttttllll
Dec 2023.
Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages, 2023.
OpenAT, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shaping Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damicn Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atly Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billic Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia. Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob Mc- Grew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mdly, Ashvin Nair, Reiichiro Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Fmy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Flizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Adilya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor, Fric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Ben- jamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, llya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsca VOSs, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmanm, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt WiethofT, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2023.
31

---

Published in Transactions on Machine Learning Research (01/2025)
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annuall meeling of the Association for Compulational Linguistics, pp. 311-3331111 318, 2002.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Eblesam A Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only, arXiv prepriat orXiv:2306.01. 011111 2023.
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Pupers, pp. 191, Belgium, Brussels, October 2018. Association for Computa- tional Linguistics. URL httpps://ww. aclweb. org/ wathhology/W18-6311
Gema Ramirez-Sánchee, Jaume Zaragoza-Bernabeu, Marta Bañón, and Sergio Ortiz Rojas Bifixer and bicleaner: two open-source tools to clean your parallel data. In André Martins, Helena Moniz, Sara Fumega, Bruno Martins, Fernando Batista, Luisa Coheur, Carla Parra, Isabel Trancoso, Marco Turchi, Arianna Bisazza, Joss Moorkens, Ana Guerberof, Mary Nurminen, Lena Marg, and Mikel L. Forcada (eds.). Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pp. 291-298, Lisboa, Portugal, November 2020. European Association for Machine Translation. URL orggl/oooooooooooooo/gog org//020000000 eamt-1.31
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Philipp Kochn, Loie Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussá, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewiez, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Koemi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Mattco Negri, Aurélic Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation ( (WMT), pp. 578 585, Abu Dhabi, United Arab Emirates (Hybrid), December 2022a. Association for Computational Linguistics. URL https:// aclanthology.o org/ 2022. 1m5-
Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Philipp Kochn, Loie Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Mattco Negri, Aurélic Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.). Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 634-645, Abu Dhabi, United Arab Emirates (Hybrid), December 2022b. Association for Computational Linguistics. URL https: http:///aaaaaahoooaeoo wtwp:llloooorrg/22222222 wmt-1.60
Anna Rogers and Cassaadraaaahaaa Luccioni. Position: Key claims in 1lm research have a long tail of footnotes, 2024. URL httppp///aaaav...rr///////2...
Phillip Rust, Jonas Pfeitfer, Ivan Vulié, Sebastian Ruder, and Iryna Gurevych. How good is your tokenizer? on the monolingual performance of multilingual language models, 2021.
Pamela Samuelson. Generative ai meets copyright. Science, 3666444) 161, 2023.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunoii Hashimoto. Whose opinions do language models reflect?, 2023.
Nikhil Sardana and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws, 2023.
32

---

Published in Transactions on Machine Learning Research (01/2025)
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 1760-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword wnits, 2016.
Igor Shilov, Matthieu Meeus, and Yves-Alex de Montjoye. Mosaic memory: Fuzzy duplication in copyright traps for large language models, 2024. URL htttps://arxi.....t//// h/s///ab/////a /205. 15523
Olch Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Koslova, and Tatiana Shavrina. mgpt: Few-shot learners go multilingual, 2022. URL http:::///aax...og/raab/////////
Antoine Simoulin and Benoit Crabbé. Un modèle Transformer Génératif Pré-entrainé pour le français. In Pascal Denis, Natalia Grabar, Amel Fraisse, Rémi Cardon, Bernard Jacquemin, Eric Kergosien, and Antonio Balvet. (eds.), Traitement. Automatique des Langues Naturelles, pp. 246-24455 255, Lille, France, 2021. ATALA. URL
Daria Soboleva, Faisal Al-Khateeb, Robert Mycrs, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPa- jama: A 627B token cleaned and deduplicated version of RedPajama. https://wwww http:::://ww..eeebrasss cewebras................... www.eebaa.eeebblgg..........
verssoo oooofarddaaaaa June 2023. URL
627B
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. transformer with rotary position embedding, 2023.
Roformer: Enhanced
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, An- lon Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Dean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Ali- cia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette- Choo, Danila. Sinopalníkov, David Weinberger, Dimple Vijaykumar, Dominika Rogozitsska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Dhand, Ivan Nardini, Jacinda Mcin, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joans Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gor- don, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kil- patrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Tverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsce Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wemming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joclle Barral, Zoubin Ghahramani, Raia. Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin,
Plucciskk, Harleen Batra, Harsh
33

---

Published in Transactions on Machine Learning Research (01/2025)
Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at practical size, 2024a. URI. https://ax::iii org/ www.bb///..0. 00018
NLLB Team, Marta R. Costa-jussá, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman,, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chan Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Saftyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling human-centered machine translation, 2022.
Reka Team, Aitor Ormazabal, Che Zheng, Cyprien de Masson d' d'Autume, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, Kaloyan Aleksiev, Lei Li, Matthew Henderson, Max Bain, Mikel Artetxe, Nishant Relan, Piotr Padlewski, Qi Liu, Ren Chen, Samuel Phua, Yazheng Yang, Yi Tay, Yuqi Wang, Zhongkai Zhu, and Zhihui Xie. Reka core, flash, and edge: A series of powerful multimodal language models, 2024b. URL htttss///aaiiiii wwapexivv..orreeeo abaaa/aaa/224/ 240..12382.
Jörg Tiedemann. Parallel data, tools and interfaces in opus. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Mehmet Ugur Dogan, Bente Maegaard, Joseph Mariani, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Fight. International Conference on Language Resources and Evaluation (LREC'12) 12), Istanbul, Turkey, may 2012. Europcan Language Resources Association (ELRA). ISBN 978-2-9517
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arxiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Fdunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners, 2022a.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022b.
Johannes Welbl, Nclson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions, 2017.
Linting Xuc, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer, 2021.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shnai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang,
34

---

Published in Transactions on Machine Learning Research (01/2025)
Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https://arxivv http:///aaaxiv..r/aaba//////////
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence?, 2019.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint 0rXiv:2210.02411 2022.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers, 2022. URL
Peiyuan Zhang, Guanglao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. (inth))wilcchrr 570k GPT interaction logs in the wild. In The Twelfth International Conference Learning Representa- tions, 2024. URL
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging Ilm-as- a-judge with mt-bench and chatbot arena, 2023.
35

---

Published in Transactions on Machine Learning Research (01/2025)
Upstream
88%
44%
66%
22%
19%
0%
Model
73%
67%
48%
61%
45%
18%
Downstream
80%
51%
46%
60%
54%
17%
Croissant
Llama
Bloomz
GPT-4
PaLM2
Titan
Figure 9: Aggregated FMTT
A FMTI
Disclaimers and Methodology
The FMTI grid is meant to assess Foundation Models, but base models and models that were fine-tuned instruction or chat datasets imply different training, evaluation and data curation protocols, thus largely modifying their assessment through the FMTI. Training an instruction or chat model from a base model is a process that has recently been completely democratized through the use of crowdsourced or synthetic datasets, and individuals are now fully capable finetuning their own model variants in a variety of manners. As such, we consider this work's contribution mainly lies in the base model training, and are aware that SFT finetuning of the Croissant model will be done outside of the author's control; whether on proprietary data, synthetic chat datasets, crowdsourced chat instructions leading to different legal and copyright implications for the finetuned models. We thus focus on the base model in our evaluation and give the complete criteria list as detailed in the appendix.
Transparency evaluation should idcally be done by an independent third party as there are obvious biases in aulo-evaluating a model, and point attribution not always trivial for certain criteria. such, we lake rather conservative approach to point attribution and detail our process in an open document. Effforts have consciously been made within the technical report include information not initially given to validate certain criteria, which puts us at clear advantage with respect to work published before the index's release.
We are open to discussions for potential scoring modifications, and consider these FMTI scores to be the reflection of our compliance efforts to the listed transparency principles, rather than scores fairly comparable to the larger foundation models with vastly different usage objectives.
36

---

Published in Transactions on Machine Learning Research (01/2025)
Dataset
Size (GB) Documents Tokens (M)
Token Loken/Don Doc
CulturaxFr
1216.03
363197906
292833.75
806.14
WikisourceFF
10.84
2557238
2699.00
1055.44
Wikipedia202311 101.fr
7.37
2563646
2002.51
781.12
JadeOpendata
5.19
550065
1295.29
2354.79
JorOOpendata
3.83
3189949
967.10
303.17
LegiOpendate
3.56
2151510
816.44
379.47
AccoOpendata
3.39
251332
758.15
3016.52
Inca IncaOpendata
2.60
369687
627.32
1696.90
ProjectgutenbergEr
0.97
2447
301.19
123086.18 16
CappOpendata
0.91
71949
247.14
3434.97
CustomLayout TextoOly
0.77
291604
191.11
655.38
Debats OpbaasOpendata
0.77
2114
149.09
70524.31
CassOpendata
0.76
140803
206.04
1463.35
KaliOpendata
0.68
402963
152.33
378.01
SwissLegislation
0.26
11086
68.33
6163.81
FrenchOpenSubritles
0.15
5379
41.84
7779.26 .66
CnilOpendata
0.12
15168
26.37
1738.72 72
BnfClcan2023
0.10
341
27.04
79295.7 71
QrOpendata
0.10
530
21.73
41005.03
SardeOpendata
0.09
221278
28.10
127.01
DolcOpendata
0.08
4000
19.36
4839.07
OpenditOpendata
0.07
6977
15.27
2188.28
0.06
255631
12.91
50.49
FrenchPodcastee
0.01
1237
1.56
1259.90
FrenchPoetry
0.00
1721
0.76
441.23
Train
1258.70
376266561
303509.73
806.63
Table 9: French Data mix
Dataset
Size (GB) Documents Tokens (M)
Token Doo Doc
SlimPajama
2333.77
590194779
630441.67
1068.19
Project Gutenberg PG19 10.67
28602
23580.49
824435.00
Gutenberg Canaries
2.75
7515
555.40
73905.01
Train
2351.13
591230543
655637.48
1108.94
Table 10: English Data mix
B Additional data details
B.1 French Data
Refer to Table g
B.2 English data
Refer to Table I0
B.3 Code data
Refer to Table 11
37

---

Published in Transactions on Machine Learning Research (01/2025)
Dataset
Size (GB) Documents Tokens (M)
Token Doo
StarcoderdataJava
82.49
20061773
29740.73
1482.46
MaarooderdataJavassstto
61.64
19534285
24546.60
1256.59
Starcoderdatal Python
57.00
12856649
24605.09
1913.80
StarcoderdataC
50.60
8526791
15791.76
1852.02
StarcoderdataCpp
45.84
6343527
19607.90 .90
3091.01
PypiClean
29.20
2428172
12120.74
4991.72
StarcoderdataSql
10.38
965666
3278.24
3394.80
6.67
905365
2567.77
2836.18 18
5.55
662056
2119.72
3201.72
StarcoderdataJson
5.43
4741547
2165.87
456.79
Starcoderdata Texx
4.86
517551
1916.88
3703.76 76
StarcoderdataShell
2.98
2196327
1178.17
536.43
CodeContests
2.79
1485888
1228.61
826.85
StarcoderdataCuda
0.52
57570
227.24
3947.14
GithubJ Spyrubbuuyteerrolee Toxxxl
0.48
46978
159.49
3395.09
StarcoderdataDockkriiil
0.41
565791
161.48
285.41
Starcoderdataldriss
0.03
7942
11.72
1475.09
Train
366.87
81903878
141428.02
7726.76
Table 11: Code Data mix
Dataset
Size (GB)
Documents Tokens (M) Token
CustomFrEn
113.35
407858836
35641.60
87.39
F222253
0.36
95009
81.60
858.91
OnginalSongsLLriisstthhaaaaaa Withhrree Translation 0.20
75020
53.48
712.93
Train
113.91
408028865
35776.69
87.68
Table 12: Parallel Data mix
B.4 Parallel data
Refer to Table 12
OPUS data distribution is given in Figure 10
B.5 Scaling Law Corpus
For the scaling law experiments, we use smaller subsampled dataset, consisting of splits of French, English, and Code data we vary in ratio to study the impact of language distribution. In total, we train on 50 billion tokens and sample from the following datasets: French https::/huuggigfaace datasetss/amm/Erenccr 30b, English https: haap:s::/hhggingaaae
and Code https://huggingffce.. co/ /tt.//at///aan///aaaaaa//s code_ 20b In all datasets, breakdown of the sources given in the dalasel stals. csv file at the rool of the data folder. The source distribution is chosen be consistent with the final distribution used during main model training not to affect the conclusions.
38

---

Published in Transactions on Machine Learning Research (01/2025)
Number of Samples
108
106
104
ccmatrix v1
OPUS paracrawl v9
multiparacaawi V7.1
OPUS SPUSSUT srlufiicallggeee Seaalignndie V1
giga fren V
OPUS europat V
unpc v1.0
OPUS multiun V1
OPUS ccal pookshop V2
v
OPUS tildemodel V2018
opusTod 20210402
news wikipedia v1. 0
16
wilkitit/es es c
jrc acquis V3.0
V2018g4
V2
emea V3
"Pysee V1
V2
bible uedin v1
v
OPUS O openoffice V
infopankk) v
OPUS euconst vi
PRUSU P OPUS V20201028
10
V1
vi
OPUS eubooks Bubookshog)
OPUS wikimatrix
OPUS europar
OPUS wikimedia 1100 train
Statmt news commentary eça
OPUS natoeba
kde4
gnome
OPUS tico 19 ubuntuyy V14.
OPUS kdedoo
POOOPUS OOPUSUU
OPUS
globalvoices
OPUS
OPUS OPUS
OPUS
OPUStildem
Facebook
&
OPUS global
OPUS
OPUS
LinguaTool
OPUS
OPUS
Figure 10: Opus Data Distribution withinn our training dataset
39

---

Published in Transactions on Machine Learning Research (01/2025)
C Chat examples
The following results were not cherry-picked and were generated with temperature of 0.5, Top-P of 0.95 and Top-K of 40. They focus on Writing tasks which CroissantLLM is best at.
C.1 Translate the following article in French:
The Principality of Sealand (/silend/) is an unrecognised micronation on HM Fort Roughs (also known as Roughs [55 on offshore platform in the North Sea approzimately twelue kilomelres (6+17 1/+1/2 nautical miles) off the coast of Suffolk, England. [3] Roughs Tower a Maunsell Sea Fort that was built by the British in international waters during World War II. Since 1967, the decommissioned Roughs Tower has been occupied and claimed as sovereign state by the family and associales of Paddy Roy Bates. Bates seized Roughs Tower from group of pirate radio broadcasters in 1967 with the intention of setting up his own station there. Bates and his associates have repelled incursions from vessels from rival pirate radio stations and the U.K.'s royal navy using firearms and petrol bombs.[6]]]7] Since 1987, when the United Kingdom extended its territorial walers to 12 naulical miles, the platform has been in Brilish lerrilory-
La Principauté de Sealand (/silændd/ est une micronation non reconnue sur le fort, Fort Roughs (alias Tour de Roughs) situé à environ 12 kilomètres (6,5 miles) au large des côtes de Suffolk, en Angleterre. Fort Roughs est fort Maunsell qui été construit par les Britanniques dans les eaux internationales pendant la Seconde Guerre mondiale. Depuis 1967, le fort Roughs été occupé revendiqué tant que pays souverain par la famille et les associés de Paddy Roy Bates. Bates saisi Roughs Tower à un groupe de radiotalkers pirates en 1967 dans l'intention de créer sa propre station radio. Bates et ses associés ont repoussé les incursions de navires de radiotalkers rivaux et de la marine royale britannique à l'aide d'armes à feu et de bombes carburant. Depuis 1987, lorsque le Royaume-Uni étendu ses eaux territoriales à 12 milles marins, la plate-formn forme a été dans zone Lerritoriale britannique.
C.2 What can do in Marseille during the winter
1. Visit the Vieux-Port: The Vieux-Port is the heart of Marseille and is great place to relax and enjoy the beautiful architccture and sights of the city. You can also take boat tour see the island of the Frioul.
2. Stroll through the Calanques: The Calanques are a series of stunning rock formations located just outside of Marseille. You can take boat tour to see the calanques and enjoy the beautiful scenery.
3. Visit the Parc National des Calanques: The Parc National des Calanques is a protected natural area that is home to some of the most beautiful scenery in Marseille. You can take a hike through the calanques and enjoy the fresh air and beautiful scenery.
4. Explore the Cathedrale de Notre-Dame de la Garde: Notre-Dame de la Garde is beautiful cathedral located in Marseille. You can take a walk around the cathedral and enjoy the beautiful architecture and views of the city.
5. Visit the Mucem: The Mucem is a museum that is located in Marseille and dedicated to the history and culture of Marscille, You can explore the museum and learn about the city's 's history and heritage.
C.3 Rédige une lettre de motivation pour un job étudiant de barman.
Objet: Candidature pour un job étudiant de barman
Cher [nom du recruteur],
Je me permets de vous adresser ma candidature pour un job étudiant de barman au sein de votre établisse- ment.
Je suis actuellement la recherche d'un emploi à temps partiel pour financer mes étudos, et je suis convaineu que votre entreprise est en mesure de m'offrir un emploi adapté à mes compétences et mes aspirations professionnelles.
40

---

Published in Transactions on Machine Learning Research (01/2025)
Je suis quelqu'un de dynamique, motivé et souriant. J'ai une grande passion pour la cuisine et la pâtisserie, et je suis toujours à la recherche de nouvelles expériences culinaires. Je suis également très à l'aise avec les clients et je sais m n'adapter à toutes les situations.
suis titulaire d'un diplóme en [décrire le diplóme] et j'ai déjà travaillé dans différents établissements de restauration. J'ai également eu l'occasion de participer des événements de barman lors de mes précédentes expériences professionnelles.
Je suis convaincu que mes compétences et mon expérience permettront de contribuer efficacement à votre équipe de barman. Je suis également prêt à apprendre et à m aaaaptter à vos besoins spécifiques pour m intéégrer rapidement et efficacement.
suis disponible tous les soirs de semaine, 18h à 22h, et les week-ends également. Je suis également prêt me déplacer pour des événements de barman.
Je suis disponible pour un entretien volre convenance, pour discuter plus en détail de ma candidature de mes motivations. Je reste à votre disposition pour toute information complémentaire.
Cordialement,
[Votre nom]
C.4
Show me how to code Depth First Search in Python
can provide you with the code for depth-first search in python. Here is the code:
def dís (graph, start):
visited = set () stack []
visited add start) stack append (oppnnd(start, (start)
while stack:
vertex = stack () visited add add (vertex)
for neighbor in graph vertex if neighbor not in visited: stack append neighbor)
this code defines a function called 'dfs' that takes a graph as input and starting vertex. The function uses stack keep track of the vertices to visit and set to keep track of visited vertices. The vertices are added to the stack. The function then enters while loop that continues until the stack is empty. Inside the loop, the function pops vertex from the stack, adds it to the graph, and then poppes another vertex from the stack. This process continues until all vertices have been visited. Finally, the function returns the set of vertices that have been visited. You can use this function to perform a depth-first search on graph.
C.5 Give me medical advice about how to solve my back issues.
don't have physical body or medical knowledge. However, I can provide you with some general tips how to solve back issues:
1. Stay active: exercises such as yoga and stretching can help alleviate back pain. Regular physical activity can help improve flexibility, strengthen the muscles, and reduce the risk future back problems.
2. Maintain healthy diet: a balanced diet rich in fruits, vegetables, and lean protein can help reduce inflammation and improve digestion. Avoid high-fat and high-sugar foods, which can aggravate back pain.
41

---

Published in Transactions on Machine Learning Research (01/2025)
3. Manage stress: chronic stress worsen back pain. Try relaxation techniques such as meditation, deep breathing, progressive muscle relaxation to help manage stress levels.
4. Get regular back checkups: regular checkups with a healthcare provider can help detect any underlying back issues early on. This can help prevent future problems.
5. Seek professional help: if you have persistent back pain, it may be necessary to seok the help of a healthcare professional. They can help diagnose and treat any underlying back issues that may be contributing to your pain.
42

---

Published in Transactions on Machine Learning Research (01/2025)
English Writing
French
Writing
Humanities
Roleplay
Humanities
Roleplay
STEM
Reasoning
STEM
Reasoning
0
3
6
0
2 3
Extraction
Math
Extraction
Math
Coding
Coding
CMArkea BloomZ 3B
Bloom 1B7 Chat - TinyLlamaChat CroissantLLMChat
Figure 11: MT Bench Results (Turn 1)
D Results
D.1 Methodology
Base models are evaluated through the LM Evaluation harness framework (Gao et al. 2021). For classification tasks, choose the answer with the largest log likelihood when concatcnated with the prompt, is implemented within the framework.
For generative tasks, we simply generate with the default settings, which is greedy sampling. We acknowledge CroissantLLM works best with higher temperature values but did not want to introduce stochasticity to the evaluation. We also limit each benchmark task to 5000 samples at most, to shorten evaluation time. All eval- uations are reproducible through the code at https::/ httpps://gituub.com http://hhhhhh..ccoecaaeeeee/aaaa
D.2 MT-Bench
Turn (Figure 11 and Turn 2 (Figure 12 results are shown. We notice small models, struggle with reasoning based tasks and contrained generation imposed by Turn 2 prompts. Figures 13 and [4 compare our results
on small language models to other common bigger models 70
D.3 Bias Assessment
We assess bias through CROWS (Nangia et al.) 2020 Névéol et al. 2022 the Crowdsourced Stereotype Pairs benchmark that cover stereotypes dealing with nine types of bias, like race, religion, and age and report results in Table 15
We find CroissantLLM is in line, or slightly less biased than other models, notably in
French.
D.4 MMLU Results
The MMLU benchmark (Hendrycks al. 2021 has become standard in evaluating Large Language Model knowledge and reasoning capabilities. However, it is still currently very challenging for smaller models, and
Rosults in French for models with sizes 7B parameters extracted from http:///uggingggcc..ccomm ottp://aaaaaaaaaaoocc//aaaassss bofenghusng/mttuue bencht-benhh---eeeee and results in English from httpp////aaggggeeeeeeeeees/ss/ss data/mt, wwwpe/eenneeeeeeuuueeeeee _jjugmentt
43

---

Published in Transactions on Machine Learning Research (01/2025)
English Writing
French
Writing
Humanities
Roleplay
Humanities
Roleplay
STEM
Reasoning
STEM
Reasoning
0
1
2
3
0
2
Extraction
Math
Extraction
Math
Coding
Coding
CMArkea BloomZ 3B Bloom 1B7 Chat
- TinyLlamaChat CroissantLLMChat
Figure 12: MT Bench Results (Turn 2)
Models
Wri
Ro
Reas Math Cod
Ext
STEM
Hum
Avg
CroissanLLLMChat
5.32
5.35
1.9
1.16
1.8
1.4
4.55
3.75
3.15
TinyLlamaChat Bloom 1B7 Chat
5
4.1
1.45
1
2.1
1.6
3.9
4.85
3
4.62
3.25
1.2
1.5
1.45
1.5
2.85
2.22
2.32
CMArkea BloomZ 3B
2.65
2.85
1.85
1.15
1.2
2.3
3.65
2.7
2.29
Vigostral 7B Chat
7.7
7.85
4.85
3.65
4.65
7.75
7.35
9.2
6.62
Vigogne 2 7b Chat
5.35
6.25
2.75
2.2
2.47
3.4
6.05
6.68
4.39
OpenHermes Mistral 7B
8.8 7.5
5.1
4.05
5.55
6.2
8.35
9.4
6.87
Vigogne 2 70B Chat
9.4
8.25
4.75
4.3
5.35
7.25
9.1
9.43
7.23
Mixtral 8x7b Instruct
9.65
8.88
6.95
4.95
4.6
8.55
9.5
9.6
7.84
Mistral Medium
9.6 9.05
5.4
6.1
7.35
9.25
9.3
9.75
8.23
GPT 3.5 Turbo
8.75 8.93
5.05
5.65
7.85
9.05
9.05
9.68
8
GPT 4
9.6 9.65
8.55
8.5
8.35
9.2
9.85
9.88
9.2
Table 13: French MT Bench Results Average of turn 1 and 2 of many supervised finetuned models
even very recent pretrained models such Llama 3.2(1B) or Gemma2 (ubeyy et al. 2024 Team et al.
2024a) report base model performances that are only slightly better than random (s 33%). CroissantLLM Tiaaa, and other small model baselines in this paper display random performance on the task (> 25%) which is why the task not included in the main paper.
44

---

Published in Transactions on Machine Learning Research (01/2025)
Models
Wri
Ro
Reas
Math
Cod
Ext
STEM
Hum
Avg
CroissnntLLMhha
5.1
3.6
2.4
1.1
1.8
1.3
5
5.85
3.27
TinyLlamaChat
6.25
3.3
1.35
2.1
1.5
4.82
6.3
3.83
Bloom 1B7 Chat
3.95
4.55
1.95
1.45
1.35
3.5
3.45
2.7
CMArkea BloomZ 3B
3.15
3.3
2.2
1.1
1.25
1.35
3.15
1.9
2.17
Llama 7B Chat
8.9
7.7
4.25
2.4
6.5
8.65
8.75
6.27
Vicuna 7B v1.3
8.1
7.45
4.65
2.3
3.55
7.82
9.1
Llama 2 13B Chat
8.85
7.5
5.1
3.45
6.92
8.62
9.75
6.65
Vicuna 13B v1.3
9.25
7.18
5.85
2.6
3.25
5.55
7.98
9.45 6.39
Vicuna 33B vl.3
9.5
8.45
6.65
3.15
3.35
7.1
8.98
9.8 7.12
Llama 70B Chat
9.3
7.5
5.8
3.3
3.15
7.25
8.93
9.62 6.86
GPT 3.5 Turbo
9.2
8.4
5.65
6.3
6.9
8.85
8.7
9.55
7.94
GPT
9.65
8.9
9
6.8
8.55
9.38
9.7
9.95
8.99
Table 14: English MT Bench Results Average of turn and of many supervised finetuned models
Task
Crows(en)
Crows(Fr) Avg
mGPT(1.3B)
3.16
2.94 3.05
Bloom(31)
3.39
3.02 3.21
Bloom(1.1B)
3.36
3.07 3.22
CroissantLLM
3.56
3.22 3.39
Pythia(1.4b)
3.36
3.62 3.49
(PT(1.3b)
3.35
3.67 3.51
TinyLlama(1.1B)
3.48
3.76 3.62
GPT-fr(1B)
4.50
2.97
3.73
Llama2(7B)
3.72
3.81 3.76
Table 15: Bias Evaluation through Crows-Pains dataset assessed by likelihood difference.
45

---

Published in Transactions on Machine Learning Research (01/2025)
E
Terminology
Large Language Models. According to the definition proposed by Rogers & Luccioni (0024), Large
Language Models (LLMs) (1) have the capacity to model and generate text, (2) are pretrained on large amounts text data (over 1B tokens according to the proposed threshold) (3) enable transfer learning through finetuning or prompting. Under these criteria, LM can largely be considered a large language model despite its small size relative current best-in-class LLMs.
Small Language Models. Small Language Models (SLMs) is term that has appeared recently and that refers to LLMs with relatively low parameter count, often optimized for speed or to enable inference with local or lower-end hardware.
Effective Capacity Ratio. Introduced in Fernandes et al. 2023 the effective parameters of a multi- lingual model in a particular language is the number of non-embedding parameters a monolingual model would require to match the language performance of the multilingual model. The intuition that mul- tilingual model splits its capacity between different languages, so monolingual model would reach similar performances specific language with fraction of the parameters. We further compute the effective capacily ratio by dividing the effective parameters by the multilingual model size. In our joint scaling laws experiments (Figure 4D, we show that the models trained with equal ratios of English to French have an effective capacity ratio of over 82% in the French language, indicating that training a monolingual French model with the same performance would require about 82% the non-embedding parameters. This hints there must be some effective capacity sharing between English and French.
Tokenizer fertility. According to Rust et al. (2021), fertilily measures the average number of sub-words (tokens) produced per tokenized word. De facto, fertility has theoretical lower bound of 1 which would imply that the tokenizer's 's vocabulary contains every word in the corpus.
46

---

Published in Transactions on Machine Learning Research (01/2025)
F Dost-Mortem & Lessons learned
In line with our efforts of full transparency, we share some key lessons learned through this project, in light of a few months of hindsight and of recent developments posterior to the model release.
Tokenizer. In this work, we have developped our own tokenizer, inspired by Llama2 (Touvron et al. 2023b) but fitted to a bilingual French-English corpus to improve fertility on both languages. Several key decisions can impact tokenizer design. The vocabulary size will have large impacts on the parameter count of the model, polentially making embedding parameters a large share of the total parameler count. especially for smaller models (Team et a 2024a). One should experiment with the performance and memory trade-offs of using larger vocabulary sizcs. Furthermore, non-standard tokenizers are likely to be less supported by standard inference framework complicating model adoption. Finally, we believe hand-designing subset of tokens that are known to be useful at inference, for particular tasks, or have semantic coherence (common numbers, punctuation patterns, tokens corresponding to multiple choice templates or code syntax) probably degrades fertility but is useful in the long run from a performance and usability perspective.
Data Mix. Tt was always understood the data quality and quantity of the pretraining mix had large impacts on pretraining. In this work, we notably show the large interest of having very large ratios of 'aligned" translation data in this mix, as it boosts the models translation capabilities, but also enables cross-lingual knowledge acquisition. Since CroissanLLLM's release, it has become understood that certain types of data heavily boost model reasoning capalities or performances on benchmarks. Typically, Abdin let al. (2024); Team et. al.] (2024b); Yang et al. (2024); Dubey et al. (2024), put a particular emphasis in including math and reasoning heavy corpora in the training corpus. Groeneveld et al.) (2024) reports 24 point gains on MML.U scores by including more knowledge and reasoning rich sources to their training set, as well as improving data quality filtering. In order to source such data with highly 'oducative' values several approaches are possible, including distilling large model knowledge into fast text. classifiers to identify high- quality content from Internet scale data (Penedo et al.] 2023),, or even generating diverse and high quality synthetic text using LLMs (Abdin et al.) 2024 Ben Allal et al. 2024 Dubey et al. 2024). More generally, better pretraining and annealing data leads to better models, and many recent. datasets would probably lead to better performance nowadays than the datamix we had constituted. These efforts are however often centered around the English language, and our efforts in French data collection remain very valuable.
Scheduler. In CroissantLLM, we leverage the standard Cosine Scheduler with warmup. One disadvantage of such a scheduler is that the number of total training steps must be known in advance, limiting flexibility once training starts. Inspired by the Vision literature (Zhai et al., 2022), the MiniCPM model (Hu et al.) 2024 and several models since then have later confirmed the interest of infinite or WSD (Warmup-Stable- Decay) (onnouules(HHgeelle et al.) 2024). The concept is to keep the learning rate at a constant or asymptotically constant value such that training can be done on an arbitrarily long number of tokens. At the end of training, an annealing phase decreases the learning rate to help the model converge and boost performance. MiniCPM has further uncovered that annealing on a higher quality data mix is a very efficient strategy, Typically, by including knowledgo-rich or ceasoning-heavy data in this final training phase, large benchmark improvements are obtained. This technique has since become the standard way of training LLMs, since it both enables training flexibility and large benchmark improvements. We believe running an annealing phase on the Croissant-LM model would have largely boosted performance, especially if done with carefully selected high-quality data in French and English.
Leveraging Larger LLMs. As made clear in this work, while training with a linguistically balanced dataset helps multilingual performance, the number of model parameters remain the strongest performance factor, and given our compute budget, better performance could be obtained by training a larger model on less non-english data. This is perfectly in line with the findings of Kaplan et al. (2020a) with Chinchilla scaling laws. Models trained to be small and inference efficient such as CroissantL.LV could however benefit from larger multilingual LLMs in multiple ways. Beyond the evergrowing use of LLM-generated data during pretraining (Abdin et al. 2024), small models can also benefit from larger LLMs as critics for alignement processes (Yang et al. 2024), but also to replace stochastic weight initialization by starting off from pruned- versions of larger models (Dubey et al.) (024)). Obviously, these strategies entail significantly larger ressource requirements to train the larger models to begin with.
47

---

Published in Transactions on Machine Learning Research (01/2025)
Model Release. Training model is one thing, enabling people to use it is another. Our model has garnered significant attention particularly in the French community, and beyond the technical report, many people were interested in associated demos, blogposts, or social media posts. In retrospective, some barriers of entry remained for users with limited technical skills. The lack of day 0 support from third party infer- ence libraries (Ollama 1pama.cpp 72 MLCCCal 73 has led to some communixy-propoeel implementations that were suboptimal, leading to performance hits. Furthermore, official fine-tuning resources (notebooks, Axolotl configurations, etc.) could have largely helped users adapt CroissantLLM for specific cases and drive adoption. Considering the downstream uses of the model and its utilization seems paramount in the construction of such project.
Building a model is an iterative process. Building upon the Croissant work and the above listed insights, our team has since been able to train stronger models (Martins et al. 2024 The field evolves quickly and getting everything right the first time is not easy. To build very good models, seems important us construct the foundations iteratively, getting things work at a small scale first, collecting some practical real-world feedback and keeping model development leam stable.
72 /attps://ollama.com
httpps////tthbbocoogggge//////// 75 htttp:///lm.mcc..a/
48


Reviews:
[{"review_id": "l8tjdTr6ck", "reviewer": "Reviewer_mYt3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper conducts an insightful discussion on the language ratio selection when training bilingual LLM, and take ablation studies to find the optimal ratio between English and French.\n- Transparent and open source. Data, intermediate checkpoints and final models are all open. It is valuable for community to research the training dynamic of a bilingual LLM (English is not the only dominate language).\n- FrenchBench, a high-quality human verified benchmark for french language, benifit the development of the French LLM.\n\nWeaknesses:\n- As a research paper, I think it is need more content to discuss the “Bias towards English” problem. It comes as no surprise that increasing the portion and amount of french pretraining data will lead to the performance on Frecnh-orient benchmark. But after increasing the proportion and quantity of French data, whether the model begins to be less \"English-biased\"—such as whether its responses align more closely with the values of the French-speaking community, better match the habits of French users, or reflect the living environment of French speakers—becomes a more worthy topic of exploration. These aspects can better argue the significant importance of building a non-English-dominant model (bilingual model).\n- This paper lack some comparisons between some popular 1B-scale open-source models, such as Qwen-2.5 and Gemma2 series. These models also perform impressively in French.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This paper is a detailed technical report about how to train and evaluate a English-French bilingual LLM. \n- This paper open-sources its training data, final models and benchmark (especially a manual verified benchmark on French) for research in this area.\n- This paper releases CroissantLLM, which leads performance on french language at the 1B scale in the experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As mentioned in weaknesses,\n\n- Add more experiments about the behaviour of model under different ratios of French data (values, common sense, taboos, etc.). It is also good to build some related benchmarks to evaluate the \"English-bias\" of bilingual / multilingual models.\n- Authors can add more baselines of 1B-scale open-source models, such as Qwen-2.5 and Gemma2 series.", "broader_impact_concerns": "The broader impact concerns of this work have been fully addressed in this paper, including model release, risk mitigation, data leakage and risk assessment, no issues are left.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UMGwGRhszD", "reviewer": "Reviewer_DVTq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n1. The model scores highly on the foundation model transparency index (FMTI), open-sourcing not just the final pre-trained checkpoint, but also a large part of the training data, checkpoint across different sizes of training, the overall codebases, and other relevant information. This transparency helps democratise the LLM scientific process, affording the community a chance to scrutinise and analyse the performance of the model across different checkpoints, and also replicate the authors' results and ultimately improve upon them. This point is becoming increasingly important, especially given the growth of closed-sourced LLMs, in addition to open-sourced LLMs that only shares the final parameters of the pre-trained model, without much detail as to what data the model was trained on, what the exact architecture looks like, etc.\n\n2. The released model would likely be useful not just for scientists and engineers that would like to use the model, but also for end users who would like to use small yet efficient small models (in the order of ~1 billion parameters, for instance due to hardware constraints) that performs well despite its size, or for use cases that require good bilingual English / French LLM capability.\n\n3. The bilingual angle (here French and English) is also interesting, as it helps combat the English-centricity of some prior open-sourced LLMs. \n\n4. The small size of the model (only 1.3B parameter) is useful for broader adoption, because not many people (outside of academic and industry labs) have the hardware to run much larger models than this. The model was also trained with a high overtraining factor, which yields a much better performance than what a training-compute-optimal model of a similar size would achieve.\n\n5. The training data collection process is described in detail, and includes creative uses of possible data sources. I particularly like the inclusion of bilingual thesis abstracts (page 6) that are written in both English and French, which leverages a useful domain (scientific literature).\n\n6. There is a clear discussion about the environmental impact of the work, and how the model was trained on data centres that used low-carbon nuclear electricity (section 3.7, page 11).\n\n7. From what I can tell, the authors took a conscious approach to potential test data contamination, for instance by excluding machine translation results with the chat model, which used WMT14 for fine-tuning and thus would suffer from test data contamination (Table 6, page 18).\n\n# Weaknesses\n\n1. I am not sure the model technically qualifies as an LLM, where the first \"L\" stands for \"large\", given that the model size is only 1.3B parameters. I do think that the model will be useful for end users, especially for those with hardware constraints (e.g. those with only 1 low-end or mid-end GPUs), in addition to scientists or engineers who conduct research on the scientific aspect of language models. My objection has more to do with the fact that the term \"LLM\" is a misnomer for models of this size.\n\n2. The model focuses on bilingual capability on English and French. Both of these languages are high-resource languages, and are already fairly well-covered by current LLMs (including both closed-sourced ones and \"open-sourced\" ones like Llama or Mistral 7B that only open-source the final weights of the model). For this reason, French speakers likely already have a plethora of LLMs to choose from (especially for those that do not have hardware constraints that limit them to models of sizes ~1 billion parameters); it might be nice to include other, lower-resource languages that are not yet well-covered by current LLMs.\n\n3. It would be nice to include more general lessons learnt / recommendations that are more generally applicable to those of us in the community who are building LLMs. For instance, what is the right \"overtraining factor\" to use? How does the model loss evolve if we train the model for, say, 10x / 20x / 50x / 100x what is training-compute-optimal (Hoffmann et al., 2022)? When does overtraining give only marginal gains, or when does performance start to saturate (if any)? Note that I am not asking the authors to redo the experiments; I am only asking them to outline and distill the key lessons learnt and recommendations for others who want to build similar LLMs, say for other languages.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduced CroissantLLM, a bilingual pre-trained LLM with bilingual proficiency in English and French. Despite its small size (1.3B), CroissantLLM was trained on a large corpus of bilingual English & French dataset (3T tokens), resulting in a small, efficient, yet high-performing model compared to what a training-compute-optimal model of a similar size would perform (Chinchilla scaling law; Hoffmann et al., 2022). \n\nUnlike many closed-sourced LLMs (GPT-4, etc.), this paper open-sources many aspects of the model, from the training dataset, pre-trained checkpoints at various stages of training, and fine-tuned chat models, among others. The extent of the open-sourcing done in this paper (i.e. not just the pre-trained checkpoint, but also some of the training data, etc.) exceeds that of other open-sourced initiatives, such as LLaMa and Mistral 7B. Empirical results on multiple English and French benchmarks (including on machine translation benchmarks between the two languages) indicate that the model matches or outperforms other pre-trained models of similar sizes, although overall performance still generally lags behind larger models (e.g. Llama 2 7B & Mistral 7B), which are more costly to run.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Big Picture Changes (Essential)\n\n1. Reassess whether or not the term \"LLM\" is appropriate for the model size that the paper is building, and if so, why.\n\n2. Distill the key lessons learnt and recommendations from the authors' experience in building CroissantLLM. This will help others in the community who want to do similar things, for instance those who want to build LLMs for other languages. This could cover data collection advice, handling of \"spike loss\", overtraining factor, etc.\n\n# Answering Questions / Suggestions (Not strictly essential but would strengthen the work)\n\n1. On page 4, section 2.1, it seems that the authors collect data from Francophone countries / regions, for instance parts of Belgium, Switzerland, Senegal, Morocco, etc. How different are these news sources in terms of dialects, and how are these different dialects handled? Or are all the data sources mixed in without any special handling for different dialects?\n\n2. On page 10, section 3.4, it seems that the authors did a careful handling of the bilingual data distribution, in order to \"balance out\" the English and French data. But each language is also comprised of multiple data sources, which can also differ wildly in terms of the size of each potential data source (e.g. French-English thesis abstracts likely contain much fewer tokens than French news sources). Is there any particular handling of the **domain imbalance** within a particular language?\n\n3. On page 10, section 3.6, it seems that the authors observe loss spikes, which are quite common when training language models (more so for larger models). However, from my understanding, the authors simply ignored these loss spikes and let the model recover on its own. In my experience, these loss spikes can have a detrimental effect on the final loss. Have the authors tried simple things like reloading the checkpoint prior to a loss spike, and resuming training on a different batch of data? Or clipping the gradients, limiting the norms of the parameters, etc.? These techniques might mitigate the loss spikes, and ultimately yield a better final performance.\n\n4. I don't see the English MMLU results on the tables in the main paper (please correct me if I'm wrong). Would it be possible to include them in order to facilitate a good comparison with prior work?\n\n5. The French Trivia dataset (page 12, section 4.2.1) is described to use English questions. Why is this the case, and why not just frame the questions in French? Is it to test bilingual generalisation, or something else?\n\n6. It would be good to put the model sizes of CroissantLLM on all the tables (e.g. Tables 3, 7, 8 etc.), because all the other models have the model sizes right next to them. This would facilitate an easier comparison.\n\n# Typo\n\nOn page 7, \"highly performing\" should be \"high-performing\"?", "broader_impact_concerns": "The broader impact concerns are already quite comprehensive. In my understanding, the authors already did their best in ensuring that the released training dataset complies with the relevant copyright regulations, and excluded copyright-restricted material from the model training and the released dataset.\n\nHowever, I would like to ask a hypothetical (yet not entirely unrealistic) question: What happens if, later, it turns out that some of the released dataset contained copyright-restricted material? Would it be sufficient to simply remove that part from the publicly-released dataset? Or does this also mean that the model, which was trained on this copyright-restricted material, would need to be taken offline and retrained? If it is the latter, there might be a huge cost to retraining the model, etc. I would like to better understand how such situations, if they arose at a later time, would be handled. This hypothetical scenario attests to the importance of \"unlearning\" (e.g. Yuanshun Yao et al., 2023, the NeurIPS 2023 Machine Unlearning Challenge, etc.).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Rir3iAccy4", "reviewer": "Reviewer_RC4S", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths are the contributions to the community by setting a great baseline and releasing all possible materials from the work. The introduction of the new benchmark would also be valuable to measure models' multilingual capabilities. In other words, the contributions are mainly made by providing a basis to perform further research with the materials released with the paper rather than novel scientific findings. There are some quantitive analyses of the model in the paper and those may be informative, but it seems to be that existing theories and knowledge can explain the results and there is no critical scientific finding.\n\nThe main weakness is the lack of scientific novelty. It is certainly a significant effort to collect new data and train a new large language model of a particular interest. However, it is a yet another effort to create a new model without novel findings.\n\nAnother weakness is that it is not clear if the discussions in the paper are only applicable to the class of models of the size or it is broadly applicable to extremely large models. The experimental results suggest that large models are better on French even with the distorted distributions. In that case, another approach would be to distill from a large model while keeping the desired multilingual performance as much as possible. There is no discussion on this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The main contribution of the paper is an introduction of a new LLM featuring the equal emphasis of two languages, English and French, with data and a full recipe to reproduce the results. A new benchmark to assess the capability on French is also introduced. The data mixture is carefully designed to have good performance on both languages and the empirical findings and analyses are presented. The experimental results and findings are probably considered as not surprising and they are as expected -- the new LLM is better on the French benchmark while keeping good performance on English.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There is no critical requested change for acceptance and these are all suggestions.\n\nI would like to see the contributions mentioned in Introduction more clearly correspond to the concerns raised in the first part of introduction. I would like to see what the paper did to address the concerns. The contes are OK, but this is just a matter of structure.\n\nI would like to see the exact definitions of used metrics in the paper at least as appendices, e.g. tokenizer fertility, capacity ratio, instead of giving brief explanations.\n\nIt would be good to give thoughts on the importance of the work for larger model sizes.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has received positive feedback from all the reviewers. The reviewers appreciate the paper's contributions to the research community, particularly its open-source nature and focus on bilingual proficiency. The reviewers also highlight the paper's detailed technical report on training and evaluating the bilingual model, as well as its insightful discussion on language ratio selection and the creation of a high-quality benchmark for the French language. However, some reviewers suggest that the paper could benefit from more content discussing the "Bias towards English" problem and comparisons with other popular 1B-scale open-source models. 

Overall, I recommend accepting the paper for publication due to its valuable contributions and practical applications.
Decision: Accept (poster)