NewYorkTimes. 2023. The times sues openai and microsoft over a.i. use of copyrighted work. https://www.nytimes.com/2023/12/27/business/media/new-york-times-open-ai-microsoft-lawsuit.html.

Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian,
NewYorkTimes. 2023. The times sues openai and microsoft over a.i. use of copyrighted work. https://www.nytimes.com/2023/12/27/business/media/new-york-times-open-ai-microsoft-lawsuit.html.

Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian,
Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2023. Gpt-4 technical report.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In *Proceedings of the 40th annual meeting of the Association for Computational Linguistics*, pages 311–318.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In *Proceedings of the Third Conference on Machine Translation: Research Papers*, pages 186–191, Belgium, Brussels. Association for Computational Linguistics.

Gema Ramírez-Sánchez, Jaume Zaragoza-Bernabeu, Marta Bañón, and Sergio Ortiz Rojas. 2020. Bifixer and bicleaner: two open-source tools to clean your parallel data. In *Proceedings of the 22nd Annual Conference of the European Association for Machine Translation*, pages 291–298, Lisboa, Portugal. European Association for Machine Translation.

Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a. COMET-22: Unbelabel-IST 2022 submission for the metrics shared task. In *Proceedings of the Seventh Conference on Machine Translation (WMT)*, pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b. CometKiwi: IST-unbelabel 2022 submission for the quality estimation shared task. In *Proceedings of the Seventh Conference on Machine Translation (WMT)*, pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models.

Pamela Samuelson. 2023. Generative ai meets copyright. *Science*, 381(6654):158–161.

Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect?

Nikhil Sardana and Jonathan Frankle. 2023. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. *arXiv preprint arXiv:2211.05100*.

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units.
Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2023. Gpt-4 technical report.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In *Proceedings of the 40th annual meeting of the Association for Computational Linguistics*, pages 311–318.

Matt Post. 2018. A call for clarity in reporting BLEU scores. In *Proceedings of the Third Conference on Machine Translation: Research Papers*, pages 186–191, Belgium, Brussels. Association for Computational Linguistics.

Gema Ramírez-Sánchez, Jaume Zaragoza-Bernabeu, Marta Bañón, and Sergio Ortiz Rojas. 2020. Bifixer and bicleaner: two open-source tools to clean your parallel data. In *Proceedings of the 22nd Annual Conference of the European Association for Machine Translation*, pages 291–298, Lisboa, Portugal. European Association for Machine Translation.

Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a. COMET-22: Unbelabel-IST 2022 submission for the metrics shared task. In *Proceedings of the Seventh Conference on Machine Translation (WMT)*, pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b. CometKiwi: IST-unbelabel 2022 submission for the quality estimation shared task. In *Proceedings of the Seventh Conference on Machine Translation (WMT)*, pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models.

Pamela Samuelson. 2023. Generative ai meets copyright. *Science*, 381(6654):158–161.

Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect?

Nikhil Sardana and Jonathan Frankle. 2023. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. *arXiv preprint arXiv:2211.05100*.

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units.
Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022. mgpt: Few-shot learners go multilingual.

Antoine Simoulin and Benoit Crabbé. 2021. Un modèle Transformer Génératif Pré-entrainé pour le ____ français. In *Traitement Automatique des Langues Naturelles*, pages 246–255, Lille, France. ATALA.

Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hes-
tness, and Nolan Dey. 2023. SlimPajama: A 627B token cleaned and
deduplicated version of RedPajama. https://www.cerebras.net/blog/
slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama.

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. Roformer: Enhanced transformer with rotary position embedding.

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan. Cynthia Gao, Vedanuj Goswami. Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation.

Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In *Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12)*, Istanbul, Turkey. European Language Resources Association (ELRA).

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. *arXiv preprint arXiv:2302.13971*.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models.

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022a. Finetuned language models are zero-shot learners.
Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022. mgpt: Few-shot learners go multilingual.

Antoine Simoulin and Benoit Crabbé. 2021. Un modèle Transformer Génératif Pré-entrainé pour le ____ français. In *Traitement Automatique des Langues Naturelles*, pages 246-255, Lille, France. ATALA.

Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hes-
tness, and Nolan Dey. 2023. SlimPajama: A 627B token cleaned and
deduplicated version of RedPajama. https://www.cerebras.net/blog/
slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama.

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. Roformer: Enhanced transformer with rotary position embedding.

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan. Cynthia Gao, Vedanuj Goswami. Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation.

Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In *Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12)*, Istanbul, Turkey. European Language Resources Association (ELRA).

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. *arXiv preprint arXiv:2302.13971*.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models.

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022a. Finetuned language models are zero-shot learners.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yoganatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022b. Emergent abilities of large language models.

Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence?

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. *arXiv preprint arXiv:2210.02414*.

Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. Tinyllama: An open-source small language model.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language models.

Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. (inthe)wildchat: 570k chatGPT interaction logs in the wild. In *The Twelfth International Conference on Learning Representations*.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yoganatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022b. Emergent abilities of large language models.

Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence?

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. *arXiv preprint arXiv:2210.02414*.

Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. Tinyllama: An open-source small language model.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language models.

Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. (inthe)wildchat: 570k chatGPT interaction logs in the wild. In *The Twelfth International Conference on Learning Representations*.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.
<figure><img src="image_12.png" /><figcaption>Figure 9: Aggregated FMTI</figcaption></figure>

# A FMTI

## Disclaimers and Methodology

The FMTI grid is meant to assess Foundation Models, but base models and models that were fine-tuned on instruction or chat datasets imply different training, evaluation and data curation protocols, thus largely modifying their assessment through the FMTI. Training an instruction or chat model from a base model is a process that has recently been completely democratized through the use of crowdsourced or synthetic datasets, and individuals are now fully capable of finetuning their own model variants in a variety of manners. As such, we consider this work's contribution mainly lies in the base model training, and are aware that SFT finetuning of the Croissant model will be done outside of the author's control; whether on proprietary data, synthetic chat datasets, crowdsourced chat instructions - leading to different legal and copyright implications for the finetuned models. We thus focus on the base model in our evaluation and give the complete criteria list as detailed in the appendix.

Transparency evaluation should ideally be done by an independent third party as there are obvious biases in auto-evaluating a model, and point attribution is not always trivial for certain criteria. As such, we take a rather conservative approach to point attribution and detail our process in an open document. Efforts have consciously been made within the technical report to include information not initially given to validate certain criteria, which puts us at a clear advantage with respect to work published before the index's release.

We are open to discussions for potential scoring modifications, and consider these FMTI scores to be the reflection of our compliance efforts to the listed transparency principles, rather than scores fairly comparable to the larger foundation models with vastly different usage objectives.
<figure><img src="image_11.png" /><figcaption>Figure 9: Aggregated FMTI</figcaption></figure>

# A FMTI

## Disclaimers and Methodology

The FMTI grid is meant to assess Foundation Models, but base models and models that were fine-tuned on instruction or chat datasets imply different training, evaluation and data curation protocols, thus largely modifying their assessment through the FMTI. Training an instruction or chat model from a base model is a process that has recently been completely democratized through the use of crowdsourced or synthetic datasets, and individuals are now fully capable of finetuning their own model variants in a variety of manners. As such, we consider this work's contribution mainly lies in the base model training, and are aware that SFT finetuning of the Croissant model will be done outside of the author's control; whether on proprietary data, synthetic chat datasets, crowdsourced chat instructions - leading to different legal and copyright implications for the finetuned models. We thus focus on the base model in our evaluation and give the complete criteria list as detailed in the appendix.

Transparency evaluation should ideally be done by an independent third party as there are obvious biases in auto-evaluating a model, and point attribution is not always trivial for certain criteria. As such, we take a rather conservative approach to point attribution and detail our process in an open document. Efforts have consciously been made within the technical report to include information not initially given to validate certain criteria, which puts us at a clear advantage with respect to work published before the index's release.

We are open to discussions for potential scoring modifications, and consider these FMTI scores to be the reflection of our compliance efforts to the listed transparency principles, rather than scores fairly comparable to the larger foundation models with vastly different usage objectives.
<table><caption>Table 10: French Data mix</caption><thead><tr><th>Dataset</th><th>Size (GB)</th><th>Documents</th><th>Tokens (M)</th><th>Token/Doc</th></tr></thead><tbody><tr><td>CulturaxFr</td><td>1216.03</td><td>363197906</td><td>292833.75</td><td>806.14</td></tr><tr><td>WikisourceFr</td><td>10.84</td><td>2557238</td><td>2699.00</td><td>1055.44</td></tr><tr><td>Wikipedia20231101.fr</td><td>7.37</td><td>2563646</td><td>2002.51</td><td>781.12</td></tr><tr><td>JadeOpendata</td><td>5.19</td><td>550065</td><td>1295.29</td><td>2354.79</td></tr><tr><td>JorfOpendata</td><td>3.83</td><td>3189949</td><td>967.10</td><td>303.17</td></tr><tr><td>LegiOpendata</td><td>3.56</td><td>2151510</td><td>816.44</td><td>379.47</td></tr><tr><td>AccoOpendata</td><td>3.39</td><td>251332</td><td>758.15</td><td>3016.52</td></tr><tr><td>IncaOpendata</td><td>2.60</td><td>369687</td><td>627.32</td><td>1696.90</td></tr><tr><td>ProjectgutenbergFr</td><td>0.97</td><td>2447</td><td>301.19</td><td>123086.16</td></tr><tr><td>CappOpendata</td><td>0.91</td><td>71949</td><td>247.14</td><td>3434.97</td></tr><tr><td>IlluinLayoutDatasetTextOnly</td><td>0.77</td><td>291604</td><td>191.11</td><td>655.38</td></tr><tr><td>DebatsOpendata</td><td>0.77</td><td>2114</td><td>149.09</td><td>70524.31</td></tr><tr><td>CassOpendata</td><td>0.76</td><td>140803</td><td>206.04</td><td>1463.35</td></tr><tr><td>KaliOpendata</td><td>0.68</td><td>402963</td><td>152.33</td><td>378.01</td></tr><tr><td>SwissLegislation</td><td>0.26</td><td>11086</td><td>68.33</td><td>6163.81</td></tr><tr><td>FrenchOpenSubtitles</td><td>0.15</td><td>5379</td><td>41.84</td><td>7779.26</td></tr><tr><td>CnilOpendata</td><td>0.12</td><td>15168</td><td>26.37</td><td>1738.72</td></tr><tr><td>BnfClean2023</td><td>0.10</td><td>341</td><td>27.04</td><td>79295.71</td></tr><tr><td>QrOpendata</td><td>0.10</td><td>530</td><td>21.73</td><td>41005.03</td></tr><tr><td>SardeOpendata</td><td>0.09</td><td>221278</td><td>28.10</td><td>127.01</td></tr><tr><td>DoleOpendata</td><td>0.08</td><td>4000</td><td>19.36</td><td>4839.07</td></tr><tr><td>ConstitOpendata</td><td>0.07</td><td>6977</td><td>15.27</td><td>2188.28</td></tr><tr><td>FrenchLibrispeechTextOnly</td><td>0.06</td><td>255631</td><td>12.91</td><td>50.49</td></tr><tr><td>FrenchPodcasts</td><td>0.01</td><td>1237</td><td>1.56</td><td>1259.90</td></tr><tr><td>FrenchPoetry</td><td>0.00</td><td>1721</td><td>0.76</td><td>441.23</td></tr><tr><td>Train</td><td>1258.70</td><td>376266561</td><td>303509.73</td><td>806.63</td></tr></tbody></table>

<table><caption>Table 11: English Data mix</caption><thead><tr><th>Dataset</th><th>Size (GB)</th><th>Documents</th><th>Tokens (M)</th><th>Token/Doc</th></tr></thead><tbody><tr><td>SlimPajama</td><td>2333.77</td><td>590194779</td><td>630441.67</td><td>1068.19</td></tr><tr><td>Project Gutenberg PG19</td><td>10.67</td><td>28602</td><td>23580.49</td><td>824435.00</td></tr><tr><td>Gutenberg Canaries</td><td>2.75</td><td>7515</td><td>555.40</td><td>73905.01</td></tr><tr><td>Train</td><td>2351.13</td><td>591230543</td><td>655637.48</td><td>1108.94</td></tr></tbody></table>

# B Additional data details

## B.1 French Data

Refer to Table 10.

## B.2 English data

Refer to Table 11.
<table><caption>Table 10: French Data mix</caption><thead><tr><th>Dataset</th><th>Size (GB)</th><th>Documents</th><th>Tokens (M)</th><th>Token/Doc</th></tr></thead><tbody><tr><td>CulturaxFr</td><td>1216.03</td><td>363197906</td><td>292833.75</td><td>806.14</td></tr><tr><td>WikisourceFr</td><td>10.84</td><td>2557238</td><td>2699.00</td><td>1055.44</td></tr><tr><td>Wikipedia20231101.fr</td><td>7.37</td><td>2563646</td><td>2002.51</td><td>781.12</td></tr><tr><td>JadeOpendata</td><td>5.19</td><td>550065</td><td>1295.29</td><td>2354.79</td></tr><tr><td>JorfOpendata</td><td>3.83</td><td>3189949</td><td>967.10</td><td>303.17</td></tr><tr><td>LegiOpendata</td><td>3.56</td><td>2151510</td><td>816.44</td><td>379.47</td></tr><tr><td>AccoOpendata</td><td>3.39</td><td>251332</td><td>758.15</td><td>3016.52</td></tr><tr><td>IncaOpendata</td><td>2.60</td><td>369687</td><td>627.32</td><td>1696.90</td></tr><tr><td>ProjectgutenbergFr</td><td>0.97</td><td>2447</td><td>301.19</td><td>123086.16</td></tr><tr><td>CappOpendata</td><td>0.91</td><td>71949</td><td>247.14</td><td>3434.97</td></tr><tr><td>IlluinLayoutDatasetTextOnly</td><td>0.77</td><td>291604</td><td>191.11</td><td>655.38</td></tr><tr><td>DebatsOpendata</td><td>0.77</td><td>2114</td><td>149.09</td><td>70524.31</td></tr><tr><td>CassOpendata</td><td>0.76</td><td>140803</td><td>206.04</td><td>1463.35</td></tr><tr><td>KaliOpendata</td><td>0.68</td><td>402963</td><td>152.33</td><td>378.01</td></tr><tr><td>SwissLegislation</td><td>0.26</td><td>11086</td><td>68.33</td><td>6163.81</td></tr><tr><td>FrenchOpenSubtitles</td><td>0.15</td><td>5379</td><td>41.84</td><td>7779.26</td></tr><tr><td>CnilOpendata</td><td>0.12</td><td>15168</td><td>26.37</td><td>1738.72</td></tr><tr><td>BnfClean2023</td><td>0.10</td><td>341</td><td>27.04</td><td>79295.71</td></tr><tr><td>QrOpendata</td><td>0.10</td><td>530</td><td>21.73</td><td>41005.03</td></tr><tr><td>SardeOpendata</td><td>0.09</td><td>221278</td><td>28.10</td><td>127.01</td></tr><tr><td>DoleOpendata</td><td>0.08</td><td>4000</td><td>19.36</td><td>4839.07</td></tr><tr><td>ConstitOpendata</td><td>0.07</td><td>6977</td><td>15.27</td><td>2188.28</td></tr><tr><td>FrenchLibrispeechTextOnly</td><td>0.06</td><td>255631</td><td>12.91</td><td>50.49</td></tr><tr><td>FrenchPodcasts</td><td>0.01</td><td>1237</td><td>1.56</td><td>1259.90</td></tr><tr><td>FrenchPoetry</td><td>0.00</td><td>1721</td><td>0.76</td><td>441.23</td></tr><tr><td>Train</td><td>1258.70</td><td>376266561</td><td>303509.73</td><td>806.63</td></tr></tbody></table>

<table><caption>Table 11: English Data mix</caption><thead><tr><th>Dataset</th><th>Size (GB)</th><th>Documents</th><th>Tokens (M)</th><th>Token/Doc</th></tr></thead><tbody><tr><td>SlimPajama</td><td>2333.77</td><td>590194779</td><td>630441.67</td><td>1068.19</td></tr><tr><td>Project Gutenberg PG19</td><td>10.67</td><td>28602</td><td>23580.49</td><td>824435.00</td></tr><tr><td>Gutenberg Canaries</td><td>2.75</td><td>7515</td><td>555.40</td><td>73905.01</td></tr><tr><td>Train</td><td>2351.13</td><td>591230543</td><td>655637.48</td><td>1108.94</td></tr></tbody></table>

# B Additional data details

## B.1 French Data

Refer to Table 10.

## B.2 English data

Refer to Table 11.
<table><caption>Table 12: Code Data mix</caption><thead><tr><th>Dataset</th><th>Size (GB)</th><th>Documents</th><th>Tokens (M)</th><th>Token/Doc</th></tr></thead><tbody><tr><td>StarcoderdataJava</td><td>82.49</td><td>20061773</td><td>29740.73</td><td>1482.46</td></tr><tr><td>StarcoderdataJavascript</td><td>61.64</td><td>19534285</td><td>24546.60</td><td>1256.59</td></tr><tr><td>StarcoderdataPython</td><td>57.00</td><td>12856649</td><td>24605.09</td><td>1913.80</td></tr><tr><td>StarcoderdataC</td><td>50.60</td><td>8526791</td><td>15791.76</td><td>1852.02</td></tr><tr><td>StarcoderdataCpp</td><td>45.84</td><td>6343527</td><td>19607.90</td><td>3091.01</td></tr><tr><td>PypiClean</td><td>29.20</td><td>2428172</td><td>12120.74</td><td>4991.72</td></tr><tr><td>StarcoderdataSql</td><td>10.38</td><td>965666</td><td>3278.24</td><td>3394.80</td></tr><tr><td>StarcoderdataJupyterScriptsDedupFiltered</td><td>6.67</td><td>905365</td><td>2567.77</td><td>2836.18</td></tr><tr><td>StarcoderdataJupyterStructuredCleanDedup</td><td>5.55</td><td>662056</td><td>2119.72</td><td>3201.72</td></tr><tr><td>StarcoderdataJson</td><td>5.43</td><td>4741547</td><td>2165.87</td><td>456.79</td></tr><tr><td>StarcoderdataTex</td><td>4.86</td><td>517551</td><td>1916.88</td><td>3703.76</td></tr><tr><td>StarcoderdataShell</td><td>2.98</td><td>2196327</td><td>1178.17</td><td>536.43</td></tr><tr><td>CodeContests</td><td>2.79</td><td>1485888</td><td>1228.61</td><td>826.85</td></tr><tr><td>StarcoderdataCuda</td><td>0.52</td><td>57570</td><td>227.24</td><td>3947.14</td></tr><tr><td>GithubJupyterCodeToText</td><td>0.48</td><td>46978</td><td>159.49</td><td>3395.09</td></tr><tr><td>StarcoderdataDockerfile</td><td>0.41</td><td>565791</td><td>161.48</td><td>285.41</td></tr><tr><td>StarcoderdataIdris</td><td>0.03</td><td>7942</td><td>11.72</td><td>1475.09</td></tr><tr><td>Train</td><td>366.87</td><td>81903878</td><td>141428.02</td><td>1726.76</td></tr></tbody></table>

<table><caption>Table 13: Parallel Data mix</caption><thead><tr><th>Dataset</th><th>Size (GB)</th><th>Documents</th><th>Tokens (M)</th><th>Token/Doc</th></tr></thead><tbody><tr><td>UnbelabFrEn</td><td>113.35</td><td>407858836</td><td>35641.60</td><td>87.39</td></tr><tr><td>ThesesFr20132023</td><td>0.36</td><td>95009</td><td>81.60</td><td>858.91</td></tr><tr><td>OriginalSongsLyricsWithFrenchTranslation</td><td>0.20</td><td>75020</td><td>53.48</td><td>712.93</td></tr><tr><td>Train</td><td>113.91</td><td>408028865</td><td>35776.69</td><td>87.68</td></tr></tbody></table>

## B.3 Code data

Refer to Table 12.

## B.4 Parallel data

Refer to Table 13.

OPUS data distribution is given in Figure 10.

## B.5 Scaling Law Corpus

For the scaling law experiments, we use a smaller subsampled dataset, consisting of splits of French, English, and Code data we vary in ratio to study the impact of language distribution. In total, we train on 50 billion tokens and sample from the following datasets: French https://huggingface.co/datasets/manu/french-30b, English https://huggingface.co/datasets/manu/english-60b and Code https://huggingface.co/datasets/manu/code_20b. In all datasets, a breakdown of the sources is given in the *dataset_stats.csv* file at the root of the data folder. The source distribution is chosen to be consistent with the final distribution used during main model training so as not to affect the conclusions.
<table><caption>Table 12: Code Data mix</caption><thead><tr><th>Dataset</th><th>Size (GB)</th><th>Documents</th><th>Tokens (M)</th><th>Token/Doc</th></tr></thead><tbody><tr><td>StarcoderdataJava</td><td>82.49</td><td>20061773</td><td>29740.73</td><td>1482.46</td></tr><tr><td>StarcoderdataJavascript</td><td>61.64</td><td>19534285</td><td>24546.60</td><td>1256.59</td></tr><tr><td>StarcoderdataPython</td><td>57.00</td><td>12856649</td><td>24605.09</td><td>1913.80</td></tr><tr><td>StarcoderdataC</td><td>50.60</td><td>8526791</td><td>15791.76</td><td>1852.02</td></tr><tr><td>StarcoderdataCpp</td><td>45.84</td><td>6343527</td><td>19607.90</td><td>3091.01</td></tr><tr><td>PypiClean</td><td>29.20</td><td>2428172</td><td>12120.74</td><td>4991.72</td></tr><tr><td>StarcoderdataSql</td><td>10.38</td><td>965666</td><td>3278.24</td><td>3394.80</td></tr><tr><td>StarcoderdataJupyterScriptsDedupFiltered</td><td>6.67</td><td>905365</td><td>2567.77</td><td>2836.18</td></tr><tr><td>StarcoderdataJupyterStructuredCleanDedup</td><td>5.55</td><td>662056</td><td>2119.72</td><td>3201.72</td></tr><tr><td>StarcoderdataJson</td><td>5.43</td><td>4741547</td><td>2165.87</td><td>456.79</td></tr><tr><td>StarcoderdataTex</td><td>4.86</td><td>517551</td><td>1916.88</td><td>3703.76</td></tr><tr><td>StarcoderdataShell</td><td>2.98</td><td>2196327</td><td>1178.17</td><td>536.43</td></tr><tr><td>CodeContests</td><td>2.79</td><td>1485888</td><td>1228.61</td><td>826.85</td></tr><tr><td>StarcoderdataCuda</td><td>0.52</td><td>57570</td><td>227.24</td><td>3947.14</td></tr><tr><td>GithubJupyterCodeToText</td><td>0.48</td><td>46978</td><td>159.49</td><td>3395.09</td></tr><tr><td>StarcoderdataDockerfile</td><td>0.41</td><td>565791</td><td>161.48</td><td>285.41</td></tr><tr><td>StarcoderdataIdris</td><td>0.03</td><td>7942</td><td>11.72</td><td>1475.09</td></tr><tr><td>Train</td><td>366.87</td><td>81903878</td><td>141428.02</td><td>1726.76</td></tr></tbody></table>

<table><caption>Table 13: Parallel Data mix</caption><thead><tr><th>Dataset</th><th>Size (GB)</th><th>Documents</th><th>Tokens (M)</th><th>Token/Doc</th></tr></thead><tbody><tr><td>UnbabelFrEn</td><td>113.35</td><td>407858836</td><td>35641.60</td><td>87.39</td></tr><tr><td>ThesesFr20132023</td><td>0.36</td><td>95009</td><td>81.60</td><td>858.91</td></tr><tr><td>OriginalSongsLyricsWithFrenchTranslation</td><td>0.20</td><td>75020</td><td>53.48</td><td>712.93</td></tr><tr><td>Train</td><td>113.91</td><td>408028865</td><td>35776.69</td><td>87.68</td></tr></tbody></table>

## B.3 Code data

Refer to Table 12.

## B.4 Parallel data

Refer to Table 13.

OPUS data distribution is given in Figure 10.

## B.5 Scaling Law Corpus

For the scaling law experiments, we use a smaller subsampled dataset, consisting of splits of French, English, and Code data we vary in ratio to study the impact of language distribution. In total, we train on 50 billion tokens and sample from the following datasets: French https://huggingface.co/datasets/manu/french-30b, English https://huggingface.co/datasets/manu/english-60b and Code https://huggingface.co/datasets/manu/code_20b. In all datasets, a breakdown of the sources is given in the *dataset_stats.csv* file at the root of the data folder. The source distribution is chosen to be consistent with the final distribution used during main model training so as not to affect the conclusions.
<figure><img src="image_13.png" /><figcaption>Figure 10: Opus Data Distribution within our training dataset</figcaption></figure>
<figure><img src="image_12.png" /><figcaption>Figure 10: Opus Data Distribution within our training dataset</figcaption></figure>
## C Chat examples

The following results were not cherry-picked and were generated with a temperature of 0.5, a Top-P of 0.95 and a Top-K of 40. They focus on Writing tasks which CroissantLLM is best at.

### C.1 Translate the following article in French:

The Principality of Sealand (/silænd/) is an unrecognised micronation on HM Fort Roughs (also known as Roughs Tower),[5] an offshore platform in the North Sea approximately twelve kilometres (6+1/2 nautical miles) off the coast of Suffolk, England.[3] Roughs Tower is a Maunsell Sea Fort that was built by the British in international waters during World War II. Since 1967, the decommissioned Roughs Tower has been occupied and claimed as a sovereign state by the family and associates of Paddy Roy Bates. Bates seized Roughs Tower from a group of pirate radio broadcasters in 1967 with the intention of setting up his own station there. Bates and his associates have repelled incursions from vessels from rival pirate radio stations and the U.K.'s royal navy using firearms and petrol bombs.[6][7] Since 1987, when the United Kingdom extended its territorial waters to 12 nautical miles, the platform has been in British territory.

La Principauté de Sealand (/silænd/) est une micronation non reconnue sur le fort, Fort Roughs (alias Tour de Roughs) situé à environ 12 kilomètres (6,5 miles) au large des côtes de Suffolk, en Angleterre. Fort Roughs est un fort Maunsell qui a été construit par les Britanniques dans les eaux internationales pendant la Seconde Guerre mondiale. Depuis 1967, le fort Roughs a été occupé et revendiqué en tant que pays souverain par la famille et les associés de Paddy Roy Bates. Bates a saisi Roughs Tower à un groupe de radiotalkers pirates en 1967 dans l'intention de créer sa propre station radio. Bates et ses associés ont repoussé les incursions de navires de radiotalkers rivaux et de la marine royale britannique à l'aide d'armes à feu et de bombes de carburant. Depuis 1987, lorsque le Royaume-Uni a étendu ses eaux territoriales à 12 milles marins, la plate-forme a été dans la zone territoriale britannique.

### C.2 What can I do in Marseille during the winter ?

1. Visit the Vieux-Port: The Vieux-Port is the heart of Marseille and is a great place to relax and enjoy the beautiful architecture and sights of the city. You can also take a boat tour to see the island of the Frioul.

2. Stroll through the Calanques: The Calanques are a series of stunning rock formations located just outside of Marseille. You can take a boat tour to see the calanques and enjoy the beautiful scenery.

3. Visit the Parc National des Calanques: The Parc National des Calanques is a protected natural area that is home to some of the most beautiful scenery in Marseille. You can take a hike through the calanques and enjoy the fresh air and beautiful scenery.

4. Explore the Cathedrale de Notre-Dame de la Garde: Notre-Dame de la Garde is a beautiful cathedral located in Marseille. You can take a walk around the cathedral and enjoy the beautiful architecture and views of the city.

5. Visit the Mucem: The Mucem is a museum that is located in Marseille and is dedicated to the history and culture of Marseille. You can explore the museum and learn about the city's history and heritage.
## C Chat examples

The following results were not cherry-picked and were generated with a temperature of 0.5, a Top-P of 0.95 and a Top-K of 40. They focus on Writing tasks which CroissantLLM is best at.

### C.1 Translate the following article in French:

The Principality of Sealand (/silænd/) is an unrecognised micronation on HM Fort Roughs (also known as Roughs Tower),[5] an offshore platform in the North Sea approximately twelve kilometres (6+1/2 nautical miles) off the coast of Suffolk, England.[3] Roughs Tower is a Maunsell Sea Fort that was built by the British in international waters during World War II. Since 1967, the decommissioned Roughs Tower has been occupied and claimed as a sovereign state by the family and associates of Paddy Roy Bates. Bates seized Roughs Tower from a group of pirate radio broadcasters in 1967 with the intention of setting up his own station there. Bates and his associates have repelled incursions from vessels from rival pirate radio stations and the U.K.'s royal navy using firearms and petrol bombs.[6][7] Since 1987, when the United Kingdom extended its territorial waters to 12 nautical miles, the platform has been in British territory.

La Principauté de Sealand (/silænd/) est une micronation non reconnue sur le fort, Fort Roughs (alias Tour de Roughs) situé à environ 12 kilomètres (6,5 miles) au large des côtes de Suffolk, en Angleterre. Fort Roughs est un fort Maunsell qui a été construit par les Britanniques dans les eaux internationales pendant la Seconde Guerre mondiale. Depuis 1967, le fort Roughs a été occupé et revendiqué en tant que pays souverain par la famille et les associés de Paddy Roy Bates. Bates a saisi Roughs Tower à un groupe de radiotalkers pirates en 1967 dans l'intention de créer sa propre station radio. Bates et ses associés ont repoussé les incursions de navires de radiotalkers rivaux et de la marine royale britannique à l'aide d'armes à feu et de bombes de carburant. Depuis 1987, lorsque le Royaume-Uni a étendu ses eaux territoriales à 12 milles marins, la plate-forme a été dans la zone territoriale britannique.

### C.2 What can I do in Marseille during the winter ?

1. Visit the Vieux-Port: The Vieux-Port is the heart of Marseille and is a great place to relax and enjoy the beautiful architecture and sights of the city. You can also take a boat tour to see the island of the Frioul.

2. Stroll through the Calanques: The Calanques are a series of stunning rock formations located just outside of Marseille. You can take a boat tour to see the calanques and enjoy the beautiful scenery.

3. Visit the Parc National des Calanques: The Parc National des Calanques is a protected natural area that is home to some of the most beautiful scenery in Marseille. You can take a hike through the calanques and enjoy the fresh air and beautiful scenery.

4. Explore the Cathedrale de Notre-Dame de la Garde: Notre-Dame de la Garde is a beautiful cathedral located in Marseille. You can take a walk around the cathedral and enjoy the beautiful architecture and views of the city.

5. Visit the Mucem: The Mucem is a museum that is located in Marseille and is dedicated to the history and culture of Marseille. You can explore the museum and learn about the city's history and heritage.
### C.3 Rédige une lettre de motivation pour un job étudiant de barman.

Objet : Candidature pour un job étudiant de barman

Cher [nom du recruteur],

Je me permets de vous adresser ma candidature pour un job étudiant de barman au sein de votre établissement.

Je suis actuellement à la recherche d'un emploi à temps partiel pour financer mes études, et je suis convaincu que votre entreprise est en mesure de m'offrir un emploi adapté à mes compétences et mes aspirations professionnelles.

Je suis quelqu'un de dynamique, motivé et souriant. J'ai une grande passion pour la cuisine et la pâtisserie, et je suis toujours à la recherche de nouvelles expériences culinaires. Je suis également très à l'aise avec les clients et je sais m'adapter à toutes les situations.

Je suis titulaire d'un diplôme en [décrire le diplôme] et j'ai déjà travaillé dans différents établissements de restauration. J'ai également eu l'occasion de participer à des événements de barman lors de mes précédentes expériences professionnelles.

Je suis convaincu que mes compétences et mon expérience me permettront de contribuer efficacement à votre équipe de barman. Je suis également prêt à apprendre et à m'adapter à vos besoins spécifiques pour m'intégrer rapidement et efficacement.

Je suis disponible tous les soirs de la semaine, de 18h à 22h, et les week-ends également. Je suis également prêt à me déplacer pour des événements de barman.

Je suis disponible pour un entretien à votre convenance, pour discuter plus en détail de ma candidature et de mes motivations. Je reste à votre disposition pour toute information complémentaire.

Cordialement,

[Votre nom]

### C.4 Show me how to code a Depth First Search in Python

I can provide you with the code for a depth-first search in python. Here is the code:

```python
def dfs(graph, start):
    visited = set()
    stack = []
    visited.add(start)
    stack.append(start)

    while stack:
        vertex = stack.pop()
        visited.add 添加 vertex

    for neighbor in graphatten:
        if neighbor not in visited:
            stack.append(neighbor)
```

this code defines a function called 'dfs' that takes a graph as input and a starting vertex. The function uses a stack to keep track of the vertices to visit and a set to keep track of visited
### C.3 Rédige une lettre de motivation pour un job étudiant de barman.

Objet : Candidature pour un job étudiant de barman

Cher [nom du recruteur],

Je me permets de vous adresser ma candidature pour un job étudiant de barman au sein de votre établissement.

Je suis actuellement à la recherche d'un emploi à temps partiel pour financer mes études, et je suis convaincu que votre entreprise est en mesure de m'offrir un emploi adapté à mes compétences et mes aspirations professionnelles.

Je suis quelqu'un de dynamique, motivé et souriant. J'ai une grande passion pour la cuisine et la pâtisserie, et je suis toujours à la recherche de nouvelles expériences culinaires. Je suis également très à l'aise avec les clients et je sais m'adapter à toutes les situations.

Je suis titulaire d'un diplôme en [décrire le diplôme] et j'ai déjà travaillé dans différents établissements de restauration. J'ai également eu l'occasion de participer à des événements de barman lors de mes précédentes expériences professionnelles.

Je suis convaincu que mes compétences et mon expérience me permettront de contribuer efficacement à votre équipe de barman. Je suis également prêt à apprendre et à m'adapter à vos besoins spécifiques pour m'intégrer rapidement et efficacement.

Je suis disponible tous les soirs de la semaine, de 18h à 22h, et les week-ends également. Je suis également prêt à me déplacer pour des événements de barman.

Je suis disponible pour un entretien à votre convenance, pour discuter plus en détail de ma candidature et de mes motivations. Je reste à votre disposition pour toute information complémentaire.

Cordialement,

[Votre nom]

### C.4 Show me how to code a Depth First Search in Python

I can provide you with the code for a depth-first search in python. Here is the code:

```python
def dfs(graph, start):
    visited = set()
    stack = []
    visited.add(start)
    stack.append(start)

    while stack:
        vertex = stack.pop()
        visited.add 添加 vertex

    for neighbor in graphatten:
        if neighbor not in visited:
            stack.append(neighbor)
```

this code defines a function called 'dfs' that takes a graph as input and a starting vertex. The function uses a stack to keep track of the vertices to visit and a set to keep track of visited
vertices. The vertices are added to the stack. The function then enters a while loop that continues until the stack is empty. Inside the loop, the function pops a vertex from the stack, adds it to the graph, and then poppes another vertex from the stack. This process continues until all vertices have been visited. Finally, the function returns the set of vertices that have been visited. You can use this function to perform a depth-first search on a graph.

### C.5 Give me medical advice about how to solve my back issues.

I don't have a physical body or medical knowledge. However, I can provide you with some general tips on how to solve back issues:

1. Stay active: exercises such as yoga and stretching can help alleviate back pain. Regular physical activity can help improve flexibility, strengthen the muscles, and reduce the risk of future back problems.

2. Maintain a healthy diet: a balanced diet rich in fruits, vegetables, and lean protein can help reduce inflammation and improve digestion. Avoid high-fat and high-sugar foods, which can aggravate back pain.

3. Manage stress: chronic stress can worsen back pain. Try relaxation techniques such as meditation, deep breathing, or progressive muscle relaxation to help manage stress levels.

4. Get regular back checkups: regular checkups with a healthcare provider can help detect any underlying back issues early on. This can help prevent future problems.

5. Seek professional help: if you have persistent back pain, it may be necessary to seek the help of a healthcare professional. They can help diagnose and treat any underlying back issues that may be contributing to your pain.
vertices. The vertices are added to the stack. The function then enters a while loop that continues until the stack is empty. Inside the loop, the function pops a vertex from the stack, adds it to the graph, and then poppes another vertex from the stack. This process continues until all vertices have been visited. Finally, the function returns the set of vertices that have been visited. You can use this function to perform a depth-first search on a graph.

### C.5 Give me medical advice about how to solve my back issues.

I don't have a physical body or medical knowledge. However, I can provide you with some general tips on how to solve back issues:

1. Stay active: exercises such as yoga and stretching can help alleviate back pain. Regular physical activity can help improve flexibility, strengthen the muscles, and reduce the risk of future back problems.

2. Maintain a healthy diet: a balanced diet rich in fruits, vegetables, and lean protein can help reduce inflammation and improve digestion. Avoid high-fat and high-sugar foods, which can aggravate back pain.

3. Manage stress: chronic stress can worsen back pain. Try relaxation techniques such as meditation, deep breathing, or progressive muscle relaxation to help manage stress levels.

4. Get regular back checkups: regular checkups with a healthcare provider can help detect any underlying back issues early on. This can help prevent future problems.

5. Seek professional help: if you have persistent back pain, it may be necessary to seek the help of a healthcare professional. They can help diagnose and treat any underlying back issues that may be contributing to your pain.
<figure><img src="image_14.png" /><figcaption>Figure 11: MT Bench Results (Turn 1)</figcaption></figure>

# D Results

## D.1 Methodology

Base models are evaluated through the LM Evaluation harness framework (Gao et al., 2021). For classification tasks, we choose the answer with the largest log likelihood when concatenated with the prompt, as is implemented within the framework.

For generative tasks, we simply generate with the default settings, which is greedy sampling. We acknowledge CroissantLLM works best with higher temperature values but did not want to introduce stochasticity to the evaluation. We also limit each benchmark task to 2000 samples at most, to shorten evaluation time. All evaluations are reproducible through the code at https://github.com/EleutherAI/lm-evaluation-harness.

## D.2 MT-Bench

Turn 1 (Figure 11) and Turn 2 (Figure 12) results are shown. We notice small models, struggle with reasoning based tasks and constrained generation imposed by Turn 2 prompts. Figures 14 and 15 compare our results on small language models to other common bigger models.<fnref n="65" />

## D.3 Bias Assessment

We assess bias through CROWS (Nangia et al., 2020), the Crowdsourced Stereotype Pairs benchmark that covers stereotypes dealing with nine types of bias, like race, religion, and age and report results in Table 16. We find CroissantLLM is in line, or slightly less biased than other models, notably in French.

<footnote n="65">⁶⁵Results in French for models with sizes over 7B parameters were extracted from https://huggingface.co/datasets/bofenghuang/mt-bench-french and results in English are from https://huggingface.co/spaces/lmsys/mt-bench/tree/main/data/mt_bench/model_judgment</footnote>
<figure><img src="image_13.png" /><figcaption>Figure 11: MT Bench Results (Turn 1)</figcaption></figure>

# D Results

## D.1 Methodology

Base models are evaluated through the LM Evaluation harness framework (Gao et al., 2021). For classification tasks, we choose the answer with the largest log likelihood when concatenated with the prompt, as is implemented within the framework.

For generative tasks, we simply generate with the default settings, which is greedy sampling. We acknowledge CroissantLLM works best with higher temperature values but did not want to introduce stochasticity to the evaluation. We also limit each benchmark task to 2000 samples at most, to shorten evaluation time. All evaluations are reproducible through the code at https://github.com/EleutherAI/lm-evaluation-harness.

## D.2 MT-Bench

Turn 1 (Figure 11) and Turn 2 (Figure 12) results are shown. We notice small models, struggle with reasoning based tasks and constrained generation imposed by Turn 2 prompts. Figures 14 and 15 compare our results on small language models to other common bigger models.<fnref n="65" />

## D.3 Bias Assessment

We assess bias through CROWS (Nangia et al., 2020), the Crowdsourced Stereotype Pairs benchmark that covers stereotypes dealing with nine types of bias, like race, religion, and age and report results in Table 16. We find CroissantLLM is in line, or slightly less biased than other models, notably in French.

<footnote n="65">⁶⁵Results in French for models with sizes over 7B parameters were extracted from https://huggingface.co/datasets/bofenghuang/mt-bench-french and results in English are from https://huggingface.co/spaces/lmsys/mt-bench/tree/main/data/mt_bench/model_judgment</footnote>
<figure><img src="image_15.png" /><figcaption>Figure 12: MT Bench Results (Turn 2)</figcaption></figure>

<table><caption>Table 14: French MT Bench Results Average of turn 1 and 2 of many supervised finetuned models</caption><thead><tr><th>Models</th><th>Wri</th><th>Ro</th><th>Reas</th><th>Math</th><th>Cod</th><th>Ext</th><th>STEM</th><th>Hum</th><th>Avg</th></tr></thead><tbody><tr><td>CroissantLLMChat</td><td>5.32</td><td>5.35</td><td>1.9</td><td>1.16</td><td>1.8</td><td>1.4</td><td>4.55</td><td>3.75</td><td>3.15</td></tr><tr><td>TinyLlamaChat</td><td>5</td><td>4.1</td><td>1.45</td><td>1</td><td>2.1</td><td>1.6</td><td>3.9</td><td>4.85</td><td>3</td></tr><tr><td>Bloom 1B7 Chat</td><td>4.62</td><td>3.25</td><td>1.2</td><td>1.5</td><td>1.45</td><td>1.5</td><td>2.85</td><td>2.22</td><td>2.32</td></tr><tr><td>CMArkea BloomZ 3B</td><td>2.65</td><td>2.85</td><td>1.85</td><td>1.15</td><td>1.2</td><td>2.3</td><td>3.65</td><td>2.7</td><td>2.29</td></tr><tr><td>Vigostral 7B Chat</td><td>7.7</td><td>7.85</td><td>4.85</td><td>3.65</td><td>4.65</td><td>7.75</td><td>7.35</td><td>9.2</td><td>6.62</td></tr><tr><td>Vigogne 2 7b Chat</td><td>5.35</td><td>6.25</td><td>2.75</td><td>2.2</td><td>2.47</td><td>3.4</td><td>6.05</td><td>6.68</td><td>4.39</td></tr><tr><td>OpenHermes Mistral 7B</td><td>8.8</td><td>7.5</td><td>5.1</td><td>4.05</td><td>5.55</td><td>6.2</td><td>8.35</td><td>9.4</td><td>6.87</td></tr><tr><td>Vigogne 2 70B Chat</td><td>9.4</td><td>8.25</td><td>4.75</td><td>4.3</td><td>5.35</td><td>7.25</td><td>9.1</td><td>9.43</td><td>7.23</td></tr><tr><td>Mixtral 8x7b Instruct</td><td>9.65</td><td>8.88</td><td>6.95</td><td>4.95</td><td>4.6</td><td>8.55</td><td>9.5</td><td>9.6</td><td>7.84</td></tr><tr><td>Mistral Medium</td><td>9.6</td><td>9.05</td><td>5.4</td><td>6.1</td><td>7.35</td><td>9.25</td><td>9.3</td><td>9.75</td><td>8.23</td></tr><tr><td>GPT 3.5 Turbo</td><td>8.75</td><td>8.93</td><td>5.05</td><td>5.65</td><td>7.85</td><td>9.05</td><td>9.05</td><td>9.68</td><td>8</td></tr><tr><td>GPT 4</td><td>9.6</td><td>9.65</td><td>8.55</td><td>8.5</td><td>8.35</td><td>9.2</td><td>9.85</td><td>9.88</td><td>9.2</td></tr></tbody></table>
<figure><img src="image_14.png" /><figcaption>Figure 12: MT Bench Results (Turn 2)</figcaption></figure>

<table><caption>Table 14: French MT Bench Results Average of turn 1 and 2 of many supervised finetuned models</caption><thead><tr><th>Models</th><th>Wri</th><th>Ro</th><th>Reas</th><th>Math</th><th>Cod</th><th>Ext</th><th>STEM</th><th>Hum</th><th>Avg</th></tr></thead><tbody><tr><td>CroissantLLMChat</td><td>5.32</td><td>5.35</td><td>1.9</td><td>1.16</td><td>1.8</td><td>1.4</td><td>4.55</td><td>3.75</td><td>3.15</td></tr><tr><td>TinyLlamaChat</td><td>5</td><td>4.1</td><td>1.45</td><td>1</td><td>2.1</td><td>1.6</td><td>3.9</td><td>4.85</td><td>3</td></tr><tr><td>Bloom 1B7 Chat</td><td>4.62</td><td>3.25</td><td>1.2</td><td>1.5</td><td>1.45</td><td>1.5</td><td>2.85</td><td>2.22</td><td>2.32</td></tr><tr><td>CMArkea BloomZ 3B</td><td>2.65</td><td>2.85</td><td>1.85</td><td>1.15</td><td>1.2</td><td>2.3</td><td>3.65</td><td>2.7</td><td>2.29</td></tr><tr><td>Vigostral 7B Chat</td><td>7.7</td><td>7.85</td><td>4.85</td><td>3.65</td><td>4.65</td><td>7.75</td><td>7.35</td><td>9.2</td><td>6.62</td></tr><tr><td>Vigogne 2 7b Chat</td><td>5.35</td><td>6.25</td><td>2.75</td><td>2.2</td><td>2.47</td><td>3.4</td><td>6.05</td><td>6.68</td><td>4.39</td></tr><tr><td>OpenHermes Mistral 7B</td><td>8.8</td><td>7.5</td><td>5.1</td><td>4.05</td><td>5.55</td><td>6.2</td><td>8.35</td><td>9.4</td><td>6.87</td></tr><tr><td>Vigogne 2 70B Chat</td><td>9.4</td><td>8.25</td><td>4.75</td><td>4.3</td><td>5.35</td><td>7.25</td><td>9.1</td><td>9.43</td><td>7.23</td></tr><tr><td>Mixtral 8x7b Instruct</td><td>9.65</td><td>8.88</td><td>6.95</td><td>4.95</td><td>4.6</td><td>8.55</td><td>9.5</td><td>9.6</td><td>7.84</td></tr><tr><td>Mistral Medium</td><td>9.6</td><td>9.05</td><td>5.4</td><td>6.1</td><td>7.35</td><td>9.25</td><td>9.3</td><td>9.75</td><td>8.23</td></tr><tr><td>GPT 3.5 Turbo</td><td>8.75</td><td>8.93</td><td>5.05</td><td>5.65</td><td>7.85</td><td>9.05</td><td>9.05</td><td>9.68</td><td>8</td></tr><tr><td>GPT 4</td><td>9.6</td><td>9.65</td><td>8.55</td><td>8.5</td><td>8.35</td><td>9.2</td><td>9.85</td><td>9.88</td><td>9.2</td></tr></tbody></table>
<table><caption>Table 15: English MT Bench Results Average of turn 1 and 2 of many supervised finetuned models</caption><thead><tr><th>Models</th><th>Wri</th><th>Ro</th><th>Reas</th><th>Math</th><th>Cod</th><th>Ext</th><th>STEM</th><th>Hum</th><th>Avg</th></tr></thead><tbody><tr><td>CroissantLLMChat</td><td>5.1</td><td>3.6</td><td>2.4</td><td>1.1</td><td>1.8</td><td>1.3</td><td>5</td><td>5.85</td><td>3.27</td></tr><tr><td>TinyLlamaChat</td><td>6.25</td><td>5</td><td>3.3</td><td>1.35</td><td>2.1</td><td>1.5</td><td>4.82</td><td>6.3</td><td>3.83</td></tr><tr><td>Bloom 1B7 Chat</td><td>3.95</td><td>4.55</td><td>1.95</td><td>1.4</td><td>1.45</td><td>1.35</td><td>3.5</td><td>3.45</td><td>2.7</td></tr><tr><td>CMArkea BloomZ 3B</td><td>3.15</td><td>3.3</td><td>2.2</td><td>1.1</td><td>1.25</td><td>1.35</td><td>3.15</td><td>1.9</td><td>2.17</td></tr><tr><td>Llama 2 7B Chat</td><td>8.9</td><td>7.7</td><td>4.25</td><td>2.4</td><td>3</td><td>6.5</td><td>8.65</td><td>8.75</td><td>6.27</td></tr><tr><td>Vicuna 7B v1.3</td><td>8.1</td><td>7.45</td><td>4.65</td><td>2.3</td><td>3.55</td><td>5</td><td>7.82</td><td>9.1</td><td>6</td></tr><tr><td>Llama 2 13B Chat</td><td>8.85</td><td>7.5</td><td>5.1</td><td>3.45</td><td>3</td><td>6.92</td><td>8.62</td><td>9.75</td><td>6.65</td></tr><tr><td>Vicuna 13B v1.3</td><td>9.25</td><td>7.18</td><td>5.85</td><td>2.6</td><td>3.25</td><td>5.55</td><td>7.98</td><td>9.45</td><td>6.39</td></tr><tr><td>Vicuna 33B v1.3</td><td>9.5</td><td>8.45</td><td>6.65</td><td>3.15</td><td>3.35</td><td>7.1</td><td>8.98</td><td>9.8</td><td>7.12</td></tr><tr><td>Llama 2 70B Chat</td><td>9.3</td><td>7.5</td><td>5.8</td><td>3.3</td><td>3.15</td><td>7.25</td><td>8.93</td><td>9.62</td><td>6.86</td></tr><tr><td>GPT 3.5 Turbo</td><td>9.2</td><td>8.4</td><td>5.65</td><td>6.3</td><td>6.9</td><td>8.85</td><td>8.7</td><td>9.55</td><td>7.94</td></tr><tr><td>GPT 4</td><td>9.65</td><td>8.9</td><td>9</td><td>6.8</td><td>8.55</td><td>9.38</td><td>9.7</td><td>9.95</td><td>8.99</td></tr></tbody></table>

<table><caption>Table 16: Code Data mix</caption><thead><tr><th>Task</th><th>Crows(en)</th><th>Crows(Fr)</th><th>Avg</th></tr></thead><tbody><tr><td>mGPT(1.3B)</td><td>3.16</td><td>2.94</td><td>3.05</td></tr><tr><td>Bloom(3B)</td><td>3.39</td><td>3.02</td><td>3.21</td></tr><tr><td>Bloom(1.1B)</td><td>3.36</td><td>3.07</td><td>3.22</td></tr><tr><td><strong>CroissantLLM</strong></td><td>3.56</td><td>3.22</td><td>3.39</td></tr><tr><td>Pythia(1.4b)</td><td>3.36</td><td>3.62</td><td>3.49</td></tr><tr><td>OPT(1.3b)</td><td>3.35</td><td>3.67</td><td>3.51</td></tr><tr><td>TinyLlama(1.1B)</td><td>3.48</td><td>3.76</td><td>3.62</td></tr><tr><td>GPT-fr(1B)</td><td>4.50</td><td>2.97</td><td>3.73</td></tr><tr><td>Llama2(7B)</td><td>3.72</td><td>3.81</td><td>3.76</td></tr></tbody></table>
<table><caption>Table 15: English MT Bench Results Average of turn 1 and 2 of many supervised finetuned models</caption><thead><tr><th>Models</th><th>Wri</th><th>Ro</th><th>Reas</th><th>Math</th><th>Cod</th><th>Ext</th><th>STEM</th><th>Hum</th><th>Avg</th></tr></thead><tbody><tr><td>CroissantLLMChat</td><td>5.1</td><td>3.6</td><td>2.4</td><td>1.1</td><td>1.8</td><td>1.3</td><td>5</td><td>5.85</td><td>3.27</td></tr><tr><td>TinyLlamaChat</td><td>6.25</td><td>5</td><td>3.3</td><td>1.35</td><td>2.1</td><td>1.5</td><td>4.82</td><td>6.3</td><td>3.83</td></tr><tr><td>Bloom 1B7 Chat</td><td>3.95</td><td>4.55</td><td>1.95</td><td>1.4</td><td>1.45</td><td>1.35</td><td>3.5</td><td>3.45</td><td>2.7</td></tr><tr><td>CMArkea BloomZ 3B</td><td>3.15</td><td>3.3</td><td>2.2</td><td>1.1</td><td>1.25</td><td>1.35</td><td>3.15</td><td>1.9</td><td>2.17</td></tr><tr><td>Llama 2 7B Chat</td><td>8.9</td><td>7.7</td><td>4.25</td><td>2.4</td><td>3</td><td>6.5</td><td>8.65</td><td>8.75</td><td>6.27</td></tr><tr><td>Vicuna 7B v1.3</td><td>8.1</td><td>7.45</td><td>4.65</td><td>2.3</td><td>3.55</td><td>5</td><td>7.82</td><td>9.1</td><td>6</td></tr><tr><td>Llama 2 13B Chat</td><td>8.85</td><td>7.5</td><td>5.1</td><td>3.45</td><td>3</td><td>6.92</td><td>8.62</td><td>9.75</td><td>6.65</td></tr><tr><td>Vicuna 13B v1.3</td><td>9.25</td><td>7.18</td><td>5.85</td><td>2.6</td><td>3.25</td><td>5.55</td><td>7.98</td><td>9.45</td><td>6.39</td></tr><tr><td>Vicuna 33B v1.3</td><td>9.5</td><td>8.45</td><td>6.65</td><td>3.15</td><td>3.35</td><td>7.1</td><td>8.98</td><td>9.8</td><td>7.12</td></tr><tr><td>Llama 2 70B Chat</td><td>9.3</td><td>7.5</td><td>5.8</td><td>3.3</td><td>3.15</td><td>7.25</td><td>8.93</td><td>9.62</td><td>6.86</td></tr><tr><td>GPT 3.5 Turbo</td><td>9.2</td><td>8.4</td><td>5.65</td><td>6.3</td><td>6.9</td><td>8.85</td><td>8.7</td><td>9.55</td><td>7.94</td></tr><tr><td>GPT 4</td><td>9.65</td><td>8.9</td><td>9</td><td>6.8</td><td>8.55</td><td>9.38</td><td>9.7</td><td>9.95</td><td>8.99</td></tr></tbody></table>

<table><caption>Table 16: Code Data mix</caption><thead><tr><th>Task</th><th>Crows(en)</th><th>Crows(Fr)</th><th>Avg</th></tr></thead><tbody><tr><td>mGPT(1.3B)</td><td>3.16</td><td>2.94</td><td>3.05</td></tr><tr><td>Bloom(3B)</td><td>3.39</td><td>3.02</td><td>3.21</td></tr><tr><td>Bloom(1.1B)</td><td>3.36</td><td>3.07</td><td>3.22</td></tr><tr><td><strong>CroissantLLM</strong></td><td>3.56</td><td>3.22</td><td>3.39</td></tr><tr><td>Pythia(1.4b)</td><td>3.36</td><td>3.62</td><td>3.49</td></tr><tr><td>OPT(1.3b)</td><td>3.35</td><td>3.67</td><td>3.51</td></tr><tr><td>TinyLlama(1.1B)</td><td>3.48</td><td>3.76</td><td>3.62</td></tr><tr><td>GPT-fr(1B)</td><td>4.50</td><td>2.97</td><td>3.73</td></tr><tr><td>Llama2(7B)</td><td>3.72</td><td>3.81</td><td>3.76</td></tr></tbody></table>