The Llama 3 Herd of Models
Authors: name: Abhimanyu Dubey, name: Abhinav Jauhri, name: Abhinav Pandey, name: Abhishek Kadian, name: Ahmad Al-Dahle, name: Aiesha Letman, name: Akhil Mathur, name: Alan Schelten, name: Amy Yang, name: Angela Fan, name: Anirudh Goyal, name: Anthony Hartshorn, name: Aobo Yang, name: Archi Mitra, name: Archie Sravankumar, name: Artem Korenev, name: Arthur Hinsvark, name: Arun Rao, name: Aston Zhang, name: Aurelien Rodriguez, name: Austen Gregerson, name: Ava Spataru, name: Baptiste Roziere, name: Bethany Biron, name: Binh Tang, name: Bobbie Chern, name: Charlotte Caucheteux, name: Chaya Nayak, name: Chloe Bi, name: Chris Marra, name: Chris McConnell, name: Christian Keller, name: Christophe Touret, name: Chunyang Wu, name: Corinne Wong, name: Cristian Canton Ferrer, name: Cyrus Nikolaidis, name: Damien Allonsius, name: Daniel Song, name: Danielle Pintz, name: Danny Livshits, name: David Esiobu, name: Dhruv Choudhary, name: Dhruv Mahajan, name: Diego Garcia-Olano, name: Diego Perino, name: Dieuwke Hupkes, name: Egor Lakomkin, name: Ehab AlBadawy, name: Elina Lobanova, name: Emily Dinan, name: Eric Michael Smith, name: Filip Radenovic, name: Frank Zhang, name: Gabriel Synnaeve, name: Gabrielle Lee, name: Georgia Lewis Anderson, name: Graeme Nail, name: Gregoire Mialon, name: Guan Pang, name: Guillem Cucurell, name: Hailey Nguyen, name: Hannah Korevaar, name: Hu Xu, name: Hugo Touvron, name: Iliyan Zarov, name: Imanol Arrieta Ibarra, name: Isabel Kloumann, name: Ishan Misra, name: Ivan Evtimov, name: Jade Copet, name: Jaewon Lee, name: Jan Geffert, name: Jana Vranes, name: Jason Park, name: Jay Mahadeokar, name: Jeet Shah, name: Jelmer van der Linde, name: Jennifer Billock, name: Jenny Hong, name: Jenya Lee, name: Jeremy Fu, name: Jianfeng Chi, name: Jianyu Huang, name: Jiawen Liu, name: Jie Wang, name: Jiecao Yu, name: Joanna Bitton, name: Joe Spisak, name: Jongsoo Park, name: Joseph Rocca, name: Joshua Johnstun, name: Joshua Saxe, name: Junteng Jia, name: Kalyan Vasuden Alwala, name: Kartikeya Upasani, name: Kate Plawiak, name: Ke Li, name: Kenneth Heafield, name: Kevin Stone, name: Khalid El-Arini, name: Krithika Iyer, name: Kshitiz Malik, name: Kuenley Chiu, name: Kunal Bhalla, name: Lauren Rantala-Yeary, name: Laurens van der Maaten, name: Lawrence Chen, name: Liang Tan, name: Liz Jenkins, name: Louis Martin, name: Lovish Madaan, name: Lubo Malo, name: Lukas Blecher, name: Lukas Landzaat, name: Luke de Oliveira, name: Madeline Muzzi, name: Mahesh Pasupuleti, name: Mannat Singh, name: Manohar Paluri, name: Marcin Kardas, name: Mathew Oldham, name: Mathieu Rita, name: Maya Pavlova, name: Melanie Kambadur, name: Mike Lewis, name: Min Si, name: Mitesh Kumar Singh, name: Mona Hassan, name: Naman Goyal, name: Narjes Torabi, name: Nikolay Bashlykov, name: Nikolay Bogoychev, name: Niladri Chatterji, name: Olivier Duchenne, name: Onur Çelebi, name: Patrick Alrassy, name: Pengchuan Zhang, name: Pengwei Li, name: Petar Vasic, name: Peter Weng, name: Prajjwal Bhargava, name: Pratik Dubal, name: Praveen Krishnan, name: Punit Singh Koura, name: Puxin Xu, name: Qing He, name: Qingxiao Dong, name: Ragavan Srinivasan, name: Raj Ganapathy, name: Ramon Calderer, name: Ricardo Silveira Cabral, name: Robert Stojnic, name: Roberta Raileanu, name: Rohit Girdhar, name: Rohit Patel, name: Romain Sauvestre, name: Ronnie Polidoro, name: Roshan Sumbaly, name: Ross Taylor, name: Ruan Silva, name: Rui Hou, name: Rui Wang, name: Saghar Hosseini, name: Sahana Chennabasappa, name: Sanjay Singh, name: Sean Bell, name: Seohyun Sonia Kim, name: Sergey Edunov, name: Shaoliang Nie, name: Sharan Narang, name: Sharath Raparthy, name: Sheng Shen, name: Shengye Wan, name: Shruti Bhosale, name: Shun Zhang, name: Simon Vandenhende, name: Soumya Batra, name: Spencer Whitman, name: Sten Sootla, name: Stephane Collot, name: Suchin Gururangan, name: Sydney Borodinsky, name: Tamar Herman, name: Tara Fowler, name: Tarek Sheasha, name: Thomas Georgiou, name: Thomas Scialom, name: Tobias Speckbacher, name: Todor Mihaylov, name: Tong Xiao, name: Ujjwal Karn, name: Vedanuj Goswami, name: Vibhor Gupta, name: Vignesh Ramanathan, name: Viktor Kerkez, name: Vincent Gonguet, name: Virginie Do, name: Vish Vogeti, name: Vladan Petrovic, name: Weiwei Chu, name: Wenhan Xiong, name: Wenyin Fu, name: Whitney Meers, name: Xavier Martinet, name: Xiaodong Wang, name: Xiaoqing Ellen Tan, name: Xinfeng Xie, name: Xuchao Jia, name: Xuewei Wang, name: Yaelle Goldschlag, name: Yashesh Gaur, name: Yasmine Babaei, name: Yi Wen, name: Yiwen Song, name: Yuchen Zhang, name: Yue Li, name: Yuning Mao, name: Zacharie Delpierre Coudert, name: Zheng Yan, name: Zhengxing Chen, name: Zoe Papakipos, name: Aaditya Singh, name: Aaron Grattafiori, name: Abha Jain, name: Adam Kelsey, name: Adam Shajnfeld, name: Adithya Gangidi, name: Adolfo Victoria, name: Ahuva Goldstand, name: Ajay Menon, name: Ajay Sharma, name: Alex Boesenberg, name: Alex Vaughan, name: Alexei Baevski, name: Allie Feinstein, name: Amanda Kallet, name: Amit Sangani, name: Anam Yunus, name: Andrei Lupu, name: Andres Alvarado, name: Andrew Caples, name: Andrew Gu, name: Andrew Ho, name: Andrew Poulton, name: Andrew Ryan, name: Ankit Ramchandani, name: Annie Franco, name: Aparajita Saraf, name: Arkabandhu Chowdhury, name: Ashley Gabriel, name: Ashwin Bharambe, name: Assaf Eisenman, name: Azadeh Yazdan, name: Beau James, name: Ben Maurer, name: Benjamin Leonhardi, name: Bernie Huang, name: Beth Loyd, name: Beto De Paola, name: Bhargavi Paranjape, name: Bing Liu, name: Bo Wu, name: Boyu Ni, name: Braden Hancock, name: Bram Wasti, name: Brandon Spence, name: Brani Stojkovic, name: Brian Gamido, name: Britt Montalvo, name: Carl Parker, name: Carly Burton, name: Catalina Mejia, name: Changhan Wang, name: Changkyu Kim, name: Chao Zhou, name: Chester Hu, name: Ching-Hsiang Chu, name: Chris Cai, name: Chris Tindal, name: Christoph Feichtenhofer, name: Damon Civin, name: Dana Beaty, name: Daniel Kreymer, name: Daniel Li, name: Danny Wyatt, name: David Adkins, name: David Xu, name: Davide Testuggine, name: Delia David, name: Devi Parikh, name: Diana Liskovich, name: Didem Foss, name: Dingkang Wang, name: Duc Le, name: Dustin Holland, name: Edward Dowling, name: Eissa Jamil, name: Elaine Montgomery, name: Eleonora Presani, name: Emily Hahn, name: Emily Wood, name: Erik Brinkman, name: Esteban Arcaute, name: Evan Dunbar, name: Evan Smothers, name: Fei Sun, name: Felix Kreuk, name: Feng Tian, name: Firat Ozgenel, name: Francesco Caggioni, name: Francisco Guzmán, name: Frank Kanayet, name: Frank Seide, name: Gabriela Medina Florez, name: Gabriella Schwarz, name: Gada Badeer, name: Georgia Swee, name: Gil Halpern, name: Govind Thattai, name: Grant Herman, name: Grigory Sizov, name: Guangyi, name: Zhang, name: Guna Lakshminarayanan, name: Hamid Shojanazeri, name: Han Zou, name: Hannah Wang, name: Hanwen Zha, name: Haroun Habeeb, name: Harrison Rudolph, name: Helen Suk, name: Henry Aspegren, name: Hunter Goldman, name: Igor Molybog, name: Igor Tufanov, name: Irina-Elena Veliche, name: Itai Gat, name: Jake Weissman, name: James Geboski, name: James Kohli, name: Japhet Asher, name: Jean-Baptiste Gaya, name: Jeff Marcus, name: Jeff Tang, name: Jennifer Chan, name: Jenny Zhen, name: Jeremy Reizenstein, name: Jeremy Teboul, name: Jessica Zhong, name: Jian Jin, name: Jingyi Yang, name: Joe Cummings, name: Jon Carvill, name: Jon Shepard, name: Jonathan McPhie, name: Jonathan Torres, name: Josh Ginsburg, name: Junjie Wang, name: Kai Wu, name: Kam Hou U, name: Karan Saxena, name: Karthik Prasad, name: Kartikay Khandelwal, name: Katayoun Zand, name: Kathy Matosich, name: Kaushik Veeraraghavan, name: Kelly Michelena, name: Keqian Li, name: Kun Huang, name: Kunal Chawla, name: Kushal Lakhotia, name: Kyle Huang, name: Lailin Chen, name: Lakshya Garg, name: Lavender A, name: Leandro Silva, name: Lee Bell, name: Lei Zhang, name: Liangpeng Guo, name: Licheng Yu, name: Liron Moshkovich, name: Luca Wehrstedt, name: Madian Khabsa, name: Manav Avalani, name: Manish Bhatt, name: Maria Tsimpoukelli, name: Martynas Mankus, name: Matan Hasson, name: Matthew Lennie, name: Matthias Reso, name: Maxim Groshev, name: Maxim Naumov, name: Maya Lathi, name: Meghan Keneally, name: Michael L. Seltzer, name: Michal Valko, name: Michelle Restrepo, name: Mihir Patel, name: Mik Vyatskov, name: Mikayel Samvelyan, name: Mike Clark, name: Mike Macey, name: Mike Wang, name: Miquel Jubert Hermoso, name: Mo Metanat, name: Mohammad Rastegari, name: Munish Bansal, name: Nandhini Santhanam, name: Natascha Parks, name: Natasha White, name: Navyata Bawa, name: Nayan Singhal, name: Nick Egebo, name: Nicolas Usunier, name: Nikolay Pavlovich Laptev, name: Ning Dong, name: Ning Zhang, name: Norman Cheng, name: Oleg Chernoguz, name: Olivia Hart, name: Omkar Salpekar, name: Ozlem Kalinli, name: Parkin Kent, name: Parth Parekh, name: Paul Saab, name: Pavan Balaji, name: Pedro Rittner, name: Philip Bontrager, name: Pierre Roux, name: Piotr Dollar, name: Polina Zvyagina, name: Prashant Ratanchandani, name: Pritish Yuvraj, name: Qian Liang, name: Rachad Alao, name: Rachel Rodriguez, name: Rafi Ayub, name: Raghotham Murthy, name: Raghu Nayani, name: Rahul Mitra, name: Raymond Li, name: Rebekkah Hogan, name: Robin Battey, name: Rocky Wang, name: Rohan Maheswari, name: Russ Howes, name: Ruty Rinott, name: Sai Jayesh Bondu, name: Samyak Datta, name: Sara Chugh, name: Sara Hunt, name: Sargun Dhillon, name: Sasha Sidorov, name: Satadru Pan, name: Saurabh Verma, name: Seiji Yamamoto, name: Sharadh Ramaswamy, name: Shaun Lindsay, name: Shaun Lindsay, name: Sheng Feng, name: Shenghao Lin, name: Shengxin Cindy Zha, name: Shiva Shankar, name: Shuqiang Zhang, name: Shuqiang Zhang, name: Sinong Wang, name: Sneha Agarwal, name: Soji Sajuyigbe, name: Soumith Chintala, name: Stephanie Max, name: Stephen Chen, name: Steve Kehoe, name: Steve Satterfield, name: Sudarshan Govindaprasad, name: Sumit Gupta, name: Sungmin Cho, name: Sunny Virk, name: Suraj Subramanian, name: Sy Choudhury, name: Sydney Goldman, name: Tal Remez, name: Tamar Glaser, name: Tamara Best, name: Thilo Kohler, name: Thomas Robinson, name: Tianhe Li, name: Tianjun Zhang, name: Tim Matthews, name: Timothy Chou, name: Tzook Shaked, name: Varun Vontimitta, name: Victoria Ajayi, name: Victoria Montanez, name: Vijai Mohan, name: Vinay Satish Kumar, name: Vishal Mangla, name: Vlad Ionescu, name: Vlad Poenaru, name: Vlad Tiberiu Mihailescu, name: Vladimir Ivanov, name: Wei Li, name: Wenchen Wang, name: Wenwen Jiang, name: Wes Bouaziz, name: Will Constable, name: Xiaocheng Tang, name: Xiaofang Wang, name: Xiaojian Wu, name: Xiaolan Wang, name: Xide Xia, name: Xilun Wu, name: Xinbo Gao, name: Yanjun Chen, name: Ye Hu, name: Ye Jia, name: Ye Qi, name: Yenda Li, name: Yilin Zhang, name: Ying Zhang, name: Yossi Adi, name: Youngjin Nam, name: Yu, name: Wang, name: Yuchen Hao, name: Yundi Qian, name: Yuzi He, name: Zach Rait, name: Zachary DeVito, name: Zef Rosnbrick, name: Zhaoduo Wen, name: Zhenyu Yang, name: Zhiwei Zhao
Published: 2024-07-31 17:54:27

Modern artificial intelligence (AI) systems are powered by foundation models.
This paper presents a new set of foundation models, called Llama 3. It is a
herd of language models that natively support multilinguality, coding,
reasoning, and tool usage. Our largest model is a dense Transformer with 405B
parameters and a context window of up to 128K tokens. This paper presents an
extensive empirical evaluation of Llama 3. We find that Llama 3 delivers
comparable quality to leading language models such as GPT-4 on a plethora of
tasks. We publicly release Llama 3, including pre-trained and post-trained
versions of the 405B parameter language model and our Llama Guard 3 model for
input and output safety. The paper also presents the results of experiments in
which we integrate image, video, and speech capabilities into Llama 3 via a
compositional approach. We observe this approach performs competitively with
the state-of-the-art on image, video, and speech recognition tasks. The
resulting models are not yet being broadly released as they are still under
development.