The Llama 3 Herd of Models
Authors: name: Abhimanyu Dubey; name: Abhinav Jauhri; name: Abhinav Pandey; name: Abhishek Kadian; name: Ahmad Al-Dahle; name: Aiesha Letman; name: Akhil Mathur; name: Alan Schelten; name: Amy Yang; name: Angela Fan; name: Anirudh Goyal; name: Anthony Hartshorn; name: Aobo Yang; name: Archi Mitra; name: Archie Sravankumar; name: Artem Korenev; name: Arthur Hinsvark; name: Arun Rao; name: Aston Zhang; name: Aurelien Rodriguez; name: Austen Gregerson; name: Ava Spataru; name: Baptiste Roziere; name: Bethany Biron; name: Binh Tang; name: Bobbie Chern; name: Charlotte Caucheteux; name: Chaya Nayak; name: Chloe Bi; name: Chris Marra; name: Chris McConnell; name: Christian Keller; name: Christophe Touret; name: Chunyang Wu; name: Corinne Wong; name: Cristian Canton Ferrer; name: Cyrus Nikolaidis; name: Damien Allonsius; name: Daniel Song; name: Danielle Pintz; name: Danny Livshits; name: David Esiobu; name: Dhruv Choudhary; name: Dhruv Mahajan; name: Diego Garcia-Olano; name: Diego Perino; name: Dieuwke Hupkes; name: Egor Lakomkin; name: Ehab AlBadawy; name: Elina Lobanova; name: Emily Dinan; name: Eric Michael Smith; name: Filip Radenovic; name: Frank Zhang; name: Gabriel Synnaeve; name: Gabrielle Lee; name: Georgia Lewis Anderson; name: Graeme Nail; name: Gregoire Mialon; name: Guan Pang; name: Guillem Cucurell; name: Hailey Nguyen; name: Hannah Korevaar; name: Hu Xu; name: Hugo Touvron; name: Iliyan Zarov; name: Imanol Arrieta Ibarra; name: Isabel Kloumann; name: Ishan Misra; name: Ivan Evtimov; name: Jade Copet; name: Jaewon Lee; name: Jan Geffert; name: Jana Vranes; name: Jason Park; name: Jay Mahadeokar; name: Jeet Shah; name: Jelmer van der Linde; name: Jennifer Billock; name: Jenny Hong; name: Jenya Lee; name: Jeremy Fu; name: Jianfeng Chi; name: Jianyu Huang; name: Jiawen Liu; name: Jie Wang; name: Jiecao Yu; name: Joanna Bitton; name: Joe Spisak; name: Jongsoo Park; name: Joseph Rocca; name: Joshua Johnstun; name: Joshua Saxe; name: Junteng Jia; name: Kalyan Vasuden Alwala; name: Kartikeya Upasani; name: Kate Plawiak; name: Ke Li; name: Kenneth Heafield; name: Kevin Stone; name: Khalid El-Arini; name: Krithika Iyer; name: Kshitiz Malik; name: Kuenley Chiu; name: Kunal Bhalla; name: Lauren Rantala-Yeary; name: Laurens van der Maaten; name: Lawrence Chen; name: Liang Tan; name: Liz Jenkins; name: Louis Martin; name: Lovish Madaan; name: Lubo Malo; name: Lukas Blecher; name: Lukas Landzaat; name: Luke de Oliveira; name: Madeline Muzzi; name: Mahesh Pasupuleti; name: Mannat Singh; name: Manohar Paluri; name: Marcin Kardas; name: Mathew Oldham; name: Mathieu Rita; name: Maya Pavlova; name: Melanie Kambadur; name: Mike Lewis; name: Min Si; name: Mitesh Kumar Singh; name: Mona Hassan; name: Naman Goyal; name: Narjes Torabi; name: Nikolay Bashlykov; name: Nikolay Bogoychev; name: Niladri Chatterji; name: Olivier Duchenne; name: Onur Çelebi; name: Patrick Alrassy; name: Pengchuan Zhang; name: Pengwei Li; name: Petar Vasic; name: Peter Weng; name: Prajjwal Bhargava; name: Pratik Dubal; name: Praveen Krishnan; name: Punit Singh Koura; name: Puxin Xu; name: Qing He; name: Qingxiao Dong; name: Ragavan Srinivasan; name: Raj Ganapathy; name: Ramon Calderer; name: Ricardo Silveira Cabral; name: Robert Stojnic; name: Roberta Raileanu; name: Rohit Girdhar; name: Rohit Patel; name: Romain Sauvestre; name: Ronnie Polidoro; name: Roshan Sumbaly; name: Ross Taylor; name: Ruan Silva; name: Rui Hou; name: Rui Wang; name: Saghar Hosseini; name: Sahana Chennabasappa; name: Sanjay Singh; name: Sean Bell; name: Seohyun Sonia Kim; name: Sergey Edunov; name: Shaoliang Nie; name: Sharan Narang; name: Sharath Raparthy; name: Sheng Shen; name: Shengye Wan; name: Shruti Bhosale; name: Shun Zhang; name: Simon Vandenhende; name: Soumya Batra; name: Spencer Whitman; name: Sten Sootla; name: Stephane Collot; name: Suchin Gururangan; name: Sydney Borodinsky; name: Tamar Herman; name: Tara Fowler; name: Tarek Sheasha; name: Thomas Georgiou; name: Thomas Scialom; name: Tobias Speckbacher; name: Todor Mihaylov; name: Tong Xiao; name: Ujjwal Karn; name: Vedanuj Goswami; name: Vibhor Gupta; name: Vignesh Ramanathan; name: Viktor Kerkez; name: Vincent Gonguet; name: Virginie Do; name: Vish Vogeti; name: Vladan Petrovic; name: Weiwei Chu; name: Wenhan Xiong; name: Wenyin Fu; name: Whitney Meers; name: Xavier Martinet; name: Xiaodong Wang; name: Xiaoqing Ellen Tan; name: Xinfeng Xie; name: Xuchao Jia; name: Xuewei Wang; name: Yaelle Goldschlag; name: Yashesh Gaur; name: Yasmine Babaei; name: Yi Wen; name: Yiwen Song; name: Yuchen Zhang; name: Yue Li; name: Yuning Mao; name: Zacharie Delpierre Coudert; name: Zheng Yan; name: Zhengxing Chen; name: Zoe Papakipos; name: Aaditya Singh; name: Aaron Grattafiori; name: Abha Jain; name: Adam Kelsey; name: Adam Shajnfeld; name: Adithya Gangidi; name: Adolfo Victoria; name: Ahuva Goldstand; name: Ajay Menon; name: Ajay Sharma; name: Alex Boesenberg; name: Alex Vaughan; name: Alexei Baevski; name: Allie Feinstein; name: Amanda Kallet; name: Amit Sangani; name: Anam Yunus; name: Andrei Lupu; name: Andres Alvarado; name: Andrew Caples; name: Andrew Gu; name: Andrew Ho; name: Andrew Poulton; name: Andrew Ryan; name: Ankit Ramchandani; name: Annie Franco; name: Aparajita Saraf; name: Arkabandhu Chowdhury; name: Ashley Gabriel; name: Ashwin Bharambe; name: Assaf Eisenman; name: Azadeh Yazdan; name: Beau James; name: Ben Maurer; name: Benjamin Leonhardi; name: Bernie Huang; name: Beth Loyd; name: Beto De Paola; name: Bhargavi Paranjape; name: Bing Liu; name: Bo Wu; name: Boyu Ni; name: Braden Hancock; name: Bram Wasti; name: Brandon Spence; name: Brani Stojkovic; name: Brian Gamido; name: Britt Montalvo; name: Carl Parker; name: Carly Burton; name: Catalina Mejia; name: Changhan Wang; name: Changkyu Kim; name: Chao Zhou; name: Chester Hu; name: Ching-Hsiang Chu; name: Chris Cai; name: Chris Tindal; name: Christoph Feichtenhofer; name: Damon Civin; name: Dana Beaty; name: Daniel Kreymer; name: Daniel Li; name: Danny Wyatt; name: David Adkins; name: David Xu; name: Davide Testuggine; name: Delia David; name: Devi Parikh; name: Diana Liskovich; name: Didem Foss; name: Dingkang Wang; name: Duc Le; name: Dustin Holland; name: Edward Dowling; name: Eissa Jamil; name: Elaine Montgomery; name: Eleonora Presani; name: Emily Hahn; name: Emily Wood; name: Erik Brinkman; name: Esteban Arcaute; name: Evan Dunbar; name: Evan Smothers; name: Fei Sun; name: Felix Kreuk; name: Feng Tian; name: Firat Ozgenel; name: Francesco Caggioni; name: Francisco Guzmán; name: Frank Kanayet; name: Frank Seide; name: Gabriela Medina Florez; name: Gabriella Schwarz; name: Gada Badeer; name: Georgia Swee; name: Gil Halpern; name: Govind Thattai; name: Grant Herman; name: Grigory Sizov; name: Guangyi; name: Zhang; name: Guna Lakshminarayanan; name: Hamid Shojanazeri; name: Han Zou; name: Hannah Wang; name: Hanwen Zha; name: Haroun Habeeb; name: Harrison Rudolph; name: Helen Suk; name: Henry Aspegren; name: Hunter Goldman; name: Igor Molybog; name: Igor Tufanov; name: Irina-Elena Veliche; name: Itai Gat; name: Jake Weissman; name: James Geboski; name: James Kohli; name: Japhet Asher; name: Jean-Baptiste Gaya; name: Jeff Marcus; name: Jeff Tang; name: Jennifer Chan; name: Jenny Zhen; name: Jeremy Reizenstein; name: Jeremy Teboul; name: Jessica Zhong; name: Jian Jin; name: Jingyi Yang; name: Joe Cummings; name: Jon Carvill; name: Jon Shepard; name: Jonathan McPhie; name: Jonathan Torres; name: Josh Ginsburg; name: Junjie Wang; name: Kai Wu; name: Kam Hou U; name: Karan Saxena; name: Karthik Prasad; name: Kartikay Khandelwal; name: Katayoun Zand; name: Kathy Matosich; name: Kaushik Veeraraghavan; name: Kelly Michelena; name: Keqian Li; name: Kun Huang; name: Kunal Chawla; name: Kushal Lakhotia; name: Kyle Huang; name: Lailin Chen; name: Lakshya Garg; name: Lavender A; name: Leandro Silva; name: Lee Bell; name: Lei Zhang; name: Liangpeng Guo; name: Licheng Yu; name: Liron Moshkovich; name: Luca Wehrstedt; name: Madian Khabsa; name: Manav Avalani; name: Manish Bhatt; name: Maria Tsimpoukelli; name: Martynas Mankus; name: Matan Hasson; name: Matthew Lennie; name: Matthias Reso; name: Maxim Groshev; name: Maxim Naumov; name: Maya Lathi; name: Meghan Keneally; name: Michael L. Seltzer; name: Michal Valko; name: Michelle Restrepo; name: Mihir Patel; name: Mik Vyatskov; name: Mikayel Samvelyan; name: Mike Clark; name: Mike Macey; name: Mike Wang; name: Miquel Jubert Hermoso; name: Mo Metanat; name: Mohammad Rastegari; name: Munish Bansal; name: Nandhini Santhanam; name: Natascha Parks; name: Natasha White; name: Navyata Bawa; name: Nayan Singhal; name: Nick Egebo; name: Nicolas Usunier; name: Nikolay Pavlovich Laptev; name: Ning Dong; name: Ning Zhang; name: Norman Cheng; name: Oleg Chernoguz; name: Olivia Hart; name: Omkar Salpekar; name: Ozlem Kalinli; name: Parkin Kent; name: Parth Parekh; name: Paul Saab; name: Pavan Balaji; name: Pedro Rittner; name: Philip Bontrager; name: Pierre Roux; name: Piotr Dollar; name: Polina Zvyagina; name: Prashant Ratanchandani; name: Pritish Yuvraj; name: Qian Liang; name: Rachad Alao; name: Rachel Rodriguez; name: Rafi Ayub; name: Raghotham Murthy; name: Raghu Nayani; name: Rahul Mitra; name: Raymond Li; name: Rebekkah Hogan; name: Robin Battey; name: Rocky Wang; name: Rohan Maheswari; name: Russ Howes; name: Ruty Rinott; name: Sai Jayesh Bondu; name: Samyak Datta; name: Sara Chugh; name: Sara Hunt; name: Sargun Dhillon; name: Sasha Sidorov; name: Satadru Pan; name: Saurabh Verma; name: Seiji Yamamoto; name: Sharadh Ramaswamy; name: Shaun Lindsay; name: Shaun Lindsay; name: Sheng Feng; name: Shenghao Lin; name: Shengxin Cindy Zha; name: Shiva Shankar; name: Shuqiang Zhang; name: Shuqiang Zhang; name: Sinong Wang; name: Sneha Agarwal; name: Soji Sajuyigbe; name: Soumith Chintala; name: Stephanie Max; name: Stephen Chen; name: Steve Kehoe; name: Steve Satterfield; name: Sudarshan Govindaprasad; name: Sumit Gupta; name: Sungmin Cho; name: Sunny Virk; name: Suraj Subramanian; name: Sy Choudhury; name: Sydney Goldman; name: Tal Remez; name: Tamar Glaser; name: Tamara Best; name: Thilo Kohler; name: Thomas Robinson; name: Tianhe Li; name: Tianjun Zhang; name: Tim Matthews; name: Timothy Chou; name: Tzook Shaked; name: Varun Vontimitta; name: Victoria Ajayi; name: Victoria Montanez; name: Vijai Mohan; name: Vinay Satish Kumar; name: Vishal Mangla; name: Vlad Ionescu; name: Vlad Poenaru; name: Vlad Tiberiu Mihailescu; name: Vladimir Ivanov; name: Wei Li; name: Wenchen Wang; name: Wenwen Jiang; name: Wes Bouaziz; name: Will Constable; name: Xiaocheng Tang; name: Xiaofang Wang; name: Xiaojian Wu; name: Xiaolan Wang; name: Xide Xia; name: Xilun Wu; name: Xinbo Gao; name: Yanjun Chen; name: Ye Hu; name: Ye Jia; name: Ye Qi; name: Yenda Li; name: Yilin Zhang; name: Ying Zhang; name: Yossi Adi; name: Youngjin Nam; name: Yu; name: Wang; name: Yuchen Hao; name: Yundi Qian; name: Yuzi He; name: Zach Rait; name: Zachary DeVito; name: Zef Rosnbrick; name: Zhaoduo Wen; name: Zhenyu Yang; name: Zhiwei Zhao
Published: 2024-07-31 17:54:27

Abstract: Modern artificial intelligence (AI) systems are powered by foundation models.
This paper presents a new set of foundation models, called Llama 3. It is a
herd of language models that natively support multilinguality, coding,
reasoning, and tool usage. Our largest model is a dense Transformer with 405B
parameters and a context window of up to 128K tokens. This paper presents an
extensive empirical evaluation of Llama 3. We find that Llama 3 delivers
comparable quality to leading language models such as GPT-4 on a plethora of
tasks. We publicly release Llama 3, including pre-trained and post-trained
versions of the 405B parameter language model and our Llama Guard 3 model for
input and output safety. The paper also presents the results of experiments in
which we integrate image, video, and speech capabilities into Llama 3 via a
compositional approach. We observe this approach performs competitively with
the state-of-the-art on image, video, and speech recognition tasks. The
resulting models are not yet being broadly released as they are still under
development.