Gemini 1.5: Unlocking multimodal understanding across millions of tokens
  of context
Authors: name: Machel Reid, name: Nikolay Savinov, name: Denis Teplyashin, name: Dmitry Lepikhin, name: Timothy Lillicrap, name: Jean-baptiste Alayrac, name: Radu Soricut, name: Angeliki Lazaridou, name: Orhan Firat, name: Julian Schrittwieser, name: Ioannis Antonoglou, name: Rohan Anil, name: Sebastian Borgeaud, name: Andrew Dai, name: Katie Millican, name: Ethan Dyer, name: Mia Glaese, name: Thibault Sottiaux, name: Benjamin Lee, name: Fabio Viola, name: Malcolm Reynolds, name: Yuanzhong Xu, name: James Molloy, name: Jilin Chen, name: Michael Isard, name: Paul Barham, name: Tom Hennigan, name: Ross McIlroy, name: Melvin Johnson, name: Johan Schalkwyk, name: Eli Collins, name: Eliza Rutherford, name: Erica Moreira, name: Kareem Ayoub, name: Megha Goel, name: Clemens Meyer, name: Gregory Thornton, name: Zhen Yang, name: Henryk Michalewski, name: Zaheer Abbas, name: Nathan Schucher, name: Ankesh Anand, name: Richard Ives, name: James Keeling, name: Karel Lenc, name: Salem Haykal, name: Siamak Shakeri, name: Pranav Shyam, name: Aakanksha Chowdhery, name: Roman Ring, name: Stephen Spencer, name: Eren Sezener, name: Luke Vilnis, name: Oscar Chang, name: Nobuyuki Morioka, name: George Tucker, name: Ce Zheng, name: Oliver Woodman, name: Nithya Attaluri, name: Tomas Kocisky, name: Evgenii Eltyshev, name: Xi Chen, name: Timothy Chung, name: Vittorio Selo, name: Siddhartha Brahma, name: Petko Georgiev, name: Ambrose Slone, name: Zhenkai Zhu, name: James Lottes, name: Siyuan Qiao, name: Ben Caine, name: Sebastian Riedel, name: Alex Tomala, name: Martin Chadwick, name: Juliette Love, name: Peter Choy, name: Sid Mittal, name: Neil Houlsby, name: Yunhao Tang, name: Matthew Lamm, name: Libin Bai, name: Qiao Zhang, name: Luheng He, name: Yong Cheng, name: Peter Humphreys, name: Yujia Li, name: Sergey Brin, name: Albin Cassirer, name: Yingjie Miao, name: Lukas Zilka, name: Taylor Tobin, name: Kelvin Xu, name: Lev Proleev, name: Daniel Sohn, name: Alberto Magni, name: Lisa Anne Hendricks, name: Isabel Gao, name: Santiago Ontañón, name: Oskar Bunyan, name: Nathan Byrd, name: Abhanshu Sharma, name: Biao Zhang, name: Mario Pinto, name: Rishika Sinha, name: Harsh Mehta, name: Dawei Jia, name: Sergi Caelles, name: Albert Webson, name: Alex Morris, name: Becca Roelofs, name: Yifan Ding, name: Robin Strudel, name: Xuehan Xiong, name: Marvin Ritter, name: Mostafa Dehghani, name: Rahma Chaabouni, name: Abhijit Karmarkar, name: Guangda Lai, name: Fabian Mentzer, name: Bibo Xu, name: YaGuang Li, name: Yujing Zhang, name: Tom Le Paine, name: Alex Goldin, name: Behnam Neyshabur, name: Kate Baumli, name: Anselm Levskaya, name: Michael Laskin, name: Wenhao Jia, name: Jack W. Rae, name: Kefan Xiao, name: Antoine He, name: Skye Giordano, name: Lakshman Yagati, name: Jean-Baptiste Lespiau, name: Paul Natsev, name: Sanjay Ganapathy, name: Fangyu Liu, name: Danilo Martins, name: Nanxin Chen, name: Yunhan Xu, name: Megan Barnes, name: Rhys May, name: Arpi Vezer, name: Junhyuk Oh, name: Ken Franko, name: Sophie Bridgers, name: Ruizhe Zhao, name: Boxi Wu, name: Basil Mustafa, name: Sean Sechrist, name: Emilio Parisotto, name: Thanumalayan Sankaranarayana Pillai, name: Chris Larkin, name: Chenjie Gu, name: Christina Sorokin, name: Maxim Krikun, name: Alexey Guseynov, name: Jessica Landon, name: Romina Datta, name: Alexander Pritzel, name: Phoebe Thacker, name: Fan Yang, name: Kevin Hui, name: Anja Hauth, name: Chih-Kuan Yeh, name: David Barker, name: Justin Mao-Jones, name: Sophia Austin, name: Hannah Sheahan, name: Parker Schuh, name: James Svensson, name: Rohan Jain, name: Vinay Ramasesh, name: Anton Briukhov, name: Da-Woon Chung, name: Tamara von Glehn, name: Christina Butterfield, name: Priya Jhakra, name: Matthew Wiethoff, name: Justin Frye, name: Jordan Grimstad, name: Beer Changpinyo, name: Charline Le Lan, name: Anna Bortsova, name: Yonghui Wu, name: Paul Voigtlaender, name: Tara Sainath, name: Charlotte Smith, name: Will Hawkins, name: Kris Cao, name: James Besley, name: Srivatsan Srinivasan, name: Mark Omernick, name: Colin Gaffney, name: Gabriela Surita, name: Ryan Burnell, name: Bogdan Damoc, name: Junwhan Ahn, name: Andrew Brock, name: Mantas Pajarskas, name: Anastasia Petrushkina, name: Seb Noury, name: Lorenzo Blanco, name: Kevin Swersky, name: Arun Ahuja, name: Thi Avrahami, name: Vedant Misra, name: Raoul de Liedekerke, name: Mariko Iinuma, name: Alex Polozov, name: Sarah York, name: George van den Driessche, name: Paul Michel, name: Justin Chiu, name: Rory Blevins, name: Zach Gleicher, name: Adrià Recasens, name: Alban Rrustemi, name: Elena Gribovskaya, name: Aurko Roy, name: Wiktor Gworek, name: Séb Arnold, name: Lisa Lee, name: James Lee-Thorp, name: Marcello Maggioni, name: Enrique Piqueras, name: Kartikeya Badola, name: Sharad Vikram, name: Lucas Gonzalez, name: Anirudh Baddepudi, name: Evan Senter, name: Jacob Devlin, name: James Qin, name: Michael Azzam, name: Maja Trebacz, name: Martin Polacek, name: Kashyap Krishnakumar, name: Shuo-yiin Chang, name: Matthew Tung, name: Ivo Penchev, name: Rishabh Joshi, name: Kate Olszewska, name: Carrie Muir, name: Mateo Wirth, name: Ale Jakse Hartman, name: Josh Newlan, name: Sheleem Kashem, name: Vijay Bolina, name: Elahe Dabir, name: Joost van Amersfoort, name: Zafarali Ahmed, name: James Cobon-Kerr, name: Aishwarya Kamath, name: Arnar Mar Hrafnkelsson, name: Le Hou, name: Ian Mackinnon, name: Alexandre Frechette, name: Eric Noland, name: Xiance Si, name: Emanuel Taropa, name: Dong Li, name: Phil Crone, name: Anmol Gulati, name: Sébastien Cevey, name: Jonas Adler, name: Ada Ma, name: David Silver, name: Simon Tokumine, name: Richard Powell, name: Stephan Lee, name: Michael Chang, name: Samer Hassan, name: Diana Mincu, name: Antoine Yang, name: Nir Levine, name: Jenny Brennan, name: Mingqiu Wang, name: Sarah Hodkinson, name: Jeffrey Zhao, name: Josh Lipschultz, name: Aedan Pope, name: Michael B. Chang, name: Cheng Li, name: Laurent El Shafey, name: Michela Paganini, name: Sholto Douglas, name: Bernd Bohnet, name: Fabio Pardo, name: Seth Odoom, name: Mihaela Rosca, name: Cicero Nogueira dos Santos, name: Kedar Soparkar, name: Arthur Guez, name: Tom Hudson, name: Steven Hansen, name: Chulayuth Asawaroengchai, name: Ravi Addanki, name: Tianhe Yu, name: Wojciech Stokowiec, name: Mina Khan, name: Justin Gilmer, name: Jaehoon Lee, name: Carrie Grimes Bostock, name: Keran Rong, name: Jonathan Caton, name: Pedram Pejman, name: Filip Pavetic, name: Geoff Brown, name: Vivek Sharma, name: Mario Lučić, name: Rajkumar Samuel, name: Josip Djolonga, name: Amol Mandhane, name: Lars Lowe Sjösund, name: Elena Buchatskaya, name: Elspeth White, name: Natalie Clay, name: Jiepu Jiang, name: Hyeontaek Lim, name: Ross Hemsley, name: Jane Labanowski, name: Nicola De Cao, name: David Steiner, name: Sayed Hadi Hashemi, name: Jacob Austin, name: Anita Gergely, name: Tim Blyth, name: Joe Stanton, name: Kaushik Shivakumar, name: Aditya Siddhant, name: Anders Andreassen, name: Carlos Araya, name: Nikhil Sethi, name: Rakesh Shivanna, name: Steven Hand, name: Ankur Bapna, name: Ali Khodaei, name: Antoine Miech, name: Garrett Tanzer, name: Andy Swing, name: Shantanu Thakoor, name: Zhufeng Pan, name: Zachary Nado, name: Stephanie Winkler, name: Dian Yu, name: Mohammad Saleh, name: Loren Maggiore, name: Iain Barr, name: Minh Giang, name: Thais Kagohara, name: Ivo Danihelka, name: Amit Marathe, name: Vladimir Feinberg, name: Mohamed Elhawaty, name: Nimesh Ghelani, name: Dan Horgan, name: Helen Miller, name: Lexi Walker, name: Richard Tanburn, name: Mukarram Tariq, name: Disha Shrivastava, name: Fei Xia, name: Chung-Cheng Chiu, name: Zoe Ashwood, name: Khuslen Baatarsukh, name: Sina Samangooei, name: Fred Alcober, name: Axel Stjerngren, name: Paul Komarek, name: Katerina Tsihlas, name: Anudhyan Boral, name: Ramona Comanescu, name: Jeremy Chen, name: Ruibo Liu, name: Dawn Bloxwich, name: Charlie Chen, name: Yanhua Sun, name: Fangxiaoyu Feng, name: Matthew Mauger, name: Xerxes Dotiwalla, name: Vincent Hellendoorn, name: Michael Sharman, name: Ivy Zheng, name: Krishna Haridasan, name: Gabe Barth-Maron, name: Craig Swanson, name: Dominika Rogozińska, name: Alek Andreev, name: Paul Kishan Rubenstein, name: Ruoxin Sang, name: Dan Hurt, name: Gamaleldin Elsayed, name: Renshen Wang, name: Dave Lacey, name: Anastasija Ilić, name: Yao Zhao, name: Lora Aroyo, name: Chimezie Iwuanyanwu, name: Vitaly Nikolaev, name: Balaji Lakshminarayanan, name: Sadegh Jazayeri, name: Raphaël Lopez Kaufman, name: Mani Varadarajan, name: Chetan Tekur, name: Doug Fritz, name: Misha Khalman, name: David Reitter, name: Kingshuk Dasgupta, name: Shourya Sarcar, name: Tina Ornduff, name: Javier Snaider, name: Fantine Huot, name: Johnson Jia, name: Rupert Kemp, name: Nejc Trdin, name: Anitha Vijayakumar, name: Lucy Kim, name: Christof Angermueller, name: Li Lao, name: Tianqi Liu, name: Haibin Zhang, name: David Engel, name: Somer Greene, name: Anaïs White, name: Jessica Austin, name: Lilly Taylor, name: Shereen Ashraf, name: Dangyi Liu, name: Maria Georgaki, name: Irene Cai, name: Yana Kulizhskaya, name: Sonam Goenka, name: Brennan Saeta, name: Kiran Vodrahalli, name: Christian Frank, name: Dario de Cesare, name: Brona Robenek, name: Harry Richardson, name: Mahmoud Alnahlawi, name: Christopher Yew, name: Priya Ponnapalli, name: Marco Tagliasacchi, name: Alex Korchemniy, name: Yelin Kim, name: Dinghua Li, name: Bill Rosgen, name: Zoe Ashwood, name: Kyle Levin, name: Jeremy Wiesner, name: Praseem Banzal, name: Praveen Srinivasan, name: Hongkun Yu, name: Çağlar Ünlü, name: David Reid, name: Zora Tung, name: Daniel Finchelstein, name: Ravin Kumar, name: Andre Elisseeff, name: Jin Huang, name: Ming Zhang, name: Rui Zhu, name: Ricardo Aguilar, name: Mai Giménez, name: Jiawei Xia, name: Olivier Dousse, name: Willi Gierke, name: Soheil Hassas Yeganeh, name: Damion Yates, name: Komal Jalan, name: Lu Li, name: Eri Latorre-Chimoto, name: Duc Dung Nguyen, name: Ken Durden, name: Praveen Kallakuri, name: Yaxin Liu, name: Matthew Johnson, name: Tomy Tsai, name: Alice Talbert, name: Jasmine Liu, name: Alexander Neitz, name: Chen Elkind, name: Marco Selvi, name: Mimi Jasarevic, name: Livio Baldini Soares, name: Albert Cui, name: Pidong Wang, name: Alek Wenjiao Wang, name: Xinyu Ye, name: Krystal Kallarackal, name: Lucia Loher, name: Hoi Lam, name: Josef Broder, name: Dan Holtmann-Rice, name: Nina Martin, name: Bramandia Ramadhana, name: Daniel Toyama, name: Mrinal Shukla, name: Sujoy Basu, name: Abhi Mohan, name: Nick Fernando, name: Noah Fiedel, name: Kim Paterson, name: Hui Li, name: Ankush Garg, name: Jane Park, name: DongHyun Choi, name: Diane Wu, name: Sankalp Singh, name: Zhishuai Zhang, name: Amir Globerson, name: Lily Yu, name: John Carpenter, name: Félix de Chaumont Quitry, name: Carey Radebaugh, name: Chu-Cheng Lin, name: Alex Tudor, name: Prakash Shroff, name: Drew Garmon, name: Dayou Du, name: Neera Vats, name: Han Lu, name: Shariq Iqbal, name: Alex Yakubovich, name: Nilesh Tripuraneni, name: James Manyika, name: Haroon Qureshi, name: Nan Hua, name: Christel Ngani, name: Maria Abi Raad, name: Hannah Forbes, name: Anna Bulanova, name: Jeff Stanway, name: Mukund Sundararajan, name: Victor Ungureanu, name: Colton Bishop, name: Yunjie Li, name: Balaji Venkatraman, name: Bo Li, name: Chloe Thornton, name: Salvatore Scellato, name: Nishesh Gupta, name: Yicheng Wang, name: Ian Tenney, name: Xihui Wu, name: Ashish Shenoy, name: Gabriel Carvajal, name: Diana Gage Wright, name: Ben Bariach, name: Zhuyun Xiao, name: Peter Hawkins, name: Sid Dalmia, name: Clement Farabet, name: Pedro Valenzuela, name: Quan Yuan, name: Chris Welty, name: Ananth Agarwal, name: Mia Chen, name: Wooyeol Kim, name: Brice Hulse, name: Nandita Dukkipati, name: Adam Paszke, name: Andrew Bolt, name: Elnaz Davoodi, name: Kiam Choo, name: Jennifer Beattie, name: Jennifer Prendki, name: Harsha Vashisht, name: Rebeca Santamaria-Fernandez, name: Luis C. Cobo, name: Jarek Wilkiewicz, name: David Madras, name: Ali Elqursh, name: Grant Uy, name: Kevin Ramirez, name: Matt Harvey, name: Tyler Liechty, name: Heiga Zen, name: Jeff Seibert, name: Clara Huiyi Hu, name: Mohamed Elhawaty, name: Andrey Khorlin, name: Maigo Le, name: Asaf Aharoni, name: Megan Li, name: Lily Wang, name: Sandeep Kumar, name: Alejandro Lince, name: Norman Casagrande, name: Jay Hoover, name: Dalia El Badawy, name: David Soergel, name: Denis Vnukov, name: Matt Miecnikowski, name: Jiri Simsa, name: Anna Koop, name: Praveen Kumar, name: Thibault Sellam, name: Daniel Vlasic, name: Samira Daruki, name: Nir Shabat, name: John Zhang, name: Guolong Su, name: Jiageng Zhang, name: Jeremiah Liu, name: Yi Sun, name: Evan Palmer, name: Alireza Ghaffarkhah, name: Xi Xiong, name: Victor Cotruta, name: Michael Fink, name: Lucas Dixon, name: Ashwin Sreevatsa, name: Adrian Goedeckemeyer, name: Alek Dimitriev, name: Mohsen Jafari, name: Remi Crocker, name: Nicholas FitzGerald, name: Aviral Kumar, name: Sanjay Ghemawat, name: Ivan Philips, name: Frederick Liu, name: Yannie Liang, name: Rachel Sterneck, name: Alena Repina, name: Marcus Wu, name: Laura Knight, name: Marin Georgiev, name: Hyo Lee, name: Harry Askham, name: Abhishek Chakladar, name: Annie Louis, name: Carl Crous, name: Hardie Cate, name: Dessie Petrova, name: Michael Quinn, name: Denese Owusu-Afriyie, name: Achintya Singhal, name: Nan Wei, name: Solomon Kim, name: Damien Vincent, name: Milad Nasr, name: Christopher A. Choquette-Choo, name: Reiko Tojo, name: Shawn Lu, name: Diego de Las Casas, name: Yuchung Cheng, name: Tolga Bolukbasi, name: Katherine Lee, name: Saaber Fatehi, name: Rajagopal Ananthanarayanan, name: Miteyan Patel, name: Charbel Kaed, name: Jing Li, name: Jakub Sygnowski, name: Shreyas Rammohan Belle, name: Zhe Chen, name: Jaclyn Konzelmann, name: Siim Põder, name: Roopal Garg, name: Vinod Koverkathu, name: Adam Brown, name: Chris Dyer, name: Rosanne Liu, name: Azade Nova, name: Jun Xu, name: Slav Petrov, name: Demis Hassabis, name: Koray Kavukcuoglu, name: Jeffrey Dean, name: Oriol Vinyals
Published: 2024-03-08 18:54:20

In this report, we present the latest model of the Gemini family, Gemini 1.5
Pro, a highly compute-efficient multimodal mixture-of-experts model capable of
recalling and reasoning over fine-grained information from millions of tokens
of context, including multiple long documents and hours of video and audio.
Gemini 1.5 Pro achieves near-perfect recall on long-context retrieval tasks
across modalities, improves the state-of-the-art in long-document QA,
long-video QA and long-context ASR, and matches or surpasses Gemini 1.0 Ultra's
state-of-the-art performance across a broad set of benchmarks. Studying the
limits of Gemini 1.5 Pro's long-context ability, we find continued improvement
in next-token prediction and near-perfect retrieval (>99%) up to at least 10M
tokens, a generational leap over existing models such as Claude 2.1 (200k) and
GPT-4 Turbo (128k). Finally, we highlight surprising new capabilities of large
language models at the frontier; when given a grammar manual for Kalamang, a
language with fewer than 200 speakers worldwide, the model learns to translate
English to Kalamang at a similar level to a person who learned from the same
content.