Gemini 1.5: Unlocking multimodal understanding across millions of tokens
  of context
Authors: name: Machel Reid; name: Nikolay Savinov; name: Denis Teplyashin; name: Dmitry Lepikhin; name: Timothy Lillicrap; name: Jean-baptiste Alayrac; name: Radu Soricut; name: Angeliki Lazaridou; name: Orhan Firat; name: Julian Schrittwieser; name: Ioannis Antonoglou; name: Rohan Anil; name: Sebastian Borgeaud; name: Andrew Dai; name: Katie Millican; name: Ethan Dyer; name: Mia Glaese; name: Thibault Sottiaux; name: Benjamin Lee; name: Fabio Viola; name: Malcolm Reynolds; name: Yuanzhong Xu; name: James Molloy; name: Jilin Chen; name: Michael Isard; name: Paul Barham; name: Tom Hennigan; name: Ross McIlroy; name: Melvin Johnson; name: Johan Schalkwyk; name: Eli Collins; name: Eliza Rutherford; name: Erica Moreira; name: Kareem Ayoub; name: Megha Goel; name: Clemens Meyer; name: Gregory Thornton; name: Zhen Yang; name: Henryk Michalewski; name: Zaheer Abbas; name: Nathan Schucher; name: Ankesh Anand; name: Richard Ives; name: James Keeling; name: Karel Lenc; name: Salem Haykal; name: Siamak Shakeri; name: Pranav Shyam; name: Aakanksha Chowdhery; name: Roman Ring; name: Stephen Spencer; name: Eren Sezener; name: Luke Vilnis; name: Oscar Chang; name: Nobuyuki Morioka; name: George Tucker; name: Ce Zheng; name: Oliver Woodman; name: Nithya Attaluri; name: Tomas Kocisky; name: Evgenii Eltyshev; name: Xi Chen; name: Timothy Chung; name: Vittorio Selo; name: Siddhartha Brahma; name: Petko Georgiev; name: Ambrose Slone; name: Zhenkai Zhu; name: James Lottes; name: Siyuan Qiao; name: Ben Caine; name: Sebastian Riedel; name: Alex Tomala; name: Martin Chadwick; name: Juliette Love; name: Peter Choy; name: Sid Mittal; name: Neil Houlsby; name: Yunhao Tang; name: Matthew Lamm; name: Libin Bai; name: Qiao Zhang; name: Luheng He; name: Yong Cheng; name: Peter Humphreys; name: Yujia Li; name: Sergey Brin; name: Albin Cassirer; name: Yingjie Miao; name: Lukas Zilka; name: Taylor Tobin; name: Kelvin Xu; name: Lev Proleev; name: Daniel Sohn; name: Alberto Magni; name: Lisa Anne Hendricks; name: Isabel Gao; name: Santiago Ontañón; name: Oskar Bunyan; name: Nathan Byrd; name: Abhanshu Sharma; name: Biao Zhang; name: Mario Pinto; name: Rishika Sinha; name: Harsh Mehta; name: Dawei Jia; name: Sergi Caelles; name: Albert Webson; name: Alex Morris; name: Becca Roelofs; name: Yifan Ding; name: Robin Strudel; name: Xuehan Xiong; name: Marvin Ritter; name: Mostafa Dehghani; name: Rahma Chaabouni; name: Abhijit Karmarkar; name: Guangda Lai; name: Fabian Mentzer; name: Bibo Xu; name: YaGuang Li; name: Yujing Zhang; name: Tom Le Paine; name: Alex Goldin; name: Behnam Neyshabur; name: Kate Baumli; name: Anselm Levskaya; name: Michael Laskin; name: Wenhao Jia; name: Jack W. Rae; name: Kefan Xiao; name: Antoine He; name: Skye Giordano; name: Lakshman Yagati; name: Jean-Baptiste Lespiau; name: Paul Natsev; name: Sanjay Ganapathy; name: Fangyu Liu; name: Danilo Martins; name: Nanxin Chen; name: Yunhan Xu; name: Megan Barnes; name: Rhys May; name: Arpi Vezer; name: Junhyuk Oh; name: Ken Franko; name: Sophie Bridgers; name: Ruizhe Zhao; name: Boxi Wu; name: Basil Mustafa; name: Sean Sechrist; name: Emilio Parisotto; name: Thanumalayan Sankaranarayana Pillai; name: Chris Larkin; name: Chenjie Gu; name: Christina Sorokin; name: Maxim Krikun; name: Alexey Guseynov; name: Jessica Landon; name: Romina Datta; name: Alexander Pritzel; name: Phoebe Thacker; name: Fan Yang; name: Kevin Hui; name: Anja Hauth; name: Chih-Kuan Yeh; name: David Barker; name: Justin Mao-Jones; name: Sophia Austin; name: Hannah Sheahan; name: Parker Schuh; name: James Svensson; name: Rohan Jain; name: Vinay Ramasesh; name: Anton Briukhov; name: Da-Woon Chung; name: Tamara von Glehn; name: Christina Butterfield; name: Priya Jhakra; name: Matthew Wiethoff; name: Justin Frye; name: Jordan Grimstad; name: Beer Changpinyo; name: Charline Le Lan; name: Anna Bortsova; name: Yonghui Wu; name: Paul Voigtlaender; name: Tara Sainath; name: Charlotte Smith; name: Will Hawkins; name: Kris Cao; name: James Besley; name: Srivatsan Srinivasan; name: Mark Omernick; name: Colin Gaffney; name: Gabriela Surita; name: Ryan Burnell; name: Bogdan Damoc; name: Junwhan Ahn; name: Andrew Brock; name: Mantas Pajarskas; name: Anastasia Petrushkina; name: Seb Noury; name: Lorenzo Blanco; name: Kevin Swersky; name: Arun Ahuja; name: Thi Avrahami; name: Vedant Misra; name: Raoul de Liedekerke; name: Mariko Iinuma; name: Alex Polozov; name: Sarah York; name: George van den Driessche; name: Paul Michel; name: Justin Chiu; name: Rory Blevins; name: Zach Gleicher; name: Adrià Recasens; name: Alban Rrustemi; name: Elena Gribovskaya; name: Aurko Roy; name: Wiktor Gworek; name: Séb Arnold; name: Lisa Lee; name: James Lee-Thorp; name: Marcello Maggioni; name: Enrique Piqueras; name: Kartikeya Badola; name: Sharad Vikram; name: Lucas Gonzalez; name: Anirudh Baddepudi; name: Evan Senter; name: Jacob Devlin; name: James Qin; name: Michael Azzam; name: Maja Trebacz; name: Martin Polacek; name: Kashyap Krishnakumar; name: Shuo-yiin Chang; name: Matthew Tung; name: Ivo Penchev; name: Rishabh Joshi; name: Kate Olszewska; name: Carrie Muir; name: Mateo Wirth; name: Ale Jakse Hartman; name: Josh Newlan; name: Sheleem Kashem; name: Vijay Bolina; name: Elahe Dabir; name: Joost van Amersfoort; name: Zafarali Ahmed; name: James Cobon-Kerr; name: Aishwarya Kamath; name: Arnar Mar Hrafnkelsson; name: Le Hou; name: Ian Mackinnon; name: Alexandre Frechette; name: Eric Noland; name: Xiance Si; name: Emanuel Taropa; name: Dong Li; name: Phil Crone; name: Anmol Gulati; name: Sébastien Cevey; name: Jonas Adler; name: Ada Ma; name: David Silver; name: Simon Tokumine; name: Richard Powell; name: Stephan Lee; name: Michael Chang; name: Samer Hassan; name: Diana Mincu; name: Antoine Yang; name: Nir Levine; name: Jenny Brennan; name: Mingqiu Wang; name: Sarah Hodkinson; name: Jeffrey Zhao; name: Josh Lipschultz; name: Aedan Pope; name: Michael B. Chang; name: Cheng Li; name: Laurent El Shafey; name: Michela Paganini; name: Sholto Douglas; name: Bernd Bohnet; name: Fabio Pardo; name: Seth Odoom; name: Mihaela Rosca; name: Cicero Nogueira dos Santos; name: Kedar Soparkar; name: Arthur Guez; name: Tom Hudson; name: Steven Hansen; name: Chulayuth Asawaroengchai; name: Ravi Addanki; name: Tianhe Yu; name: Wojciech Stokowiec; name: Mina Khan; name: Justin Gilmer; name: Jaehoon Lee; name: Carrie Grimes Bostock; name: Keran Rong; name: Jonathan Caton; name: Pedram Pejman; name: Filip Pavetic; name: Geoff Brown; name: Vivek Sharma; name: Mario Lučić; name: Rajkumar Samuel; name: Josip Djolonga; name: Amol Mandhane; name: Lars Lowe Sjösund; name: Elena Buchatskaya; name: Elspeth White; name: Natalie Clay; name: Jiepu Jiang; name: Hyeontaek Lim; name: Ross Hemsley; name: Jane Labanowski; name: Nicola De Cao; name: David Steiner; name: Sayed Hadi Hashemi; name: Jacob Austin; name: Anita Gergely; name: Tim Blyth; name: Joe Stanton; name: Kaushik Shivakumar; name: Aditya Siddhant; name: Anders Andreassen; name: Carlos Araya; name: Nikhil Sethi; name: Rakesh Shivanna; name: Steven Hand; name: Ankur Bapna; name: Ali Khodaei; name: Antoine Miech; name: Garrett Tanzer; name: Andy Swing; name: Shantanu Thakoor; name: Zhufeng Pan; name: Zachary Nado; name: Stephanie Winkler; name: Dian Yu; name: Mohammad Saleh; name: Loren Maggiore; name: Iain Barr; name: Minh Giang; name: Thais Kagohara; name: Ivo Danihelka; name: Amit Marathe; name: Vladimir Feinberg; name: Mohamed Elhawaty; name: Nimesh Ghelani; name: Dan Horgan; name: Helen Miller; name: Lexi Walker; name: Richard Tanburn; name: Mukarram Tariq; name: Disha Shrivastava; name: Fei Xia; name: Chung-Cheng Chiu; name: Zoe Ashwood; name: Khuslen Baatarsukh; name: Sina Samangooei; name: Fred Alcober; name: Axel Stjerngren; name: Paul Komarek; name: Katerina Tsihlas; name: Anudhyan Boral; name: Ramona Comanescu; name: Jeremy Chen; name: Ruibo Liu; name: Dawn Bloxwich; name: Charlie Chen; name: Yanhua Sun; name: Fangxiaoyu Feng; name: Matthew Mauger; name: Xerxes Dotiwalla; name: Vincent Hellendoorn; name: Michael Sharman; name: Ivy Zheng; name: Krishna Haridasan; name: Gabe Barth-Maron; name: Craig Swanson; name: Dominika Rogozińska; name: Alek Andreev; name: Paul Kishan Rubenstein; name: Ruoxin Sang; name: Dan Hurt; name: Gamaleldin Elsayed; name: Renshen Wang; name: Dave Lacey; name: Anastasija Ilić; name: Yao Zhao; name: Lora Aroyo; name: Chimezie Iwuanyanwu; name: Vitaly Nikolaev; name: Balaji Lakshminarayanan; name: Sadegh Jazayeri; name: Raphaël Lopez Kaufman; name: Mani Varadarajan; name: Chetan Tekur; name: Doug Fritz; name: Misha Khalman; name: David Reitter; name: Kingshuk Dasgupta; name: Shourya Sarcar; name: Tina Ornduff; name: Javier Snaider; name: Fantine Huot; name: Johnson Jia; name: Rupert Kemp; name: Nejc Trdin; name: Anitha Vijayakumar; name: Lucy Kim; name: Christof Angermueller; name: Li Lao; name: Tianqi Liu; name: Haibin Zhang; name: David Engel; name: Somer Greene; name: Anaïs White; name: Jessica Austin; name: Lilly Taylor; name: Shereen Ashraf; name: Dangyi Liu; name: Maria Georgaki; name: Irene Cai; name: Yana Kulizhskaya; name: Sonam Goenka; name: Brennan Saeta; name: Kiran Vodrahalli; name: Christian Frank; name: Dario de Cesare; name: Brona Robenek; name: Harry Richardson; name: Mahmoud Alnahlawi; name: Christopher Yew; name: Priya Ponnapalli; name: Marco Tagliasacchi; name: Alex Korchemniy; name: Yelin Kim; name: Dinghua Li; name: Bill Rosgen; name: Zoe Ashwood; name: Kyle Levin; name: Jeremy Wiesner; name: Praseem Banzal; name: Praveen Srinivasan; name: Hongkun Yu; name: Çağlar Ünlü; name: David Reid; name: Zora Tung; name: Daniel Finchelstein; name: Ravin Kumar; name: Andre Elisseeff; name: Jin Huang; name: Ming Zhang; name: Rui Zhu; name: Ricardo Aguilar; name: Mai Giménez; name: Jiawei Xia; name: Olivier Dousse; name: Willi Gierke; name: Soheil Hassas Yeganeh; name: Damion Yates; name: Komal Jalan; name: Lu Li; name: Eri Latorre-Chimoto; name: Duc Dung Nguyen; name: Ken Durden; name: Praveen Kallakuri; name: Yaxin Liu; name: Matthew Johnson; name: Tomy Tsai; name: Alice Talbert; name: Jasmine Liu; name: Alexander Neitz; name: Chen Elkind; name: Marco Selvi; name: Mimi Jasarevic; name: Livio Baldini Soares; name: Albert Cui; name: Pidong Wang; name: Alek Wenjiao Wang; name: Xinyu Ye; name: Krystal Kallarackal; name: Lucia Loher; name: Hoi Lam; name: Josef Broder; name: Dan Holtmann-Rice; name: Nina Martin; name: Bramandia Ramadhana; name: Daniel Toyama; name: Mrinal Shukla; name: Sujoy Basu; name: Abhi Mohan; name: Nick Fernando; name: Noah Fiedel; name: Kim Paterson; name: Hui Li; name: Ankush Garg; name: Jane Park; name: DongHyun Choi; name: Diane Wu; name: Sankalp Singh; name: Zhishuai Zhang; name: Amir Globerson; name: Lily Yu; name: John Carpenter; name: Félix de Chaumont Quitry; name: Carey Radebaugh; name: Chu-Cheng Lin; name: Alex Tudor; name: Prakash Shroff; name: Drew Garmon; name: Dayou Du; name: Neera Vats; name: Han Lu; name: Shariq Iqbal; name: Alex Yakubovich; name: Nilesh Tripuraneni; name: James Manyika; name: Haroon Qureshi; name: Nan Hua; name: Christel Ngani; name: Maria Abi Raad; name: Hannah Forbes; name: Anna Bulanova; name: Jeff Stanway; name: Mukund Sundararajan; name: Victor Ungureanu; name: Colton Bishop; name: Yunjie Li; name: Balaji Venkatraman; name: Bo Li; name: Chloe Thornton; name: Salvatore Scellato; name: Nishesh Gupta; name: Yicheng Wang; name: Ian Tenney; name: Xihui Wu; name: Ashish Shenoy; name: Gabriel Carvajal; name: Diana Gage Wright; name: Ben Bariach; name: Zhuyun Xiao; name: Peter Hawkins; name: Sid Dalmia; name: Clement Farabet; name: Pedro Valenzuela; name: Quan Yuan; name: Chris Welty; name: Ananth Agarwal; name: Mia Chen; name: Wooyeol Kim; name: Brice Hulse; name: Nandita Dukkipati; name: Adam Paszke; name: Andrew Bolt; name: Elnaz Davoodi; name: Kiam Choo; name: Jennifer Beattie; name: Jennifer Prendki; name: Harsha Vashisht; name: Rebeca Santamaria-Fernandez; name: Luis C. Cobo; name: Jarek Wilkiewicz; name: David Madras; name: Ali Elqursh; name: Grant Uy; name: Kevin Ramirez; name: Matt Harvey; name: Tyler Liechty; name: Heiga Zen; name: Jeff Seibert; name: Clara Huiyi Hu; name: Mohamed Elhawaty; name: Andrey Khorlin; name: Maigo Le; name: Asaf Aharoni; name: Megan Li; name: Lily Wang; name: Sandeep Kumar; name: Alejandro Lince; name: Norman Casagrande; name: Jay Hoover; name: Dalia El Badawy; name: David Soergel; name: Denis Vnukov; name: Matt Miecnikowski; name: Jiri Simsa; name: Anna Koop; name: Praveen Kumar; name: Thibault Sellam; name: Daniel Vlasic; name: Samira Daruki; name: Nir Shabat; name: John Zhang; name: Guolong Su; name: Jiageng Zhang; name: Jeremiah Liu; name: Yi Sun; name: Evan Palmer; name: Alireza Ghaffarkhah; name: Xi Xiong; name: Victor Cotruta; name: Michael Fink; name: Lucas Dixon; name: Ashwin Sreevatsa; name: Adrian Goedeckemeyer; name: Alek Dimitriev; name: Mohsen Jafari; name: Remi Crocker; name: Nicholas FitzGerald; name: Aviral Kumar; name: Sanjay Ghemawat; name: Ivan Philips; name: Frederick Liu; name: Yannie Liang; name: Rachel Sterneck; name: Alena Repina; name: Marcus Wu; name: Laura Knight; name: Marin Georgiev; name: Hyo Lee; name: Harry Askham; name: Abhishek Chakladar; name: Annie Louis; name: Carl Crous; name: Hardie Cate; name: Dessie Petrova; name: Michael Quinn; name: Denese Owusu-Afriyie; name: Achintya Singhal; name: Nan Wei; name: Solomon Kim; name: Damien Vincent; name: Milad Nasr; name: Christopher A. Choquette-Choo; name: Reiko Tojo; name: Shawn Lu; name: Diego de Las Casas; name: Yuchung Cheng; name: Tolga Bolukbasi; name: Katherine Lee; name: Saaber Fatehi; name: Rajagopal Ananthanarayanan; name: Miteyan Patel; name: Charbel Kaed; name: Jing Li; name: Jakub Sygnowski; name: Shreyas Rammohan Belle; name: Zhe Chen; name: Jaclyn Konzelmann; name: Siim Põder; name: Roopal Garg; name: Vinod Koverkathu; name: Adam Brown; name: Chris Dyer; name: Rosanne Liu; name: Azade Nova; name: Jun Xu; name: Slav Petrov; name: Demis Hassabis; name: Koray Kavukcuoglu; name: Jeffrey Dean; name: Oriol Vinyals
Published: 2024-03-08 18:54:20

Abstract: In this report, we present the latest model of the Gemini family, Gemini 1.5
Pro, a highly compute-efficient multimodal mixture-of-experts model capable of
recalling and reasoning over fine-grained information from millions of tokens
of context, including multiple long documents and hours of video and audio.
Gemini 1.5 Pro achieves near-perfect recall on long-context retrieval tasks
across modalities, improves the state-of-the-art in long-document QA,
long-video QA and long-context ASR, and matches or surpasses Gemini 1.0 Ultra's
state-of-the-art performance across a broad set of benchmarks. Studying the
limits of Gemini 1.5 Pro's long-context ability, we find continued improvement
in next-token prediction and near-perfect retrieval (>99%) up to at least 10M
tokens, a generational leap over existing models such as Claude 2.1 (200k) and
GPT-4 Turbo (128k). Finally, we highlight surprising new capabilities of large
language models at the frontier; when given a grammar manual for Kalamang, a
language with fewer than 200 speakers worldwide, the model learns to translate
English to Kalamang at a similar level to a person who learned from the same
content.