RecurrentGemma: Moving Past Transformers for Efficient Open Language
  Models
Authors: name: Aleksandar Botev; name: Soham De; name: Samuel L Smith; name: Anushan Fernando; name: George-Cristian Muraru; name: Ruba Haroun; name: Leonard Berrada; name: Razvan Pascanu; name: Pier Giuseppe Sessa; name: Robert Dadashi; name: Léonard Hussenot; name: Johan Ferret; name: Sertan Girgin; name: Olivier Bachem; name: Alek Andreev; name: Kathleen Kenealy; name: Thomas Mesnard; name: Cassidy Hardin; name: Surya Bhupatiraju; name: Shreya Pathak; name: Laurent Sifre; name: Morgane Rivière; name: Mihir Sanjay Kale; name: Juliette Love; name: Pouya Tafti; name: Armand Joulin; name: Noah Fiedel; name: Evan Senter; name: Yutian Chen; name: Srivatsan Srinivasan; name: Guillaume Desjardins; name: David Budden; name: Arnaud Doucet; name: Sharad Vikram; name: Adam Paszke; name: Trevor Gale; name: Sebastian Borgeaud; name: Charlie Chen; name: Andy Brock; name: Antonia Paterson; name: Jenny Brennan; name: Meg Risdal; name: Raj Gundluru; name: Nesh Devanathan; name: Paul Mooney; name: Nilay Chauhan; name: Phil Culliton; name: Luiz GUStavo Martins; name: Elisa Bandy; name: David Huntsperger; name: Glenn Cameron; name: Arthur Zucker; name: Tris Warkentin; name: Ludovic Peran; name: Minh Giang; name: Zoubin Ghahramani; name: Clément Farabet; name: Koray Kavukcuoglu; name: Demis Hassabis; name: Raia Hadsell; name: Yee Whye Teh; name: Nando de Frietas
Published: 2024-04-11 15:27:22

Abstract: We introduce RecurrentGemma, an open language model which uses Google's novel
Griffin architecture. Griffin combines linear recurrences with local attention
to achieve excellent performance on language. It has a fixed-sized state, which
reduces memory use and enables efficient inference on long sequences. We
provide a pre-trained model with 2B non-embedding parameters, and an
instruction tuned variant. Both models achieve comparable performance to
Gemma-2B despite being trained on fewer tokens.