Jamba-1.5: Hybrid Transformer-Mamba Models at Scale
Authors: name: Jamba Team; name: Barak Lenz; name: Alan Arazi; name: Amir Bergman; name: Avshalom Manevich; name: Barak Peleg; name: Ben Aviram; name: Chen Almagor; name: Clara Fridman; name: Dan Padnos; name: Daniel Gissin; name: Daniel Jannai; name: Dor Muhlgay; name: Dor Zimberg; name: Edden M Gerber; name: Elad Dolev; name: Eran Krakovsky; name: Erez Safahi; name: Erez Schwartz; name: Gal Cohen; name: Gal Shachaf; name: Haim Rozenblum; name: Hofit Bata; name: Ido Blass; name: Inbal Magar; name: Itay Dalmedigos; name: Jhonathan Osin; name: Julie Fadlon; name: Maria Rozman; name: Matan Danos; name: Michael Gokhman; name: Mor Zusman; name: Naama Gidron; name: Nir Ratner; name: Noam Gat; name: Noam Rozen; name: Oded Fried; name: Ohad Leshno; name: Omer Antverg; name: Omri Abend; name: Opher Lieber; name: Or Dagan; name: Orit Cohavi; name: Raz Alon; name: Ro'i Belson; name: Roi Cohen; name: Rom Gilad; name: Roman Glozman; name: Shahar Lev; name: Shaked Meirom; name: Tal Delbari; name: Tal Ness; name: Tomer Asida; name: Tom Ben Gal; name: Tom Braude; name: Uriya Pumerantz; name: Yehoshua Cohen; name: Yonatan Belinkov; name: Yuval Globerson; name: Yuval Peleg Levy; name: Yoav Shoham
Published: 2024-08-22 17:38:59

Abstract: We present Jamba-1.5, new instruction-tuned large language models based on
our Jamba architecture. Jamba is a hybrid Transformer-Mamba mixture of experts
architecture, providing high throughput and low memory usage across context
lengths, while retaining the same or better quality as Transformer models. We
release two model sizes: Jamba-1.5-Large, with 94B active parameters, and
Jamba-1.5-Mini, with 12B active parameters. Both models are fine-tuned for a
variety of conversational and instruction-following capabilties, and have an
effective context length of 256K tokens, the largest amongst open-weight
models. To support cost-effective inference, we introduce ExpertsInt8, a novel
quantization technique that allows fitting Jamba-1.5-Large on a machine with 8
80GB GPUs when processing 256K-token contexts without loss of quality. When
evaluated on a battery of academic and chatbot benchmarks, Jamba-1.5 models
achieve excellent results while providing high throughput and outperforming
other open-weight models on long-context benchmarks. The model weights for both
sizes are publicly available under the Jamba Open Model License and we release
ExpertsInt8 as open source.