Jamba: Hybrid Transformer-Mamba Language Models
Authors: Barak Lenz
Opher Lieber
Alan Arazi
Amir Bergman
Avshalom Manevich
Barak Peleg
Ben Aviram
Chen Almagor
Clara Fridman
Dan Padnos
Daniel Gissin
Daniel Jannai
Dor Muhlgay
Dor Zimberg
Edden M. Gerber
Elad Dolev
Eran Krakovsky
Erez Safahi
Erez Schwartz
Gal Cohen
Gal Shachaf
Haim Rozenblum
Hofit Bata
Ido Blass
Inbal Magar
Itay Dalmedigos
Jhonathan Osin
Julie Fadlon
Maria Rozman
Matan Danos
Michael Gokhman
Mor Zusman
Naama Gidron
Nir Ratner
Noam Gat
Noam Rozen
Oded Fried
Ohad Leshno
Omer Antverg
Omri Abend
Or Dagan
Orit Cohavi
Raz Alon
Ro'i Belson
Roi Cohen
Rom Gilad
Roman Glozman
Shahar Lev
Shai Shalev-Shwartz
Shaked Haim Meirom
Tal Delbari
Tal Ness
Tomer Asida
Tom Ben Gal
Tom Braude
Uriya Pumerantz
Josh Cohen
Yonatan Belinkov
Yuval Globerson
Yuval Peleg Levy
Yoav Shoham
Venue: ICLR 2025 Poster (2025)
Keywords: language models
state-space models
mamba
hybrid architecture
foundation models
Primary area: foundation or frontier models, including LLMs
TL;DR: Hybrid Transformer-Mamba models give better throughput and long-context capabilities at large scale

Abstract: We present Jamba, a novel hybrid Transformer-Mamba mixture-of-experts (MoE) architecture. Jamba interleaves blocks of Transformer and Mamba layers, enjoying the benefits of both model families. MoE is added in some of these layers to increase model capacity while keeping active parameter usage manageable. This flexible architecture allows resource- and objective-specific configurations. We implement two configurations: Jamba-1.5-Large, with 94B active parameters, and Jamba-1.5-mini, with 12B active parameters. Built at large scale, Jamba models provide high throughput and small memory footprint compared to vanilla Transformers, especially at long-context tasks, with an effective context length of 256K tokens, the largest amongst open-weight models. At the same time, they are also competitive on standard language modeling and chatbot benchmarks. We study various architectural decisions, such as how to combine Transformer and Mamba layers, and how to mix experts, and show that some of them are crucial in large scale modeling. To support cost-effective inference, we introduce ExpertsInt8, a novel quantization technique that allows fitting Jamba-1.5-Large on a machine with 8 80GB GPUs when processing 256K-token contexts without loss of quality. We also describe several interesting properties of this architecture that the training and evaluation of Jamba have revealed. The model weights are publicly available.

Published as a conference paper at ICLR 2025
JAMBA: HYBRID TRANSFORMER-MAMBA LANGUAGE MODELS
Jamba Team
A121 labs
ABSTRACT
We present Jamba, novel hybrid Transformer-Mambe mixture-of-experts (MoE) architecture. Jamba interleaves blocks of Transformer and Mamba layers, enjoying the benefits of both model families. MoE is added in some of these layers increase model capacity while keeping active parameter usage manageable. This flexible architecture allows resource- and objective-specific configurations. We implement two configurations: Jamba-1.5 1.5-aaee,, with 94B active parameters, and
with 12B active parameters. Built at large scale, Jamba models provide high throughput and small memory footprint compared to vanilla Transformers, especially at long-context tasks, with an effective context length 256K tokens, the largest amongst open- weight models. At the same time, they are also competitive on standard language modeling and chatbot benchmarks. We study various architectural decisions, such as how to combine Transformer and Mamba layers, and how to mix experts, and show that some of them are crucial in large-scale modeling. To support cost-effective inference, we introduce ExpertsInt8, a novel quantization technique that allows fitting Jamba-1.5-Larg on a machine with 80GB GPUs when processing 256K- token contexts without loss of quality. We also describe several interesting properties of this architecture that the training and evaluation of Jamba have revealed. The model weights are publicly available.
Models: http::///uggiinfffcccc/////
INTRODUCTION
We introduce Jamba, a novel hybrid architecture, which combines Transformer layers (Vaswani et al., 2017) with Mamba layers (Gu & Dao, 2023), recent state-space model (Gu et al., 20222022a0a as well as a mixture-of-eeperts (MoE) module (Shazeer et al., 2017; Fedus et al., 2022). Jamba thus combines two orthogonal architectural designs that together give it improved performance and higher throughput, while maintaining a manageable memory footprint. We release two models based on this architecture: Jamba-1.5 1.5MMnn (12B active parameters, 52B total available parameters) and Jamba-1 L.5-Le (94B active, 398B total available parameters), which are respectively designed to fit on a single 80GB GPU and a single 8x80GB GPU machine. However, the Jamba architecture supports other design choices, depending on one's 's hardware and performance requirements.
The fundamental novelty of Jamba is its hybrid immense popularity of the Transformer the predominant architecture for language models, it suffers from two main drawbacks. First, its high memory and compute requirements hinder the processing of long contexts, where the key-value (KV) cache size becomes limiting factor. Second, its lack a single summary state entails slow inference and low throughput, as each generated token performs computation on the entire context. In contrast, older recurrent neural networks (RNN) (Elman, 1990; Hochreiter & Schmidhuber, 1997; Mikolov et al., 2010), which summarize an arbitrarily long context in single state, do not suffer from these limitations. RNN models have their own shortcomings, however. They are costly to train as training cannot be parallelized across time. And they struggle with long distance relationships, which the hidden state captures limited extent.
Beansfooer--Mabbee architecture. Despite the

---

Published as a conference paper at ICLR 2025
Recent state space models (SSMs) like Mamba are more efficient to train than RNNs and are more capable at handling long distance relationships, but still lag behind the performance of comparably sized Transformer language models. Taking advantage of both model families, Jamba combines Transformer and Mamha layers, at a certain ratio. Varying the ratio of Transformer/Mambl layers allows balancing memory usage, efficient training, and long context capabilities.
A few other recent attempts to combine Attention and SSM modules are worth noting. Zuo et al. (2022) mixes an S4 layer (Gu et al., 2021a) with a local attention layer, followed by sequence of local attention layers; it shows experiments with small models and simple tasks. Gu & Dao (2023) report that interleaving Mamba and attention layers is only slightly better than pure Mamba in terms of perplexity, with models up to 1.3B parameters. Pilault et al. (2023) start with an SSM layer followed by chunk-based Transformers, with models up to 1.3B showing improved perplexity. Fathullah et al. (2023) adds an SSM layer before the self-attention in Transformer layer, while Saon et al. (2023) add the SSM after the self-attention, both showing improvements on speech recognition. Park et al. (2024) replace the MLP layers in the Transformer by Mamba layers, showing benefits in simple tasks. These efforts are different from Jamba both in the particular way in which the SSM component is mixed with the attention one, and in the scale of implementation. Closest are perhaps H3 (Fu et al., 2022), a specially designed SSM that enables induction capabilities, and a generalization called Hyena (Poli et al., 2023a). The former proposed hybrid architecture that replaces the second and middle layers with self-attention, and was implemented with up to 2.7B parameters and 400B training tokens. However, its perfomance lags behind that of pure Mamba (Gu & Dao, 2023). Based on Hyena, StripedHyena (Poli et al., 2023b) intericaves attention and SSM layers in a 7B parameter model. However, lags behind the Attention-only Mistral-7B (Jiang et al., 2023). All of this renders Jamba the first production-gin -grade Attention-SSM. hybrid model." Scaling the hybrid Jamba architecture required overcoming several obstacles, dicsussed in Section
Jamba also includes MoE layers (Shazeer et al., 2017; Fedus et al., 2022), which allow increasing the model capacity (total number of available parameters) without increasing compute requirements (number of active parameters). MoF. is flexible approach that enables training extremely large models with strong performance (Jiang ct al., 2024). In Jamba, MoE is applied to some of the MLP layers (similar to Anthony et al. (2024), but they applied MoE to all MLPs). The more MoE layers, and the more experts in each MoE layer, the larger the total number of parameters. In contrast, the more experts we use at each forward pass, the more active parameters and the higher the compute.
To support cost-effective inference with large Jamba models, we introduce ExpertsInt8, a novel quantization technique that allows fitting lamba-1.5-Lart on single machine with 80GB GPUs when processing texts with 256K-token contexts without loss of quality (Section 4).
We evaluated the Jamba models on a wide range of benchmarks and found they performs comparably to saate-of-the-eee open-weight models of a similar or greater number of parameterts, while offering much better throughput. Notably, our models support a context length of 256K tokens- the longest supported context length for production-gret long-context evaluations, making them the only models with an effective length of 256K on the RULER benchmark (Hsich et al., 2024), while offering 10x reduction in KV cache memory as well as superior throughput and latency.
publicly available models. Jamba-1.5 1.5 models shine
We make the Jamba models publicly available under the Jamba Open Model License to support further study, experimentation, and optimization of this novel architecture by the community: 1.5 http:///hhhgggg.ee...///aa/aa/// 1amba-1.5-Large.
wot.aa/aaa////aa11111-----1 Jamba - 1.5-Mini wtt.:faccc.cccaaa.aeaeeeea
Jamba- 1. 5- Large
2 ARCHITECTURE
Jamba is hybrid decoder architecture that mixes Transformer layers (Vaswani et al., 2017) with Mamba layers (Gu & Dao, 2023), a recent state-space model (SSM) (Gu et al., 202 lb;a), in addition to mixture-offexppert (MoE) module (Shazeer et al., 2017; Fedus al., 2022). We call the combination of these three elements a Jamba block. See Figure for an illustration.
'A couple of other attempts to scale hybrid models were announced after an earlier release of Jamba (Lieber et al., 2024). confirming our design choices (Dao & Gu, 2024; Waleffe et al., 2024): we discuss this more below. Other recent attempts include Samba (Ren et al., 2024), which combines Mamba with sliding-window window attention, and YOCO (Sun al., 2024), which introduces a decnder-decoder architecture.
2

---

Published as a conference paper at ICLR 2025
MLP
Mamba MoE layer
RMSNorm
RMSNor
Mamba layer
Attention
Attention
Mamba MoE layer
RMSNorm
RMSNorm
Transformer layer
Transformer layer
Attention MoE layer
Mamba MoE layer
MLP
McE
RMSNorm
Mamba layer
Mamba MoE layer
Mamba
Mamba
RMSNorm
RMSNomm
Mamba layer
Mamba layer
Mamba MoE layer
(a) Jamba block
(b) Different types of layers
Figure 1: (a) single Jamba block. (b) Different types of layers. The implementation shown here is with = 8, m = 1 ratio of attention-to-Mabbee layers, and MoE applied every = 2 layers.
Combining Transformer, Mamba, and MoE elements allows flexibility in balancing among the sometimes conflicting objectives of low memory usage, high throughput, and high quality. In terms of memory usage, comparing the total size of the model parameters can be misleading. In an MoE model, the number of active parameters that participate any given forward step may much smaller than the total number of parameters. Another important consideration is the KV cache-the memory required store the attention keys and values the context-whice becomes limiting factor when scaling Transformer models to long contexts. Trading off attention layers for Mamba layers reduces the total size of the KV cache. Our architecture provides not only small number of active parameters but also an 8x smaller KV cache compared to vanilla Transformer.
In terms of throughput, with short sequences, attention operations take up a small fraction of the inference and training FLOOPS (Chowdhery et al., 2023). However, with long sequences, attention hogs most of the compute. In contrast, Mamba layers are more compute-efficient. Thus, increasing the ratio of Mamba layers improves throughput especially for long sequences.
Here a description of the main configuration, which provides improved performance and efficiency. Section C contains results from ablation experiments supporting the design choices.
The basic component is a Jamba block, which may be repeated sequence. Each block combines Mamba and Attention layers. Each such layer contains either attention Mamba module, followed by an MLP. The different possible types of layers are shown in Figure 1b.² A Jamba block contains layers, mixed at ratio of a m, meaning attention layers for every Mamba layers. "The figure shows potential Attention MoB layer, which Jamba does use, but future variants could.
3

---

Published as a conference paper at ICLR 2025
In Jamba, some of the MLPs may be replaced by MoE layers, which increase model capacity while keeping the active number of parameters, and thus the compute, small. The MoE may be applied to MLPs every layers. When using MoE, there are possible experts per layer; router chooses the top- K' experts at each token. In summary, the different degrees of freedom the Jamba architecture are:
t: The number of layers.
a : m: ratio of attention-to- -o-an------aaaa layers.
e: how often use MoE instead of single MLP.
nt total number of experts per layer.
K: number of top experts used at each token.
Given this design space, Jamba provides flexibility in preferring certain properties over others. For instance, increasing the ratio of Mamba layers (m) at the expense of attention layers (a), reduces the required memory for storing the KV cache. This reduces the overall memory footprint, which is espe- cially important for processing long sequences. Increasing the ratio of Mamba layers also improves throughput, especially at long sequences. However, decreasing might harm model capabilities.
Additionally, balancing v,, K, and e affects the relationship between active parameters and total available parameters. A larger increases model capacity at the expense of memory footprint, while a larger K increases the active parameter usage and the compute requirement. In contrast, larger decreases the model capacity, while decreasing both compute (when K >> 1) and memory requirements, and allowing for fewer communication dependencies (decreasing memory transfers as well as inter-GPU communication during expert- -arealll training and inference).
Jamba's 's implementation Mamba layers incorporate several normalizations that help stabilize train- ing at large scales. In particular, apply RMSNorm (Zhang & Sennrich, 2019) in the Mamba layers.
We experimented also with Mamba-2 2 (Dao Gu, 2024), a faster and improved version of Mamba, which was reported to outperform Mamba and Transformers separately. However, shown in Appendix C.2, we found that a hybrid architecture, the Mamba-1 Meamba-l-Auie -AAttentin combination works better than Mamba-2-Attentee
so we use Mamba-1 in the released models.
We found that with the Mamba layer, positional embeddings or mechanisms like RoPE (Su et al., 2024) are not necessary (Appendix C), and do not use any explicit positional information. Other architecture details are standard, and are detailed in Appendix B.
MODEL CONFIGURATION
The specific configurations in our implementation were chosen to fit in single 80GB GPU (for Jamba-1.5 M.5-M or a single 8x80GB GPU machine (for Jamba- 1.5--115-a) while achieving best performance in the of quality and throughput. It is based on the following main insights, which stem from ablations experiments (Appendix C):
Hybrid Attention-Men Maniia performs better than vanilla Attention Mamba models.
There is no substantial difference in performance between 1:3 and 1:7 ratios of Attention-to-
Mamba layers, while 1:7 is more compute-efficient.
Pure Mamba struggles with in-context learning (ICL) while the hybrid Iieenton-Mamble succeeds ICL and acquires induction heads (Olsson al., 2022).
MoE improves on hybrid Attention-Man Mamba at large scale.
Concretely, our implementation of Jamba uses sequence of Jamba blocks (4 blocks in L.5-Large). Each block has the following configuration:
Jamba-1.5-Ming 9 in
Number of layers ())
8
Total number of experts (n) # top experts used at each token (K)
16
Ratio of coontiiontooMambbe layers (a m) Use MoE instead of MLP every e layers
1 7
2
2
The MoE configuration was chosen to enable the model to fit in single 80GB GPU (or an 8-GPU ma- chine for Jamba-1.5-Lan L.s-aae, with appropriate quantization; Section 4.1), while including enough mem- ory for the inputs. Thus, and were balanced to have average of ~88 experts per layer. We also balanced n, K, and e to allow for high quality, while reducing both compute requirements and com- munication dependencies (memory transfers). Thus, other layer, and we have a total of experts, with two used at each token. These choices were inspired by prior work MoE (Zoph al., 2022; Clark et al., 2022) and verified in preliminary experiments.
replace the MLP module with MoE every
4

---

Published as a conference paper at ICLR 2025
Table Comparison of Jamba- Maabbaa-.5--Mii 1.5-Mini, Jamba-1.5-Large. and recent open models in terms of total available parameters, active parameters, and KV cache memory on long contexts. Jamba-1.5 1.5-Mini and Jamba-1.5- L.5- provide substantial reductions in the KV cache memory requirements.
Available params
Active params
KV cache (256K context, 16bit)
Mistral
7.2B
7.2B
32GB
Mixtral 8x7B
46.7B
12.9B
32GB
Llama-3.1 8B
8B
8B
32GB
Mixtral 8x22B
141B
39B
56GB
Mistral-Large-2
123B
123B
88GB
Llama-3.1 70B
70B
70B
80GB
Llama-3, 3.1 405B
405B
405B
252GB
Hamba-1.5-Minn
52B
12B
4GB
Camba-1.5-Large
398B
94B
9GB
Table I compares the Jamba-1. 1.5 models to publicly available models of similar sizes. Jamba- 1.5- -5-Mini has a similar number of active parameters as Mixtral 8x7B, while lamba-1.5-Larg. 's active parameter count is between Llama-3.1-700
and Mistral-Large-2. At the same time, both Jamba models have a much smaller KV cache memory usage (at 256K tokens) compared to all other models, with roughly an order of magnitude reduction compared their respective counterparts.
With these settings, and our specialized quantization (Section 4.1), Jamba- L.5-L can be served on single machine with 8 80GB GPUs with context lengths up to 256K tokens.
4 SERVING CONSIDERATIONS AND IMPROVEMENTS
We share few insights and improvements that allow for efficient serving of large Jamba models.
4.1 EXPERTSINTS QUANTIZATION
To support efficient serving of Jamba- L.rge, we developed new quantization technique, which we dub ExpertsInt8. We observe that over 85% of the model weights are the MoE layers, and over 90% are in MoE or MLP layers. We wish to quantize these weights while still enjoying the benefits of fast BF16 kernels. To do so, we quantize the MoB and MLP weights to INT8, save them in INT8, and dequantize them back to BF16 before the actual computation. Importantly, the dequantization step happens directly inside the fused moe kernel in vLLM (Kwon et al., 2023). In this way, the dequantization process adds negligible overhead, and even leads to improved latency over BF16. We have contributed our modified fused_moe kernel to vLL.M.
Our ExpertsInt8 method has several advantages. First, it is fast; quantization only takes few seconds at model loading. Second, unlike most other techniques in vLLM, it does not rely on calibration, which can take hours days and be unstable. Third, we can still BF16 hold large activations. Fourth, it is available to use on A 100 GPUs, unlike FP8, which is only available on H100. Finally, our quantization matches FPS in latency, while surpassing other quantization techniques, without a loss in quality (Appendix C.7).
Figure compares the latency with different quantization techniques using Jamba-1.5 1.5-Mini, Jamba- L.reL and two Mixtral models (8x78B and 8x22B). On H100 GPUs, ExpertsInt8 matches the latency of FP8. On A A100, where FP8 is unavailable, ExpertsInt8 shines, outperforming GPTQ (Frantar et al., 2023) by a large margin. Finally, ExpertsInt8 comes with no noticeable degradation in quality (Appendix C.7). Together with the advantages of ExpertsInt8 explained above, this makes an attractive quantization technique for serving large MoE models.
We attribute this to the kernel operating on relatively small blocks of weights and activations, which moves from GPU HBM to SRAM prior to performing the computations. In our implementation, the weights move from HBM to SRAM when they in int8, it takes less time their memory footprint is cut by half. httpps: : // thub. com/v1 1m- porpprojee pnpproeeettvvi 1 pu /7415
'Other MoE quantization attempts are Kim et al. (2023) and the recent work of Li al. (2024a), both focused on very low-bit guantization, which downgrades performance.
5

---

Published as a conference paper at ICLR 2025
Eapenees) FPI GPTO None
ExpartabriB GPTQ Narw
Expertsint8 FP8
Latenno (800)
Laaery 190
10
20
40
50
30
Darch eDe
Botch 92D
Batch size
(a) Jamba, 2xH100
(b) Jamba, 2xA100
(c) Jamba- L.5-aaeee, 8xH100
Exparishi8 FP8 None
Expertantt
Batch
(d) Mixtral-8x7B, 2xH100
(c) Mixtral-8x22B, 8xH100
Comparison different quantization techniques. End-to-end latency with 1024-token context and 128-token decoding. ExpertsInt8 performs similar to FP8, while being fast and simple to apply, and allowing BF16 activations, as well as applicable to A100 GPUs, where FP8 is unavailable.
Figure 2:
4.2 ACTIVATION LOSS
During pre-training, we found that certain activations, namely outputs of specific experts and the last Mamba layers, were gradually increasing in magnitude for certain input tokens, reaching values as high as 4 x 105. Although this did not hurt the pre-training itself, which was done in BF16 precision, the activations' magnitude could cause numerical issues during inference as some quantization libraries support only FP16 precision for activations, which has maximum range of 64K. To alleviate these concerns, we added an "Activation Loss" term, proportional to the mean-square of activations, with configurable a factor, which penalizes larger activation values. We found via experimentation that this auxilary loss has no affect on training even with a values up to at least 10-3 For 1amba-1.5-Large. we used a 10-3 5, which reduced the activations an acceptable range (2K-3K max). Moreover, adding this auxilary loss reduced the activations almost instantly, allowing be added only towards the end of training without any affect on training speed and quality.
To validate this approach, we ran our full evaluation suite on the model using FP16 activations and obtained the same results as the BF16 evaluations without any nans/overflows.
THROUGHPUT AND LATENCY ANALYSIS
Thanks to the hybrid architecture, Jamba-1. 1.5 models provide excellent throughput and latency. As Figures and 4 show, our models obtain much better latency and throughput than similarly-sized mod- els. Their advantage shines long contexts, with substantial gaps. Importantly, Jamba-1.5 1.5-Large runs efficiently even at long contexts, where the large LLaMA3-405B cannot run on the same hardware.
TRAINING
6.1 TRAINING INFRASTRUCTURE AND DATA
Jamba-I 1. Large was trained on NVIDIA H100 GPUs using our in-house proprietary framework, which includes FSDP, tensor parallelism, sequence parallelism, and expert parallelism. For the latter we have adapted MegaBlocks (Gale al., 2023).
6.2 TRAINING STAGES
The model was trained in three stages. During pre-training, it was first trained on an in-house dataset last updated in March 2024. Our pre-training dataset is mixture of publicly available web documents, code, books, scientific articles, and Wikipedia. Our pre- processecs pipeline includes parsing, quality filters, and deduplication. To make the best of publicly available data, developed our in-house parser, and used it to extract text and formatting. The exact data mixture was determined through various ablations. This stage included multilingual data with emphasis on the following

---

Published as a conference paper at ICLR 2025
Latency at different context lengths
Throughput at different context lengths
Jamba 1.5 Miri
Liama 3.1 8B
Mixtiul-Ba78
Matual Nemo 128
Jamba 1.5 Mini
Uama 3.188
Mixtral 0x78
Mistral Nemo 128
80
75
Throughput (US)
Latency (sit)
50
40
25
20
4098
8192
16394
32768
65633
131072
262144
0
50000
100000
150000
200000
250000
Total context
Total context
(a) Jamba-1 1.5- 5-Mini, end-to-ent latency.
(b) Jamba- 1.5ba--155.. 1.5-Mini, output tokens throughput.
Figure 3: Comparison of Jamba- L.5-Mini to other models in terms of latency and throughout. All measurements are on 2xA 100 80GB GPUs, with batch size and output length 512 tokens. Jamba- 1.5-Mini exhibits better latency, especially at large contexts, with only a slight reduction in throughput.
Latency at different context lengths
Throughput at different context lengths
1.5
Liama 3.1 70B
Mistral Large
Liama 4063
Jambe 1.5 Large
Laama 3.1 70B
Mistral Large
Liara 3.1 4058
200
50
150
Throughpue (U1s)
40
30
Lalency
100
20
50
10
0
4096
6192
16384
32768
65536
131072
282164
50000
100000
150000
200000
250000
Total context
Total context
(a) Jamba- 1.5 Laree and-to-end lateney.
(b) Jamba- 1.5 L.r-L, output tokens throughput
Figure 4: Comparing Jamba- 1. L.5eL to other models in terms of latency and throughout. All measurements are on 8xA 100 80GB GPUs, with batch size and output length 512 tokens. Jamba- 1.5-Large exhibits better latency, especially at long contexts, with only slight reduction in throughput.
Llama-3. 3-45B results stop at 64K as it cannot fit contexts over ~ 100K tokens on 8 80GB GPUs.
languages:
English, Spanish, French, Portueguse, Italian, Dutch, German, Arabic, and Hebrew. The pre-training context length was 4K for Jamba-1.5- M.5- and 8K for Lamba-1.5-LLagge It was then trained for short phase of mid-training with a high proportion of long documents to emphasize its long-range capabilities. Finally, the model went through post-training described in the next section.
6.3 POST-TRAINING
Our approach to post-training aims to achieve two objectives simultaneously: (i) provide the model with various skills and conversational capabilities; (ii) retain capabilities from pre-training and especially the long-context capabilities from mid-training. These two objectives are partly conflicting, since most of the available post-training datasets consist of relatively short examples.
Given these considerations, our post-training process involves supervised fine-tuning (Wei et al., 2022; Sanh et al., 2022) on high-quality conversational data, skill-specific data, and long-context data. Mix- ing these different types of data aims to retain long-context capabilities and acquire desired skills. As shown in the evaluations below, we find that our models perform very well in long-context evaluations.
In supervised fine-tuning, we use three kinds of datasets: general eestructionfollowinng code and math instructions, and specific skills. For the latter, we make heavy use of synthetic data, as common in recent foundation models (Dubey et al., 2024) and reflecting our approach for constructing structured data for building compound AI systems (Lenz et al., 2024). We develop multiple data synthesis pipelines, targeting different capabilities, which apply the following pattern: (i) Sample or generate prompts in a target distribution; (ii) Generate responses from language models; (iii) Filter rank responses by quality using automatic validation and scoring; and (iv) Post-edit to remove artifacts and fit desired formatting. We use different models, prompting, sampling, filtering, and editing for different pipelines that compose the final data mixes. Appendix D provides more details and examples for synthetic data generation processes, well as several observations from our post-training work.
7

---

Published as a conference paper at ICLR 2025
7 EVALUATION
This section performs a comprehensive evaluation of Jamba-1.5 models. First, we report results on long-context settings, where the advantages of the Jamba arhcitecture most pronounced. Then, we show that our models are competitive in on standard academic benchmarks, as well as chatbot benchmarks. We conclude with small multilingual evaluation.
We mainly compare with recent open-weight models of the same size range: Llama-3.1 70B and Mistral-Large-22 Laall--arrre--12333 2-12BB when comparing with Jamba-1.55 1a5ba--.--Lage. Llama-3.1-8 and Gemma-2-9B when comparing with Camba-1.5-Minn
7.1
LONG-CONTEXT EVALUATIONS
The benefits of Jamba models shine in long contexts, where they offer superior throughput (Section 5). This section evaluates them on synthetic and naturalistic long-context benchmarks.
7.1.1 RULER
We evaluate on the RULER benchmark (Hsich et al., 2024), set of synthetic tasks aimed long-context capabilities of language models. RULER includes 8 variants of -aallliin---ayttck retrieval tasks (Kamradt, 2023; Mohtashami & Jaggi, 2023; Li al., 2023; Liu et al., 2023), including multiple 'needles' (Arora et al.). It also has one variable tracking task, two aggregation tasks where one should return the most common words, and two question-answering tasks, where paragraphs cotraining answers from naturalistic datasets (Rajpurkar et al., 2018; Yang et al., 2018) are inserted into random paragraphs to simulate long contexts.
The results for key proprictary and open-weight models shown in Table 2; full results given in Appendix E.I. Among all publicly available and proprietary models, Jamba-1.5-Mins and Jamba-1.5-Larg are the only ones with a confirmed effective length of 256K tokens.
Table 2: Comparison of Jamba-1.5 1.5 models with other publicly available and proprietary models on the RULER benchmark. Results for other models are from the RULER Github. Jamba-1.5 models the only ones with confirmed effective length of 256K tokens.
Claimed Length
Effective Length
4k
8k
16k
32k
64k
128k
256k
Avg-
Jamba-1.5-Large
256K
256K
96.7
96.6
96.4
96.0
95.4
95.1
93.9
95.7
Jamba-1.5-Mini
256K
256K
95.7
95.2
94.7
93.8
92.7
89.8
86.1
92.6
Gemini-1.5-pro
1M
>128
96.7
95.8
96
95.9
95.9
94.4
91.4
GPT-4-1 106-preview
128K
64K
96.6
96.3
95.2
93.2
87
81.2
91.6
Llama 3.1 70B
128K
64K
96.5
95.8
94.8
88.4
66.6
89.6
Llama 3.1 8B
128K
32K
95.5
93.8
91.6
87.4
84.7
77
88.3
Mistral Large 2
128K
32K
96.2
96.1
95.1
93
78.8
23.7
80.5
7.1.2
BBBENC
Next we evaluate on BcBENCH (Zhang et al., 2024), which has an average length of 100K tokens. We focus two English tasks on understanding long novels: question answering (EN, (EN.QA) and multiple-choice question answering (EN.MC). As Table shows, Jamba-1.3 1.5 models perform very well in this case, outperforming similarly sized Llama-3.1 and Mistral- models. (We do not report results with Gemma-2 9B due to its short context window 8K.)
Table 3: Jamba-1.5 1.5 models outperform similarly sized Llama-3 and Mistral Mistral-Large-2 models in long-context evaluations. evaluation run by us.
Jamba-1.5 Mini
Llama-3.1 8B
Gemma-2 9B
Jamba-1.5 Large
Llama-3.1 70B
Mistral-L-2 123B
Benchmark
EN.MC
76.9
65.1
80.4
78.2
36.9f
EN.QA
40.6
27.1
34.9
36.7
15.4
'In the official results, Gemini-pro has effective length >128K but RULER did confirm longer contexts.
8

---

Published as a conference paper at ICLR 2025
7.2 ACADEMIC BENCHMARKS
While not our main focus, we report results range of standard academic benchmarks: MMLU (Hendrycks et al., 2020), MMLU-Pro (Wang ct al., 2024), GPQA (Rein ct al., 2023), ARC-Challenge (Clark et al., 2018), BBH (Suzgun al., 2023), and HvmanEval (Chen et al., 2021). We also evaluate on the IFEval instruction following dataset (Zhou et al., 2023) and the BFCL v1 function calling dataset (Yan et al., 2024). Finally, report safety evaluations RealToxicity (Gehman al., 2020) and TruthfulQA (Lin et al., 2022).
Table 4 compares Jamba-1.5 1.5-Large to several publicly available models at similar sizes. All results are either taken from official sources or evaluated by us, as indicated in the table.? We observe that the Jamba-1.5 models perform similarly recent state-of-the-art publicly available models on standard academic benchmarks, including knowledge, reasoning, instruction following and function calling capabilities. We also observe similar safety metries those reported in the literature.
Importantly, the Jamba-l. 1.5 models achieve these results while providing much better throughput and latency, discussed above.
Table 4: Jamba-1.3 models obtain similar performance to similarly sized models while enjoying better throughput and latency. evaluation run by us. reported the HuggingFace OpenLLM leaderboard. Lacking function calling capabilities. Strict/flexible evaluation.
Jamba-1.5 Mini
Llama-3.1
Gemma-2
Jamba-1.5 Large
Llama-3.1
Mistral-L-2
Benchmark
Metric
8B
9B
70B
123B
MMLU
5-shot
69.7
69.4
71.3
80.0
83.6
82,51
MMLU Pro
5-shot
39.8
38.0P
39.0
48.3
53.0"
54.2
GPQA
O-shot
32.3
27.0"
36.0
36.9
36.0
40.71
ARC-C
O-shot
85.7
83.4
68.4
93.0
94.8
65.0
BBH
3-shot
53.4
51.0%
60.0
65.5
69
70.8
HumanEval
pass @
62.8
72.6
40.2
71.3
80.5
92
GSM8K
5-shot
75.8
75.2/83.7*
68.6
87.0
71,5/94.2*
91.0
IFEval
O-shot
75.8
80.4
74.3
81.5
87.5
87.81
BECL
O-shot
80.7
76.1
85.5
84.8
85.1
Realiixicyty TruthfulQA
avg lox
8.1
6.7
0-shot
54.1
51.5t
50.2
58.3
60.71
50.41
7.3 CHATBOT EVALUATIONS
In this section we evaluate the Jamba-1.5 models on two chatbot scenarios: Arena-Hard (Li et al., 2024b), set of 500 challenging user queries that uses GPT4-Turbo as a judge, and WildBench (Lin et al., 2024), which uses GPT4-Turbo as a judge with length bias mitigation. We then report results from live interactions with users the Chatbot Arena. As Table 5 shows, Jamba-1.5 models obtain excellent results in these evaluations, with Comba-1.5--art surpassing Llama-3.1 70B, but somewhat trailing behind Mistral-Large 123B, which has about 30% more active parameters.
Table 5: Comparison of Jamba-1 1.5 models to similarly sized models chatbot benchmarks. Jamba- 1.5 models obtain similar performance with better throughput and latency. Tevaluation run by us.
Jamba-1.5 Mini
Llama-3.1
Gemma-2
Jamba-1.5 Large
Llama-3.1 70B
Mistral-L--2
Benchmark
8B
9B
123B
Arena-Hard
46.1
21.3
43.2
65.4
55.7
70.4
Wild-Bench
42.4
33.6t
42.7
48.5
49.8t
56.3
II two cases we failed to obtain good results: Mistral-Large- fails to obtain good scores on ARC- C despite multiple attempts. Llama-3.1 models perform poorly GSM8K with the standard strict evaluation mode, so we also report for them flexible evaluation, which allows higher results.

---

Published as a conference paper at ICLR 2025
Figure provides ELO scores from the Chatbot Arena (as of September 10, 2024), where humans make live pairwise comparisons of models. Jamba models perform comparably to state-of-the-aee open-weight models of similar size, while enjoying the efficiency benefits of the Jamba architecture.
1300
1275
1250
Rating (ELO)
1225
1200
1175
1150
1125
1100
LLaMA-3.1-40BB
Mistral-Large Large-
LLaMA-3.1-700 1-708
Jamba-1. mamba-1.5-Larg. 5-Large
Gemma-2-277
Gemma-2-90 2-98
Jamba-1 Mini
LLaMA-3.1 1-88
Mixtral- Mixtral-8x228 -x228
Model
Figure 5: Chatbot Arena ELO scores of Jamba with gpen-weight models of similar size. Jamba models obtain good performance while enjoying better throughput and latency.
7.4 MUL.TH.NNGUA CAPABILITIES
We perform basic evaluation of Jamba-1.5 abilities in non-English languages on the multilingual MMLU dataset (Lai et al., 2023), available via the LM Evaluation Harness (Gao et al., 2024). As Table 6 shows, Jamba-I 1.aobaa-.---iii Mini performs similarly better than its points of comparison. Jamba-1.5 Large is slightly behind its comparable models, but still exhibits good multilingual capabilities.
Table 6: Comparison of Jamba- 1.5 with other models on the multilingual MMLU dataset.
Spanish
Portuguese
French
German
Arabie
Italian
Dutch
Avg
Jamba-1.5-Min
66.3
66.7
65.9
63.8
57.3
65.1
65.0
64.30
Llama-3.1-89
59.5
59.1
59.5
57.2
46.9
58.4
57.2
56.83
Gemma-9B
66.0
59.9
66.7
64.3
55.9
65.8
64.8
63.34
Jamba-1 1.5- L.5-Larree
75.5
75.5
75.8
73.9
67.1
75.2
74.6
73.94
Llama-3.1-90 3.1-0BB
79.5
79.4
79.1
78.4
70.4
79.1
78.4
77.76
Mistral-Large-2
78.7
78.4
78.4
77.4
65.9
78.3
76.2
76.19
CONCLUSION
We presented Jamba, a novel architecture that combines Attention and Mamba layers, with MoE modules. and an open implementation of in two sizes: 1.5ma-i.5-Mimi with 12B active and 52B total parameters and 1imba--.55-Larg with 94B active and 398B total parameters. We showed how Jamba offers flexibility for balancing performance and memory needs, while maintaining a high throughput. We experimented with several design choices such as the ratio of iicention-to-Mamble layers and discussed some discoveries made during the development process, which will inform future work on hybrid models. We also developed ExpertsInt8, novel quantization to support cost-effective serving such models at large scale, even on a single machine with long contexts. Both models achieve excellent performance in long-context evaluations, academic benchmarks, and chatbot evaluations, while offering improved latency and throughput, especially for long contexts. We release the model weights for use by the community in hopes that others build this technology.
10

---

Published as a conference paper at ICLR 2025
CONTRIBUTIONS
Pre- and Post- Trst-Training
Data
Alan Arazi
Ben Aviram
Barak Lenz"
Dor Zimberg
Chen Almagor
Ido Blass
Dan Padnos"
Ohad Leshno
Daniel Gissin"
Rom Gilad
Daniel Jannai
Tom Ben Gal
Dor Muhlgay
Edden M Gerber
Evaluation
Erez Safahi
Clara Fridman
Gal Cohen
Julie Fadlon
Gal Shachaf
Maria Rozman
Hofit Bata
Naama Gidron
Inbal Magar
Ro'i Belson
Itay Dalmedigos
Tal Ness
Jhonathan Osin
Matan Danos
Michael Gokhman
Project & Product Management
Nir Ratner
Or Dagan
Noam Gat
Roi Cohen
Noam Rozen
Shaked Meirom"
Omer Antverg
Tal Delbari
Omri Abend
Yoav Shoham
Opher Lieber"
Orit Cohavi
Raz Alon
Shai Shalev-Shwartz
Shaked Meirom
Tom Braude
Uriya Pumerantz
Yonatan Belinkov
Yuval Globerson
Yuval Peleg Levy
Serving & Infrastructure
Amir Bergman
Avshalom Manevich
Barak Peleg
Elad Dolev
Eran Krakovsky
Erez. Schwartz
Haim Rozenblum
Mor Zusman
Oded Fried
Roman Glozman
Shahar Lev
Tomer Asida
Yehoshua Cohen
Projecct leads
11

---

Published as a conference paper at ICLR 2025
REFERENCES
Glaive AL Glaive function calling v2. http:////h/pgggae....../aaaaaa/a www.--nnnnnnnniiiieeee
Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of mamba models. arXiv preprint arXiv:2403.01590, 2024.
Quentin Anthony, Yury Tokpanov, Paolo Glorioso, and Beren Millidge. BlackMamba: Mixture of experts for state-space models. arXiv preprint arXiv:2402.01771, 2024.
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Re. Zoology: Measuring and improving recall in efficient language models. In The Twelfth International Conference on Learning Representations.
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, et al. Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405.1478. 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint vrXiv:2107.03374 2021.
Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 2174-2184, 2018
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference on machine learning, pp. 4057-4086 PMLR, 2022.
Christopher Clark, Kenton Lee, Ming-Wei Chang. Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ:
Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1924-2936, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the A12 reasoning challenge. arXiv preprint arXiv: 1803.08033.555 2018.
Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. ArXiv, abs/2405.21060, 2024. URL https://////... :/app. semanticscholart wwwpchhllar..g//orrreoeooo www./ooposooo--o-oooooooo 12:711997222
Abhimanyu Dubey, Abhinay Jauhri, Abhinay Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Jeffrey Elman. Finding structure in time. Cognitive Science, 14:179-211, 1990. URL https: //api. i cscholar Cr.
Hugging Face. Open LLM leaderboard.
attps://aagginffaee.cc//ppapp
HuggingFaceR4/ 1m 1aateeen_mm___eeee_eeeee leadebboart 2024.
12

---

Published as a conference paper at ICLR 2025
Yassir Fathullah, Chunyang Wu, Yuan Shangguan, Junteng Jia, Wenhan Xiong, Jay Mahadeokar, Chunxi Liu, Yangyang Shi, Ozlem Kalinli, Mike Seltzer, and Mark J. F. Gales. Multi-head state space model for speech recognition. In Proceedings of INTERSPEECH 2023, pp. 241-245, 2023. doi: 10.2 00021447//neeespeech.2.0. 2023: 1036.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39,
2022.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023, URL https: aepenreviees net/ forum cid.ttddddttntntnmm EwxS.
Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re. Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh International Conference Learning Representations, 2022.
Philip Gage, A new algorithm for data compression. The C Users Journal, 12(2):23-38, 1994.
Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. MegaBlocks: Training with Sfxxture-of-Exerrtt Proceedings of Machine Learning and Systems, 5, 2023.
Efficient Sparse
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds. Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot language model evaluation, 07 2024. URL hwwpt:tps://om zenooo.ooooomm org/
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxici- typrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp.
2020.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arX10:2312.00752, 2023.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2021a.
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with lincar state space layers. Advances in neural information processing systems, 34:572-585, 2021b.
Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1382-1390, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt, Measuring massive multitask language understanding. In International Conference on Learning Representationss 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997. URL htt.:p://appl.smmtttccccootl
www.stiisceooeeeeeeoeeeo OrggCorpusID:11000
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What's 's the real context size of your long-context language models? The 1st Conference on Language Modeling (COLM), 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401. 040088, 2024.
13

---

Published as a conference paper at ICLR 2025
Greg Kamradt. Needle in haystack pressure testing llms. gkamradt CoaaraattLLAAN LLLMtest....tt. Maw.iieeetteNeeeeeAnlleee Ntw.eeenAlaaystacckaaaaa 2023.
https://github.cem
Young Jin Kim, Raffy Fahim, and Hany Hassan Awadalla. Mixture of quantized experts (moqe): Complementary effect of low-bit quantization and robustness. arXiv preprint arXiv:2310.02410,
2023.
Tomas Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gabor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. Okapi: Instruction tuned large language models in multiple languages with reinforcement learning from human feedback. Yansong Feng and Els Lefever (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 318-327, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ URL https: htt/////aaaattttte
w1/2023.emnlp-demm
org/ 2022
demo 28.
Barak Lenz, Raz Alon, Noam Rozen, Omri Abend, Yonatan Belinkov, Kevin Leyton- and Yoav Shoham. Structured data as a key element of ai systems: test case on table understanding. In Compound AI Systems Workshop, 2024.
Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can context length of open-source Ilms truly promise? In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
Pingzhi Li, Xiaolong Jin, Yu Cheng, and Tianlong Chen. Examining post-training quantization for mixture-of-experts: benchmark. arXiv preprint MxXi::2406.08/555 2024a.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024b. URL
https:://mmt /ttp:///mmmssss//// org/ blog/ 2024 4 04 1 1e--arrnaaaaaaa
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer- mamba language model. arXiv preprint arXiv:2403. 2024.
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Joseph Le Bras, and Yejin Choi. Wildbench: Benchmarking Iims with challenging tasks from real users in the wild. ArXiv, abs/2406.04770, 2024. URL
https:/ / /api..app///// semant icscholarr ar. orgg/Corpun IDr 2700377771.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3214-3252, 2022.
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2023. URL https: //api. Semanticschole
otg/ http/oorpuslii ID: 22:2259350 666...6
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang. Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguisties: Human language technologies, pp. 142-150, 201
14

---

Published as a conference paper at ICLR 2025
Sabrina J Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Lee, Benoît Sagot, et al. Between words and characters: A brief history of open-vocabulary modeling and tokenization in NLP. arXiv preprint arXtv:2112.10508, 2021.
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cemocký, and Sanjeev Khudanpur. Re- current neural network based language model. Interspeech, 2010. URL https://api. semanti cschol ar. .org/ wwtp///ooo/sor..o//oooommm Corpustt: 1D:
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048-11064, 2022.
Amirkeivan Mohtashami and Martin Jaggi. Random-access infinite context length for transformers. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL https: // sepenreview.nete wwt.:nruuteetateeeooooo.oo..ouat
forum? -eeedddddeeeenn deteiiiiwwewenww
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11899 2022.
Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kang- wook Lee, and Dimitris Papailiopoulos. Can mamba learn how to learn? comparative study on in-context learning tasks. arXiv preprint arXiv:2402.0 04248, 2024.
Jonathan Pilault, Mahan Fathi, Orhan Firat, Christopher Pal, Pierre-Luc Bacon, and Ross Goroshin. Block-state transformers. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Maciej Pióro, Kamil Ciebiera, Krystian Król, Jan Ludziejewski, and Sebastian Jaszczur. MoE-Mamba: Efficient selective state space models with mixture of experts. arXiv preprint arXiv:2401.0400 2024.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pp. 28043-28075 PMLR, 2023a.
Michael Poli, Jue Wang. Stefano Massaroli, Jeffrey Quesnelle, Ryan Carlow, Eric Nguyen, and Armin Thomas. StripedHyena: Moving Beyond Transformers with Hybrid Signal Processing Models.
https: /ttpp:://gtthnn comp wtt/tcooo/t//teeeecce htt.:/tttceccocpeeeeeeeeeeeeeeee wtwp://eeees.t.iieeeeeee...e 2023b.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, 2018.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. arXiv preprini
2023.
Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. arXiv preprint arXiv:2406.07552.. 2024.
Keisuke Sakaguchi, Ronan Le Bras. Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8732-8740, 2020.
15

---

Published as a conference paper at ICLR 2025
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, et al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
George Saon, Ankit Gupta, and Xiaodong Cui. Diagonal state space augmented transformers for speech recognition. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1725-1725, 2016.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gaaste mixture-of-experts layer. In International Conference Learning Representations, 2017.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: ses127763, 2024.
Yutao Sun, Dong, Yi Zhu, Shaohan Huang, Wenhui Wang. Shuming Ma. Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. arXiv preprint arXiv:2405.05254 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, et al. Challenging BIG-Bench tasks and whether chain-of- can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 13003-13051, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXi::2307.09288 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Anand Korthikanti. Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi, and Bryan Catanzaro. An empirical study of mamba- based language models. ArXiv, abss2406600787,, 2024. URL https: //api. semant wes.httiscccarrr org/ Corpus 27039:2:7225555
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.015: 405401 2024.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing, 2018. URL https:/ //api semanticscool ar. wttp///rrpss1::222222222
16

---

Published as a conference paper at ICLR 2025
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang. Zhiyuan Liu, and Maosong Sun. oobench: Extending long con- text evaluation beyond 100k tokens. ArXiv, abs/2402. 13718, 2024. URL https: /api.
semant ionticclcclaa...... :ttp:arrorr//oorr.a10007777
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-folloiingg evaluation for large language models. arXiv preprint vrXiv:2311.07911, 2023.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. ST-MoE: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906, 2022.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Denis Charles, Eren Manavoglu, Tuo Zhao, and Jianfeng Gao. Efficient long sequence modeling via state space augmented transformer. arXiv preprint 1rXiv:2212.08136. 2022.
17

---

Published as a conference paper at ICLR 2025
A LIMITATIONS
One limitation of this work is that some details about the training process not disclosed, such as the total training budget in terms of tokens and compute, and the data sources for pre-training and post-training. However, we provide ablations of the Jamba architecture with 1.3B/7B parameter models trained for 250B/50B tokens with the exact same data and compute budget, which reveal insights about the novel aspects of the Jamba architecture a fairly large scale.
B ARCHITECTURE AND IMPLEMENTATION DETAILS
The main features of the Jamba architecture are explained in Section 2, with ablations supporting our design choices in Appendix
Other details are standard in recent language models. Concretely, we use grouped -upry attention (GQA), SwiGLU activation function (Shazeer, 2020; Chowdhery et al., 2023; Touvron et al., 2023), load-balancing loss to balance the routing decisions in the MoE and router z-loss to avoid large router logits (Fedus al., 2022: Zoph et al., 2022). The vocabulary size is 64K. The tokenizer trained with SentencePiece BPE (Gage, 1994; Sennrich et al., 2016; Mielke et al., 2021) and each digit is a separate token (Chowdhery al., 2023). We also remove the dummy space used in Llama and Mistral tokenizers for more consistent and reversible tokenization.
We have made sure to use the same evaluation setup all models compared this work, including prompts, official splits, and number of shots. When evaluating Jamba and self-reporting results of other models, we always used the official repositories. We mainly used the LM Evaluation Haresss (Biderman al., 2024) whenever possible. Regarding evaluations not in LM Harness: For HumanEval, BFCL, and IFEval, we used the official dataset and metrics, as mentioned in their official repos. For RealToxicity, we used the official dataset and used their API for judgment.
To evaluate the efficiency of Mamba and Attention layers during pre-training, we measured the models FLOPPs utilization metric (MFU), proposed by Chowdhery al. (2023). In our measurements, the MFU with vanilla attention drops from 60% to 20% at 32K token length, while the Mamba MFU remains close to 60%. In longer contexts, the difference would be even larger.
C ABLATIONS AND INSIGHTS
This section discusses ablation experiments we ran for different design choices in our implementation of the Jamba architecture. First we show the benefit of combining attention and Mamba layers, at which ratio they should be combined, and how to interleave them." We investigate cases where pure Mamba fails, suggesting that it struggles to develop in-context learning capabilities, while the Attention-Mambl hybrid exhibits in-context learning similar to vanilla Transformers. Then we show the benefit of adding MoE on top of a hybrid Attention-Mambee model. Finally, we share two additional learnings that found useful: explicit positional information is not needed in Jamba, and Mamba layers necessitate special normalization to stabilize training at large scale."
For these ablations, we report the following measures, which exhibit informative performance even at small data or model scale. (In Appendix C.8 we report ablations of the architecture in terms of latency.)
Academic benchmarks: HellaSwag (10-shot) (Zellers et al., 2019), WinoGrande (5-shot) (Sak- aguchi et al., 2020), Natural Questions closed-book (NQ; 5-shot) (Kwiatkowski et al., 2019), ARC-Challenge (25-shot) O. BoolQ (10-shots) (Clark al., 2019), IMDB (Maas et al., 2011), and NarrativeQA (NQA; Kocisky et al., 2018).
HuggingFace OpenLLM leaderboard (OLLM) (Face, 2024): a summary statistic of several datasets. We report results with our reproduction.
Perplexity evaluations: we report log-prob (per byte) texts from three domains: C4, Books, and code.
"Our design choices were later confirmed hy independent experiments of others (Dao & Gu, 2024; Waleffe ct al,, 2024), which followed an carlier Jamba release. In all the ablations, "pure Mamba" refers to models with Mamba layers interleaved with MI.P layers.
18

---

Published as a conference paper at ICLR 2025
C.I BENEFITS OF COMBINING ATTENTION AND MAMBA
We first investigate the ratio of Atrention-to-Manin layers (a m.) with 1.3B parameters models trained for 250B tokens. As Table 7 shows, the hybrid Jamba model outperforms the pure attention Mamba models. The ratio of attention-to-Manii performance difference. Figure 6 shows the training loss these models, where Jamba exhibits improved loss during training. Given that 1:7 ratio is more compute-efficie and shows similar performance, we opt for it in our larger-scale experiments.
layers may be 1:3 1:7 with virtually
Table 7: Results on academic benchmarks and log-probability evaluations showing an improved performance of hybrid Altention-Mamba (no MoE) compared to vanilla Attention and Mamba models. There is no substantial difference between 1:3 and 1:7 ratios of Attention-to-Mant to-Mamba layers. Models 1.3B parameters, trained for 250B tokens.
log-prob
Hella Swag
Wino Grande
OLLMM
NQ
BoolQ
NQA
ARC-C
IMDB
C4
Books
Code
Attention
36.4
62.4
59.6
14.5
60.9
34.6
84.1
-0.543
-0.659
-0.331
Mamba
36.1
62.6
59.4
14.5
61.1
34.1
48.8
-0,543
-0.661
-0.334
Hybrid, 1:
37.2
65.1
61.7
60.6
44.3
36.8
90.1
-0.333
-0.649
-0.321
Hybrid, 1:
37.2
65.1
61.7
16.0
64.4
41.9
34.8
90.5
-0.533
+0.650
-0.321
Attention-Manin Ratio
attention:
only
only attention
1:3 attention ratio
3.4
3.2
3
2.8
2.4
22
Step
10k 20k 30k
40k
50k
601
Figure 6: Training loss curves for pure Attention, pure Mamba, and Attention Mamiona hybrids (no MoE), with ratios m of 1:3 and 1:7. All models are 1.3B parameters. The two hybrids achieve better loss throughout this training run, without any noticeable difference between the different Attention/Mamba ratios.
Next, we compare performance of vanilla Transformer, vanilla Mamba, and Attention-Mamba hybrid models, at 7B model size, after training on 50B tokens. As Table 8 shows, the pure Mamba model is quite competitive, but lags slightly behind pure Attention. The hybrid Attention-Mant Mamba (without MoE) outperforms the pure models while obtaining better throughput than vanilla Transformers (Section 5).
Table 8: Results on academic benchmarks and log-prob evaluations, comparing pure Attention, pure Mamba, and Attention Mamba hybrid (a m = 1 7, no MoE). Models are 7B parameters, trained for 50B tokens.
log rre-prob
Hella Swag
Wino Grande
OLLM
NQ
BoolQ
NQA
ARC-C
IMDB
C4
Books
Code
Attention
36.1
60.4
59,7
13.7
60.6
40.0
32.4
87.6
-0.555
.0.666
-0.347
Mamba
35.3
60.2
55.8
14,0
52.0
19.4
33.8
64.0
-0.554
-0.667
-0.355
Hybrid
36.6
62.5
58.8
15.4
59.4
38.3
35.0
87.1
-0.547
40.658
-0.340
19

---

Published as a conference paper at ICLR 2025
Figure shows the training loss of the three architectures. While the pure Transformer and Mamba
models have a similar convergence, the hybrid Jamba (no MoE) has lower loss throughout this run.
Architectures Comparison
Transformer
Mamba
Jamba
3.2
2.6
22
2k
4k
6k
8k
10k
Figure 7: Training loss curves for pure Attention, pure Mamba, and an Attention- Attention-Mamaa hybrid (no MoE). All models 7B parameters. The hybrid achieves better loss throughout this training run.
C.2 MAMBA-1 VS. MAMBA-2
We experimented also with Mamba-2 (Dao Gu, 2024), a faster and improved version of Mamba, which was reported to outperform Mamba and Transformers separately. However, Figure shows, we found that in a hybrid architecture, the Mamba-l-Attention combination works better than Mamba-2-Attention. (We also found the hybrid architecture to outperform pure Mamba-2.) We hypothesize this is because some of the advantages of Mamba-2 over Mamba-1, in particular the ability to much larger state size, are less significant when have full attention layers interleaved between the Mamba layers, as they can pool information from the entire context.
Training Loss
Mamba-1
Mamba-2
Mamba-2-Ai
Mamba-1-Ate
2.55
135
350M.
Training Loss
Mamba-2-Attn
Mamba-1-Atth
2.5
2/5
2.4
20k
(b) 1.3B.
Figure 8: Comparison Mamba-1. Mamba-2, Mamba- 1-Attention, and Mamba-2-Attenin 2-ttentti on models trained on 100B tokens. While Mamba-2 outperforms Mamba-1 without attention, the hybrid Mamba-I-Are l-Attention performs better.
20

---

Published as a conference paper at ICLR 2025
C.3 WHY DOES THE COMBINATION WORK?
The pure Mamba model showed fairly good results in most tasks early on, including in general perplexity evaluations. However, it performed substantially worse than the pure Attention model in three common benchmark tasks: LMDB (Maas et al., 2011), QuAC (Choi et al., 2018), and NarrativeQA (Kocisky et al., 2018). In contrast, the hybrid Attention-Mambe performed similarly to the Attention model these datasets. Table 9 shows the results for 1.3B models after 250B tokens.
Table 9: Mamba performs poorly on certain datasets, while the Attention-Mamba hybrid performs on par with the Attention model.
IMDB
QuAC
NarrativeQA
Attention
84.1
27.9
45.8
Mamba
48.8
20.2
27.7
Attention-Mamba 90.9
26.6
43.7
Looking into these results further, we found out that the pure Mamba model often does not follow the correct format. For instance, in the IMDB dataset, answer choices are "Positive" or "Negative". While the Attention model adheres to this format, the pure Mamba model often produces other answers, such as "Very Good", "Very Positive", "Funny", "Bad", "Poor", and "3/10". While these may be considered correct answers, the difficulty of Mamba to adhere the format suggests a potential problem. Indeed, to perform successful in-context learning, it is important for models to capture the input-output format (Min et al., 2022). The hybrid Attention Mamba model follows the format successfully, just like the pure Attention model.
We hypothesize that this phenomenon points to a limitation of SSMs a potential difficulty in in-context learning (ICL). Indeed, the ability to perform ICL has been linked to the emergence of so- called induction heads in Transformer language models during training, which perform approximate copying operations that are supportive of ICL (Olsson et al., 2022). We conjecture that the lack of an attention mechanism in the pure Mamba model makes it difficult for it to learn in-context. While Mamba may learn copy and perform simple ICL when explicitly trained to do (Gu & Dao, 2023; Park et al., 2024), it is not clear if ICI. is an emergent capability in SSM as is typical of Transformer Cecentio-Maabll model does perform successful ICL., even when models. In contrast, the hybrid
only 1 out of 8 layers is an Attention one.
As anecdotal evidence of an emergent induction mechanism, we visualize in Figure the attention of an example head from a 1.3B Attention-Mamba hybrid model (no MoE), on an IMDB example where the pure Mamba failed and the hybrid succeeded. Clearly, the attention from the last token ("t") is focused on the labels from the few-shot examples. We have found 12 such heads in our hybrid model, in all three attention layers (which correspond to layers 4, 12, 20 in the model).
Future work can further investigate the emergence of ICL in hybrid models at large scale. Finally, recent work has attempted to extract attention-like like scores from state-space models like Mamba (Ali et al., 2024), which opens another direction to search for induction capabilities in state-space models.
C.4 THE EFFECT OF MIXTURE-OF-ER EXPERRSS (MoE)
Recent work has shown that MoE improves Transformer language models while keeping compute manageable (Jiang et al., 2024).* However, it is not clear if MoE integrates well with state-space models a large scale, and specifically with our hybrid Attention- architecture. Indeed, Table 10 shows that MoE improves the performance of the hybrid Attention-Mambs architecture at large scale (7B parameters trained on 50B tokens). The MoE variant has 16 total experts, K 2 experts used at each token, and MoE is applied every
2 layers, as described in Section 3.
In terms of throughput/latency. the effects of MoE are linear with sequence length, as expected. Interestingly, compared to regular MLP layers, MoE has only about a 1.3-1.6x effect on through- put/latency, even though two experts are used (meaning 2x active parameters). This indicates that
*There is also initial evidence that MoE helps Mamba layers, alheit at small model and data scale (Pióro al., 2024).
21

---

Published as a conference paper at ICLR 2025
[B0S| _.assage __oo _call _this fiimm _ disaster _will be _an __nderstttteeett _don _were _oo _eegin _I _havee _questions _though _andd _lots _of then _I wouud [...]
to _eentiin _ few _of _course _This film _wa _ _painful _experience for _ee _and _I_ _advise Leveryone _t _skip _it _by _all _means _necessary _and _possible Bollywood _shou'ld be _terribly _ashamed _of this _iidd _ef _filn naking <[newline|> _Sent iment [Negative 
 </newline|> Passage _woke _and _it _was _beautiful _day __the _snn was _shining the _irrd _eere _singing _and _i fanc _getting _ _ovie _something [...]
_i _nothing _else _KK iit _is _nothing _else Enjoy but _a little _advice -- _beefore pressinn _he _play _bttonn _on _your _DDD _player _throw _it _out _of tte _window <nnewline]> _ent inent Negative 
 
 Passage _This _ _an _excellent Janes Bond _novie _Although _it _is _oo _part _of the _riiginal _and mooe famous _eeries _and
_standalone _file _it _very _one _En tic ing _Sean Conner [...]
different _version _of _hunder ball _updated with _nwwer _technology Regardless _of the repeated theme there _are suffffcinnt _ifferenccs _o _aaee_ it _ost _entertaining _wlll _watch this _one _frequently 
 __entt
_Positive 


 Passage I _am fan _of _Ed _Harris _work _and _I _really _hadd _high expectations _about _this _film _HHaving _0 _good _actors _as HHrris _Von __y [...]
mavies for the WHS _era Lof the 90 Whatever _the reason _ws _toough _ovie _was_ _very bad _choice for _anyone _nnvolved 
 Sent iment
this
Figure 9: Example induction head (H3, first attention layer) from hybrid Attention Mambba model. Highlighted words reflect strong attention from the last token, ":",, just before the model is about to predict the label. We see that the attention focused on label tokens from the few-shot examples.
MLP/MoE layers are I/O bounded and since two experts can be computed independently. batching and fusing save time. In general, increasing MLP compute via MoE helps reduce the relative effect of Attention layers, which are quadratic in sequence length. In Jamba-1.55 1.5-Min,,. Attention layers become dominant around context length of 500K tokens, until then MoE is dominant. Jamba- 1 1.5 L.--Large, where the matrices are larger, I/O is less of problem, and MoE is the dominant factor. See Appendix C.8 for more results.
Table 10: Mixture-of-expprts improves the IneentionMaabhee hybrid.
log-prob
Hella Swag
Wino Grande
OLLM
NQ
BoolQ
NQA
ARC-C
IMDB
C4
Books
Code
w/o MoE
36.6
62.5
58.8
15.4
59.4
38.3
35,0
87.1
-0.547
-0.658
-0.340
w/ MoE
38.1
66.0
61.2
18.9
64.9
40.0
38.5
89.6
-0.534
-0.645
-0.326
C.5 STABILIZING MAMBA AT LARGE SCALE
When training Jamba models of up to 1.3B parameters, we observed stable training without special problems. However, when scaling to the largest model released here (aB-based, which has 12B/52B active/total parameters), we encountered large loss spikes. Investigating this revealed that inner parts of the Mamba layers suffer from large activation values, leading to the spikes. We therefore added RMSNorm (Zhang& & Sennrich, 2019) to internal activations. As Figure 10 shows, this stabilized training and prevented additional loss spikes.
22

---

Published as a conference paper at ICLR 2025
Mamba RMSnorm impact
Jamba 78 RMSnorm Jamba without RM
Stop
Figure 10: Adding RMSNorm Mamba layers prevents loss spikes.
C.6 JAMBA DOES NOT REQUIRE EXPLICIT POSITIONAL INFORMATION
Table 11 shows results of the Jamba architecture (with MoE) with no positional information and when applying RoPE (Su et al., 2024) in the attention layers (1.3B parameter models, 250B tokens). The results are similar, suggesting that explicit positional information may not be required for the hybrid architecture. Presumably, the Mamba layers, which are placed before attention layers, provide implicit position information.
Table 11: Comparison of Jamba with and without explicit positional information.
log- nigpprob
Hella Swag
Wino Grande
OLLLM
NQ
BoolQ
NQA
ARC-C
IMDB
C4
Books
Code
w/o RoPE
39.6
71.5
64.2
22.2
68.9
50.5
40.7
93.1
-0.516
-0.623
-0.299
w/ RoPE
40.1
71.8
65.5
22.2
67.9
46.2
40.4
90.7
-0.516
-0.623
-0.299
C.7 ADDITIONAL QUANTIZATION RESULTS
Table 12 provides results (with Jamba- 1.5 1.5-Mini) with ExpertsInt8 compared to quantization, show- ing that our quantization technique does not hurt quality. Figure 11 shows throughput measurements to complement the latency measurements in Section 4.1.
Table 12: Comparison of Jamba- M.ni with and without ExpertsInt8 on academic benchmarks, showing no noticeable performance degradation.
GSMSK
MMLU
NarrativeQA
log-prob (C4)
w/o ExpertsInt8 59.5
67.3
67.8
-0.4301
w/ ExpertsInt8 59.8
67.3
68.7
-0.4301
C.8 ARCHITECTURE EFFECTS ON LATENCY AND THROUGHPUT
In this section we provide ablations of various aspects of the Jamba architecture terms of latency and memory uasge. We compared multiple variants to the Jamba architecture reported in the main paper along three metrics: encoding time, decoding time, and number active parameters. In each variant, we changed one of the following: ratio of Attention-to-Mant every e layers, total number of experts per layer n, number of top experts to use K, number of layers in Jamba block I, and using MoE vs. dense layers.
layers a applying MoE
Some prior evidence suggested that Transformer decoder models do not need positional encodings (Haviy al., 2022). However, all existing large-scale models do
of explicit position information.
23

---

Published as a conference paper at ICLR 2025
Expertiiné FRB Note
Expartabrit GPTQ Nerw
Expertsint8 FP8
Thioug Thooughout (19)
(33)
Thoughes (99)
793
10
10
20
50
50
62
Dalch sow
Batch 5ize
Batch size
(a) Jamba, 2xH100
(b) Jamba, 2xA100
(c) Jamba- Lamba-1.5-Large. L.rge 8xH100
Esparishi8
FP8 None
Expertaet
Thouughoat@l)
2000
700
30
Batch
(d) Mixtral-8x7B, 2xH100
(c) Mixtral-8x228, 8xH100
Figure 11: Comparison of different quantization techniques. Output throughput with 1024-token context and 128-token decoding.
The main conclusions from these ablations are:
Encoding latency seems consistent across different context lengths.
Decoding times are flat to about 64K-token context, starting to vary after that.
Dropping MoE leads to gains at decoding time in longer contexts, up to 30% at 128K-token context.
Using MoE at every layer instead of every two layers increases the active parameter count by 23% while slowing encoding/decoddng by 11/14% respectively.
Doubling the attention-to-Manii ratio slows decoding by 50% with context of a 64K tokens.
Using 16 MoE experts at every other has similar memory requirements to 8 experts at every layer, with lower latency and similar quality.
Each MoE layer adds about 2.7B parameters. Overall, memory is 5.5x higher with MoE.
Using MoE at every layer instead of every other layer increases paramerter count from 52B to 95B in Jamba- 1.5-Mini. Using experts instead of 16 lowers this number to 29B. Using 8 experts at every layer (similar to Mixtrla-8x7B) slightly lowers memory requirement (49B compared to 52B parameters), but with higher latency.
The attention-to-Mamba ratio
has small effect on parameter count; attention-only takes 49.8B instead of 51.6B parameters. However, attention obviously has a high memory requirement inference, discussed in detail in Sections 2 and 3.
D POST-TRAINING DETAILS
During training, trained with the normal next-token prediction loss, taken only the assistant's tokens.
D.I SYNTHETIC DATA GENERATION
Section 6.3 described the general processes for post-training, including the use of synthetic data. In programmatic tools to verbalize
general, for synthetic completions when we know the answer, it into a textual answer. In more complex cases, such as when generating a document, we use a mix of early in-house versions of Jamba and open models with permissive weights. We filter generated completions using both programmatic metrics and reward models, which allows for using smaller models by repeated sampling and filtering.
24

---

Published as a conference paper at ICLR 2025
Here give several notable examples of synthetic data generation:
Table-based QA. We generate tabular data and accompanying question-answer pairs, as demon- strated in our work on table understanding (Lenz et al., 2024). We then convert the tables into natural language paragraphs using language model. Our generated training examples include extraction, aggregation, and attribution tasks vis-a-vis text corresponding to specific rows or columns in a given table.
Document QA. Given document, we prompt language model to generate question-answer pairs, for both single and multiple paragraphs. We sometimes embed these examples within longer context by adding similar texts, to encourage long-context understanding with attribution.
Tool use. We use the open-source Glaive function-calll calling dataset (AI) as starting point, filtered with various heuristics and validations on the output schemas. To support parallel function calling, first generate multiple valid parameter assignments for each function in Glaive. Next, we sample subsets of these valid parameter assignments, for the same function and across different functions, to generate requests corresponding to the set of function calls. Finally, we prompt function-calling language model to respond to these generated user requests and retain only responses where the function calls matched the original parameter assignments.
Steerability. We defined a of instructions that can be easily validated and synthesized prompts that include a generic document-drafting task with one or more constraints added to it. We generated completions for these prompts from language model and used rejection sampling based on the vali- dations of our fine-grained instructions plus ceneral-purpose reward model. To support instructions in system messages, we chose multiple prompts of this kind that share a fine- fineggriieed instruction instance and reformatted these prompts into a multi-turn conversation, with the instruction moved to the system message.
D.2 SOME OBSERVATIONS
We share few observations from the development of Jamba-1.5 1.5. While these are not fully explored, we hope they would inspire the community to look further into these issues.
First, while we included only a very small fraction of non-english data, for a few languages and only for specific skills in the post-training phase, our Jamba-1.5 1.5 models perform quite well in multiple languages. We did include multilingual data in the pre-training phase, as mentioned above. Thus we speculate that the models are able to use the learned knowledge from that phase when being post-trained mostly English.
Second, our efficient Jamba architecture lowers the cost of fine-tuning long contexts, allowing us to experiment more with given budget. Thus we could experiment with multiple different training recipes the post-training stage.
Finally, while preference tuning algorithms like PPO (Schulman et al., 2017) or DPO (Rafailov et al., 2024) improve alignment between model ouiputs and human intent, we found that the combination of careful synthetic data generation, data filtering, and supervised fine-tuning is crucial for obtaining strong post- trained model.
E ADDITIONAL RESULTS
E. FULL RULER RESULTS
Table 13 provides the full results on the RULER benchmark, to complement the results in Section 7.1.
25

---

Published as a conference paper at ICLR 2025
Table 13: Comparison of Jamba-1.5 1.5 models with other publicly available and proprietary models on the RULER benchmark. Results for other models are from the RULER Github. Jamba- 1.5 models are the only ones with confirmed effective length of 256K tokens.
Claimed Length
Effective Length
4k
8k
16k
32k
64k
128k
256k
Avg.
Jamba-1.5-Larg
256K
256K
96.7
96.6
96.4
96.0
95.4
95.1
93.9
95.7
Jamba-1.5-Mini
256K
256K
95.7
95.2
94.7
93.8
92.7
89.8
86.1
92.6
Gemini-1.5- 1.5-pr
1M
>128
96.7
95.8
96
95.9
95.9
94.4
91.4
GPT-4-1100 1111--1066ppeieeee
128K
64K
96.6
96.3
95.2
93.2
87
81.2
91.6
Llama 3.1 70B
128K
64K
96.5
95.8
95.4
94.8
88.4
66.6
89.6
Qwen2 72B
128K
96.9
96.1
94.1
79.8
53.7
85.9
Command R+
128K
32K
95.6
95.2
94.2
92
84.3
63.1
87.4
Llama 3.1 8B
128K
32K
95.5
93.8
91.6
87.4
84.7
77
88.3
Command-R
128K
32K
93.8
93.3
92.4
89.5
84.9
76
88.3
Mistral Large
128K
32K
96.2
96.1
95.1
93
78.8
23.7
80.5
Mixtral 8x22B
64K
32K
95.6
94.9
93.4
90.9
84.7
31.7
81.9
Yi 34B
200K
32K
93.3
92.2
91.3
87.5
83.2
77.3
87.5
Phi3 mini 3.8B
128K
32K
92.2
91.5
90.7
87.5
80.6
66.7
84.8
Phi3 medium 14B
128K
32K
93.3
93.2
86.8
78.6
46.1
81.5
Mixtral 8x7B
32K
32K
94.9
92.1
92.5
85.9
72.4
44.5
80.4
Mistral Nemo 12B
128K
16K
87.8
87.2
87.7
69
46,8
19
66.2
DBRX
32K
8K
95.1
93.8
83.6
63.1
2.4
0
56.3
26


Reviews:
[{"review_id": "m3yHS0zGev", "reviewer": "Reviewer_bBMs", "summary": "The paper introduces Jamba, a hybrid language model combining Transformer, Mamba, and Mixture-of-Experts (MoE) layers to enhance memory efficiency, throughput, and long-context handling, supporting up to 256K tokens. With ExpertsInt8 quantization, Jamba achieves scalable, cost-effective deployment on 8-GPU setups. Experimental results demonstrate competitive performance across benchmarks in language modeling, chatbot, and multilingual tasks, highlighting its adaptability to various hardware and resource constraints.", "questions": "In Section 3.2, the authors mention that “Jamba leverages a balanced mix of Transformer, Mamba, and Mixture-of-Experts (MoE) layers to optimize both throughput and model capacity, particularly in long-context scenarios.” It would be helpful if the authors could provide further insights into how the specific ratio of these layers was determined. An ablation study examining the impact of varying the number or sequence of each layer type (Transformer, Mamba, MoE) on key performance metrics such as latency, memory usage, and accuracy would lend greater clarity to the efficacy of this particular configuration and substantiate its contribution to Jamba’s overall efficiency.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1.Hybrid Architecture with Enhanced Flexibility : Jamba’s combination of Transformer layers, Mamba layers, and Mixture-of-Experts (MoE) modules offers a unique balance of efficiency and performance, addressing the limitations of each component individually. This hybrid design allows for configurable trade-offs between memory usage, throughput, and model capacity, making Jamba adaptable to diverse hardware configurations.\n\n2. Long-Context Capabilities: Jamba supports an effective context length of 256K tokens, one of the longest among open-weight models. This is particularly advantageous for long-context tasks, as demonstrated in benchmarks like RULER and ∞BENCH, where Jamba-1.5 models perform competitively, underscoring their suitability for tasks requiring extensive memory.\n\n3. Cost-Effective Inference via ExpertsInt8 Quantization\nThe introduction of ExpertsInt8 quantization allows Jamba-1.5-Large to fit on hardware with 8 80GB GPUs without compromising quality, even for long-context processing. This quantization technique shows latency advantages on A100 GPUs, where FP8 isn’t available, thereby providing a cost-efficient alternative to traditional approaches.\n\n4.Competitive Performance Across Benchmarks\nJamba models perform comparably to state-of-the-art models on standard language modeling, chatbot evaluations, and multilingual benchmarks. This includes maintaining high throughput and latency efficiency, particularly at large context lengths, further demonstrating the model’s real-world applicability.", "weaknesses": "1. Minimal Discussion of MoE and Mamba Layer Interaction : While the hybrid design leverages both MoE and Mamba layers, the paper provides limited analysis of how these components interact to optimize performance. An in-depth exploration of how each component contributes to throughput, especially in long-context scenarios, would clarify the architectural benefits and limitations.\n\n2. Sparse Performance Analysis on Edge and CPU: Although Jamba is designed to balance memory and compute efficiency, its performance on edge devices and CPUs is not evaluated. Given the trend toward deploying models in low-resource environments, this data would offer practical insights into the model’s viability beyond high-end GPU setups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NFbe9ky6GW", "reviewer": "Reviewer_x4G4", "summary": "The paper introduces Jamba-1.5-{Large, Mini} models based on a novel hybrid of Transformer, Mamba, and MoE architectures. Combining the architectural design and the novel ExpertsInt8 quantization technique, the models are efficient in serving in terms of latency, throughput, and memory footprint. Evaluations on long-context benchmarks demonstrate that the model has strong performance in its effective 256K context length. Academic and chat benchmark evaluations indicate that Jamba-1.5 models perform similarly to SOTA public models.", "questions": "- Is it possible to provide details on training procedures? (e.g., pre-training: data source, number of tokens, training throughput, training time, training context length, training batch size; post-training: strategies used and tokens trained)\n- Is it possible to provide experimental results of comparison concerning the Jamba/Mamba/Attention/MoE architectures under the same training setting at scale (like more tokens and larger model size) and for more comprehensive tasks (like all tasks in Section 7)?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Serving the proposed Jamba architecture is efficient in terms of low latency, high throughput, and reduced memory footprint, especially in longer context length. \n2. The paper shares some insights found in the pre-training process, which can be beneficial to the community.\n3. The performance of Jamba-1.5 models is close to other public models of similar activated parameters.", "weaknesses": "1. The details of model training are not revealed, including data source, data mixture ratio, number of tokens trained, context length, and post-training techniques. This makes the training process unclear and not reproducible.\n2.  A direct comparison between Jamba, Llama, and Mistral is insufficient to demonstrate the effectiveness of the proposed Jamba architecture. \n\t- The training of Jamba, Llama, and Mistral models are different in data, model size, and computation, making it difficult to attribute the evaluation results. A comparison between architectures (like Jamba, Mamba, and Attention; with and without MoE) under the same pre/mid/post-training condition (like with the same data and same computation or number of tokens) can better reveal the effectiveness. \n\t- Although Appendix B provides some ablation studies in architecture, the training computation, model size, and reported evaluation datasets are rather limited, and evaluation after mid/post-training is not included. This is insufficient to demonstrate the architectural effectiveness at scale and in comprehensive downstream tasks.\n3. It would be better to demonstrate the scaling ability of the Jamba architecture. For instance, the relationship of LM loss/downstream task performance wrt. training computation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yRiUix5sCm", "reviewer": "Reviewer_7oKN", "summary": "This paper presents Jamba, a hybrid MoE model with interleaved Attention and Mamba layers. They open-source Jamba-1.5-Large (94B active parameters) and Jamba-1.5-Mini (12B active parameters). The authors conduct extensive experiments on various design choices, including different ways to interleave attention and Mamba layers and mix experts. Jamba demonstrates strong results and inference efficiency on long-context tasks, and is competitive on standard language tasks (e.g., MMLU). They introduce ExpertsInt8 to save time and memory of model loading for MoE models.", "questions": "Add more comparsion against the other hybrid attention-SSM models under a fair setting.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1)\tThe proposed model, Jamba, is open sourced for research use.\n2)\tJamba achieves strong results on long-context tasks, while offering better inference efficiency on throughput and memory footprint.\n3)\tThe authors provide detailed ablations and insights on different ways of combining attention and Mamba layers, including the ratio of attention and Mamba layers, using Mamba-1 or Mamba-2 layers.", "weaknesses": "Lack of a fair comparison on the effectiveness of Jamba against the other hybrid attention-SSM architectures. Although Jamba achieved promising results, the paper lacks a comparison with other hybrid attention-SSM models, such as YOCO [1] and Samba [2], using the same training data and parameter scale, particularly on the long-context and standard language tasks (e.g., MMLU).\n\n\n[1] Sun, Yutao, et al. \"You only cache once: Decoder-decoder architectures for language models.\"\n[2] Ren, Liliang, et al. \"Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TdhrqhtGOl", "reviewer": "Reviewer_pW77", "summary": "The paper introduces a model family of Transformer-Mamba models, augmented with Mixture of Expert (MoE) layers. Due to the Mamba and MoE components, the model has throughput and memory benefits on long context tasks. The authors describe a quantization method for the MoE layers which further enhances the inference performance. Finally, the authors describe some evaluations for long context, chat, and natural language benchmarks.", "questions": "- Is there a regression downstream performance when using ExpertsInt8 Quantization?\n- Why adding Gemma on Table 3 if it does not support the context window for this task?\n- Section 7.2 begins with “While not our main focus” — what’s the main focus? This seems important to contextualize to what degree this model is applicable for future work.\n- What is the rationale to self-report some metrics and models in table 4?\n- What does strict/flexible mean in table 4?\n- Why are jamba models having higher error bars than the other models in figure 5?\n- There's a couple of ablations in the appendix comparing Jamba to vanilla transformers (with and without MoE) but it lacks the details to assess if it was a fair comparison. For example, a transformer and mamba models are notably different architectures, are the optimization parameters tuned for each model in figure 6 and 7? (Learning rate, scheduler, beta values if Adam, etc). Also, are the two models parameter count equivalent? If so, how is this compensated?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper proposes a large-scale Hybrid of Mamba, Transformers, and MoE layers. The large-scale aspect of it with respect to model size is certainly novel.\n- Although quantizing the Expert layers on MoE models has been explored before [1] [2] the quantization on Int8 is novel and interesting.\n\n[1] https://neurips2023-enlsp.github.io/papers/paper_81.pdf \n\n[2] https://arxiv.org/abs/2406.08155", "weaknesses": "Unfortunately there are many key aspects missing from all the sections of the paper which makes it hard to quantify if the proposed approach is better than the state of the art in an apples to apples setup, or for future reproductions and/or comparisons. The most important weakness is the lack of disclosing the pre-training size (or FLOPs budget in general), which is known to be indicative of scaling trends and allows for comparing models in an standardised way [5]. While I believe the proposed protocol is novel and interesting enough, I think the following weaknesses should be addressed so that this paper is useful for the ICLR community.\n\n**Efficient Inference details**\n\n- Section 4.1 and 5 focus on the latency aspects of the proposed quantization method, but there is no empirical evidence to quantify the loss in model performance (if any).\n- Figure 2 from section 4.1 seems focused on latency and missing throughput, which is an important consideration to compare to the other approaches.\n- Relevant very similar work in MoE quantization on the experts [2] is missing from section 4.1, there should be at least a discussion, at best a latency comparison.\n- Section 5 lacks important details of what software stack was used to run the latency and throughput comparisons. It is not possible to assess to what degree the comparison is fair, or if it paints a good or bad picture for the proposed method. Concretely: what quantization is used in the baseline models if at all? What’s the software stack used for running these comparisons?\n\n**Pre-training details**\n\n- There are no meaningful details about the composition of the pre-training dataset or the ablations conducted, which makes it hard for future work to compare to this model (e.g. was Jamba trained strongly on X or Y language, or domain).\n- There is no indication on section 6 or the rest of the paper about either the pre-training dataset size, or the amount of tokens the model saw during pre-training. Without this information, it is simply not possible to understand if the proposed model outperforms in a FLOPs apples-to-apples comparison with other models. This makes it hard for future work to make a call of whether or not it is worth it to implement the Jamba architecture with a given FLOPs budget — which is a tradeoff often times researchers in Academia and Industry must think about.\n- MoE models are known to be finicky to pre-train [3], there’s no details about the MoE pre-training protocol. Such as for example, routing loss function and/or hyperparameters. Broadly speaking, there’s no discussion about pre-training hyperparameters at all, which makes it hard to reproduce Jamba in the future, or to run sensible comparable ablations.\n- There’s no information about the tokenizer training either: the mixture of corpus size used, or software stack which is known to matter in practice [4].\n- There is no mention at all about the pre-training effective throughput and FLOPs per step. This makes it hard for future work and practitioners to decide whether the proposed approach scales once the model performance is viewed as a function of FLOPs and wall-clock time.\n- [1] is a very close recent work but is not discussed or ablated throughout the paper.\n\n**Post-training details**\n\n- There are no details at all about what algorithm was used during Post-training and what are their hyper-parameters and compute budget, which makes it hard to reproduce and to compare to other models.\n- There’s a small mention of synthetic data, but there are no details about what model was used to bootstrap the synthetic data, the size, and under what protocol. This makes it hard to assess to what degree this model can be a distillation of other models.\n\n**Evaluation**\n\n- Section 7 has this statement “We mainly compare with recent open-weight models of the same size range” . This is problematic because the delta in performance is dictated by both model size **and** data budget [5]. Since the full compute budget is not disclosed, it’s hard to draw conclusions that can inform future research ideas and questions (Eg is the Jamba architecture better than transformers for a given compute budget).\n- There are no details about the inference used to compute the self-reported numbers in table 4. This is important since some models/tasks are sensible to the use of different floating point precisions and quantization schemes. This is also important to guarantee a fair comparison.\n- A seemingly arbitrary set of (model, tasks) are self-reported, whereas others are drawn from previous papers (without citation) or from the Huggingface OpenLLM Leaderboard. It is not clear why this is the case, which makes it harder for future work to reference this table or to compare to it.\n- Since some numbers are self-reported and for the rest there’s no citation of where they come from (except for the leaderboard) it is not clear at all if this is a fair comparison under the same setup: prompts, split sets, same shot number. While this is a very challenging area of the LLM literature, there should be at least an explanation of the setup used to evaluate Jamba so that future work is somewhat comparable.\n- The claim on Figure 5 seems overreaching, since Jamba 1.5-large is both larger in total and active parameters than llama 3.1 70b, and is well above taking into account error bars, so is not either of a similar size (the other models are smaller) or having competitive performance.\n\n**Typos**\n\n- quantization technique that **allows**\n- Section 7.1.1: The results for key **proprietary**\n\n[1] https://arxiv.org/abs/2402.01771\n\n[2] https://neurips2023-enlsp.github.io/papers/paper_81.pdf\n\n[3] https://arxiv.org/abs/2202.08906\n\n[4] https://aclanthology.org/2024.findings-naacl.247/\n\n[5] https://proceedings.neurips.cc/paper_files/paper/2022/file/c1e2faff6f588870935f114ebe04a3e5-Paper-Conference.pdf\n\n[6] https://arxiv.org/abs/2404.06654\n\n[7] https://arxiv.org/abs/2403.05530", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)