xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference
Authors: Maximilian Beck
Korbinian Pöppel
Phillip Lippe
Richard Kurle
Patrick M Blies
Günter Klambauer
Sebastian Böck
Sepp Hochreiter
Venue: ICML 2025 poster (2025)
Keywords: xLSTM
LLM
inference
inference time
inference speed
Transformer
Primary area: deep_learning->large_language_models
TL;DR: We introduce xLSTM 7B a Large Language Model based on the xLSTM architecture with targeted optimizations for fast and efficient inference.

Abstract: Recent breakthroughs in solving reasoning, math and coding problems with Large Language Models (LLMs) have been enabled by investing substantial computation budgets at inference time. Therefore, inference speed is one of the most critical properties of LLM architectures, and there is a growing need for LLMs that are efficient and fast at inference. Recently, LLMs built on the xLSTM architecture have emerged as a powerful alternative to Transformers, offering linear compute scaling with sequence length and constant memory usage, both highly desirable properties for efficient inference. However, such xLSTM-based LLMs have yet to be scaled to larger models and assessed and compared with respect to inference speed and efficiency. In this work, we introduce xLSTM 7B, a 7-billion-parameter LLM that combines xLSTM’s architectural benefits with targeted optimizations for fast and efficient inference. Our experiments demonstrate that xLSTM 7B achieves performance on downstream tasks comparable to other similar-sized LLMs, while providing significantly faster inference speeds and greater efficiency compared to Llama- and Mamba-based LLMs. These results establish xLSTM 7B as the fastest and most efficient 7B LLM, offering a solution for tasks that require large amounts of test-time computation. Our work highlights xLSTM’s potential as a foundational architecture for methods building on heavy use of LLM inference. Our model weights, model code and training code are open-source.
Model: https://huggingface.co/NX-AI/xLSTM-7b
Code: https://github.com/NX-AI/xlstm and
https://github.com/NX-AI/xlstm-jax.

XLSTM 7B: A Recurrent LLM for Fast and Efficient Inference
Maximilian Beck
Korbinian Pöppel 12 Phillip Lippe 11 Richard Kurle"
Patrick M. Blies
Günter Klambauer' 12 Sebastian Böck Sepp Hochreiter"
Abstract
1. Introduction
Recent breakthroughs in solving reasoning, math and coding problems with Large Language Mod- els (LLMs) have been enabled by investing sub- stantial computation budgets at inference time. Therefore, inference speed is one of the most crit- ical properties of LLM architectures, and there is a growing need for LLMs that are efficient and fast at inference. Recently, LLMs built on the XLSTM architecture have emerged as powerful alternative Transformers, offering linear com- pute scaling with sequence length and constant memory usage, both highly desirable properties for efficient inference. However, such xLSTM- based LLMS have yet to be scaled to larger mod- els and assessed and compared with respect to this work,
Recent breakthroughs in test-time compute scaling have unlocked significant improvements in solving complex rea- soning and math problems. By sampling multiple promising solutions, the best answers can be provided to the user or used as training targets (Yao et al., 2023; Hao et al., 2023; Guan et al., 2025). However, as state-of-the-art models such as OpenAl o1 and DeepSeek-RI (DeepSeek-Al et al., 2025) leverage these methods to push the capabilities of language models new heights, the significantly increased computational overhead of test-time compute methods re- quires more efficient architectures that provide greater in- ference speeds. A promising path involves linear recurrent neural networks with gating mechanisms, including GLA (Yang et al., 2024b), Mamba (Gu & Dao, 2024; Dao & Gu, 2024), RWKV (Peng et al., 2023; 2024). RetNet (Sun et al., 2023), and xI.STM (Beck et al., 2024). Compared to Trans- formers, these models offer a parallel mode for efficient training (e.g. Yang al., 2024b) and recurrent generation mode that both scale linearly with context length. The in- creased compute efficiency combined with constant memory usage during inference allows spending more compute at test-time, but also enables running models locally on edge devices acting as an interface to the user with fast response times.
inference speed and efficiency. we introduce XLSTM 7B, f-billion-parametee LLM that combines xI.STM's architectural ben- efits with targeted optimizations for fast and ef- ficient inference. Our experiments demonstrate that XLSTM 7B achieves performance on down- stream tasks comparable to other similar-sized LLMs, while providing significantly faster infer- ence speeds and greater efficiency compared Llama- and Mamba-based LLMss. These results establish xLSTM 7B as the fastest and most effi- cient 7B LLM, offering a solution for tasks that re- quire large amounts of test-time computation. Our work highlights xLS''ss potential as a founda- tional architecture for methods building on heavy use of LLM inference. Our model weights, model code and training code are open-source.
xLSTM has shown competitive performance compared to alternative recurrent models and even Transformers in controlled experimental setting using the same data and sim- ilar parameter counts (Beck et al., 2024). Moreover, this architecture also excelled in other domains, such as com- puter vision (Alkin et al., 2025), robotics (Schmied et al., 2024), molecular biology (Schmidinger et al., 2025), and time series (Kraus et al., 2024). However, far, SLM has not been scaled to datasets beyond 300B tokens and 1.3B parameters. It therefore remains uncertain whether this architecture can match the Transformer's ability to scale effectively with larger model sizes and extract meaningful patterns from ever-larger datasets.
Model: https: http://hhggnngggcccee . co /Wt.//IIIIIII-----II00
Code: https: /// http///ttttttttte//////tt htw.:/...oo//////////// /ww-///--////// /x1 stm and bttps://githnn con/NXXAI/xxistm fax.
'Equal contribution NXAI GmblI, Linz, Austria JJohannes Kepler University, Linz, Austria 'Now at Google Deepmind. Corre- spondence to: Maximilian Beck <maximilian Korbinian Pöppel <korbinian, koorbiniannppoppeeel wwweeeix.xnnnmnnaaeeaiee ai.com>, Sebastian Böck <sebastian boeck nx ai.com>.
@xx-aicomm>,
In this work, we scale the xLSTM to 7B parameters and present xLSTM 7B, large language model trained on 2.3T tokens from the DCLM dataset (Li et al., 2024) with
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 202.5 by the author(s).
'Etpps:// //wpenat.....//oooooooooo index int coducing- ngg openai- ol-provisw/

---

XLSTM-7B
context length 8192 using 128 H100 GPUs To achieve this, improve and optimize the initial xLSTM architecture from Beck et al. (2024) for optimal training efficiency and stability, without sacrificing performance in downstream tasks. Our new architecture fully relies on mLTT cells with parallel training mode to achieve maximum speed at high language modeling performance. We further optimize the throughput by modifying the surrounding block archi- tecture. By operating the mLSTM in lower dimensional space and adding position-wise feedforward MLP layers similar to the default Transformer blocks, we increase the amount of compute spent for highly optimized linear layers. Additionally, we discard components such as channel- wise convolutions or learnable skip connections to increase the GPU utilization during training. We find that this optimized block architecture has 2x to 4x higher token throughput compared to the previous xLSTM architecture of Beck et al. (2024), while achieving similar performance on language modeling. In addition to the efficiency optimizations, we optimize the new xLSTM architecture for improved training stability, focusing specifically on the gating mechanism of the mLSTM cell. By introducing soft-capping for input and forget gates and improved initializations for the input gate effectively mitigate high gradient norm spikes and variance, and improve the performance of our XLSTM 7B.
large-scale model training while maintaining fast recurrent inference with constant memory.
Generation Mode. During inference, when generating tokens, the mLSTM cell processes the series of input vec- tors 2e E IR for time steps & E {l,..., in a recurrent manner, mapping a state (hz-1: Ct-1. T2- m,-1) to a successor state (hr, C, Re, m)) given an input Ht. Here, h E Rdda denotes the hidden state, C, E R4 denotes the cell state responsible for long-term memory, TI Rd denotes the normalizer state. and m, E R de- notes the max state controlling the magnitude of the expo- nential input gate.
In the recurrent mode (generation), the mLSTM cell
h mLSTMCell (a h Ct-1, 1 m,-1), (1)
is defined by the following state update equations:
m. max { (log (ft) + m }
(2)
C. [, C1-1 + i, kc v/
(3)
n, f, f--1 + is kr,
(4)
C, (4t/ /dga))
he = max n (Ql//dght .ex((-ma)}
(5)
In our evaluations on language downstream and long- context tasks, xLSTM 7B shows comparable performance to Transformers and Mamba models the same size, but with our optimized block architecture it achieves the high- est prefill and generation throughput with the lowest GPU memory footprint our inference efficiency benchmarks.
h, = or ④ Norm(
(6)
The gate activations are computed as:
f. = exp (oogo(( + m m -m²)
(7)
i, exp(ie - m.).
(8)
To summarize, in this work we present targeted modifica- tions to the XLSTM architecture in order to (i) improve training and inference efficiency, and (ii) ensure training stability at large scales. (iii) We introduce a new language model with 7B parameters based on the xLSTM architec- ture trained on 2.3 T tokens with 8k context length demon- strating the highest inference speed and efficiency in our benchmarks.
o, = o (o.).
The query, key, and value vectors qı, ky E RRd , v. E Rd are computed as (qt, kt, v,} W (ight.) at + blgk..)- The sealar input and forget gates i,, fu E IR are computed from the pre-activations fie,
(i.)) Dy + b(.r) and the vector output gate o: E IRdaa is computed from the pre-activation o = W. H; + b. with the sigmoid func- tion a. The normalization layer Norm in (6) can be either R.SNNrm (Zhang & Sennrich, 2019) or LayerNorm (Ba et al., 2016).
We release the pre-trained model xLSTM 7B on Hugging- face² and provide the code? including optimized triton kernels for fast training and inference.
model implementation and training
Training Mode. In training, the mLSTM cell processes a full sequence of input vectors X E R and computes the hidden states H E Tx for all time steps T in parallel. We denote the mLSTM cell in parallel mode (training) as
2. Background: xLSTM with Matrix Memory
In this section, we reassess the mLSTM (Beck et al., 2024), on which we build our XLSTM 7B. The mLSTM cell is fully parallclizable, and, therefore, enables highly efficient
H = mLSTMCell (X).
(10)
hhttpss//hugginggsce..oo co/XX BAIAIXXAIIXXXIIII AI/XISTM-7B 'https: httttp:://gthuub.com cowp:x///////////eee /x1l///xxmmmmmmmmm stm- jax 'https: www.://w////o/eeeeeeoo. /ml stt_/ernemmmmeeeeeeememm
Due to the linear nature of the recurrence in equations (2)- (9), the hidden states H can be computed in chunks without materializing the intermediate memory states (C,, ne, m.).
2

---

XLSTM-7B
3.1. Optimizing for Efficiency
The core of the xLSTM 7B architecture, the mLSTM cell, with its recurrent and parallel mode enable efficient infer- ence and training. To leverage its full potential, we revisit the design of the surrounding block structures.
SwiGLU
Previous mLSTM Block. Similarly to other linear RNNs like Mamba (Gu & Dao, 2024; Hua et al., 2022), the pre- vious xLSTM architecture places the mLSTM cell com- bined with channel-win convolutions in between linear up-projection and down-projection, which is referred to as pre up-projection block (Beck et al., 2024). These blocks combine sequence mixing and channel mixing in one block and are therefore stacked homogeneously without interleav- ing position- wise feed- forward MLP layers. Although the pre up-projection block architecture has proven competi- tive language modeling performance for the xLSTM up to 1.4B parameters, it comes with a substantial trade-off in computational efficiency for the following reasons:
mLSTM
Figure 1. Sketch of the updated xLSTM Block. The lower part an output-gated sequence-mix layer with the mL.TM at its core, whereas the upper part is gated MLP (SwiGLU) as feature/channel-min layer. See Fig. 8 for details.
This mamnlwiise-pralllee form enables highly efficient train- ing kernels, analogous to MaahLinnerAAtettiii (Yang et al., 2024b; Yang & Zhang, 2024), surpassing the training speeds of FlashAttention (Dao, 2024; Shah et al., 2024). For details on the chunkwise-parallle training kernels for the mLSTM cell, refer Beck et al. (2025).
1.
Within the pre up-projection block, the mLSTM oper- ates in a significantly higher dimension than the em- bedding dimension of the model. This leads to a sub- stantially higher computational cost and GPU memory usage for the mLSTM operation.
Multi-Head mLSTM. Similar to multi-head attention in Transformers (Vaswani et al., 2017), the xLSTM has Nhead = d/dn. different mLSTM cells (LSTMCel1 The hidden states (6) of every head are then concatenated and once again projected, resulting in the mLSTM layer
Omitting position-wis wise feed-forwand MLP layers re- sults in decreased proportion of highly efficient linear layer FLOPs in the model.
3.
The previous xLSTM architecture uses several addi- tional components such as learnable skip connections, channel-wise convolutions, and small (block-diagonal) projection layers to compute queries, keys and val- ucs. Without custom kernel fusion, these small opera- tions result in multiple short kernel calls on the GPU, which cannot effectively utilize tensor cores' and, con- sequently, significantly reduce GPU utilization.
()) (oncat(HI)) H (Mraa))
T proj (11)
where H mLSTMMall
(X). We discuss key con- siderations for choosing the number of parallel heads or in other words the head dimension day in Sec. 3.1.
4. Previously, the input and forget gate pre-activations were computed from concatenated query, key and value projections. In a large-scale tensor-part caasor-arrallel training setup this requires an additional all-reduce operation per mLSTM block, which increases the overall com- munication cost.
3. Optimized XLSTM 7B Architecture
The emerging paradigm of increasing test-time computation necessitates i) the development of novel architectures opti- mized for efficient inference. Additionally, new architectures must ii be viable in large-scale pre- traininiig setups, thus be highly efficient during training, and iii) exhibit stable con- vergence. Our xLSTM 7B is designed to meet these three challenges by offering an architecture that can be trained efficiently and with stable convergence and is also highly efficient at inference. In Sec. 3.1, we detail our optimiza- tion of the XLSTM architecture for efficiency during both inference and training. We then describe in Sec. 3.2 our ac- tions to improve and ensure stable convergence for training large XLSTM models, focusing specifically on the gating mechanism of the mL.TTM cell.
These limitations prevent efficient scaling of the xLSTM architecture as introduced by Beck et al. (2024) beyond 1.4B parameters. To scale the xL.TTM even larger model sizes, we optimize the mLSTM block for maximal efficiency by addressing these four limitations.
Optimizing the mLSTM Block. To begin, we operate the mLSTM cell in the models' embedding dimension, in-
Tensorr cores are specialized compute units that accelerate matrix multiplications on GPUs.
3

---

XLSTM-7B
stead of a higher dimensional space and place position-wise feed-forward MIP layers after each mLSTM layer. This modification increases the proportion of highly optimized linear layer (i.e. matrix multiplication) FLOPs and reduces the computation cost of the mLSTM operation (see App. E for details on the FLOP computation). The significantly reduced GPU memory usage enables larger batch sizes dur- ing training, which also increases training efficiency. The result is the default dense Transformer block configuration referred to as post up-projection block by Beck et al. (2024);
slightly longer train step time to make sure the model is not constrained by lack of memory. We elaborate further on this in Sec. 5. We choose 8 heads with head dimension dne = 512 for xLSTM 7B.
Fused Generation Kernels for the mLSTM Cell. Dur- ing autoregressive generation, the hidden state outputs of the ML.TMM cell are computed, with its recurrent formulation given by (1) - (9). The recurrent formulation consists of a combination of an outer-product, dot-products and several pointwise operations. which translates to individual con- secutive GPU kernels. Since each kernel loads its inputs from and stores its outputs to GPU memory, this increases the amount of slow memory operations. To ensure that intermediate results of equations (2)(5) are not unneces- sarily transferred to GPU memory, but instead remain on the GPU's compute chips, we write fused GPU kernels for the mLSTM generation mode. This results in significantly faster generation as shown in speed benchmarks in Sec. 5.2.
z = x + mI..TM(NNrm((z)) y - + MLP((Nrm(=))
(12a)
(12b)
where is the input to the block, x is the intermediate output of the mLSTM layer defined in (11), and y is the block output. The MLP is SwiGLU (Shazeer, 2020) (see Fig. 1).
Moreover, we discard operations like the chamnel-wist con- volution and the learnable skip-connection, and replace the block-wise query, key and value projections by dense linear layers. This again increases linear layer FLOPs and ensures effective usage of tensor cores within the mLSTM layer.
3.2. Optimizing for Stability
We find that the previous xLSTM architecture at the 7B parameter scale often becomes unstable in early stages of training. In particular, we noticed that training at higher learning rates leads to large spikes in the gradient magnitude and loss value, similar to reports from previous works on Mamba- -basd models (Lieber et al., 2024; Dao & Gu, 2024; Zuo et al., 2024). We further observed and attribute these spikes to very large outlier features, i.c. individual feature values that are significantly larger than the average feature value (He et al.). We address these stability issues by (i) the use of RMSNorm instead of LayerNorm, (ii) soft-capping of the input and forget gates, and (iii) a negative initialization of the input gate bias.
Finally, we ensure that the gate pre-activations for every head are computed independently as outlined in (11). This allows us to apply the model parallelization strategies opti- mized for Transformers with self atf-ntttenton (Shoeybi al., 2020) to our XL. 7B architecture and therefore minimize additional communication cost.
These optimizations result in our optimized mLSTM block described in Fig. and Fig. 8 in the appendix, of which we stack 32 in our xLSTM 7B architecture. We observe that our optimizations achieve a 3.5x speedup in training for 1.4B models, with a slight trade-off in validation perplexity that can be mitigated by few more training steps (see Tab. 2).
Pre-Norm with RMSNorM. Many works report that re- placing the LayerNorm by RMSNorm at the input of each layer ge.g. in the pre-norm setting (Xiong et al., 2020)) improves training stability for Transformers (OLMo et al., 2025; Touvron et al., 2023; Gemma Team, 2024a; Yang et al., 2024a) and Mamba models (Zuo et al., 2024). Our experiments in App. C.2, Fig. 9 confirm that this also ap- plies to the pre-norm normalization layers in (12) in our XLSTM architecture. Therefore, we replace the LayerNorm by RMSNorm in our XLSTM architecture.
Although the modified block structure reduces the size of the ML.TTM cell memory states C, we find that it does not compromise the language modeling quality of our model.
Optimizing the Memory Capacity. The overall memory capacity of the XLSTM, i.e. the amount of information that can be stored from an input sequence, is related to the phys- ical size of its memory cell states C' of shape dyk x dnu in GPU memory. By choosing either the number of heads or the head dimension da., the other is given by the rela- tion to the embedding dimension d #heads x dny- For the XLSTM 7B we set dyt dnv similar Sun et al. (2023). We can then compute the total memory state size by #blocks x #heads x dyk x dr.. x 4 bytes, assuming that the state is stored in Float32 format. In Tab. 3 we show the memory state size for different number of heads as well as their trade-offs with language modeling performance and training efficiency. We use a larger memory state size and
Gate Soft-Capping. To reduce potential large outlier fea- tures and related loss spikes, we apply soft-capping to the input and forget gate pre-activations i, and f,, such that their values stay between -a and a for specific cap value a. We cap the gates using a = 15 with the function
softcap, (x) = a tanh(x/a).
(13)
In Sec. 5.3 and App. Sec. C.2, we confirm that this sig- nificantly improves the stability and performance of our
4

---

XLSTM-7B
Maximum (over 50 steps)
Mean (over 50 steps)
3.0
10
2.8
10a
Gradient Norm (logscale)
Training Loss
2.6
10
2.4
10
2.2
10
100
200
300
400
500
100
200
300
400
500
Step
x10²
Step
x10
Figure 2. Loss and Gradient Norm during Pretraining of xLSTM 7B. We show the mean and maximum value over 50 steps. Our enhanced architecture and initialization enable stable pretraining of xLSTM 7B, exhibiting only two brief loss spikes early in training. both of which were rapidly recovered.
Table 1. Model Performance on Huggingface Leaderboard v2.
indicates larger values are better.
MODEL.
BBH
MMLU-PRO
MATH 1
MUSR
GPQA
IFEVAL
AVERAGE
TRANSFORMERS
Llama-3.1-80
0.465
0.325
0.042
0.379
0.312
0.125
0.275
Llama-2-788 2-
0.349
0.186
0.013
0.363
0.269
0.264
OLMo-7B-hf
0,330
0.118
0.010
0.357
0.257
0.280
0.225
Gemma-7B
0,426
0.293
0,061
0.408
0.295
0.272
0.292
Ministral-SBB-IIstrut2222
0.496
0.350
0.151
0.430
0.319
0.322
0.345
Bloom-7B1
0.311
0.111
0.000
0.354
0.264
0.138
0.196
Gpt-j-6B
0,321
0.125
0.009
0,363
0.261
0.250
0.222
Pythia 6.9B
0.326
0.116
0,006
0.355
0.270
0.232
0.217
Qwen2.5-7BB
0.541
0.435
0.165
0.446
0.329
0.359
0.379
Gemma-2-9B
0.543
0.414
0.117
0.453
0.334
0.217
0.346
DCLM-7B
0.426
0.312
0.030
0.392
0,303
0.228
0.282
TRANSFORMER RECURRENT HYBRIDS
SERFFRRMERRRECRRENE
7amba2-7B
0.489
0.319
0.114
0.402
0.318
0.375
0.336
RECURRENT MODELS
7B (pre-decay)
0,373
0.177
0.024
0.387
0.275
0.252
0.248
0.429
0.229
0.039
0.412
0.299
0.335
0),990
MambaCodestral-7B (v0.1)
0.405
0.191
0.023
0.359
0.266
0.322
0.261
NKWV-v5-Eagle-2
0.325
0.121
0.007
0.322
0.243
0.266
0.214
RWKVAv6-Finch-77
0.342
0.154
0.014
0.338
0.265
0.264
0,230
7L
0,381
0,242
0.036
0.379
0.280
0.244
0.260
XLSTM 7B LCTX
0.390
0.252
0.040
0.374
0.253
0.234
0.257
5

---

XLSTM-7B
XLSTM architecture. Additionally, we apply soft-capping with a 30 to the final layer logits, similar to Gemma Team (2024b).
5. Experiments
5.1. Language Modeling Performance
Huggingface Leaderboard. We start by benchmarking XLSTM 7B against state-of-the-art Transformer and recur- rent LLMs on the 7B parameter scale. To this end, we evaluate the performance on the Open LLM Leaderboard v2 using the LM Evaluation Harness (Gao et al., 2024; Fourrier et al., 2024). The results are summarized in Tab. 1, show- ing that xLSTM 7B ranks in the mid-range among 7B-scale models, several of which benefited from substantially larger training datasets. We believe that with a larger and better curated training dataset, including a greater emphasis on math and code data in earlier training phases, xLSTM 7B could match the performance of the strongest 7B models.
Negative Input Gate Bias Initialization. We observe that early on in training our xLSTM models experience large gradient norm spikes, which affect the final performance of our model (see Fig. 11 in App. C.2). Initializing the input gate at large negative values (e.g. -10) effectively mitigates these gradient norm spikes and improves performance. We analyze the impact of the input gate further in Sec. 5.3.
In summary, our optimizations enable remarkably stable pre-training of xLSTM 7B, as we show in Figure 2.
We outline the detailed block architecture of our xLSTM 7B in Appendix A and our training recipe in Appendix B.
Evaluation and Fine-Tuning To evalu- ate long-context capabilities, we use the RULER bench- mark (Hsich et al.. 2024), which consists of set of syn- thetic beedle-in-a-hyystcck question- able tracking tasks, with varying context length from 4K to 131K tokens. For this benchmark, we consider both our standard XLSTM 7B and a long-context version (XLSTM 7B LCTX), where we replace the standard cool-down phase described in App. with a long-context variant. For the long-context cool-down phase, we add long-context data (see App. Tab. 5) to the training corpus and train the model with a context length of 32K, while adjusting the batch size to maintain the number of tokens per batch. We compare to Llama 2 7B (not long-context fine-tuned) and Llama 3.1 8B (long-context context fine-tuned up to 131K tokens) as Trans- former baselines, CodestralMamba and FalconMamba as State Space Model baselines, and RWKV-5/6 as additional RNN baselines.
4. Related Work
Although the largest language models to date have predomi- nantly relied on Transformer-based architectures, recurrent LLMs and hybrid models have recently gained traction as alternative architectures due to their enhanced efficiency in processing long contexts. Many recent efforts have targeted the 7B parameter scale (or nearby), striking a balance be- tween model capacity and resource constraints. Griffin (De et al., 2024) is one of the first hybrid recurrent models that was trained with up to 14B parameters. Later, the same architecture was used to train RecurrentGemma with 9B pa- rameters (Botev et al., 2024). The Griffin architecture uses a ID temporal convolution of size 4 before the sequence mixing part, similar to H3 (Fu et al., 2023) and Mamba (Gu & Dao, 2024), but the hidden state is vector valued with independent updates per each (scalar) dimension. In contrast, Eagle-BB (Peng al., 2024) builds the RWKV architecture and uses matrix-valuee hidden state similar to linear attention and gated linear attention (Katharopoulos et al., 2020; Yang et al., 2024b).
and vari-
The results on RULER are shown in Fig. 3. As expected, Llama 3 provides the strongest baseline, since it is heavily fine-tuned on very long contexts and with a more advanced and optimized approach (Grattafiori et al., 2024). On the other hand, Llama 2 fails entirely for context lengths be- yond 4k, for which it has not been trained. For xLSTM 7B, the long-context cool-down stage in pre-training largely im- proves long-context capabilities, resulting in competitive performance compared to state-space models and outper- forming RWKV-5/6. Notably, the long-contert xLSTM 7B achieves 20% average accuracy at a context length 131k, although it was trained only with a context length up to 32k during the cool-down phase. This is particularly remarkable given that, unlike Transformers with a growing KV cache, XLSTM 7B must store information from the entire sequence in a fixed-size memory with limited capacity (see Tab. 3), We assume that xLSTM 7B's 's performance could be pushed further by explicitly training on even longer sequences and with a more advanced finettning protocol as it was used in the training of Llama 3 (Grattafiori et al., 2024).
Among the Mamba models at the 7B parameter scale, Wal- effe et al. (2024) provided the first comparative analysis of Mamba 1, Mamba 2, and hybrid Mamba architecture. In their experiments, the performance of both Mamba and Mamba 2 significantly lagged behind Transformers, while the hybrid architecture was shown to surpass the performance of Transformers. Aligned with this finding, several new hybrid Mamba architectures have been pro- posed, including Samba (3.8B) (Ren et al., 2024), Zamba (7B) (Glorioso et al., 2024), and the 12B parameter mixture- of eeexperts-model Jamba (Lieber et al., 2024). More re- cently, Mamcon (Zuo et al., 2024) based on Mamba and Codestral Mamba (Mistral AI Team, 2024) based on Mamba 2 have shown that a purely recurrent architecture is capable of exceeding the performance of both hybrid Mamba models and Transformers.
6

---

XLSTM-7B
Llama 88
Mamibaa 78
KLSTM
Liama 8B
FalconMamba 7B
XLSTM 7B
78
RWKV-5
LCTX
Liama 2 78
CodestralMamba 7B
Falcan 7B
RWKV-6 6 7B
160
100
140
Average Accuracy [%]
Tokens per Second
120
00.0
40
60
20
40
4096
8192
16384
32768
65536
131072
2000
10000 12000 14000 16000 Prefill Length [Tokens]
6000
8000
Context Length [Tokens]
Figure 3. RULER results of xLSTM 7B in comparison to Trans- fomers (with and without long context finetuning) and State Space Models, with and without medium context cooldown.
Figure 4. Throughput for generating 100 tokens with batch size I at varying prefill lengths.
the Transformer models can generate text significantly drops for longer prefill lengths. In contrast, recurrent architectures with constant cost per generated token have a constant gen- eration speed independent of the input context length.
In Sec. 5.3, we further investigate the effect of the memory state size and the input gate on the long context capabilities of 7B.
We find that xLSTM 7B is about 50% faster in text gener- ation than Mamba, which we attribute mostly to our opti- mized block design (see Sec. 3), and even faster than Llama- based Transformer models with similar block design at prefill length 0.
5.2. Speed Benchmarks
The constant memory size and linear compute scaling with context length of our XLSTM architecture enable highly ef- ficient generative inference in large scale-inference serving environments as well as local inference running on edge
devices.
Generation Time and Memory Consumption. We mea- sure the token generation time and GPU memory usage (without pre-fill) for different generation lengths. Fig. 5 (left) demonstrates the linear scaling of recurrent models vs. the quadratic scaling of Transformers in compute (run- time), while Fig. 5 (right) shows the constant memory size of recurrent models compared to the linear growth of the Transformer KV-cache. Since Llama 3 uses grouped query attention (Ainslic et al., 2023) the memory usage grows slower compared to Llama 2, which uses default multi-head attention.
We focus on the local single user inference setting, which is common when models are deployed on edge devices. Therefore, we benchmark generative inference with our XLSTM 7B model on a single NVIDIA H100 GPU with batch size I, unless specified otherwise. We compare our 7L Llama 2 and Llama models as Trans- former baselines and Falcon Mamba (Manba architec- ture) and Codestral Mamba (Mamba 2 architecture) Mamba baselines. We use model implementations from Huggingface transformers library and optimize each with torch. compile 6 and Py ToTchh CUDA Graphs (Nguyen et al., 2021). In Appendix D. we additionally compare our optimized Huggingface xL.STM 7B to Llama2, Llama3, Fal- con Mamba and Codestral Mamba in the vLLM (Kwon et al., 2023) inference framework.
With our optimized block design, we operate the mLSTM in a lower dimensional space. This results in significantly lower memory footprint (Fig. 5 (right)) and lower generation times (Fig. 5 (left)) of our xLSTM 7B model compared to the Mamba models.
Generation Throughput. The generation throughput measures the generation speed in tokens per second at vary- ing prefill lengths, i.e., varying length of documents the model gets to read before starts to generate text. In Fig. 4, we observe that due to the quadratic scaling with input con- text length of the attention mechanism, the speed at which
Time To First Token. In applications, where the language model operates interface to the user (potentially on edge devices), it is important to have short response times. In Fig. 6, we measure this response time or latency as the time the model takes to generate or 100 token after consuming varying prefill lengths. Our xLSTM 7B achieves the fastest response times for all prefill lengths.
htt pie /f / /tggttttttt hubb conn/ guu ing farce/ ranss ransforners.
7

---

-LST--7B
Liama 3 8B
FalconMamna 7B
XLSTM 7B
Liama 3 88
Falcon 7B
XLSTM 78
Liama 2
CodestralMamba 7B
Liama 2 7B
CodestralMamba 78
500
x10+
Generation Merrary [GB]
22
47580
47543
Generation Time [s]
400
42609
300
20
Tokens per Second
37741
36343
33912
37864
33211
18
30000
200
27237
27166
27354
28016
26224
20110
16-
24
100
16878
16840
16449
16309
14-
15000 Generation Length [Tokens]
5000
10000
5000
10000
15000
Generation Length [Tokens]
Figure 5. Time and GPU memory used for generation of single sequence of varying lengths for generation without prefill.
BS-32 CTX=2040
B5 116 CTX=4096
85-8 CTX-8192 Tokens
B5-4 CTX-16384
CTX=32768
B5-2
Liama 3 8B
FalconMamba 7B
XLSTM 78
Figure 7. Prefill throughput varying batch size and context length.
Llama 2 78
CodestralMambba 78
1.0
Time First 100 Tokens [s]
post-up projection block xLSTM 7B, deviating from the pre-up projection Mamba (Gu Dao, 2024; Dao & Gu, 2024) and the previous xLSTM architecture (Beck al., 2024).
Time to First Token
0.8
0.6
0.4
Memory State Size. The memory state size as well as the training step time directly influenced by the number of heads (see Sec. 3.1 and Tab. 3). In this experiment we investigate how the memory state size affects the perfor- mance of the xLSTM in validation perplexity, on down- stream tasks as well as on long context tasks. To do so, we train xLSTM models with 7B parameters and different number of heads on 160B tokens of our pre-trate pre-training dataset. In our evaluations perplexity (Tab. 3) and on downstream tasks (Tab. 7 and 8), we find that the performance remains stable across different the number of heads, i.e., memory state sizes, with a slight improvement for more heads (e.g. 16). In contrast, our long context evaluation in Fig. 13 sug- gests that at very long contexts 4 and 8 heads (i.e., larger memory states) seem to perform better. While this line with our intuition that larger memory state size corresponds to better long-context capabilities, we believe that larger study (e.g., training more tokens) than ablation at 7B parameters and 160B tokens would be necessary to fully explore this connection.
0.0
0
5000
10000
15000
0
15000 Prefill Length [Tokens]
5000
10000
Prefill Length [Tokens]
Figure 6. Time first (1) token and time first 100 tokens at varying prefill lengths for batch size 1.
Prefill Throughput. Finally, we measure the prefill throughput in tokens per second for 65,536 tokens at varying batch size and context length. Due to the quadratic scaling with context length, the throughput of the Llama models decreases with longer contexts. In contrast, our xLSTM 7B achieves the highest throughput (about 70% higher than Codestral Mamba) independent of the context length.
5.3. Ablation Studies
Finally, we validate our design choices to optimize the train- ing stability and efficiency of our XLSTM 7B architecture.
Norm Layer Types. Our update on the xLSTM block architecture has two normalization layers, pre-norm at the block entry and a head-wise norm layer after the mLSTM cell. In this ablation, we test the effect of the types of these normalization layers training stability and performance, with I LayerNorm (Ba et al., 2016) and RMSNorm (Zhang & Sennrich, 2019) as the options. In Fig. 9 in App. C.2 we confirm that, for the pre-norm the RMSNorm type has a strong stabilizing effect, whereas for the mLSTM cell state norm there is no impact on stability and performance.
Pre-Up vs. Post-Up Projection Block. We compare the pre-up projection block architecture against our optimized mLSTM block in terms of validation perplexity and training step time for three model sizes. For both block architec- tures, we apply gate soft-capping and the input gate bias initialization described in Sec. 3. The results in Tab. 2 show only a slight performance difference terms of validation perplexity at the largest model size. However, the 3.5x speedup training step time confirms our choice for the
8

---

XLSTM-7B
6. Conclusion
Table Comparison hetween the previous xL.TTM architec- ture (Beck et al., 2024) and our XLSTM 7B architecture in terms of step time and perplexity for different number of parameters. Mod- els of size 160M and 400M use batch size 128 distributed over 16 GPUs, and 1.4B parameter models use batch size 256 (32 GPUs). For the 7B parameter model, our architecture uses batch size 512 (128 GPUs), whereas the previous architecture uses only batch size 256 (128 GPUs) because of the architecture's 's increased GPU memory requirements. Due to the expensive computational costs, we only compute the token throughput and did not fully train the 7B parameter models for this ablation.
In this work, we demonstrate how our targeted modifica- tions enable the xLSTM architecture to scale to models with 7B parameters, trained on 2.3 T tokens. By switching to a post-up-proyye projection structure, gate soft-capping and proper initialization, we largely improve training stability and to- ken throughput, making the xLSTM the fastest RNN-based architecture at the 7B scale, while competitive in perfor- mance with Transformers and other recurrent models. We believe that xL.STM's very high decoding speeds in combi- nation with its good performance highlight its potential as foundational architecture for methods investing substantial compute at inference time.
1 indicates larger/ smaller values better.
MODEL
TIROUGIPUT t IK TOKENS/SEC
SPEEDUP t
PPL J.
A PPL
160M
PREVIOUS
76.20
20.43
OURS
225.99
X2.97
21.34
+0.91
Impact Statement
400M
PREVIOUS
28.13
15.26
OURS
102.40
x3.64
15.74
+0.48
This paper presents a novel architecture for fast and effi- cient language modeling, reducing computational costs and energy consumption without sacrificing performance. By making high-quality language models more accessible, approach helps bridge the digital divide, enabling equitable AI deployment in low-resource settings. Additionally, the efficiency gains contribute to environmental sustainability by lowering the carbon footprint of large-scale NLP systems. However, there might be both positive and negative societal impacts. We are aware of the risks, but believe that our and the overall advancements in the field of machine learning technology provide net benefit to society and the world.
1.4B
PREVIOUS
10.57
12.46
OURS
37.03
x3.50
12.68
+0.22
7B
PREVIOUS
3.46
OURS
9.15
x 2.64
Table Head dimension ablation for a 7B parameter xLSTM model with 32 blocks, embedding dimension 4096 and training context length 8192. KV Cache in Tokens shows how many tokens in a similar sized Transformer correspond to our state size. FLOPs forward are the mLSTM cell forward FLOPs for a full sequence. 4 indicates smaller values are better.
#Heads
dap
Total Memory State MB
KV Cache Tokens
FLOPs forward
Val PPL.
Train Step Time +
4
1024
268.4
256
7,6c11
9.58
3,97
8
512
134.2
128
4.1c10
9.52
3.63
Acknowledgements
256
67.1
64
2.4c10
9.52
3,51
128
33.6
1.5e10
3.41
The ELLIS Unit Linz, the LIT AT Lab, and the Institute for Machine Learning are supported by the Federal State Upper Austria. The NXAI Lab is supported by NXAI GmbH, Linz, Austria.
Soft-capping. Soft-capping (Eq. (13)) of the output logits and the input and forget gate pre-activations, is important for training stability. In Fig. 10 of the appendix, we visualize the validation loss and gradient norms during training on 1 160B tokens with and without soft-capping. The run without soft- soft-capping shows a higher variance in the gradient norms and an overall worse validation loss.
Input Gate. We initialize the input gate with larger negative values (e.g. -10) to mitigate large gradient norm spikes and variance (see Sec. 3.2). This suggests that the input gate is important for the performance of the xLSTM architec- ture. Therefore, in App. C.2 we test the effect of having the input gate non-trainable. We compare a version with fixed input gate at one (i.e. setting weights and biases to zero) with a version, where the input gate bias is fixed at our low default initialization value of -10. We find that, while the learnable input gate only slightly improves perfor- mance of our xLSTM over the fixed input gate versions on our standard downstream tasks (App. C.2, Tab. 7 and 8), it significantly improves performance on long-com long-context evalua- tions (App. C.2, Fig. 13).
9

---

XLSTM-7B
References
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao,
L., Golding, I.., He, H., Leahy, C.,, McDonell, K., Phang, J., Pieler, M., Prashanth, U. S., Purohit, S., Reynolds, L.. Tow, J., Wang, B., and Weinbach, S. GPT-NeoX- 20B: An open-source autoregressive language model. In ACL Workshop on Challenges & Perspectives in Creating Large Language Models, 2022. URL https: / //arxiv. org/ abs/ 22002224..00000 06745.
Ainslie, J., Lee- Lee-Thorp. J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. GQA: Training generalized multi-query transformer models from multi-head check- points, 2023. URL https::////... /ttps///axxvvoooo abs/
2305. 13245.
Alkin, B., Beck, M., Pöppel, K., Hochreiter, S., and Brand- stetter, J. Vision-LSTM: xLSTM as generic vision back- bone. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. URL https: //openreview het.:iiwwwee//t..... /tw//foror//rt. forum? 1=517DDNN22
Botev, A., De, S., Smith, S. L., Fernando, A., Muraru, G.- C., Haroun, R., Berrada, L.,, Pascanu, R., Sessa, P. G., Dadashi, R., and et al. RecurrentGemma: Moving past transformers for efficient open language models, 2024. URL https: / arxiv. org. /abs///////// 2404. 07839.
Allal, L. B., Lozhkov, A., Bakouch, E., Blázquez, G. M., Tunstall, I.., Piqueres, A., Marafioti, A., Zakka, C.,, von Werra, L., and Wolf, T. SmolLM2 with great data, comes great performance, 2024.
Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., and Jia, J. Longlora: Efficient fine-tuning of long-context large language models. In Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2024.
Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S., Jiang, A. Q.. Deng, J., Biderman, S., and Welleck, S. Llemma: An open language model for math- ematics, 2023.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C.,, and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv: 10. 14168, 2021.
Ba, J., Kiros, J. R., and Hinton, G. E. Layer normalization. CoRR, abbs//607.00/ 607.06450, 2016.
Bai, Y., Lv, X., Zhang, J., He, Y., Qi, J., Hou, L., Tang, J., Dong, Y., and Li, J. LongAlign: A recipe for long context alignment of large language models. In Findings of the Association for Computational Linguis- fics: EMNIP 2024, pp. 1376-1395, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.18655/w120044.ndddnn----m 74. URL https: Eindings-emnip.2
Dao, T. Fttenooten Faster attention with better par- allelism and work partitioning. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https: /pwpneeveeeeeeeee//e/eeeeemoe..e net/ forum? forumiid=m2n2XyhsEE.
Dao, T. and Gu, A. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Proceedings of the International Confer- ence on Machine Learning (ICML), 2024. URL https: / //opereview. net/ (tcorummi==ztn@FFFF
Beck, M.. Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., and Hochreiter, S. xLSTM: Extended long short-term mem- ory. In Proceedings of the Conference on Neural In- formation Processing Systems (NeurIPS), 2024. URL https::/arxx: org/ aass//////a// 2205..200 04517.
De, S., Smith, S. L., Fernando, A., Botev, A., Cristian- Muraru, G., Gu, A., Haroun, R., Berrada, L., Chen, Y., Srinivasan, S., Desjardins, G., Doucet, A., Budden, D., Teh, Y. W., Pascanu, R., Freitas, N. D., and Gulcehre, C. Griffin: Mixing gated linear recurrences with local attention for efficient language models, 2024.
Beck, M., Pöppel, K., Lippe, P., and Hochreiter, S. Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kemels. arXiv, 015503.14376, 2025. URL https: leaaxiv.org/aaee 2503 10000..14355 14376.
DeepSeck-AL AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., and et al. DeepSeek-RI: Incentivizing reason- ing capability in LLMs via reinforcement learning, Jan- uary 2025. URL http: /aa /aaiiiiiii. www.wiww..o....o........o.. harg/abe. aaa//b////00 /2001. 12948. arXiv:2501. [cs].
Beeching, E., Fourrier, C., Habib, N., Han, S., Lambert, N., Rajani, N., Sanseviero, O., Tunstall, L., and Wolf, T. Open LLM Leaderboard. https: httttps://uugiiii htw.ghugggggacccccceeocc co / spaces /otpccce//ppp-eee 11m leaderoaard-oooddddd.. open 11m lttp_limm_eeee_ee haatm__eaaeebbbbbr 2023.
Ben Allal, L., Lozhkov A., Penedo, G., Wolf, T., and von Werra, L. Cosmopedia, Febru- ary 2024. URL https: httgg:ss///gggnggaaee... co/ datasets aatt cosmopedia.
Fourrier, C., Habib, N., Wolf, T. and Tunstall, L. Lighteval: A lightweight framework for Ilm evaluation, 2023. URL https: h http//gttubbbcc. com/
huggingface/lightenn
10

---

XLSTM-7B
Fourrier, C., Habib, N., Lozovskaya, A., Szafer, K. and Wolf, T. Open IIm leaderboard v2. https: htt.gggggggggceeeeeee/c cttcco//saaaooe /ospcces// Seen--iim-eaderbbareee haaedd/oppeepppeeeeeeee leaderboard, 2024.
He, B., Noci, L., Paliotta, D., Schlag, I., and Hofmann, T. Understanding and minimising outlier features in trans- former training. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS).
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de las Casas, D., Hendricks, L. A.. Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J. W., and Sifre, L. An empirical analysis of compute-optimat large language model training. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https: / //oppnreview net/
Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., and Re, C. Hungry hungry hippos: Towards language modeling with state space models. In Proceedings of the International Conference on Learning Representations (ICLR), 2023. URL https: https:://ppnreeeeee http:///pppeeeeeeetett..eeeeeeee forum? d=Co.Dy0WYGg..
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu. J., Le Noac"h, h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang. B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL https: : zenodo. org/ records// 12608602.
Hsieh, C.-P.. Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.0665. 2024.
Hua, W., Dai, Z., Liu, H., and Le, Q. Transformer qual- ity in linear time. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Pro- ceedings of the International Conference on Machine Learning (ICML), volume 162, pp. 9099-9117. PMLR, 17-23 Jul 2022. URL https: /ppppeeeeednggssse mlr. press/vl 62 hwwpaaaaaaaaaaaaaa html.
Gemma Team. Gemma: Open models based on gem- ini research and technology. 2024a. URL https: //arxiv.org// abs/ /4t../243..22 08295.
Gemma Team. Gemma 2: Improving open language mod- els at a practical size, 2024b. URL https:: /tttp:://axiv.. org/ / abs/ 24088 00118.
Katharopoulos, A.,, Vyas, A., Pappas, N., and Fleuret, F, Transformers are RNNs: fast autoregressive transformers with linear attention. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
Glorioso, P., Anthony, Q., Tokpanov, Y., Whittington, J., Pi- lault, J., Ibrahim, A., and Millidge, B. Zamba: compact 7b ssm hybrid model, arXiv preprint arXiv:2405. 16712, 2024.
Kocetkov, D., Li, R., allal, L. B., LI, J., Mou, C., Jernite, Y., Mitchell, M., Ferrandis, C. M., Hughes, S., Wolf, T., Bahdanau, D., Werra, L. V., and de Vries, H. The Stack: 3 TB of permissively licensed source code. Transac- tions on Machine Learning Research, 2023. ISSN 2835- 8856. URL https: i dd=pxpbTdUFpp.
Grattafiori, A., Dubey, A., Jaubri. A., Pandey, A., Kadian,
A., Al-Dahle, A., Letman, A., Mathur, A,, Schelten, A., Vaughan, A., and et al. The Llama 3 herd of models. 2024. URL https: arxiv vi.. org/ 2407 21783.
https::/ /ttp://veeeeeeeeeeeeeeeeeeeeeeee
net forum?
Gu, A. and Dao, T. Mamba:
Linear-time sequence modeling with selective state spaces. In First Conference on Lan- guage Modeling, 2024. URL htthst:::: httt://ppppeee openreview. net/ forum? d=tEY
Kraus, M., Divo, F., Dhami, D. S., and Kersting, K. XLSTM- Mixer: Multivariate time series forecasting by mixing via scalar memories. arXiv preprint, 2024. URL https:
//arxiv. wwt.:oo/gg///o////o/////// 16928..
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Sym- posium on Operating Systems Principles, pp. 611-626. 2023.
Guan, X., Zhang, L. L., Liu, Y., Shang, N., Sun, Y., Zhu, Y., Yang, F., and Yang, M. rStar-Math: Small LLMs can master math reasoning with self-evolved deep thinking, 2025.
Hao, S., Gu, Y., Ma, H., Hong, J., Wang, Z.,, Wang, D., and Hu, Z. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8154-8173, 2023.
Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H.,, Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, S., and et al. Tülu 3: Pushing frontiers in open language model post-training. 2024.

---

XLSTM-7B
LI, J., Beeching, E., Tunstall, L.., Lipkin, B., Soletskyi, R., Huang, S. C.,, Rasul, K., Yu, I.., Jiang, A., Shen, Z., Qin, Z., Dong, B., Zhou, L., Fleureau, Y., Lample, G. and Polu, S. NuminaMath. /tttps:/ggithub... project-numt na hww.r/uumnaaaamaaaaaaaaaa im--progress--rriee http://manneeptttuinneeeeeeeeeee 2024.
Pan, B. Anti-Haystack, February 2024. https://hugginggtt cttcco///aaaooo datasssssss//
URL
wenbopan/ant haysiichhya.....
Peng, B., Alcaide, E., Anthony, Q. Albalak, A., Arcad- inho, S., Biderman, S., Cao. H., Cheng, X., Chung, M., Derezynski, L., and et al. RWKV: Reinventing RNNs for the transformer era. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computa- tional Linguisties: EMNLP 2023, pp. 14048-14077, Sin- gapore, December 2023. Association for Computational Linguistics. doi: (0.186633/11/2023.fnmiiii 936. URL htt.s: httptttts///caaaaie https/://alaaathhloocooo findings 1p. 936.
pdf,
Li, J., Fang, A., Smymiss G., Ivgi, M., Jordan, M., Gadre, S., Bansal, H., Guha, E., Keh, S., Arora, K., and et al. Datacomp-lm: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406. / 1794, 2024.
2023.
Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedi- gos, I., Safahi, E., Meirom, S., Belinkov, Y., Shaley- Shwartz, S., Abend, O., Alon, R., Asida, T., Bergman, A., Glozman, R., Gokhman, M., Manevich, A., Rat- ner, N., Rozen, N., Shwartz, E., Zusman, M., and Shoham, Y. Jamba: A hybrid transformer-mane mamba lan- guage model, 2024. URL https: /ttp:///axxvvvorr/ abs/2403.1 19887.
Peng. B., Goldstein, D., Anthony, Q. G., Albalak, A., Al- caide, E., Biderman, S., Cheah, E., Ferdinan, T., GV, K. K., Hou, H., Krishna, S., Jr., R. M., Muennighoft, N., Obeid, F,, Saito, A., Song, G., Tu, H., Zhang, R., Zhao, B., Zhao, Q., Zhu, J., and Zhu, R.-J. Eagle and finch: RWKV with matrix-values states and dynamic recurrence. In First Conference on Language Modeling, 2024. URL https: peetttss//opprreeeee //pp//peeevvieeeeeee net forum? id=soz1SEiPes
Loshchilov, and Hutter, F. Decoupled weight de- cay regularization. In Proceedings of the Interna- tional Conference Learning Representations (ICLR), 2019. URL https: wwweenrrvieennet/frrrnee id=Bkg6RicqY7.
Ren, L., Liu, Y., Lu, Y., Shen, Y., Liang, C., and Chen, W. Samba: Simple hybrid state space models for effi- cient unlimited context language modeling, 2024. URL
httess:/aarsiv..rr/bbbb/// wttp://///a////...//./ 22405.0522...
Lozhkov, A., Ben Allal, L., von Werra, L., and Wolf, T. FineWeb-Edu: the finest collection educational content, 2024. URL httth:////hggg datasets /ttgg///ggaaceeee//eeeeee
Schmidinger, N., Schneckenreiter, L., Seidl, P., Schimunek, J., Hoedt, P.-.., Brandstetter, J., Mayr, A., Luukkonen. S., Hochreiter, S., and Klambauer, G. Bio-XLSTM: Genera- tive modeling, representation and in-context learning of biological and chemical sequences. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. URL https: / openreview net/ forum i dd
co/ edu.
Mistral Al Team. Codestral Mamba. Mamba.httss: /mistrall.. ai /t//e///eeee wow.estral-mammeeee... 2024. Accessed: 2025-01-30.
Nguyen, V., Carilli, M., Eryilmaz, S. B., Singh, V., Lin, M., Gimelshein, N., Desmaison, A., and Yang, E. Accelerating PyTorch with CUDA graphs, Octo- ber 2021. URL https: /ttps:::/pytthhhhhpph org/blog ingeeleetttng--yyoorr---tttttaae Accessed: 2025-01-30.
Schmied, T., Adler, T., Patil, V., Beck, M., Pöppel, K., Brandstetter, J., Klambauer, G., Pascanu, R., and Hochre- iter, S. A large recurrent action model: xLSTM en- ables fast inference for robotics tasks, 2024. URL https: http://axxvv.ggg.//
2410 22391.
Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, FlashAttentionn-:: Fast and accurate attention with asynchrony and low-precision, 2024. URL https: http://wwwwwwwwaa///////// http://aab/220700883880
OLMo, T., Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., Lambert, N., Schwenk, D., Tafjord, O., Anderson, T., Atkinson, D., Brahman, F., Clark, C., Dasigi. P., Dziri, N., Guerquin, M., Ivison, H., Koh, W., Liu, J., Malik, S., Merrill, W. Miranda, J. V., Morrison, J., Murray, T., Nam, C., Pyatkin, V., Rangapur, A., Schmitz, M., Skjonsberg, S., Wadden, D., Wilhelm, C., Wilson, M., Zettlemoyer, L., Farhadi, A., Smith, N. A., and Hajishirzi, H. 2 OLMo furious, 2025. URL https: //arxiv.. org/ prg/abs//a abs
Shazeer, N. Glu variants improve transformer, 2020. URL http://wwwrrr.aab////....0
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-LM: Training multi- billion parameter language models using model par- allelism, 2020. URL https: /aaaiaivv.o/ org/ /bs// 1909. 1909.08053.
2501 1.0.1..00011 00656.
12

---

XLSTM-7B
Sun, Y., Dong, L.,, Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to trans- former for large language models. ArXiv, abs/2307.08621, 2023. URL httpss://ap.... tettatp///aapiiieeaeaaaaaaaaoaaa stw.apppsssttttccscoooooooo
Yang, S., Wang, B., Shen, Y., Panda, R., and Kim, Y. Gated linear attention transformers with hardware-efficient train- ing. In Proceedings of the International Conference on Machine Learning (ICML), 2024b. URL https: / //opprrevii ew net. forum? d=1a5XvxFUJT.
org/CorpusID: 255993745599
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist Ilm assistants, 2023. URL https: https://huggingface. catpee.c//daseeeaeeeeee teknium/Oppettermss- 2.5.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 11809-11822. Curran Associates, Inc., 2023. URL https: httLtttt////peeeeiiiii //pp:////e.........eemm.m net/ forum
TogetherCompute.
LongDataCollections Octo- ber 2023. URL bctps://hugginn: /hwggingggccc.... to//aattasttssoogthheooompp Long-Data-Collections
Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. arXiv preprint arXiv:2309. /2284, 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., and et al. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023. doi: 10.4855/ARKXI..330..00222 doi. owg.orrg11000 100 /at/50//rrxvvv..
URL https://
Zhang, B. and Sennrich, R. Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems 32, Vancouver, Canada, 2019. URL https: /tttpp:://ppeeee /ttp///eeeeeeeeeeeeeeeeeeeeeeeee pdf?id=SlopAffi::
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Pro- cessing Systems (NeurIPS), volume 30, pp. 5998-6008. Curran Associates, Inc., 2017.
net/referrnces//
Zuo, J., Velikanov, M., Rhaiem, D. E., Chahed, I., Belkada, Y., Kunsch, G., and Hacid, H. Falcon Mamba: The first competitive attention-free language model. 2024. URL
Waleffe, R., Byeon, W., Riach, D., Norick, B., Ko- rthikanti, V. A., Dao, T. Gu, A.. Hatamizadeh, A., Singh, S., Narayanan, D., Kulshreshtha, G., Singh, V., Casper, J., Kautz, J., Shoeybi, M., and Catanzaro, B. An empirical study of Mamba- based language models. ArXiv, abs/2406.078877, 2024. URL https:/ seapiissemmtiiiccieeee otr//CorpusSD::27039133
Xiong, R. Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C. Zhang, H., Lan, Y., Wang. L., and Liu, T. On layer normalization in the Transformer archi- tecture. In III, H. D. and Singh, A. (eds.), Proceed- ings of the International Conference on Machine Learn- ing (ICML), volume 119 of Proceedings of Machine Learning Research, pp. 10524-10533, PMLR, 13-18 Jul 2020. URL https::::: http:s::://rcceeeeeeeeee peprroeeeiings... mlr press/ v119/xiong20b. html.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., and et al. Qwen2 technical report. 2024a. URL htthstts::/ httpttss//.ai.....././tt// abs/ 2els/247..00 10671.
Yang, S. and Zhang, Y. FLA: A triton-based library for hardware-efficien implementations of linear attention mechanism, January 2024. URL https: //gtt/////htb/////// com/fla com/fla-org/flastt nea rattentioo..
13

---

XLSTM-7B
A. XL Architecture Summary
The xLSTM 7B architecture consists of 32 post-up projection blocks and is described in Fig. and Tab. 4. We use GPT-NeoX-206 tokenizer (Black et al., 2022) with vocabulary size 50257 and do not tie the weights for imput layers (embedding) and output layers (logits).
Table 4. Hyperparameters of XLSTM 7B.
NUM PARAMS
VOCAB SIZE
NUM BLOCKS
MODEL DIM
NUM HEADS
6,865, 8,865,424,896
50257
32
4096
8
Swish
Norm
Headwise Norm
mLSTM
Sigmoid
C+-1
k
V
k
0
Norm
Figure 8. Improved XL.TMM Block. The lower part output-gated sequence-mix layer with the mLSTM its core, whereas the upper part is a Gated MLP (SwiGLU) as feature/chauncl-mixs layer. Multiple Heads are shown in depth, larger light gray boxes without are linear layers. For the SwiGLU we use projection factor 2.66 matching common Transformers. For the query/key dimension we factor of 0.5. The Norm layers are RMS norms (Zhang & Sennrich, 2019), the Hcadwise Norm is a Layemorm (Ba et al., 2016).
14

---

XLSTM-7B
B. Training Recipe
Optimization. Pre-training was conducted on a high-performance computing cluster comprising 128 NVIDIA H100 GPUs. We use Fully Sharded Data Parallel (FSDP) and activation checkpointing to reduce the parameter and activation memory footprint. We pre-train xLSTM 7B for total of 550K (thousand) training steps with batch size 512 and context length 8192, encompassing total 2.3T (trillion) training tokens. We apply batch size ramp-up with batch size 128 for the first 2000 steps, 256 for the next 2000 steps, and the full batch size (512) afterward. We use the Adam W optimizer (Loshchilov & Hutter, 2019) with (peak) = 5 x 10- 31 = 0.99, 32 = 0.95, e = 10-*, weight decay 0.1 and gradient clipping norm 0.5. The learning rate schedule comprises a linear warm-up over 3000 training steps, an exponential decay phase that spans 540,000 steps, and a linear cool-down lasting 7000 steps. The exponential decay factor is chosen so that 0.1 x a reached after 500,000 steps.
Sequence packing. Language datasets come with documents of highly varying lengths. To efficiently train model by processing fixed sequence length sequences (e.g. 8192 tokens), multiple shorter documents are typically packed into sequence, and the different documents are separated by an end-of-document (EOD) token. In order to avoid leaking information between independent documents that are packed into the same sequence, we reset the memory states of each MLSTM cell at the document borders signified by the EOD token. This can be easily achieved by explicitly setting the forget gate value to zero, resetting the memory state to the zero matrix.
datasets for pre-training, The dataset selection is divided into two training stages: In the first stage lasting 500K (thousand) training steps, we train exclusively the DCLM dataset (Li et al., 2024). In the second stage (50K steps) towards the end of the training, we use combination of datasets that prioritizes math, coding, and question-and-answed (Q&A) data. The dataset proportions for the second stage are listed in the second column of Tab. 5.
Dataset selection. We only use publicly available
Similarly to Zuo et al. (2024), the second training stage includes a collection of small supervised fine-tuning (SFT) Q&A datasets to improve the model's s understanding of texts involving questions and answers. These SFT datasets are all publicly available and consist of NuminaMath CoT (LI et al., 2024), MetaMathQA (Yu et al., 2023). Tulu v3.1 (Lambert et al.. 2024), OpenHermes 2.5 (Teknium, 2023), GSM8K (Cobbe et al., 2021), and Smoltalk (subsets magpie-ultra longalign, and self-oss-instruct) (Allal et al., 2024).
For longer context training we replace the high-quality data cool-down by a longer context version keeping the number of tokens per step and the number of steps fixed. The batch size is reduced from 512 to 128, while increasing the context length to 32768. We replace a large share of the DCLM dataset part with long context text collections, namely LongDataCollec- tions (TogetherCompute, 2023), LongAlign 10k 10k (Bai et al., 2024), AntiHayStacck (Pan, 2024) and LongAlpacal 2k (Chen al., 2024), see third column of Tab. 5.
Table 5. Dataset Proportions for second training stage in standard and longer context mode,
DATASET NAME
PROPORTION STANDARD
PROPORTION LONGCTX
DCLM (Li et al., 2024)
40%
20 %
FineWeb- Edu (Lozhkov et al., 2024)
15%
15%
Cosmopedia (Ben Allal et al., 2024)
10%
10%
ProofPile-2 (Azerbayev et al., 2023)
15%
15%
TheStack (Kocetkov et al., 2023)
15%
15%
SFT datasets (see Sec. B)
5%
5%
LongDataCollections (TogetherCompute, 2023)
15%
LongAlign1 Ok (Bai et al., 2024)
1%
AntiHayStack (Pan, 2024)
1%
LongAlpaca122 12k (Chen al., 2024)
2%
Ablation Training For hyperparameter tuning and ablation trainings ("-abl") at the 7B scale, we use a shorter training cycle with 76,000 training steps at context length 8192 and batch size 256, resulting in 160B tokens. We use a linear warmup of 3000 steps, cosine decay to 10% of the peak learning rate at 75,000 steps and a linear cooldown of 1,000 steps to learning rate 0 at the end. Here, we only train on a subset of the DCLM dataset, without high-quality data in the late phase of pre-training. Peak learning rate and other training hyperparameters are the same as for the main training.
15

---

XLSTM-7B
C. Experiments
C.1. Extended Evaluation
To enable comparability to older models, we evaluate our models on the task selection from the first version of the HuggingFace leaderboard using HuggingFace's lighteval (Beeching et al., 2023; Fourrier et al., 2023). The results Tab, show that there is trend upwards in metries from older (e.g. Llama 2) differences and ordering between models vary across the tasks.
models (e.g. Llama 3.1), but that the
Table 6. Model Performance Huggingface Leaderboard vI based on lighteval by HuggingFace. 1 indicates larger values are better.
MODEL
ARC-C1 MMLU1 1 HELLASWAG
WINOGRANDE
TRUTHFUI.QA
OPENBookQA
PIQA f AVERAGE
TRANSFORMERS
Llama-3.
0.663
0.720
0.745
0,362
0.447
0.818
0.617
Llama-2 Llama-2-7B-he 2-7B-hf
0.468
0.687
0.706
0.318
0.412
0.786
0.555
OLMo-7B-hr Bh
0.443
0.286
0.673
0.661
0,301
0.383
0.801
0.507
Qwen2.5.70 7B
0.617
0.753
0.700
0.717
0.478
0.458
0,804
0.647
Gemma-7H
0.593
0.640
0.721
0.740
0,381
0.436
0.813
0.618
HYDRID MODELS
Zambu2-7B
0.672
0.683
0.740
0.801
0.479
0.468
0.802
0.664
RECURRENT MODELS
0.622
0.709
0.743
0.459
0.460
0,822
0.631
Matcon-Mamha-77
(pre-decay)
0.520
0.573
0.699
0.719
0.312
0.430
0.801
0.579
Mamba-Coodettal-78 (v0.1)
0.486
0.501
0.626
0.618
0.358
0,380
0,771
0,534
RWKVVvS-Eagle-78
0.449
0.313
0.622
0.663
0.330
0,393
0,772
0.506
RWKVVv6-Finch-77
0.471
0.442
0.656
0.696
0.347
0,399
0,792
0.543
7B
0.578
0.714
0.448
0.819
0.613
XI.TTM 7B LCTX
0.516
0.588
0.715
0.740
0.374
0.429
0.819
0.597
C.2. Ablation Experiments
Effect of the Pre-norm Layer Choice (Fig. 9). Here we asses the effect of different normalization layer choices for the pre-norm in (12) and the state-norm in (6), both for XLSTM with a pre-up projection block Beck et al. (2024) and our new post-up projection architecture used for xLSTM We use soft-capping and the negative input bias initialization (see Sec. 3.2 and 5.3) for both architectures. For this experiment, we train models with 1.4B parameters for 31,000 steps using context length 8192 and batch size 256. Fig. 9 shows the validation loss and gradient norm for the different architectures and normalization layer choices over the course of training (only the 15,000 steps are shown). As can be seen, using LayerNorm as the pre-norm layer leads to very large gradient norms and diverging validation loss after few training steps, whereas models with RMSNorm train stably. For the state-norm layer, the norm type has no impact on the training dynamics.
Post-Up Projection with Pre-LN State-LN
Pre-Up Projection with Pre-LN State-LN
Post-Up Projection with Pre-RMSN & State-LN
Pre-Up Projection with Pre-RMSN State-RMSN
Post-Up Projection with Pre-RMSN State-RMSN
4.0
10a
3.8
3.6
10a
Paldation Loss
Gradient Norm
3.2
2.8
10a
2.6
2.4
10-1
2.2
0.0
0.2
0.4
0.8
1.0
1.2
1.4
0.0
0.4
0.6
0.8
1.0
1.2
1.4
Steps
1e4
Steps
1e4
Figure Comparison pre-up projection and post-up projection blocks with different combinations of RMSNorm and LayerNorm. At each step, the plot shows the maximum gradient norm observed within the previous 50 steps.
16

---

XLSTM-7B
Effect of Soft-Capping (Fig. 10). The in Fig. 10 show the effect of soft-capping for two 7B sized xLSTM models trained for 76,000 steps at batch size 256 and context length 8192, for effective 160B tokens.
No Softcap
With Softcap
2.7
0.35
0.30
Valldation Loss
Gradient Norm
2.5
2.4
0.20
0.15
2.3
2.2
0.05
Steps
1e4
Steps
1e4
Figure Effect of softcapping. Two 7B sized xLSTM models are trained and without caft-capping 160B tokens. The lower gradient norm noise on the right is clear indicator for better model performance on the left of the model trained with softcapping. At each step, plot shows the maximum gradient norm observed within the previous steps.
Effect of Negative Input Gate Bias Init (Fig. 11). In this experiment we train 160M parameter models with batch size 128 and context length 4096 and vary the input gate bias initialization [0, -2, -5, -10.. The weights of the input gates initialized to O.
In Figure 1l we observe that initializing the input gate biases at -10 effectively mitigates gradient norm spikes and reduces gradient norm variance during training. In our experiments up to 7B parameters we observed this behavior transfers across model scales.
We therefore initialize the input gate biases -10. For extensive discussion of this behavior we refer concurrent work by? ?.
Biasinit 0
Biasinit -5
Blasinit -2
Blasinit -10 10a
3.6
3.5
Validation Loss
3.4
Gradlent Norm
101
3.3
3.2
109
3.1
10-1
0.0
0.5
1.0
1.5
2.0 Steps
2.5
3.0
3.5
0.0
0.5
1.0
2.0
2.5
3.0
3.5
1e4
Steps
1e4
Figure 11. Effect of the Bias Initialization. We conduct experiments with four different input gate biases at the 160M parameter scale, with validation loss on depicted to left and gradient show large gradient norm spikes, which results in worse training results. Only the lowest initialization can maintain smooth and low gradient norms with the best validation perplexities. The reason for this behavior is studied more detail (?). each step, the plot shows the maximum gradient norm observed within the previous 50 steps.
the right, along the training steps. The higher input gate bias initializations
17

---

XLSTM-7B
Effect of the Learning Rate Scheduler (Fig. 12). In largest experiments, choose linear warmup followed by an exponential decay as learning schedule in order enable continued pre-training with more tokens and without additional warmup. However, smaller-scale experiments in Fig. 12 show the benefit of cosine schedule over exponential one.
Exponential 76k Steps
Cosine 76k Steps
Exponential 150k Steps
Cosine 150k Steps
1e-4
8
14.5
7
6
14.0
Leaming Rate
5
Perplexity
13.5
13.0
3
2
11.5
0
11.0
0.0
0.2
0.4
0.6
0.8 Steps
1.0
12
14
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
le5
Steps
1e5
Figure 12. Effect of Learning Rate Scheduler. The tested learning rate schedules shown the left, with the corresponding training perplexities on the right. While the exponential learning rate schedule can be continued trivially, the cosine schedule actually works slightly better given fixed number of iterations. The learning cooldown zero the end gives similar and significant benefit in hoth cases.
Effect of Memory State Size and Input Gate Long Context Evaluations (Fig. 13, Tab. 7 and 8). In order the influence of the head numbers (cell dimensions) and input gate long context abilities, test the ablation models trained in Sec. 5.3 for their performance in the RULER benchmark (Hsieh et al., 2024). The results in Fig. 13 show that, while the effect of the head number and equivalently the recurrent memory is inconclusive, the models strongly benefit from the learnable, exponential input gate for the long context performance.
Default
NH8
NH8 IGateFixed -10
LCTX
NH16
NH8 IGateFixed
NH4
NH32
70
60
Average Accuracy (%)
50
40
16384
32768
65536
Context Length [Tokens]
Figure 13. RULER average accuracies for different number of heads/cell dimensions, and fixed input gate. The ablations are trained on 160B tokens 8k context.
Additionally, we evaluate our ablation versions trained for 160B tokens and evaluated on the current and old HuggingFace LLM Leaderboard as Tab. and 6, respectively. Results in Tab. 7, 8 show only slight influence of the head dimensions or fixing input gate. Only fixing the input gate to the very small value of its standard bias initialization has a stronger impact on the Leaderboard vl.
18

---

XLSTM-7B
Table Model Performance for different number of heads and non-trainable imput gate the Huggingface Leaderhoard v2 tasks. indicates larger values are better.
MODEL
BBH
MMLU-PRO
MATH
MUSR
GPQA t
IFEVAL f
AVERAGE f
xLSTM 7B abl NH4
0.306
0.114
0.004
0.363
0.253
0.160
0.200
XLSTM 7B abl NH8
0,304
0.115
0.002
0.363
0.248
0.173
0.201
XLSTM 7B abl NH16
0.317
0.119
0.002
0.390
0.258
0.161
0.208
XLSTM 7B abl NHH2
0.327
0.120
0.001
0.379
0.256
0.171
0.209
xLSTM 7B abl NHE IGateFixed 0
0.303
0.117
0.004
0.381
0.229
0.149
0.197
xLSTM 7B abl NH8 IGateFixed -10
0.308
0.109
0.000
0.357
0.253
0.165
0.199
XLSTM 7B
0.381
0.242
0.036
0.379
0.280
0.244
0.260
XLSTM 7B LCTX
0.390
0.252
0.040
0.374
0.253
0.234
0.257
Table 8. Model Performance for different number of heads and non-trainable input gate the Huggingface Leaderboard v1 tasks. 1 indicates larger values are better.
MODEL
ARC-C1
MMLU
HELLASWAG
WINOGRANDE?
1 OPENBOOKQA TRUTHFULQA
XLSTM 7B abl NH4
0.492
0.296
0.665
0.672
0.282
0,405
0.798
0,516
XI.STM 7B ahl NH8
0.487
0,292
0.669
0.680
0.302
0.426
0.791
0,521
XL.TMM 7B ahl NH16
0,505
0,351
0.668
0.701
0.294
0.409
0.796
0.532
abl
0.666
xI.STM 7B ahl NH8 IGateFixed
0.464
0,292
0.658
0.672
0,280
0.415
0.788
0.510
XLSTM 7B abl NHB 1GateFised -10
0.241
0.250
0.340
0.519
0.286
0,226
0.681
0,363
xLSTM 7B xL.TM 7B LCTX
0.574
0.578
0.714
0.738
0.419
0.448
0.819
0.613
0,516
0.588
0.715
0.374
0,429
D. Speed Comparison to Optimized Inference Frameworks
While tune the baseline models' HuggingFace implementations for speed in similar way as for our xLSTM 7B architecture, there other frameworks, e.g. vLLM (Kwon et al., 2023), which optimized further for inference speeds. In Figures (18, 19), we compare our HuggingFace generation speeds to vL.LM generation speeds. In Figures (14, 15, 16, 17), we show the respective comparison for prefill, prefill+generation time and prefill throughput. We compare xLSTM in in both the HuggingFace implementation to Llama-3. 1.1-a- Llama-2-78 2-7B---2-B-hh-- Galcon-Mamba 7B and Maiitiall--aaabbbbb whereas optimized HuggingFace and vLLM. For all vLLM speeds, we use PoTorch 2.6.0 to enable Codestral- for the HuggingFace speed experiments, we use Py ToTchh 2.5.1. 1. Although there are speed improvements in vLLM, especially for brasee Llama models, XLSTM 7B continues to be the fastest model compared to others in both frameworks.
Liama 3 86
FalconMamba 78
XLSTM 76
Llama 3 BB
FalconMamba 78
XLSTM 78
Liama 2 7B
CodestralMamba 7B
Liama 2 7B
CodestralMamba 78
1.0
4
1.0
Time to First 100 Tokens [s]
Tolens
[s]
Time to First Token
0.8
0.8
0.6
0.4
0.2
15000 Profill Length [Tokens]
5000 Profil Length [Takens)
15000 Prefill Length [Tokenns]
5000
5000 10000 15000 Prefil Length [Tokens]
Figure 14, Prefill time (time first and first 100 tokens) for HuggingFace implementations. Repetition of Figure 6 for comparison vI..MM the right.
Figure 15. Prefill time (time to first and to first 100 tokens) for vL.LM implementations, except xI.STM 7B (HuggingFace implementation).
19

---

XLSTM-7B
Liama 88
FalconMamba 78
XLSTM 78
FalconMamba
Liama 78
Codestra iMamba 7B
Llama 2
CodestralMaaba
x10*
x10*
47580
47543
47627
47596
47E42
47543
47642
40029
Tokens per Second
373-1
36348
33938
37864
Tokens per Seco Second
37990095
35580
28014
26320
16870
16540
16449
36027
17830
19240 17520
1634
16977
16644
1968
325 CTX-2908
B5=16 CTX-4096
85
B5md CTX-16384
BS CTX=52768
BS-32 CTX=2068
B5=16 CTX=4096
85- B
Bbed
DS=2
CTX-8192 Tokens
CTX-8192
CTX-16386
CTX-52768
Tokens
Figure Pre-fill throughput FugeingFace implementa- tions. Repetition of Figure 7 to compare to vLLM on the right.
Figure 17. Pre-fil fill throughput for HuggingFace implementa- tions, except for xLSTM 7B (HuggingFace implementation).
Llama 8B
FalconMamba
XLSTM 7B
Liama 2 78
CodestralMambe 78
Liama 3 8B
FalconMamba 78
XLSTM 78
Liama
CodestralMamba 78
500
500
200
200
Tokens per Second (VLLM)
Generation Time (VILLM)
Ceneration Time 1s]
400
400
Tokens Second
300
300
100
100
200
200
50
50
100
100
10000 15000
10000 15000 Generation Length [Takens]
5000
5000
10000 15000
5000 10000 15000
Generation Length [Tokens]
Prefill Length [Tokens]
Prefill Length [Tokens]
Figure Generation Time comparison HuggingFace to vLLM. While for short generation sequences, all models are close in their generation time, for vLLMM backend is faster for Transformers than for Mamba models within the 16k gen- crated tokens. XLSTM in the HuggingFace backend continues be the fastest model from short to long sequences.
Figure 19. Generation Throughput comparison HuggingFace to vLLM. Transformers (Llama models) show strong speed up vLLM, still xLSTM the fastest model throughout the prefill context lengths. There is odd transition for Mamba models from 4k 8k prefill context.
20

---

XLSTM-7B
E. FLOP Counting
count the number of FLOPs in a forward pass of the mLSTM. use a factor of 2 to describe the multiply accumulate cost.
We factors denoted F_X to describe the number of FLOPs for operation X (e.g. F_exp for the exponential function).
By default we set all of these factors to 1.
E.1. FLOPs for the mLSTM Operation
Inter-chunk recurrent:
Chhnkwise gates: num_heads num-chunks
( (0.5xchunk size x (chunk size + 1) 2xchunk size
Gattss max state: num_heads x num_chunks
(3 + max exp + chunk size (3 + exp))
NNmerator: num num_heads x num_chunks
x (2xd qk x d v 4xchunk size x d qk x d v + 3xchunk size x d qk)
Denominator: num_heads x num_chunks x d_qk 4xchunk_size d.gk) )
Intra-chunk parallel:
Gate matrix: num heads num chunks
0.5 chunk_size (chunk_sizz 1)
chunk size chunk size (3 + F mask F max exp) chunk_size (1 + F__mx)
- Gated Attn logits: num heads x num chunks
2xchunk size x chunk size x ( (1+d.q)
Numerator: num heads x num chunks
2xchunk-size x chunk x d_v
Denominator: num_heade x num-chunks x 2 x chunk x chunk size
Output combination: num heads
chunks
( chunk size x ( + F_max)
+ chunk size x ( (2+ F abs F exp + max v))
E.2. FLOPs for the mLSTM in Transformer Backbone
For computing the number of FLOPs follow the procedure from Hoffmann al. (2022). We include the FLOPs contributed the embedding matrices. We do not include RMS- Layer-Norm and skip connection FLOPs We assume that the backward pass has times the number of FLOPs of the forward pass. For the forward pass, the number of FLOPs of the mLSTM for single sequence can be approximated by:
Embeddings
- x len vocab_size x momodel
mL.STM (single layer)
Query, key, value, input and forget gate projections:
2 x seq len model num heads x (2 qk + d v 2)
Output gate and projection:
4 x sen d.model x num num-heads d_v
+ seq_len x num num_heads x d_y x F_sig
mLSTM cell: See above.
Gated Feedforward (single layer)
21

---

XLSTM-7B
6 x seq_len d_model d_model proj_factor_ft _ff
2 x seq len model F swiish
Final Logits
2 x seq x d.model x vocab_size
Total forward pass FLOPs:
embeddings + num_layers (mLSTM feedforward) final_logits
E.3. FLOPs for the Transformer with Self-Attention
We use the FLOP computations from Hoffmann et al. (2022), with the difference that we use gated feedforward blocks.
Embeddings
X seq_len vocab_size d_model
Attention (single layer)
Key, query and value projections:
2 x seq_len x d_model num_heads x (2 x d_qk + d_v)
Key @ query logits:
seq_len seq_len (d.qk num hea_heads)
Softmax: x seqllen x seq_len x num heads
Softmax @ query reductions: seq_len x sen (num heads d.gk)
Final linear: 2 x seq d_model (num-heads x d_v)
Gated Feedforward (single layer)
6 X leq d_model d_model froj-factor.ft _ff
+22 x seq len model F.swish
Final Logits
- 2 x seqleen d.model vocab_size
Total forward pass FLOPs:
embeddings + namelayers (attention feedffoward) + final_logits
F. Parameter Counting
In this section we count the number of paramters in the mLSTM and compare it to the number of parameters in Transformer with self-attention. We assume that the model does not use weight tying and omits biases.
F.1. Parameter Counting for the mLSTM
Embeddings
vocab_size d_model
mLSTM (single layer)
qkv: d-model num heads x (2 x d_qk + d.v)
Input and forget gate: 2 d_model num_heads + num_heads
Output gate: d_model x d_model
Output projection: d_model d_model
Norm: d model
Gated Feedforward (single layer)
22

---

XLSTM-7B
3 X d_model x d_model proj_factor_ft
Norm (single layer)
d.model
Final Logits:
- d_model x vocab_size
Total number of parameters:
embeddings + num_layers (mLSTM feedforward 2 x norm) norm final logits
E22 Parameter Counting for the Transformer with Self-Attention
Embeddings
vocabbsiie moddel
Attention (single layer)
-qkv: d_model heads x (2 d.qk + d.v)
Output projection: d_model x d_model
Gated Feedforward (single layer)
3 x d_model d_model frojofactor
Norm (single layer)
d.model
Final Logits:
d_model x vocab_size
Total number of parameters:
embeddings + numelayers (attention feedforward 2 x norm) norm + Limallogiis
23


Reviews:
[{"review_id": "mTTvWbRn8e", "reviewer": "Reviewer_WcdK", "summary": "This paper mainly scales XLSTM to 7B using some optimization technique", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for highlighting our optimizations as a clear contribution and seeing strengths in our results in the trade offs we make, our good latency, and the interesting xLSTM framework. \n\n### Discussion on the mLSTM cell\n\nWe agree with the reviewer that there is only a brief discussion on the mLSTM cell in Section 2 of our paper. \nThe reason is that our goal in Section 2 is to only review the fundamentals of the mLSTM.  For a more in depth discussion, we would like to point the reviewer to the original xLSTM paper (https://arxiv.org/abs/2405.04517), where the mLSTM has been introduced. \n\nThe main motivation for using the mLSTM in the xLSTM 7B is its high efficiency due to its full parallelizability. Moreover, in the xLSTM 7B we still rely on two main features of the mLSTM: The improved gating with exponential gating and the enhanced memory capacity with its matrix memory, which we found to be beneficial in language modeling.\n\n### Open-Sourcing and Code Release\n\nUnfortunately, we did not upload the code as part of the submission, but we can assure you that it will be open sourced.\n\nWe thank the reviewer again for the helpful feedback and hope they find their main concerns addressed.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "not many theoretical claims.", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "another LLM models", "essential_references_not_discussed": "yes", "other_strengths_and_weaknesses": "Strengths:\n\n1. The contriution of this paper is clear, scaling xLSTM model to 7B model using optimization techinqies.\n\n2. The experimental results show the trade-off, it seems that its performance is not as good as mamba, but its lentency is good.\n\n3. This framework may be interesting for LLM community.\n\nWeakness:\n1. lack discuss about the core design  mLSTMCell , any deep insight or theoretical analysis.\n\n2. it is better to open source the project", "other_comments_or_suggestions": "no", "questions_for_authors": "1. lack discuss about the core design  mLSTMCell , any deep insight or theoretical analysis.\n\n2. it is better to open source the project", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xVno81CMD1", "reviewer": "Reviewer_Em5h", "summary": "In this paper, the authors introduce a new 7B LLM xLSTM 7B. The model is built upon optimized xLSTM architecture to achieve better training stability and efficiency. Extensive experiments show that xLSTM 7B is memory and computation efficient compared to attention-based models and Mamba-based models, and achieves comparable performance to recurrent models of a similar size.\n\n## update after rebuttal\nI thank the effort the authors made in the rebuttal about comparison with optimized transformer inference. I keep my positive score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful feedback. \nWe appreciate that the reviewer finds our paper well-written, easy to follow and that your claims are supported by evidence as well as that our experimental design is valid and reasonable. \n\n### Architecture Illustrations\nWe agree with the reviewer, our illustrations of the architecture could be improved by adding names to the respective blocks. \nThis is a good suggestion and we will update our figures in the final version of the paper. \n\n### Metrics with Optimized Inference Engines (like vLLM)\nWe expect that the speed of xLSTM will only get faster than our numbers currently reported in  the paper with the current optimizations of the Huggingface models with torch.compile and CUDA graphs.\n\nWe acknowledge that our current benchmark setup does not use any production grade inference serving frameworks (like for e.g. vLLM) and we are open about this in the paper. \nWe did our best to optimize the pure PyTorch Huggingface implementations as much as we can. \n\nFor a fair comparison, we compare baselines from Huggingface with the same optimizations as for xLSTM (torch.compile and CUDA graphs).\n\nHowever, we agree with the reviewer it would be interesting to measure the speed also in these optimized inference environments. Currently, xLSTM is not yet integrated in these inference frameworks, but we intend to do so in the future.\n\nSince some of our baselines are already integrated in vLLM, we added additional benchmark results for FalconMamba, CodestralMamba, Llama2 and Llama3 in vLLM. \n\nWe observe a speed-up of Llama 3 for longer context lengths, also small speed-ups of Mamba-based models compared to our optimized HuggingFace versions. However, xLSTM 7B is still the fastest model (even when compared to baselines from optimized frameworks like vLLM), both in generation and prefill, with larger margin towards longer contexts, due to the quadratic scaling of Transformers. See the results at: [https://i.postimg.cc/1XcMCyQV/Rebuttal-Plots.jpg](https://i.postimg.cc/1XcMCyQV/Rebuttal-Plots.jpg) \n\nWe thank the reviewer again for their valuable comments that help to improve our paper.", "claims_and_evidence": "- The authors claim the xLSTM is efficient in inference GPU memory consumption and computation.\n    - This claim is supported by experiments in Figure 4, 5, 6.\n- The authors claim XLSTM 7B is comparable to existing LLMs.\n    - This is evidenced by results in Table 1 and Table 6, showing close or superior performance to SOTA recurrent LLMs while surpassing some attention-based LLMs.\n- The optimized xLSTM architecture is claimed to improve training stability.\n    - This is evidenced in Appendix C.2. For example, with the softcap, the training gradient norm is smaller.", "methods_and_evaluation_criteria": "The methods and evaluation criteria make sense. For both the performance and the efficiency, the adopted metrics are widely used. And the compared baseline modes are also common ones.", "theoretical_claims": "The paper does not provide theoretical claims.", "experimental_designs_or_analyses": "I checked the validity of all experimental designs, and they make sense to me.\n\n- The benchmarks in Table 1 and the models chosen are widely used ones, which leads to a valid comparison.\n- The speed benchmarks consider long context token generation speed, memory consumption, and time to first token. All these metrics are reasonable for evaluation.", "relation_to_broader_scientific_literature": "The work is related to all LLM architectures, especially recurrent LLMs.", "essential_references_not_discussed": "### \n\nI am familiar with general LLM architectures. For example,\n\n> [1] Grattafiori, Aaron, et al. \"The llama 3 herd of models.\" *arXiv preprint arXiv:2407.21783* (2024). \\\n[2] Gu, Albert, and Tri Dao. \"Mamba: Linear-time sequence modeling with selective state spaces.\" *arXiv preprint arXiv:2312.00752* (2023).\n>", "other_strengths_and_weaknesses": "Strength:\n\n- The paper is well-written and easy to follow.\n\nWeakness:\n\n- The Figure 1 and Figure 7 are confusing. While I can guess the meaning of most blocks without their names, it is better to add names.", "other_comments_or_suggestions": "None.", "questions_for_authors": "In the paper, you do experiments for speed metrics with Huggingface transformers, so I wonder what would the speed of xLSTM inference be like when compared to attention-based LLMs with vLLM or other inference engines, that provide optimized inference speed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qUZyXroZxY", "reviewer": "Reviewer_MywF", "summary": "This work builds on the work by Beck et al. on language modeling with xLSTMs. It introduces several architecture modifications and tricks that support efficiency and training stability, and it introduces an (open sourced) pre-trained 7B parameter language model based on mLSTM. The paper contains a detailed description and evaluation of these modifications with ablations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful feedback. We appreciate that the reviewer finds that our claims are supported by evidence, that our experimental results are thorough and detailed and that our strong model is a valuable contribution to the open-source community. \n\n### Attempt on Suggested Improvements\nStarting from the models in the xLSTM paper, this study already improves the pre-training dataset by switching from SlimPajama in the xLSTM paper to the DCLM dataset. \nAt the time of training the xLSTM 7B, this was the probably best open source pre-training dataset, to the best of our knowledge. \n\nOther competitor models used custom internal pre-training datasets (Codestral-Mamba, Falcon-Mamba), which make a re-training unfeasible.\n\nWhile the main focus of this study are architecture optimizations, improving the pre-training data is an exciting future opportunity to train even better xLSTM models and we will continue to work towards this goal.\n\nFigure 5: Time to first token includes the prefill time\nYes, for the time to first token metric in Figure 5 the prefill time is included.\nFigure 5 (left) measures the time to the first token (or latency), i.e. the time it takes until the model answers with the first token. \nFigure 5 (right) measures the time until the model has answered with 100 tokens. This corresponds to a mix between prefill and generate time. \n\n### Soft-capping parameters\nThis is an interesting question that we did not investigate in our work yet. The logit soft capping values were taken from the Gemma 2 technical report. For the gate soft-capping the range is intended to be well outside the interesting initialization range (see attached additional paper).\n\n### Input gate & long context evaluations\nMany of the long context tasks in RULER contain only small parts of highly relevant text among a lot of unnecessary \"filling\" text. The exponential input gate can increase the magnitude of the important parts within the linear matrix memory, and this seems to help improve the task performance. Still, it might be interesting to look at some examples qualitatively to support this mechanistically.\n\n### Non-linear RNNs\nWhile non-linear models have the theoretical advantage of state tracking capabilities, in our preliminary experiments, we did not see any benefit of including those for human language modeling tasks. Our xLSTM-7B architecture tries to maximize speed in both training and inference at high performance in language modeling, which is why sLSTM was not included here. In recent work, researchers have shown how to maximize speed for sLSTM and other non-linear RNN variants, but the training speeds reported there are still far behind what can be achieved with mLSTM (as a linear RNN) and Transformers. [1]\n\nWe agree with the reviewer, it is an interesting question whether state-tracking capable architectures such as non-linear RNNs show benefits on math and code downstream tasks, which should be investigated further in future work. \n\n\n\nWe thank the reviewer again for their valuable comments that help to improve our paper.\n\n[1] Pöppel, Korbinian, Maximilian Beck, and Sepp Hochreiter. \"FlashRNN: I/O-Aware Optimization of Traditional RNNs on modern hardware.\" The Thirteenth International Conference on Learning Representations. 2025.", "claims_and_evidence": "Yes, claims are supported with detailed empirical evidence (mostly in the form of ablation studies).", "methods_and_evaluation_criteria": "Yes, models are evaluated on standard language modeling and long context benchmark tasks.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Experimental results are thorough and detailed.", "relation_to_broader_scientific_literature": "This is an empirical study, and prior work is discussed (as it should be) primarily through quantitative comparisons to existing models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper introduces a strong new open source pre-trained model, which is as valuable contribution. \n\nAs stated in the paper, the performance of the model ranks somewhere in the mid-range among other, similar size models. The paper also states that with a better training set mix (more data, better curation, more emphasis on math and code in early training phase) performance could match stronger models. This is a bit of a strange claim to make, given that the main aim of this paper is to provide an empirical study and improvement of the existing architecture (from Beck et al) Why not attempt and report on those suggested improvements?", "other_comments_or_suggestions": "There should be citations for RMSNorm and Layernorm the first time they occur (line 187 or before). \n\nFigure 5: to clarify: these results all include the time it takes to consume the prefill tokens? \n\nHow sensitive is model performance with respect to choice of soft-capping parameters? Also, why was it set to 15 and 30, for gates and logits, respectively? How sensitive is that choice? \n\nIt is interesting that learnable input gates do not matter much, except for long-context evaluations. Any insights into why that is? \n\nPre- and post-up projection should be described in the paper to make it more self-contained. Currently, the paper relies on definitions in Beck et al. and Gu & Dao. \n\nWhat about non-linear models? RNNs hold the promise to beat transformers in tasks that rely heavily on state-tracking (as opposed to natural language modeling, where this does not seem to be the case as much), but they require non-linearity (or at least special considerations regarding the eigenvalues of the transition matrix). Have the authors considered exploring sLSTM blocks over just mLSTM in this study? Related to this point, as well as the authors' point on the dependence of performance on training set amount and mix: I am wondering how much the introduction of non-linearities at the cost of parallelism during training would really reduce efficiency, and how much this would really matter with respect to downstream metrics - in particular, in tasks that are heavy on state tracking requirements (such as code and math)?", "questions_for_authors": "See previous section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper improves the xLSTM architecture and scales it to the 7B parameter model size. It proposes several ways to increase the efficiency and stability of xLSTM, especially at scale. The resulting model performs well compared to related recurrent models as well as transformers with strong efficiency at inference time. All reviewers are positive about this work.
Decision: Accept (poster)