xLSTM: Extended Long Short-Term Memory
Authors: Maximilian Beck
Korbinian Pöppel
Markus Spanring
Andreas Auer
Oleksandra Prudnikova
Michael K Kopp
Günter Klambauer
Johannes Brandstetter
Sepp Hochreiter
Venue: NeurIPS 2024 spotlight (2024)
Keywords: LSTM
LLM
Language modeling
NLP
Memory
Primary area: deep_learning_architectures
TL;DR: We extend the LSTM architecture with exponential gating and new memory structures and show that this new xLSTM performs favorably on large-scale language modeling tasks.

Abstract: In the 1990s, the constant error carousel and gating were introduced as the central ideas of the Long Short-Term Memory (LSTM). Since then, LSTMs have stood the test of time and contributed to numerous deep learning success stories, in particular they constituted the first Large Language Models (LLMs). However, the advent of the Transformer technology with parallelizable self-attention at its core marked the dawn of a new era, outpacing LSTMs at scale.  We now raise a simple question: How far do we get in language modeling when scaling LSTMs to billions of parameters, leveraging the latest techniques from modern LLMs, but mitigating known limitations of LSTMs? Firstly, we introduce exponential gating with appropriate normalization and stabilization techniques. Secondly, we modify the LSTM memory structure, obtaining: (i) sLSTM with a scalar memory, a scalar update, and new memory mixing, (ii) mLSTM that is fully parallelizable with a matrix memory and a covariance update rule. Integrating these LSTM extensions into residual block backbones yields xLSTM blocks that are then residually stacked into xLSTM architectures. Exponential gating and modified memory structures boost xLSTM capabilities to perform favorably when compared to state-of-the-art Transformers and State Space Models, both in performance and scaling.

xLSTM: Extended Long Short-Term Memory
Maximilian Beck" 1,2,3
Korbinian Pöppel
1,2,3
Markus Spanring
Andreas Auer 1,2
Oleksandra Prudnikova
Michael Kopp
Günter Klambauer 1,2,3
Johannes Brandstetter 1,2,3
Sepp Hochreiter 1,2,3
"Equal contribution
ELLIS Unit, LIT AI Lab, Institute for Machine Learning, JKU Linz, Austria NXXI Lab, Linz, Austria, NNXAI GmbH, Linz, Austria
Abstract
In the 1990s, the constant carousel and gating introduced the central ideas of the Long Short-Term Memory (LSTM).. Since then, LSTMs have stood the of time and contributed to numerous deep learning success stories, in particular they constituted the first Large Language Models (LLMs). However, the advent of the Transformer technology with parallelizable self-attention its core marked the dawn of new era, outpacing LSTMs at scale. We now raise a simple question: How far do we get language modeling when scaling LSTMs billions of parameters, leveraging the latest techniques from modern LLMs. but mitigating known limitations LSTMs? Firstly, introduce exponential gating with appropriate normalization and stabilization techniques. Secondly, we modify the L.STM memory structure, obtaining: (i) sLSTM with sealar memory, a scalar update, and new memory mixing. (ii) mLSTM that is fully parallelizable with a matrix memory and a covariance update rule. Integrating these LSTM extensions into residual block backbones yields xLSTM blocks that are then residually stacked into XLSTM architectures. Exponential gating and modified memory structures boost xLSTM capabilities to perform favorably when compared to state-of-the the-art Transformers and State Space Models, both in performance and scaling. Code available at: https: // thub. Com/NX-aa AI/xlstm
Introduction
The Long Short-Term Memory (LSTM) ideas (Hochreiter, 1991; Hochreiter & Schmidhuber; 1997b,a), i.e., the constant error carousel and gating, introduced overcome the vanishing gradient problem of recurrent neural networks (Hochreiter, 1991; Hochreiter et al., 2000):
Ct
Ct-1
+
it
2t
he
Oz
v Cz
(I)
The constant error carousel is the additive update of the cell state c--1 (green) by cell inputs and moderated by sigmoid gates (blue). The input gate i, and the forget gate f, control this update, while the output gate o, controls the output of the memory cell, i.e. the hidden state ht. The cell state is normalized or squashed by t and then output gating gives the hidden state.
LSTMs have been successfully applied various domains (Hochreiter et al., 2001, 2007; Schmid- huber, 2015). and prevailed over text generation until the dawn of Transformers in 2017 (Vaswani al., 2017). The effectiveness of LSTMs has been demonstrated at numerous sequence-related tasks such as generating text (Graves, 2013; Karpathy, 2015), generating handwritings (Graves, 2013), loquence-to-squeene translation (Sutskever et al., 2014), evaluating computer programs (Zaremba & Sutskever, 2014), generating image captions (Karpathy & Fei-Fei, 2015; Hossain et al., 2019), generating source code (Karpathy, 2015), rainfall-runoff modeling (Kratzert al., 2018, 2019).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

or hydrological models for flooding warnings (Nearing et al., 2024). In reinforcement learning, LSTMs are the best performing sequence models, e.g., the AlphaStar model for StarCraft II (Vinyals et al., 2017), the OpenAl Five model for Dota 2 (Karpathy, 2019), and models of the magnetic controller for nuclear fusion (Degrave et al., 2022). LSTMMs excel at learning abstractions, i.e., adeptly extracting semantic information and storing it in their memory cells (Karpathy, 2015), which for example became evident by number and syntax neurons (Lakretz et al., 2019), linguistic neurons (Bau et al., 2019), and sentiment neurons (Radford et al., 2017). LSTMs are still used in highly relevant applications (Degrave et al., 2022; Nearing et al., 2024) and have stood the test of time.
Despite their tremendous successes, LSTMs have three main limitations: (i) Inability to revise storage deci- sions. We exemplify this limitation via the Nearest Neighbor Search prob- lem (see also Appendix D): With a ref- erence vector given, a sequence must be scanned sequentially for the most similar vector in order to provide its attached value at sequence end. The left panel of Figure shows the mean squared error at this task. LSTM strug- gles to revise stored value when more similar vector is found, while new xLSTM remediates this limi- tation by exponential gating. (ii) Lim- ited storage capacities, i.e., informa- tion must be compressed into scalar cell states. We exemplify this limita- tion via Rare Token Prediction. In the right panel of Figure 1, the perplex- ity of token prediction on Wikitext- 103 (Merity et al., 2017) given for partitions of different token frequency. storage capacities. Our new xLSTM solves this problem by matrix memory. (iii) Lack of paralleliz- ability due to memory mixing, i.e., the hidden-hidder connections between hidden states from one time step to the next, which enforce sequential processing.
LSTM
600
LSTM
0.15
xLSTM
xLSTM
Transformer
Transformer
0.10
200
0.05
20
MSE
PPL <10 100-10
>10
Figure 1: LSTM limitations. Left: Nearest Neighbor Search problem in terms of mean squared error (MSE). Given reference vector, a sequence is scanned sequentially for the most similar vector with the objective to return its attached value at sequence end. LSTM struggles to revise a stored value when more similar vector is found. Our new xLSTM overcomes this limitation by exponential gating. Right: Rare Token Prediction. The perplexity (PPL) of token prediction on Wikitext- 103, in partitions of token frequency. LSTM performs worse on predicting rare tokens because of its lim- ited storage capacities, whereas our new xLSTM solves this problem via matrix memory.
LSTM performs worse on rare tokens because of its limited
These limitations of LSTM have paved the way for the emergence of Transformers (Vaswani et al., 2017) in language modeling. What performances can we achieve in language modeling when overcoming these limitations and scaling LSTMs to the size of current Large Language Models?
2 Extended Long Short-Term Memory
To overcome the LSTM limitations, Extended Long Short-Term Memory (xLSTM) introduces two main modifications to the LSTM idea of Equation (1). Those modifications exponential gating and novel memory structures enrich the LSTM family by two members:
(i) the new sLSTM (see Section 2.2) with scalar memory, a scalar update, and memory mixing, and (ii) the new mLSTM (see Section 2.3) with a matrix memory and covariance (outer product) update rule, which is fully parallclizable. Both sLSTM and mLSTM enhance the LSTM through exponential gating. To enable parallelization, the mLSTM abandons memory mixing, i.c., the hidden-hidden recurrent connections. Both mLSTM and sLSTM can be extended to multiple memory cells, where sLSTM features memory mixing across cells. Further, the sLSTM can have multiple heads without memory mixing across the heads, but only memory mixing across cells within each head. This introduction of heads for sLSTM together with exponential gating establishes new way of memory mixing. For mLSTM multiple heads and multiple cells are equivalent.
Integrating these new LSTM variants into residual block modules results in xLSTM blocks (see Section 2.4). Residually stacking those XTM blocks in architectures provides STM architectures (see Section 2.4). See Appendix Figure 6 for the xLSTM architecture with its components.

---

2.1 Review of the Long Short-Term Memory
The original LSTM idea (Hochreiter, 1991; Hochreiter Schmidhuber, 1997b.a) introduced the scalar memory cell as a central processing and storage unit that avoids vanishing gradients (Hochreiter, 1991; Hochreiter al., 2000) through the constant error carousel (cell state update). The memory cell contains three gates: input, output, and forget gate. The latter was introduced by Gers et al. (2000). The LSTM memory cell update rules at time step are:
f
Ct-1
Z1
cell state
(2)
hr
O2
h2
hi
= vp
hidden state
(3)
Zt
(1
ãe = w
Wt. + r r- + by
cell input
(4)
it
it m
Et h4-1 + bi
input gate
(5)
f = wi St + rs h4-1 + bf
forget gate
(6)
Ot (ox)
6. = !n at + ro 14-1 + bo output gate
The weight vectors W2. Wi, wr, and correspond to the input weight vectors between inputs and cell input, input gate, forget gate, and output gate, respectively. The weights rz, ri, rr, and ro correspond to the recurrent weights between hidden state h and cell input, input gate, forget gate, and output gate, respectively. bg. b, b,, and bo are the corresponding bias terms. &p and i are the cell input and hidden state activation functions (typically tanh). is used to normalize squash the cell state, which would unbounded otherwise. All gate activation functions are sigmoid, i.c., (x) = 11(11 exp(-x)). In later formulations, multiple scalar memory cells e E R were combined in vector e₂ E Rd, which allows the usage of recurrent weight matrices R E Rdxd for each gate to mix the cell outputs memory cells (Greff et al., 2015), for more details see Appendix B. Ablation studies showed that all components of the memory cell are crucial (Greff et al., 2015).
2.2 sLSTM
To empower LSTMs with the ability revise storage decisions, we introduce exponential gates (red) together with normalization and stabilization. In particular, input and forget gates can have exponential activation functions. For normalization, we introduce a normalizer state that sums up the product of the input gate times all future forget gates. The scalar sLSTM forward pass is:
Ct
ft
Ct
Zt
cell state
(8)
fg
+
is
normalizer state
(9)
he
O2
h4
h
Ci n
hidden state
(10)
Ze
(z)
& w
+ T r= + be
cell input
(11)
it
exp
(1z)
i w. x.
+ r h--1 + by
input gate
(12)
f,
a
OR
exp
ig
f wi + rf h + by
forget gate
(13)
O2 = o (ox)
o w + To hr-1 + bo output gate
(14)
We transfer the original LSTM gating techniques, i.e., input- and/or hidden-dependent gating plus bias term, to the new architectures. Exponential activation functions can lead to large values that cause overflows. Therefore, we stabilize gates with an additional state m, (Milakov & Gimelshein, 2018). Equations (49) - (51) in the appendix.
New Memory Mixing. sLSTM can have multiple memory cells like original LSTM (see Appendix B.2). Multiple memory cells enable memory mixing via recurrent connections Rz, Rí. Rr, R. from hidden state vector to memory cell input z and the gates i, f, o, respectively. A new aspect in memory mixing is the effect of exponential gating. The new sLSTM can have multiple
3

---

heads with memory mixing within each head but not across heads. The introduction of heads for sLSTM together with exponential gating establishes a new way of memory mixing.
2.3 mLSTM
To enhance storage capacities of LSTMs. we increase the LSTM memory cell from scalar E IR to a matrix C E Rixd Hence, retrieval performed via a matrix multiplication. At time t, we want to store a pair of vectors, the key k, R and the value 2 B (we use the Transformer terminology). Later at time t + т, the value v should be retrieved by a query vector qt+r E R.. This is the setting of Bidirectional Associative Memories (BAMs) (Kohonen, 1972; Anderson, 1972; Nakano, 1972; Anderson et al., 1977). The covariance update rule (Sejnowski, 1977; Dayan & Willshaw, 1991) for storing key-value pair is
C, = C + Ve k;
(15)
We assume a layer-norm before projecting inputs to keys and values, therefore they have zero mean. The covariance update rule is optimal (Dayan & Willshaw, 1991) for a maximal separability of retrieved binary vectors, which equivalent to a maximal signal/noise ratio. Higher separability is possible when limiting retrieval to pairwise interactions and conceding quadratic complexity like attention (Krotov Hopfield, 2016, 2017; Ramsauer et al., 2021). The covariance update rule is equivalent to Fast Weight Programmers (Schmidhuber, 1992; Schlag al., 2021), which have later been equipped with a constant decay rate multiplied to C and a constant learning rate multiplied to o, (Ba et al., 2016a). In this spirit, integrate the covariance update rule into the LSTM framework, where the forget gate corresponds to decay rate and the input gate to the learning rate, while the output gate scales the retrieved vector.
For this matrix memory, the normalizer state the weighted sum of key vectors, where each key vector is weighted by the input gate and all future forget gates. Again, the normalizer state keeps record of the strength of the gates. Since the dot product between query and normalizer state can be close to zero, use the absolute value of this dot product and lower bound it by threshold (typically 1.0) as done previously (Sun et al., 2023). The mLSTM forward pass is:
C
C
+
it
cell state (16)
n,
n
1
l+
normalizer state (17)
h
Or
he,
h
C₁
q
max
ni
q:
hidden state (18)
q: Wg g
Dtt
b4
query input (19)
1
k
WR
at
+
by
key input (20)
Vd
W. at b
value input (21)
exp
ie wi
8z + bi
input gate (22)
f
OR exp
i,
wi Rt + bf
forget gate (23)
Or (o.)
o. W. C2 + bo
output gate (24)
mLSTM can have multiple memory cells like the original LSTM. For mLSTM, multiple heads and multiple cells are equivalent as there is no memory mixing. In order to stabilize the exponential gates of ML.TMM we use the same stabilization techniques for sL.TTM (see Equation 49). Since the mLSTM has no memory mixing, this recurrence can be reformulated in parallel version. For more details we refer Appendix B.3.
2.4 XLSTMM Architecture
XLSTM Blocks. An XLSTM block should non-linearly summarize the past in high-dimensional space to better separate different histories or contexts. Separating histories is the prerequisite to cor- rectly predict the next sequence element such the next token. We resort to Cover's 's Theorem (Cover,

---

1965), which states that in a higher dimensional space non- non-linearly embedded patterns can more likely be linearly separated than in the original space. We consider two residual block architectures: (i) A residual block with post up-projection (like Transformers), which non-linearly summarizes the past in the original space, then linearly maps into a high-dimensional space, applies non-linear activation function, and linearly maps back to the original space; see Appendix Figure 7 for details. (ii) residual block with pre up-projection (like State Space Models), which linearly maps to a nigh-dimensional space, non-linearly summarizes the past in the high-dimensional space and then linearly maps back to the original space. See Appendix Figure 8 for more details. For an XLSTM block containing an sLSTM, mostly use the post up- op-projection block. For an xLSTM block containing an mLSTM, we use the pre up-uroop up-projection block since the memory capacity becomes larger in the sighndimensional space.
XLSTM Architecture. An xLSTM architecture is constructed by residually stacking build- ing blocks (Srivastava et al., 2015; He et al., 2016). We rely on the most commonly used pre- LayerNorm (Ba et al., 2016b) residual backbones as used in contemporary Large Language Models. See last two columns (from the left) in Figure 6.
2.5 Memory and Speed Considerations
Contrary to Transformers, xLSTM networks have a linear computation and a constant memory complexity with respect the sequence length. Since the xLSTM memory is compressive, it is well suited for industrial applications and implementations on the edge. The memory of mLSTM does not require parameters, but computationally expensive through its d d matrix memory and x d update. We trade off memory capacity against computational complexity. Nevertheless, the computations be done in parallel on GPUs, therefore these computations have only minor effect on the wall clock time.
While mLSTM is parallelizable analog FlashAttention (Dao et al., 2022: Dao, 2024) GLA (Yang et al., 2023), sLSTM not parallelizable due to the memory mixing hhidenn-hideen connections). However, we developed a fast CUDA implementation with GPU memory optimizations the register level which is typically less than two times slower than mLSTM.
3 Related Work
Conceptually, the closest models to xLSTM are Retention (Sun et al., 2023), RWKV (Peng et al., 2023, 2024), GLA (Yang et al., 2023), HGRN2 (Qin et al., 2024) and Mamba (Gu & Dao, 2024). These models share the concepts matrix memory and/or gating. However, in contrast to the new sLSTM, they do not allow memory mixing. Memory mixing enables to solve state tracking problems, and therefore LSTMs are more expressive than State Space Models (SSMs) and Transformers (Merrill et al., 2024; Delétang et al., 2023). Other closely related work is discussed further in Appendix C.
4 Experiments
We experimentally evaluate xL.STM and compare it existing methods with a focus on language modeling. We investigate XL-ST'ss specific capabilities on synthetic tasks in Section 4.1. In Section 4.2, we compare the validation set perplexity of various current language modeling methods that have been trained on 15B tokens from SlimPajama (Soboleva et al., 2023). On the same dataset, we perform ablation studies for xLSTM. Then, we compare XLSTM and the best performing methods from Section 4.2 after being trained on 300B tokens from SlimPajama (Soboleva et al., 2023) on downstream tasks, assess their scaling behavior analogous to Kaplan et al. (2020) and Brown et al. (2020), and compare the text generation times and the maximal throughput of the xLSTM in Section 4.3. For all experiments, we use the notation XLSTM[a:b] for the ratio a/b of mL.STM-base versus SLSTTM-aased xL.STM blocks. For example, XLSTM[[:1] means that out of eight blocks, seven are MoL.STM-aase blocks and one is an sl -absed block. For common total block number of 48, this translates to 6 sLSTM-baa basee blocks and 42 ALLSTM-basee blocks.
5

---

4.1 Synthetic Tasks and Long Range Arena
Firstly, we test the effectiveness of xLSTM's new exponential gating with memory mixing on formal languages (Delétang et al., 2023). Then, we assess the effectiveness of XLSTM's 's new matrix memory on the Multi- Senlti-Queryaaua Associative Recall task (Arora al., 2023). Finally, xLSTM's performance at processing long sequences in the Long Range Arena is evaluated (Tay et al., 2021).
Test of XLSTM's Exponential Gating with Memory Mixing, We test xI.ST''s new exponential gating with memory mixing, which should enable it to solve state tracking problems (Merrill et al., 2024; Merrill & Sabharwal, 2023). We implement and extend the formal language tasks from Delétang et al. (2023) to enable multi-length training for length extrapolation. For detailed description of all tasks and extended results see Appendix D.I.1. We compare xLSTM other methods including Transformers, State Space Models, and Recurrent Neural Networks. The accuracy of the tested methods is evaluated on those tokens relevant the task. The accuracy scaled between 0 (random) and (perfect). We compare 2-block architectures of the following methods on these tasks: x1 1. (i.e., only M),,), X[SSTM[1:0] (i.e., only mL.STM, xL.STM[1:11 IJ, Llama, Mamba, RWKV, Retention, Hyena, LSTM, and LSTM in Transformer blocks (LSTM (Block)). The results of this experiment are shown in Figure 9. Models such as Transformers or State Space Models without memory mixing (no state tracking) cannot solve, e.g. regular grammars like the parity task. This result is in agreement with findings that Transformers and State Space models are fundamentally less powerful than RNNs (Merrill et al., 2024; Merrill & Sabharwal, 2023; Delétang et al., 2023).
Test of xLSTM's Memory Capacities on Associative Recall Tasks. In this experiment, we test xLSTM's new matrix memory in terms of the memory capacity on the Multi Multi-Query Associative Recall task (Arora et al., 2023): For each sequence, key-value pairs are randomly chosen from a large vocabulary, which must be memorized for later retrieval. To enhance the difficulty of the original task, we increase the number of key-value pairs up to 256 and enlarge the context length up to 2048, obtaining extended tests for the memory capacities of different models. We compare 2-block architectures of Llama, Mamba, RWKV-5, RWKV-6, xLSTM[1:1] and xLSTM[[:0]] The models are evaluated by the accuracy at recalling the pairs. Since Transformers (e.g. Llama) have a memory that is exponential in the coding dimension (Ramsauer et al., 2021), they constitute the gold standard at this task. Results are shown in Figure 2. XLSTM[1:1] performs best among all non Transformer models, also for small models. Interestingly, the sLSTM block does not diminish the memory capacity but rather leverages it, which becomes evident at the most difficult task with 256 key-value pairs. Additional results of Appendix D.1.2, indicate that xLSTM's s enhanced memory capacities also allow for extrapolating to contexts that are longer than those seen during training. xL
Llama
Mamba
RWKV-5
RWKV-6
xLSTM[[10]
KV Pairs 48
KV Pairs 96
Pairs 256
1.00
0.75
Accuracy
0.50
0.25
0.00
32
128
256
512
Model Dim
Model Dim
Model Dim
Figure 2: Test of memory capacities of different models at the Multi-Query Associative Recall task with context length 2048. Each panel is dedicated to different number of key-value pairs. The r-axis displays the model size and the y-axis the validation accuracy.
Test of xLSTM's Long Context Capabilities on Long Range Arena. To assess xLSTM's per- formance on long sequences and large contexts, we compare different methods on the Long Range Arena (Tay et al., 2021), 1). XLSTM demonstrates consistent strong performance on all of the tasks, suggesting that the xLSTM architecture remarkably efficient in handling different aspects of long context problems. For more details, see Appendix 1.1.3.
4.2 Method Comparison and Ablation Study
To address the main question of our paper, i.e. what can our new LSTM variants achieve when scaled up in language modelling, we train xLSTMs, Transformers, State Space Models, and other methods on 15B tokens from SlimPajama in the same auto-regressive setting. We compare the trained models on the validation set and perform ablation studies for the xLSTMs.

---

Comparing xLSTM to Other Methods. We train models on 15B tokens from SlimPajama (Soboleva al., 2023), and evaluate their perplexity on the validation set. We compare the following methods: XLSTM, GPT-3 (Transformer) (Brown et al., 2020), Llama (Transformer) (Touvron et al., 2023), H3 (SSM) (Fu et al., 2023), Mamba (SSM) (Gu & Dao, 2023), RWKV-4 (RNN) (Peng et al., 2023), V-KV-5 (RNN) (Peng et al., 2024). RWKV-6 (RNN) (Peng et al., 2024), GLA (linear Transformer) (Yang et al., 2023), HGRN2 (RNN) (Qin et al., 2024). RetNet (linear Transformer) (Sun et al., 2023), Hyena (linear Transformer) (Poli et al., 2023), XLSTM[I:0], and XLSTM[7:1]. The models were trained with mixed precision, for RWKV-5, RWKV-6, GLA, HGRN2, the mixed- precision training did not utilize the PyTorch automated mixed precision (see Appendix Section D.2). We categorize the methods into (a) Transformers, (b) State Space Models (SSMs), and (c) Recurrent Neural Networks (RNNs) together with linear Transformers, i.., linear methods that substitute the Transformer's s attention mechanism. The models match a GPT-3 model with 350M parameters in size, i.e. embedding dim 1024 and 24 residual blocks. Only GPT-3 uses shared weights for token and output embeddings, therefore has fewer parameters.
Table I:
Method comparison next token prediction when trained 15B tokens from SlimPajama. Best validation perplexities within model classes, i.e., linear Transformers, RNNs, Transformers, SSMs, and XLSTMs are underlined and overall best is in bold. For each model class, the best per- forming methods are used in Section 4.3 for LLM training. xLSTMs with new memory (XLSTM[110) and xISTM((:11) perform best.
Model
#Params M
SlimPajama (15B) ppl +
Model
#Params M
SlimPajama (15B) ppl 4
Hyena
435
17.59
GPT-3 Llama
356
14.26
RWKV-4
430
15.62
407
14.25
RWKV-5
456
14.25
H3
420
18.23
RWKV-6
442
15.03
Mamba
423
13.70
RetNet
431
16.23
GLA
412
16.15
XLSTM[1:0]
409
13.43
HGRN2
411
14.32
XLSTM[7:1]
408
13.48
The results in Table show that xLSTM outperforms all existing methods in validation perplexity. For details see Appendix D.2. Figure 15 in the appendix shows the scaling behaviour for this experiment, indicating that xLSTM will also perform favorably for larger models.
Ablation Studies. Table and Figure 15 demonstrate that xLSTM achieves excellent results at language modeling when being trained on 15B tokens from SlimPajama. To ablate the changes from L.TM to XLSTM, we morph vanilla I.STM architecture step-by-step into an xL.TTM architecture. Firstly, we integrate LSTM layers into pre-LayerNorm residual backbones. Secondly, we extend this to post up-projection block. Finally, we add exponential gating and matrix memory. The results are shown in Appendix Table (top). The ablation studies attribute the strong performance improvement to both the exponential gating and the matrix memory. Additionally, due to the importance gating in RNNs and State Space Models, ablate different gating mechanisms. In Appendix Table (bottom), we conclude that having each gate learnable and influenced by the input has an incrementally positive effect. Additional studies the individual backbone components are discussed in Appendix D.2.
4.3 XL.TMM as Large Language Model
Next, we increase the amount of training data to 300B tokens from SlimPajama, the same number of tokens as used in e.g., Mamba (Gu & Dao, 2023) and Griffin (De al., 2024). We compare xLSTM to RWKV-4, Llama, and Mamba one method from each respective method class in Section 4.2. We select RWKV-4 as RNN representative since for RWKV-5, RWKV-6 and HGRN2 a reasonable training precision setting (Appendix Section D.2) has been found only after the training start of the 300B token experiments (Peng et al., 2024). We train different model sizes (125M, 350M, 760M, 1.3B), test all models for length extrapolation capabilities and evaluate their performance on the validation set, on downstream tasks, on 471 text domains of the PALOMA benchmark, and, finally, investigate their scaling law behavior.
7

---

Sequence Length Extrapolation. Firstly, we test the sequence length extrapolation for 1.3B-sized, large models xLSTM, RWKV-4, Llama, and Mamba. All models are trained on context length 2048, and then tested for context lengths up 16384. See Figure 3 for the results. In contrast to other methods, xL.STM models maintain low perplexities for longer contexts.
Validation Perplexity
Llama
SlimPajama (300B) ppl at 16k
Mamba
Model
RWKVV4
10
xLS [Ta]
Llama
337.83
xLSTM
Mamba
RWKV-4
13.75
XLSTM[7:1]
8.92
10
xLSTM[1:0]
9.01
2048
4096
8192
16384
Token Position
Figure 3: Sequence extrapolation in language modeling. This is comparison of 1.3B- large models XLSTM, RWKV-4 Llama, and Mamba at next token prediction on the SlimPajama validation after training 300B tokens from SlimPajama. Models trained with context length 2048 (gray) and then tested for context lengths up to 16384. Left: Token perplexities evaluated at different context lengths. In contrast other methods, xLSTM models remain low perplexities for longer contexts. Right: Prediction quality when extrapolating to long context sizes in terms of validation perplexity (PPL). xLSTM yields the best PPL values (best in bold, second best underlined).
Table 2: Validation perplexity and downstream tasks. Comparison of xLSTM, RWKV-4, Llama, and Mamba on the validation set at next token prediction and on downstream tasks after training on 300B tokens from SlimPajama. Model sizes 125M, 350M, 760M, and 1.3B. The first column shows the methods and the second the actual number of parameters. The third column lists the validation perplexities, while the remaining columns show the performance on downstream tasks. Best model per model size is depicted bold and the second best is underlined. In the vast majority of tasks and across all model sizes xLSTM is the best method only on the ARC task Mamba is in some cases the best method. XLSTM[[:0] and XLSTM[7:1] are the two best models with respect to validation set perplexity.
Model
AParams M
SlimPjjama (300B) pel
LAMBADA
LAMBADA
HelnSwag
PIQA
ARC-E
ARC-C
WinoGrande acc 1
Average
RWKV-4
169.4
16.66
54.72
47.94
Llama
162.2
15.89
39.21
31.54
34.09
65.45
45.33
23.63
50.67
41.78
Mamba
167.8
15.08
27.76
34.14
36.47
66.76
48.86
24.40
51.14
43.63
xLSTM[L:0]
163.8
14.63
25.98
36.52
36.74
65.61
47.3 47.81
24.83
$1.85
43,89
KLSTM[7:1]
163.7
14.60
26.59
36.08
36.75
66.87
48.
25.26
$1.70
44.16
RWKV-4
430.5
12.62
21.57
36.62
42.47
69.42
$4.46
25.
$1.22
46,60
350M
Mamba
423.1
11.64
12.83
46.24
47.55
69.70
$5.47
27.56
$4.30
50.14
409.3
55.72
xLSTM/7:11
408.4
11.37
12.11
47.74
47.89
71.16
56.61
27.82
53.28
50.75
RWKV-4
891.0
10.55
10.98
47.43
$2.29
72.69
58,
28.84
55.41
52.58
760M
Llama
834.1
10.60
9.90
51.41
52.16
70.95
56.48
28.75
56.67
52.74
Mamba
870.5
10.24
9.24
50.84
53.97
71.16
60.44
29.78
56,99
53.86
XLSTM[L:0]
840.4
9.86
8.09
$4.78
$5.72
72.69
62.75
32.59
58.17
56.
x1.STM[7=1]
839.7
29.
$5.26
RWKV-4
1515.2
9,83
9.84
49.78
56.20
74.70
61.83
30.63
55.56
54.78
Llama
1420.4
9.44
7.23
$7.44
57.81
73.12
62.79
31.74
$9.04
56.99
Mamba
1475.3
9.14
7.41
$5.64
60.45
74.43
66.12
33.70
60.14
58.41
x1STMM1:0]
1422.6
8.89
6.86
57.83
60.91
74.59
64.31
32.59
60.62
58.48
XLSTM[7:11
1420.1
9.00
7.04
56.69
60.26
74.92
65.11
32.34
59.27
58.10

---

Validation Perplexity and Downstream Tasks. Secondly, for all model sizes, we evaluate the performance of xLSTM, RWKV-4, Llama, and Mamba models on the SlimPajama validation set for next token prediction and on downstream tasks that measure common sense reasoning. The third column of Table lists the validation set perplexities of different methods. Both xLSTM[[1: and XLSTTM[7:1] are the best models for all model sizes with respect to the validation set perplexity. The other columns of Table 2 provide the performance on downstream tasks. In the vast majority of tasks and across all model sizes xLSTM is the best method - only on the ARC task Mamba is in some cases the best method. For details see Appendix D.3.
Performance on PALOMA Language Tasks. Thirdly, for all model sizes, we test the next token prediction performance of xLSTM, RWKV-4, Llama, and Mamba models on PALOMA language tasks (Magnusson et al., 2023). We measure the performance by the perplexity for next token prediction on 571 text domains, which range from nytimes.com com to r/depression on Reddit. Appendix Table 8 shows token prediction perplexity grouped into language modeling (first seven columns) and fine-grained domain benchmarks (last 5 columns). x ALSTM[1:0 has 568 out of 571 (99.5%) text domains lower perplexity than Mamba, in 486 out of 571 (85.1%% a lower perplexity than Llama, in 570 out of 571 (99.8%) a lower perplexity than RWKV-4, see Appendix D.3.
Scaling Laws. Fourthly, we assess the power-law scaling behavior, which allows to extrapolate the performance to larger model sizes (Kaplan al., 2020: Brown al., 2020). Figure 4 presents the scaling behavior over the number of model parameters. All models share a similar scaling behavior but with different offsets. RWKV-4 performs worst, followed by Llama and Mamba. xLSTM better than Mamba with similar margin to Mamba as Mamba has to Ilama. In Figure 16 in Appendix D.3 we plot the scaling behavior over the number of training FL.PPs for the I.lama baseline and both XLSTM variants. For the xLSTM variants we calculate the FLOPs for the recurrent (see Section 2 and parallel (see Appendix B.3) formulation. The scaling behavior indicates that for larger models XLSTM will continue to perform favourable compared to Transformers and State-Space models.
17-
Liama
16
Mamba
15-
RWKV-4
Validation Perplexity
14
xLSTM(7:1]
XLSTM[1:0]
13
12
11
10
300B Tokens
9
0.2
0.4
1.0
1.4
Number of Parameters
x10
Figure 4: Scaling laws. Next token prediction perplexity of XLSTM, RWKY-4, Llama, and Mamba. The models with sizes 125M, 350M. 760M, and 1.3B parameters are trained on 300B tokens from SlimPajama. The scaling laws indicate that for larger models xL.STM will perform well too.
Generation Times and Maximal Throughput. Finally, we measure the text generation time in Figure (left) and the maximal throughput Figure (right) for our xLSTM variants at 1.3B scale. We compare against similar sized Mamba, Llama and RWKY implementions from HuggingFace including a static key-value cache for the Llama model. At the time of the experiments, both full cache compilation of the Transformer model and compilation of the Mamba model with torch. compile le did not work. For the text generation experiments all of the models are tested at batch size 1 and pre-fill 16. This fie-fill should be maximally favorable for the Transformer. Figure 5 shows the linear scaling of the xLSTM and the other recurrent models Mamba and RWKV-4 compared to the quadratic scaling of Llama. For the decoding throughput we measure different batch sizes and prefill for the I. Llama model. Figure 5 (right) shows that XL. can use much higher batch sizes than Llama due to its constant memory and thus achieves the highest throughput.

---

1200
Liana
Llama FP16 (prelll 256)
Momba
90000
Ulama FP16 (prefll 2005)
RWKV-4
KLS XLSTM[LO FP10 (t.c)
RWKV (tarch.oomple)
25000
Mamba FP16
+LSTM[]
XL5TM]] of (torch compile)
200000
(3)
X
Tokens
xLSTM (torch campli)
16000
10000
200
Out
4126
30384
61
1.25
1024
Generation Length
Batch Size
Figure 5: Generation Times and Maximal Throughput. Left: Generation times of different 1.3B models for pre-fill context of 16 tokens (to mitigate cache initialization). The recurrent models (XLSTM[[100) XL Mamba and RWKV-4) show linear behavior, whereas the Transformer (Llama) inference/decoding time is quadratic in sequence length. Right: Token throughput for different batch sizes on a A A100-80GB GPU for 1.3B sized models. Note that the Transformer Llama model goes out of memory (OOM) already for small batch sizes, whereas xLSTM and Mamba can sustain very large batch sizes. XLSTM[[1:] consistently outperforms Mamba throughput. Beyond batch size 2048, all models go OOM.
Limitations
(i) In contrast mLSTM, memory mixing of the sLSTM prohibits parallelizable operations, and thus prevents fast parallel implementation. However, we developed a fast CUDA kernel for sLSTM, which is currently less than two times slower than the parallel mLSTM implementation. (ii) The mLSTM CUDA kernels are not optimized, and therefore the current implementation is about four times slower than FlashAttention or the scan used in Mamba. Faster CUDA kernels could be obtained in the vein of Atteninn (iii) The matrix memory of mLSTM has high computation complexity since d d matrices must be processed. Still, the memory update and retrieval is parameter- free, i.e., parallelizable when using standard matrix operations. Thus, the wall clock time overhead due to the complex memory is minor. (iv) The forget gate initialization must be chosen carefully. (v) Since the matrix memory is sequence length independent, increasing the sequence length might overload the memory for longer context sizes. Still, this does not appear be a limitation for contexts up to 16k, Section 4.3. (vi) Due the expensive computational load for LLM experiments, we did neither fully optimize the architecture nor the hyperparameters, especially for larger xLSTM architectures. We anticipate that extensive optimization process is needed for xLSTM to reach its full potential.
Conclusion
We have partly answered our simple question: How far do get in language modeling when scaling LSTM to billions of parameters? So far, we can answer: "At least as far as current technologies like Transformers or State Space Models". We have enhanced L.TTM to xL.STM by exponential gating with memory mixing and new memory structure. XLSTM models perform favorably on language modeling when compared state-of-the-arr methods like Transformers and State Space Models. The scaling laws indicate that larger xLSTM models will be serious competitors to current LLMs that are built with the Transformer technology. xLSTM has the potential to considerably impact other fields like Reinforcement Learning, Time Series Prediction, or the modeling of physical systems.
References
Achiam, S. Adler, Agarwal, al. GPT-4 technical report. ArXiv, 2303.08774, 2023.
J. Anderson, J. Silverstein, S. Ritz, and R. Jones. Distinctive features, categorical perception, and probability learning: Some applications of a neural model. Psychological Review, 84:413-451, 1977. doi:
10

---

J. A. Anderson. A simple neural network generating an interactive memory. Mathematical Bio- sciences, 14, 1972. doi: 00.101160025-554422)9007777
S. Arora, S. Eyuboglu, A. Timalsina, I. Johnson, M. Poli, J. Zou, A. Rudra, and C. Ré. Zoology: Measuring and improving recall in efficient language models. ArXiv, 2312.04927, 2023.
J. Ba, G. E. Hinton, V. Mnih, J. Z. Leibo, and C. Ionescu. Using fast weights to attend to the recent past. In D. D. Lee, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4331-4339. Curran Associates, Inc., 2016a.
J. Ba, R. Kiros, and G. Hinton. Layer normalization. ArXiv, 1607.064357 2016b.
A. Bau, Belinkov, H. Sajjad, Durrani, Dalvi, and J. Glass. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Repre- sentations (ICIR), 2019. URL https: /tpp:////////eeei/eeeeee
Y. Bisk, R. Zellers, R. LeBras, J. Gao, and Y. Choi. Piqa: Reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence, volume 34, pp. 7432 7432-7439, 2020.
L. Blodgett, L. Green, and B. O'Connor. Demographic dialectal variation in social media: A case study of African-American English. In Conference on Empirical Methods in Natural Language Processing, pp. 1119-1130, 2016. doi: 10.18653/v1/D1--
T. Brown, B. Mann, N. Ryder, et al. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlós, P. Hawkins, J. Q. Davis, A. Mohiuddin, L. Kaiser, D. B. Belanger, L. J. Colwell, and A. Weller. Rethinking attention with performers. In 9th International Conference on Learning Representations (ICLR). OpenReview.net, 2021. URL https: /tt://oppppe /ppnrevieeeeeeeeeeee net met/forummicee (orumm?id-UUG2uuuORH.
A. Chowdhery, S. Narang, J. Devlin, et al. PaIM:: scaling language modeling with pathways. ArXiv, 2204.02311, 2022.
A. Chronopoulou, M. Peters, and J. Dodge. Efficient hierarchical domain adaptation for pretrained lan- guage models. In Conference of the North American Chapter of the Association for Computational Linguistics, pp. 1336-1351, 2022. doi: 10. http3//1120222maall-mo
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. ArXiv, 1803.05457, 2018.
T. M. Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. Electronic Computers, IEEE Transactions on, EC-14(3):3263344 1965.
T. Dao. Flashattention-2:: Faster attention with better parallelism and work partitioning. In In- ternational Conference on Learning Representations (ICLR), volume 12, 2024. URL https: wwwpeeeeewweeeeeefefeeeeeeee hitp:/oorrummidmm22222y23333 mx22222..
T. Dao, Y. Fu, S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory-efficient exact attention with IO-awareness. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (eds.), Advances net/ in Neural Information Processing Systems (NeurIPS), 2022. URL /tttp:://operreee ionmmddd=HHSPSSSS bmx.
P. Dayan and D. J. Willshaw. Optimising synaptic learning rules in linear associative memories. Biological Cybernetics, 65, 1991. doi: 10. 10.100776900200223..
S. De. S. L. Smith. A. Fernando, A. Botev, G. Cristian-Murant A. Gu, R. Haroun, L. Berrada, Y. Chen, Srinivasan, G. Desjardins, A. Doucet, D. Budden, W. Teh, Pascanu, N. DeFreitas, and C. Gulcehre. Griffin: Mixing gated linear recurrences with local attention for efficient language models. ArXiv, 4402.19427. 2024.
Degrave, F. Felici, Buchli, al. Magnetic control tokamak plasmas through deep reinforcement learning. Nature, 602:414-419, 2022. doi: 10. 15844441----00300000

---

G. Delétang, A. Ruoss, J. Grau-Moya, T. Genewein, L. K. Wenliang, E. Catt, C. Cundy, M. Hutter, S. Legg, J. Veness, and A. Ortega. Neural networks and the Chomsky hierarchy, In International Conference Learning Representations (ICLR), volume 11, 2023. URL https::/oppenreieee het//forumidd@WxxHHzzce.
N. Du, Y. Huang, A. M. Dai, et al. GLaM: efficient scaling of language models with mixture-of- experts. ArXiv, 2112.06905, 2021.
D. Y. Fu, T. Dao, K. K. Saab, A. W. Thomas, A. Rudra, and C. Re. Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh International Conference Learming Representations, 2023. URL WRI.https://openeeviee wwt//eetruuu DierrmmmiddOOOOOOYYGGG
L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leaby. The Pile: An 800gb dataset of diverse text for lan- guage modeling. ArXiv, 2101.00027 2021.
A. Gers, J. Schmidhuber, and Cummins. Learning to forget: Continual prediction with LSTM. Neural Compututation, (2(10)24451-271) 2000.
Gemini Team Google. Gemini: family of highly capable multimodal models. ArXiv, 2312.11805. 2023.
A. Graves. Generating sequences with recurrent neural networks. ArXiv, 1308.03035 2013.
S. Greenbaum and G. Nelson. The international corpus of English (ICE) project. World Englishes, 15(1):3-15, 1996.
K. Greff, R. K. Srivastava, Koutník, R. Steunebrink, and J. Schmidhuber. LSTM: A search space odyssey, ArXiv, 1503.04069 2015.
A. Gu and Dao. Mamba: Linear-time sequence modeling with selective state spaces. ArXiv, 2312.00752, 2023.
A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces. In Interna- tional Conference on Learning Representations (ICLR), 2024. URL httpss:/opeerevv/// net/ Ao1g0077
A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. ArXiv, 2111.00396, 2021.
A. Gupta, A. Gu, and J. Berant. Diagonal state spaces are as effective as structured state spaces. ArXiv, 2203. 2203.14343. 14343, 2022.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Master's thesis, Technische Universität München, 1991.
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 98): 1997a.
S. Hochreiter and J. Schmidhuber. LSTM can solve hard long time lag problems. In M. C. Mozer, M. I. Jordan, and T. Petsche (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 9, pp. 473-479. MIT Press, Cambridge MA, 1997b.
S. Hochreiter, Y. Bengio, P. Frasconi, and Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In J. Kolen and S. Kremer (eds.), A Field Guide to Dynamical Recurrent Networks. IEEE, 2000.
S. Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning to learn using gradient descent. In G. Dorffner, Bischof, and K. Horiik (eds.), Proc. Int. Conf. on Artificial Neural Networks (ICANN 200I), pp. 87-94. Springer, 2001.
12

---

S. Hochreiter, M. Heusel, and K. Obermayer. Fast model-based protein homology detection without alignment. Bioinformatics, 138147:178--176,, 2007.
Hoffmann, S. Borgeaud, A. Mensch, et al. Training compute-optimal large language models. ArXiv, 2203. 15556, 2022.
M. D. Hossain, F. Sohel, M. F. Shiratuddin, and H. Laga. A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CSUR), 51(6):118, 18, 2019.
Kaplan, S. McCandlish, Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. ArXiv, 2001.08361, 2020.
A. Karpathy, http://karpathy S0t htffe////mmeeecttteeeeesss/ 2015.
The unreasonable effectiveness of recurrent neural networks.
Five A. Karpathy. OpenAI Five defeats Dota world champions. https///pponiiicmm///sscccpppppp
2019.
A. Karpathy and L. Fei-Fei. Deep Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 3128 12333137, 2015.
alignments for generating image descriptions, In
A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In E. H. Daumé III and A. Singh (eds.), International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pp. 5156-5165. PMIR, 2020.
T. Katsch. GateLoop: Fully data-controlled linear recurrence for sequence modeling. ArXiv, 2311.01927, 2023.
D. Kocetkov, R. Li, L. BenAllal, J. Li, C. Mou, C. Mu nozFerrandis, Y. Jernite, M. Mitchell, S. Hughes, Wolf, D. Bahdanau, L. verWerra, and Vries. The Stack: 3 TB of permissively licensed source code. ArXiv, 2211.15533, 2022.
T. Kohonen. Correlation matrix memories. IEEE Transactions on Computers, C-21 1972. doi: 10. 10.1109/tc.1992.5000000
Kratzert, D. Klotz, C. Brenner, K. Schulz, and M. Hermegger. Rainfall-runoff modelling using long short-term memory (LSTM) networks. Hydrology and Earth System Sciences, 22(1 1):006560022 2018.
Kratzert, D. Klotz, G. Shalev, G. Klambauer, S. Hochreiter, and G. Nearing. Benchmarking a catchment-aware long short-term memory network (LSTM) for large-scale hydrological modeling. ArXiv, 1907.08458 2019.
A. Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, Deptartment of Computer Science, University of Toronto, 2009.
D. Krotov and J. Hopfield. Dense associative memory for pattern recognition. In D. D. Lee, Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances Neural Information Processing Systems, pp. 172-1180. Curran Associates, Inc., 2016.
D. Krotov and J. J. Hopfield. Dense associative memory is robust to adversarial inputs. ArXiv, 7701000939 2017.
Y. Lakretz, G. Kruszewski, T. Desbordes, Hupkes, S. Dehaene, and M. Baroni. The emergence of number and syntax units in LSTM language models. In J. Burstein, C. Doran, and T. Solorio (eds.), Conference of the North American Chapter of the Association for Computational Linguistics, pp. 11-20. Association for Computational Linguistics, 2019. doi: 10. 10.18653/11/111-10000
Y. Li, т. Cai, Y. Zhang, D. Chen, and D. Dey. What makes convolutional models great on long sequence modeling? ArXiv, 2210.09298, 2022.
Liang, R. Bommasani, T. Lee, et Holistic evaluation of language models. Annals of the New York Academy of Sciences, 1525:140-146 2023.
13

---

J. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, Zhang, X. Zou, Z. Li, X. Deng, J. Liu, Xue, H. Zhou, Ma. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A Chinese multimodal pretrainer. ArXiv, 2103.0082) 2021.
D. Linsley, J. Kim, V. Veerabadran, C. Windolf, and T. Serre Learning long-range spatial dependen- cies with horizontal gated recurrent units. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
1. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019. UKl.attpee//pppeeevieee..eeee net.o//m/de//ffo Bkg6RiCqY7.
X. Ma, C. Zhou, X. Kong. J. Hc, L. Gui, G. Neubig, May, and L. Zettlemoyer. Mega: Moving average equipped gated attention. ArXiv, 2209. 10655, 2022.
A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Annual Meeting of the Association for Computational Linguistics, volume 49, pp. 142-150, 2011.
I. Magnusson, A. Bhagia, V. Hofmaann, et al. Paloma: A benchmark for evaluating language model fit. ArXiv, 2312. 2023.
H. Mchta, A. Gupta, A. Cutkosky, and B. Neyshabur. Long range language modeling via gated state spaces. ArXiv, 2206. 13947, 2022.
S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. In Interna- tional Conference on Learning Representations (ICRL), 2017. URL / net/ Borum?id-Byj72udde
W. Merrill and A. Sabharwal. The parallelism tradeoff: Limitations of log-precision transformers. Transactions of the Association for Computational Linguistics, 11:531-5 2023. doi: 10.1 10.1162/ tacl a 00562.
W. Merrill, J. Petty, and A. Sabharwal. The illusion of state in state-space models. ArXiv, 2404.08819, 2024.
M. Milakov and N. Gimelshein. Online normalizer calculation for softmax. ArXiv, 1805.02867, 2018.
K. Nakano. Associatron a model of associative memory. IEEE Transactions on Systems, Man, and Cybernetics, SMCC2(3):300-888 1972. doi: 10.1 10.1109/TMMC.11924433099
G. Nearing, D. Cohen, V. Dube, M. Gauch, O. Gilon, S. Harrigan, A. Hassidim, D. Klotz, Kratzert, A. Metzger, S. Nevo, F. Pappenberger, C. Prudhomme, G. Shalev, S. Shenzis, T. Y. Tekalign, D. Weitzner, and Y. M. B. Kosko. Global prediction of extreme floods in ungauged watersheds. Nature, 627:559-55 563, 2024. doi: 10. 10.1038/s41586 -22-8---2----15111
C. Olsson, N. Elhage, N. Nanda, al. In-context learning and induction heads. ArXiv, 2209. 2022.
A. Orvieto, S. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De. Resurrecting recurrent neural networks for long sequences. In Proceedings of the 40th International Conference on Machine Learning (ICML). JMLR.org, 2023. doi: 01555/3618408.36195555
A. Papasavva, S. Zannettou, E. DeCristofaro, G. Stringhini, and J. Blackburn. Raiders of the lost KeK: 3.5 years of augmented 4chan posts from the politically incorrect board. In International AAAI Conference on Web and Social Media (ICWSM), volume 14, pp. 885-894, 2020.
D. Paperno, G. Kruszewski, A. Lazaridou, N.-Q. Pham, R. Bernardi, Pezzelle, Baroni, Gemma G. Boleda, and R. Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Annual Meeting of the Association for Computaaional Linguistics, volume 1, pp. 1525-1534, 2016.
14

---

G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli, H. Alobeidli, B. Pannier, E. Al- mazrouei, and J. Launay. The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data, and web data only. ArXiv, 2306.01 11.001116,,,, 16, 2023.
B. Peng, E. Alcaide, Q. Anthony, al. RWKV: Reinventing RNNs for the transformer era. ArXiv, 2305.1 13048, 2023.
B. Peng, D. Goldstein, Anthony, et al. Eagle and Finch: RWKV with matrix- matrix-valued valued states and dynamic recurrence. ArXiv, 2404.05892, 2024.
M. Poli, S. Massaroli, E. Nguyen, D. Y. Fu, T. Dao, S. Baccus, Y. Bengio, S. Ermon, and C. Ré. Hyena hierarchy: Towards larger convolutional language models. In Proceedings of the 40th International Conference Machine Leurning (ICML). JMLR.org, 2023, doi: 10.5555/3618406 84083661572
M. Poli, A. W. Thomas, E. Nguyen, P. Ponnusamy, Deiseroth, K. Kersting, T. Suzuki, B. Hie, S. Er- mon, C. Ré, C. Zhang, and S. Massaroli. Mechanistic design and scaling of hybrid architectures. ArXiv, 2403.17844 2024.
Z. Qin, S. Yang. and Y. Zhong. Hierarchically gated recurrent neural network for sequence modeling. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2023. URL htttp::///penrevier www.ee//oooummddPPPttttat.a
Z. Qin, S. Yang, W. Sun, X. Shen, D. Li, W. Sun, and Y. Zhong. HGRN2: Gated linear RNNs with state expansion. ArXiv, 2404.07904, 2024.
D. R. Radev, P. Muthukrishnan, and V. Qazvinian. The ACL anthology network corpus. In Workshop on Text and Citation Analysis for Scholarly Digital Libraries (NLPIR4DL), pp. 54 61. Association for Computational Linguistics, 2009.
A. Radford, R. Jozefowicz, and I. Sutskever. Learning to generate reviews and discovering sentiment. ArXiv, 1704.01444, 2017.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and Sutskeerr. Language models are unsupervised language-models. 2019.
multitask learners. https: /oppnaii/ com
J. W. Rae, S. Borgeaud, T. Cai, et al. Scaling language models: Methods, analysis & insights from training Gopher. ArXiv, 21 1446, 2021.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, 1910.10683, 2019.
H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, L.. Gruber, M. Holzleitner, M. Pavlović, G. K. Sandve, V. Greiff, D. Kreil, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter. Hopfield networks is all you need. In International Conference Learning Representations (ICLR). OpenReview, 2021.
M. Reid, V. Zhong, S. Gururangan, and L. Zettlemoyer. M2D2: A massively multi-domain language modeling dataset. In Conference on Empirical Methods in Natural Language Processing, pp. 964 975, 2022.
M. Reid, N. Savinov, D. Teplyashin, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv, 2403.05530, 2024.
M. H. Ribeiro, J. Blackburn, B. Bradlyn, E. DeCristofaro, G. Stringhini. S. Long, Greenberg, and
S. Zannettou. The evolution of the manosphere across the web. In Proceedings of the international AAAI conference on web and social media, volume 15, pp. 196-207, 2021.
K. Sakaguchi, R. I.. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9).99-166 2021.
I.. Scao, A. Fan, C. Akiki, et al. BLOOM: A I -araameee open-access multilingual language
model. ArXiv, 2211.05 2022.
15

---

I. Schlag, K. Irie, and J. Schmidhuber. Linear transformers are secretly fast weight programmers. In M. Meila and T. Zhang (eds.), Proceedings of the 38th Interaational Conference Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pp. PMIR, 2021.
J. Schmidhuber. Learning to control fast- festhteight memories: An alternative to recurrent nets. Neural Computation, 4(1):131-139, 1992.
J. Schmidhuber. Deep learning in neural networks: An overview, Neural Nenworks, 61:85-117, 2015. doi: 10.10166j.cuume221144..000000
J. Schulman, B. Zoph, C. Kim, J. Hilton, et al. ChatGPT: Optimizing language models for dialogue. https://openai.com https//openai..om//ll///// www.blog//coahpp/ 2022. OpenAl Research.
T. J. Sejnowski. Storing covariance with nonlinearly interacting neurons. Journal of Mathematical Biology, 4, 1977. doi: 10. 10000//0000275079
M. Shoeybi, M. Patwary, R. Puri, LeGresley, J. Casper, and B. Catanzaro. Megatron-LMM Training multi-billion parameter language models using model parallelism. ArXiv, 1909.08053, 2019.
J. T. H. Smith, A. Warrington, and S. W. Linderman. Simplified state space layers for sequence modeling. ArXiv, 2208.04933, 2022.
D. Soboleva, F. Al-Khateeb, R. Myers, J. R. Steeves, Hestness, and Dey. SlimPajama: 627B token cleaned and deduplicated version of RedPajama. https: /erbeerra/ slimpaj
cerebras net/blog/ jaaaajamaaaaa ama-a a 6276 wwweettoeeeeeeeeeeeeeeeeeeeeeeee cleaned- and deduplicatedd versoono redaaredpaaaaaa
2023. URL https::: //ugging/ http:///gggggggoooo/ooooaaa/////
am2--27B.......... 627B.
L. Soldaini, Kinney, A. Bhagia, et al. Dolma: an open corpus of three trillion tokens for language model pretraining research. ArXiv, 2306.011 1100111111011110011111111 2023.
Soltan, Ananthakrishnan, J. FitzGerald, R. Gupta, W. Hamza, H. Khan, C. Peris, Rawls, A. Rosenbaum, A. Rumshisky, C. S. Prakash, M. Sridhar, F. Triefenbach, A. Verma, G. Tur, and P. Natarajan. AMexaTM 20B: Few-shot learning using a large-scale multilingual Seq2Seq model. ArXiv, 2208.01448, 2022.
R. K. Srivastava, K. Greff, and Schmidhuber. Training very deep networks. C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 28. Curran Associates, Inc., 2015.
Y. Sun, L. Dong. S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang. and F. Wei. Retentive network: A successor to transformer for large language models. ArXiv, 2307.08621, 2023.
L. Sutawika, L. Gao, H. Schoelkopf, al. CeeutherA/Im--ealuuuatt -aamn Major refactor, 2023.
I. Sutskever, O. Vinyals, and Q. V. V. Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27 (NIPS' 13), pp. 3104-3112. Curran Associates, Inc., 2014.
Y. Tay, D. Bahri, D. Metzler, D.-C. Juan, Z. Zhao, and C. Zheng, Synthesizer: Rethinking self- attention in transformer models, ArXiv, 2005.00743, 2020.
Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler. Long range arena: benchmark for efficient transformers. In International Conference on Learning Representations (ICRI.), 2021. URL https htt.thtt///peeeerrrreee //ppr//viie../.../e/eeee
grC2k.
R. Thoppilan, deFreitas, J. Hall, al. LaMDA: Language models for dialog applications. ArXiv, 2201.08239, 2022.
TogetherComputer. Redpajama: an open dataset for training large language models, 2023. URL jama---aaaaaa-amm ama-Data.
16

---

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, Rodriguez, A. Joulin. E. Grave, and G. Lample. Llama: Open and efficient foundation language models. ArXiv, 2302. 2023.
D. Vadas and J. R. Curran. Parsing noun phrases in the Penn Treebank. Computational Linguistics, 37(4):753- 809, 2011.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, pp. 5998-6008. Curran Associates, Inc., 2017.
O. Vinyals, T. Ewalds. S. Bartunov, et al. Starcraft II: A new challenge for reinforcement learning. ArXiv, 1708.04782, 2017.
Wang, J. Yan, A. Gu, and A. M. Rush. Pretraining without attention. ArXiv, 2212. 2054.10544, 2022.
Wang, B. Z. Li, M. Khabsa, H. Fang, and Ma. Linformer: Self-attention with linear complexity, ArXiv, 2006.04766 2020.
S. Wang, Y. Sun, Y. Xiang, et al. ERNIE 3.0 Titan: Exploring larger-scale knowledge enhanced pre-training for language understanding and generation. ArXiv, 2112.12731, 2021.
Y. Wu and K. He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
L. Xue, N. Constant, A. Roberts, M. Kale, Al-Rfou, A. Siddhant, A. Barua, and C. Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Conference of the North American Chapter of the Association for Computational Linguistics, pp. 483 498, 2021. doi: 10.18653/v1/2021 12.16633/112221..ammmmmmm maincl-
S. Yang and Y. Zhang. FLA: A Triton-based library for hardware- implementa- tions of linear attention mechanism, 2024. URL https: com/ sutt//ognccon flash- www---nnner-rre attention.
S. Yang, B. Wang, Y. Shen, R. Panda, and Y. Kim. Gated linear attention transformers with hardware- efficient training. ArXiv, 2312.06635, 2023.
S. Zannettou, B. Bradlyn, E. DeCristofaro, H. Kwak, M. Sirivianos, G. Stringini, and J. Blackburn. What Gab: A bastion of free speech an alt-right echo chamber. In The Web Conference, pp. 1007-1014, 2018. doi: /411144/414558311911
W. Zaremba and I. Sutskever. Learning to execute. ArXiv, 1410.4615, 2014.
R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics, pp. 4791 4800, 2019.
A. Zeng, X. Liu, Z. Du, al. GLM-130B: An open bilingual pre-trained model. ArXiv, 2210.02414, 2022.
S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, Simig, P. Koura, A. Sridhar, T. Wang, and
L. Zettlemoyer. OPT: Open pre-trained transformer language models. ArXiv, 2205.01066 2022.
17

---

Contents
Broader Impacts
19
Extended Long Short-Tere Term Memory
19
B.1
Vanilla Long Short-Term Memory Formulation: Vector Notation
20
B.2
sLSTM
20
B.3
mLSTM
22
B.4
Detailed Block Structure
26
C Extended Related Work
28
D Experiments
29
D.1 Synthetic Tasks and Long Range Arena
29
D.1.1 Test of xLSTM's Exponential Gating with Memory Mixing.
29
D.1.2 Test of xLSTM's Memory Capacities on Associative Recall Tasks.
32
D.1.3 Test of xLSTM's Long Range Capabilities on the Long Range Arena
34
D.2 Method Comparison and Ablation Study on SlimPajama (15B)
38
D.3 XLSTM Large Language Models SlimPajama3008
41
E Detailed Results on PALOMA Language Model Evaluation
44
18

---

A Broader Impacts
Our work introduces novel LSTM architectures that perform favorably compared to Transformers or State Space Models on language modeling. As our novel LSTM models are inherently recurrent and demonstrably extrapolate well large contexts, they have the potential for near constant scaling at inference. This compares to linear scaling in the context length for Transformer architectures, representing a vast potential in energy and emission savings when deploying such models to real world applications. Thus, in an ever more connected, data-rich world with ubiquitous compute, our work has the potential to increase general accessibility state-of-the-art machine learning models whilst making them more environmentally sustainable.
However, any novel machine learning technique that has the potential impact real world applications can be used for harm, not just for good. Our models with their beneficial inference economics and their ability extrapolate to large contexts no overhead could be used, for example, to generate and spread disinformation on a grander scale. Moreover, savings in energy and emissions afforded by deploying our novel xLSTM models might be outweighed by an increase in demand for such models, leading to overall increase in energy consumption and environmentally harmful emissions. Whilst our work focuses on new, more efficient Language Model architectures, does not directly address the issues of fairness and bias of such models.
B Extended Long Short-Term Memory
LSTM
Memory Cells
XLSTM Blocks
XLSTM
Memory Cells
SLSTM
+ Constant Error Carousel
Exponential Gating
Sigmoid Gating
New Memory Mixing
Recurrent Inference
Recurrent Training
4 a-1 + 4
hr oe v( (+)
mLSTM
Exponential Gating
Matrix Memory
Parallel Training
Covariance Update Rule
Figure 6: The extended LSTTM (xLSTM) family. From left right: The original LSTM memory cell with constant error carousel and gating. 2. New sLSTM and mL.STM memory cells that introduce exponential gating. sLSTM offers new memory mixing technique. mLSTM is fully parallelizable with a novel matrix memory cell state and new covariance update rule. 3. mLSTM and sLSTM in residual blocks yield xLSTM blocks. 4. Stacked xLSTM blocks give an xLSTM architecture.
19

---

B.1 Vanilla Long Short-Term Memory Formulation: Vector Notation
The vanilla LSTM memory cell update rules (Greff al., 2015) at time step t extend the scalar cell state formulation to vector of cell states:
C₁
f
Ct-1
Zt
cell state (25)
h
O
h
h
C₁
hidden state (26)
4
2x)
Σ Wz + R₂ h4-1 + b
cell input (27)
i
=
d
i
i, Wi at + Rí h + bi
input gate (28)
f
ft
f Wr a: + Ry hi-1 + br
forget gate (29)
O
0 (ö:)
o, Wo Tt + Ro hi-1 bo output gate (30)
The matrices W., Wi, Wr. and W. correspond to the input weights between inputs æ, and cell input, input gate, forget gate, and output gale, respectively. The matrices Rg, R2 RG, and R. correspond to the recurrent weights between hidden state h and cell input, input gate, forget gate, and output gate, respectively. bz. bj. br. and b, the corresponding bias vectors. ip and e the cell input and hidden state activation functions (typically tanh). & used to normalize or squash the cell state, which would be unbounded otherwise.
B,2 sLSTM
Similar to the LSTM in Section B. also the sLSTM can be vectorized to multiple cells:
C₁
Ct-1
+
i,
Zt
cell state (31)
nh
f,
+
ip
normalizer state (32)
hr
O
h
h
C + n2
hidden state (33)
Ze
4p
(Z+)
2. W₂ w.
R2 he-1 + b
cell input (34)
exp
ir
i Wi + Rí hi-1 b
input gate (35)
fc
exp
f
OR
fg
f Wr + R h + br
forget gate (36)
(ö:)
6 Wo at + Ro hi-1 + bo output gate (37)
Here, the cell input activation function yo is tanh, the hidden state activation function is the identity. x helps stabilizing the recurrence.
Considering external gradient contribution 6oh from subsequent layers and recurrent gradient contri- bution 5ft from gradients from future states flowing over the cell interaction matrix R. we obtain the recursive backward pass STM, where 5, indicates gradients with respect to parameter internal variable
20

---

on. gar 0h:
(38)
8 f ① be, + 0:-1 74-1 -1 8h.-1
(39)
:n fl ① dn. - -i-1 ② C-1 ① n, 8h.-1
(40)
6f f/ C4-10 Ra-10de. Oda + fr o nn-1 C on,
(41)
6j i C Oz ① dc + ij o 6n
(42)
6R. i, @ ('x))) de,
(43)
dar 0f 0
Sr.
(44)
8x.
E
W
T 0g.
(45)
(f.iizo,))
5R
Σ
0g.
(46)
B
ge (f.i,o)))
Rg
Σ h
E (i,f,2,0) oo
(47)
o
Σ
g (i,f,,o))
(48)
Rt
with the derivatives the respective gate activation function i = exp' (ix) = exx(ii) = i, of = a' do,), and f/ = of;;, or f = f, depending on the forget gate activation. (0(z) is the derivative of the cell input activation function p(z).
The matrices Rg, R. R2. R. are block- -oaekona which is analogous to multiple heads in the mLSTM. This way, the parameters reduce d² /NN,,, where N₂ is the number of heads, limiting the cell interactions to individual heads. This parameter efficient formulation of cell interactions together with the exponential gating is called the new memory mixing. Finally, stabilize the backward pass, we clip the magnitude of R to 10, as means to prohibit exploding gradients for long context lengths.
sLSTM Stabilized Version. The stabilized version of sLSTM introduces a new stabilizer state m applied as:
m
max
log
+ m4-1 1
log(
stabilizer state
(49)
exp
(log
exp
(ir
m/
stabil. input gate
(50)
exp
log
fe
+
74-1
m
stabil. forget gate
(51)
We show that replacing f, by f/ and i, by it in the forward pass does neither change the output of the whole network nor the derivatives of the loss with respect to the parameters.
The stabilization state m has no gradient, and hence does not influence the other gradients. We re-define (() and n.(²) stabilized cell and normalizer states:
Ci
exp
m4
(52)
T4e n, exp
(53)
21

---

Inserting Equation 49 into Equation 8 yields:
h1 c m
(s)
(54)
exp log (ft) + m
m4
q-1 + exp
log (ir)
m 2t
(55)
exp
(log (fc)
M4-1
m+
n + exp (log (ix)
me
exp (log (fi) + m
(o) exp (log (ix)
G--1
(56)
exp (log (fc) + m4-1 14-1 + exp (log (i+)
exp (log (f)) C--1 + exp (log (i+)) 2t
(57)
exp (log (G)) n4-1 + exp (log (ic)
fa--1 + le c//n he
(58)
[ +
Therefore, since the loss solely depends on hy, there's no dependency m,, and consequently, no gradient exists for this stabilization state. Note that my can be chosen arbitrarily. We choose m max (log (fe) + m1-1, log (iz)), which stabilizes the exponential function. One even find m,, such that the normalizer state nt can be eliminated, but this version was experimentally found to be numerically unstable in the backward pass.
B.3 mLSTM
Throughout this section, 1 E I denotes column vector of ones and I RIxT row vector of ones, where T is the dimension of this vector space.
Recurrent mLSTM Backward Pass. The recurrent formulation of the mLSTM cell in Equation 16 yields the following backward pass recurrence, where 6g indicates gradients with respect to parameter or internal variable and 8ext denotes gradients from subsequent layers:
T% O, gen
(59)
9--10 hi-1
gc, fide,
(60)
max {
|2, at
0--CT_10 1
8n_ = ff8n. max [| 91-1/1 1]
(n.-1qc-1) q0-1) q-1
(61)
2
ST. inki 80.
TT
(62)
6. (v, 80, st.)
(63)
C. T/.
q: C. 8i.
6q. max {IrI gel, 13
Ω (n. q:)
(64)
z
max { nn. q: 1]
Sx. Σ W 6g.
(65)
6w, Σ
215T ge:
9 E v)
(66)
_b, Σ 8g.1
g E {g. kg
(67)
6g (1T (C C dc, (+c (n--1 2 Sn.)) 7
(68)
6 (1 ((v,k)) @ (,6,,)111 (kr ón.)) exp (ix)
(69)
5.. h 3 0 ((r) ② on.
(70)
22

---

and 92 (2) = - 1) (z) being the Heaviside step function. 7 (2) is either a' (z)
or exp (z). depending on the forget gate activation.
Parallel mLSTM Forward Pass. The mLSTM recurrence in Equations (16-24) can be reformulated in parallel form, which is used to speed up training. After training we can still use the recurrent formulation for fast text generation.
Instead of processing each input at; IR time step sequentially, the parallel version processes all timesteps of full sequence X E RTxxd once, where T is the sequence length and is the head dimension. We present the forward pass of the mLSTM for single head and drop the head dimension for simplicity.
Let f E IR' be the forget gate pre-activations and E I be the input gate pre-activations for full sequence. We construct the forget gate activation matrix E RR by
0
for i < j
F
1
for i = j
(71)
Iii=
fi
for i >
and the input gate pre-activation matrix E TxT by
Ls for <
(72)
for }
By applying the elementwise exponential input gate activation function naively, we obtain the unstabilized gate activation matrix D TR''
D = F exp(I)
(73)
In order to avoid overflow due to the exponential function we apply the same stabilization as in the recurrent sLSTM, Equation 49. In the parallel formulation of the mLSTM get numerically stable gate activation matrix D' E Txx by taking the logarithm D element-wise and subtracting the row-wise maximum value of D from each element:
log = log
F (xx((() = log F +
(74)
D' exp(DD max D)
(75)
Given the queries, keys and values Q, K, V E Rxxd for a full sequence we can compute all hidden pre-activation states H E Rxd in parallel for the un-stabilized version by
and Ö = QK Vd
H = CV. with C= max Ef 1.
OD.
(76)
Note that we extract the J/2 factor for explicitly here and further on. For the stabilized version this yields
QKT H CV. with C= max Σj=1 Ö%y1 exp(- max D)) and E'
C'
o D', (77)
Vd
where for both versions the hidden pre-activation states H are identical.
With the output gate pre-activations Ô E Txx timesteps by applying the output gate in parallel for each timestep dement-wwse:
xd we can compute the hidden states H E xxd for all
H = o (ō) о H
(78)
This gives the parallel forward pass of the mLSTM for full input sequence X Txx
23

---

Parallel mLSTM Backward Pass. We present the backward pass of the mLSTM for the stabilized version only. For completeness we summarize the forward pass in the stabilized version before we present the backward pass.
Given the forget gate matrix F E RT*T" the logarithm of the forget gate matrix F log F E RTXT, and the input gate matrix I E RTTT as introduced above, together with the queries, keys and values Q, K, V E R'
xd we can write the forward pass of the mLSTM in the stabilized version as:
D = F +1
(79)
m max Dis j
row-wise maximum
(80)
D' = exp D m 1
(81)
δ QK T
+ D'
(82)
Vd
T
b ==
C Č² 1,
row-wise sum
(83)
j=1
max{|b] exp(-m))
(84)
C = C o (n
)
(85)
C
(86)
With this forward pass we can compute the gradients 6₂ for all intermediate and input variables to the mLSTM forward pass in the backward pass. We denote the gradient with respect to variable a as 6,.
Given the output gradient oū E Rxx we can compute the backward pass for the intermediate gradients as:
T V8
(87)
8n
C
④ (n 1
3 5c) 1
(88)
( 00)
1)
2 n -2
(89)
1 if b > exp(-m)
5 sign (n) 2 6n
(90)
0
otherwise
8 c (n" IT) 2 dc,
column-wise broadcast
(91)
TT δ b 1 00
column-wise broadcast
(92)
&g. = δōn
+ 8
(93)
QK
6D
Vd 3 G6..
(94)
6 exp D m) @ dD = D' の &D
(95)
We do not compute the gradients for m as they cancel out (see the proof in the recurrent sLSTM).
With these intermediate gradients the gradients for the logarithmic forget gate matrix 6p E RTxT, the input gate matrix ō1 E RTTT and the queries, keys and values åQ 8x, ōv E Rxxd are given by
bp 6D
(96)
6 6D
(97)
K
QQ D'
(98)
Vd
Q
kk = D
(99)
Vd
5v = C T 8
(100)
24

---

Having computed the gradients for the logarithmic forget gate matrix of, we can compute the gradients for the forget gate pre-activations óg = [ 6,..... 8r, E ERT.
Recall the logarithmic forget gate matrix F = log F is computed by
for i <
0
for i =
F iy log Fs kj
Eies+1 logo fs = =i
F
for i >
(101)
With the substitution f = log (f) we compute the gradients for the logarithmic forget gate activations or 16, 61,- .... 1t,1
T E IR. as
k- T
61. ΣΣ (6F)
(102)
j= t=k
di = o [e)) o7.
(103)
where the last equation makes use the following:
(log (x)) = (1 + exp(-x)) eep(-=x) (-1)
dx
exp(-I)
1
(104)
1+ 1+exp(-x)) 1 exp(x)
= o(-x)
Finally, we compute the input gate pre-activations' gradients 6j = 1&, or.... ogg]² E R as the column-wise sum over the rows of the input gate matrix òj:
(105)
This completes the backward pass of the parallel mLSTM for a full input sequence X E Rxx
25

---

B.4 Detailed Block Structure
PF=
GeLU
PF=
PF=
LN
GN
sLSTM
NH=4
NH=4
NH=4
NH=4
NH=4
NH=4
NH=4
NH=4
Swish
Conv4
LN
Figure 7: Schematic representation of an sLSTM Block - post up-projection: Embedded in a pre- LayerNorm residual structure, the input is optionally passed through a causal convolution of window size 4 that includes a Swish activation for input and forget gates. Then, for all input, forget and output gates i, f, o, and the cell update the input is fed through a block-diagonal linear layer with four diagonal blocks or "heads". These diagonal blocks coincide with the recurrent gate pre-activations from the last hidden state, which corresponds to sLSTM with four heads depicted with the circular arrows. The resulting hidden state goes through a GroupNorm layer (Wu & He, 2018) a head-wird I LayerNorm for each of the four heads. Finally, the output is up- and down-projected using gated MLP with GeLU activation function and projection factor 4 /3 to match parameters.
26

---

PF=1/
GN
Swish
mLSTM NH=4
LSkip
B5=4
B5=4
B5=4
Swish
Conv4
PF=2
PF=2
LN
Figure 8: Schematic representation of an mLSTM block pre up-projection: Embedded in a pre- LayerNorm residual structure, the input is up-projected first with projection factor 2, once for an externalized output gate and once as input for the mLSTM cells. The mLSTM cell input is dimension- wise causally convoluted (kernel size 4), before entering a learnable skip connection. We obtain input g and & via block-diagonal projection matrices of block size The values v are fed directly, skipping the convolution part. After the MLSTM sequence mixing, outputs are normalized via GroupNorm (Wu He, 2018) a head- wise layer norm for each of the four heads. Finally, the learnable skip input is added and the result is gated component- wise with the external output gate, The output is drojeprojccted.
27

---

C Extended Related Work
Linear Attention. Several methods have been suggested to overcome the quadratic complexity in terms of context length of the Transformer and make attention linear in the context length. The Synthesizer learns synthetic attention weights without token- token-token interactions (Tay et al., 2020). Linformer realizes self-attention by a low-rank matrix and even linearly approximates it (Wang et al., 2020). Linear Transformer linearizes the attention mechanism (Katharopoulos et al., 2020).
Performer linearly approximates the attention softmax by positive orthogonal random features approach (Choromanski et al., 2021). Attention has been replaced by fast long convolutions in the Structured Global Convolution (SGConv) (Li et al., 2022) and the Hyena Hierarchy (Poli et al., 2023).
State Space Models. Recently, State Space Models (SSMs) became very popular since they are linear in the context length and show promising performance compared to Transformers. One of the first proposed models was Structured State Space sequence model (S4) (Gu et al., 2021), followed by Diagonal State Space (DSS) model (Gupta et al., 2022), Gated State Space (GSS) models (Mehta et al., 2022), S5 model (Smith et al., 2022), Bidirectional Gated SSM (BiGS) (Wang et al., 2022), H3 model (Fu et al., 2023), and Mamba (Gu & Dao, 2023).
Recurrent Neural Networks. Recurrent Neural Networks (RNNs) have been suggested to replace Transformer and attention due their linearity in the context length. RNNs with Deep Linear Recurrent Units (LRUs) showed promising results for language modeling (Orvieto et al., 2023; De et al., 2024), as did Hierarchically Gated Linear RNN (HGRN) (Qin et al., 2023) and HGRN2 (Qin et al., 2024). A well-known RNN approach to large language modeling is RWKV (Peng et al., 2023, 2024), showcasing competitive performance Transformers.
Gating, One of the key ideas of LSTM is gating, which was rediscovered and reinterpreted in many recent approaches. Gating was used in HGRN (Qin et al., 2023), HGRN2 (Qin et al., 2024), Gated Linear Attention (GLA) (Yang et al., 2023), Gated State Space (GSS) models (Mehta et al., 2022), Bidirectional Gated SSM (BiGS) (Wang et al., 2022), Moving Average Equipped Gated Attention (MEGA) (Ma et al., 2022), RWKV (Peng et al., 2023), and Mamba (Gu & Dao, 2023).
Covariance Update Rule. To enhance storage capacities, we equipped the mLSTM cell with a matrix memory with a covariance update rule. Other methods which build on such an update mechanism are Fast Weight Programmers (Schmidhuber, 1992; Schlag et al., 2021), RWKV-5 and RWKV-6 (Peng et al., 2024), Retention (Sun et al., 2023), Linear Transformer (Katharopoulos et al., 2020), and HGRN2 (Qin al., 2024).
Most Related. Conceptually the closest models to xLSTM are Retention (Sun et al., 2023), RWKV (Peng et al., 2023, 2024), and HGRN2 (Qin et al., 2024). These models share the con- cepts matrix memory and/or gating. However, in contrast to the new sLSTM, these approaches do not allow memory mixing. Memory mixing enables solve state tracking problems, and therefore LSTMs are more expressive than State Space Models (SSMs) and Transformers (Merrill et al., 2024; Delétang et al., 2023). State tracking is required evaluate code or to track entities in long narrative.
Residually Stacking Architectures. Like almost all contemporary large deep learning models, XLSTM architectures are constructed by residually stacking building blocks (Srivastava et al., 2015; He et al., 2016). This construction enabled deep convolutional networks (He et al., 2016) and Transformers (Vaswani et al., 2017). Transformers are the ultimate force behind Large Language Models (LLMs) like GPT-3 (Brown et al., 2020), ChatGPT (Schulman et al., 2022), GPT-4 (Achiam et al., 2023), Megatron-LM (Shoeybi et al., 2019), Gopher (Rae et al., 2021), ERNIE 3.0 Titan (Wang et al., 2021), GLaM (Du et al., 2021), Chinese M6 (Lin et al., 2021). mutilingual AlexaTM 20B (Soltan et al., 2022), OPT (Zhang ct al., 2022), Chinchilla (Hoffmann et al., 2022), BLOOM (Scao et al., 2022), GLM-130B (Zeng al., 2022), LaMDA (Thoppilan al., 2022), PaLM (Chowdhery et al., 2022), Llama (Touvron et al., 2023), Gemini (Google, 2023; Reid et al., 2024).
28

---

D Experiments
Training Setup. For all experiments, use Python' 3.11 with PyTorch 2.2.0 and CUDA 12.1². We developed and trained all our models and baselines over the course of three months on a cluster with 128 nodes of eight NVIDIA A100 GPUs each. More than 95% of this compute were used for the Language Modeling experiments in sections 4.2 and 4.3.
Nearest Neighbor Search Task. For this auxiliary task, we use randomly sampled feature vectors of dimension 2 and unit norm. The attached value a uniformly distributed random number from [0, 1]. leading to inputs vectors of dimension 3. The first feature vector serves as search key, with the first value being ignored. Then the model has to prediet the value of the nearest neighbor so far the sequence. We train on 8192 sequences of context length up to 64 (uniformly sampled) and validate on 8192 different samples. All models have two blocks and embedding dimension 128. We use a dropout of 0.1, 10% linear warm-up steps and cosine decay to 1c-7 for 100k total training steps. We sweep over learning rates le-4, le-3, le-2, le-1 and seeds each. The reported values in Figure I are mean values for the best learning rate and 99% confidence intervals. Note that LSTM requires very high learning rates, whereas Transformers (Llama) perform best at the smallest learning rate. The XLSTM[[01] reaches similar performance across all learning rates.
Wikitext- 103 Rare Token Prediction. For this exemplary experiment on rare token prediction, we trained 125M-sized models on Wikitext- 103 (Merity et al., 2017). All models have an embedding dimension of 768 in post up-projection structure of 12 residual blocks. The Transformer model (Llama) uses Multi-Head Attention, for what is called LSTM the Multi-Head Attention is replaced by an LSTM and the XL: contains mLSTM layers with matrix memory. Models were trained with maximum learning rate le-3, 4k steps linear warm-up and cosine decay for in total 50k steps, using a batch size of 256 and context length of 512. We use the validation perplexity as a stopping criterion and evaluate the test set.
D.1 Synthetic Tasks and Long Range Arena
D.1.1 Test of xLSTM's Exponential Gating with Memory Mixing.
We evaluate xL.STM on a suite of formal language tasks to test its exponential gating and memory mixing mechanism. Formal languages provide a framework to probe the generalization capabilities of models. They allow specifically test different expressivity levels, e.g. along the Chomsky hierarchy. Typical language model architectures do not necessarily fit perfectly in these hierarchies (Delé- tang et al., 2023) nevertheless these languages allow to illustrate differences in generalization expressivity between different architectures. Our evaluation tasks are based on Delétang et al. (2023).
Experiment Setup. The different formal language tasks in the experiment (see individual tasks description below) encompass different levels of the Chomsky hierarchy as well as additional counting and eeemory-focusee tasks. We use different lengths per sample, which allows to validate in length ex extrapolation setting. We train on a varying task length up to 40. The evaluation done for task lengths between 40 and 256 as we are only interested in the "task generalization capabilities" of the models.
In all experiments, we use two blocks (or layers for the pure LSTM) for all models. We compare Llama, Mamba, Retention, Hyena, RWKV-4, RWKV-5, RWKV-6, LSTM, XLSTM[0:1], XLSTM[1:0] and XLSTM[1:1]. The sLSTM block is used without convolution and with normal weight initializa- tion. LSTM (Block) refers to an architecture where a vanilla LSTM used instead of self-attention inside a Transformer block.
All models are trained with 3 different learning rates (le-2, 1e-3, Ie-4), each with two seeds. Batch size is 256 - cosine annealing (min Ir: lc-5) with 10% warm-up steps is applied. We use AdamW (Loshchilov & Hutter, 2019) (81 0.9, 82 = 0.99) and a weight decay of 0.1 for training. In each experiment we train for 100k steps the samples are generated randomly, however, all experiments are trained and evaluated on the same samples.
'nttps://python org
hhttpe://pytorch org hhtpps:/docs nvidia. www.iia..om///ddd/aree....
29

---

Deterministic Context Free
Context Sensitive
Regular
Missing Duplicate
Arithmetic trasus
Solve Equation
Arithmetic Octo Bracee)
Majority
Bucket Sort
Cycle Nav
Even Pairs
Parity
Majority
Count
Llama-
0.92
0.08
0.02
0.02
0.04
1.0
0.03
0.03
0.37
0.13
-01
66.
-33
Mamba-
0.69
0.15
0.04
0.06
0.06
1.0
0.05
0.13
0.69
0.45
Retention-
0.08
0.05
6.1
1:00
1.000
1:00
Hyena
0.3
0.06
0.05
0.02
0.06
0.83
0.04
0.04
0.36
0.18
0.0
-122
1:30
1000
1.02
-03
1.00
0.0
RWKV-4
0.54
0.21
0.05
0.07
0.13
1.0
0.07
0.06
0.63
0.13
1:33
-.00
1.00
-33
1.06
RWKV-5
0.08
1.0
0.06
RWKV-6-
1.0
0.22
8.0
x:901
20000
2:00
9.112
LSTM
0.99
0.15
0.76
0.5
0.97
1.0
0.91
1.0
0.58
0.27
(Block)
300
1:30
1:305
8.033
1:00
9:35
1.0
000
0.94
0.2
0.72
0.38
0.93
1.0
1.0
1.0
0.82
0.33
1.90
1.6
XLSTM[0
0.84
xLSTM[1:0]-
0.97
0.33
0.03
0.03
1.0
0.04
0.46
..00
-8.0
+99
-.000
-030
-93
+0m
XLSTMD:1]
0.7
0.2
0.15
0.24
0.8
1.0
0.6
1.0
0.64
0.5
0.00
-010
-.6
0.3
Figure 9: Supplementary results given by scaled accuracy of different models at solving formal language tasks. Tasks are grouped by the Chomsky hierarchy.
Context Sensitive
Deterministic
Context Free
Reverse String
Stack Manipulation
Odds First
Repetition
Set
Llama
0.07
0.06
0.11 bd::
0.08
0.04
1000
Retention
0.03
0.11
0.03
0.02
0.02
:.00
+000
+
# 0.0
RWKV-4-
0.08
0.12
0.2
0.1
Hyena
0.04
0.15
0.07
0.07
0.03
1.00
1.00
bi
1.50
1.30
RWKV-5
0.09
0.16
RWKV-6
0.13
0.11
0.23
0.15
0.19
2.03
0.0
5.0
2.01
XLSTM[0:1]
0.14
0.13
0.09
0.17
Mamba
0.08
0.13
0.21
0.15
0.12
LSTM
0.08
D.17
0.25
0.15
0.18
(Block)
0.0
# 0.22
6.01
# 5.0
3.0L
xL:T1[
0.09
0.14
0.09
0.17
XLSTM[L:0]
0.15
0.22
0.25
0.28
0.17
0:22
b.01
4 s.
2.01
XLSTM[1:1]
0.09
Figure 10: Supplementary results given by scaled accuracy of different models at solving formal language tasks. Tasks are grouped by the Chomsky hierarchy.
Additional Formal Language Results. Figure 9 and Figure 10 showcase supplementary results of the formal language tasks. The former extends the results with additional models. The latter details tasks where model attained minimum scaled accuracy of 0.3. Although model achieves proper extrapolation of the task to larger context length, xLSTM performs best among the evaluated models.
Individual Task Description. The majority of tasks are based on Delétang et al. (2023). We provide the vocabulary size [V] and the random accuracy Brand (for accuracy scaling), used in the
30

---

evaluation. As we evaluate different task lengths each task has padding token which is used to pad the sequence to the given context length. In Listing 1 there is an example for each task.
Bucket Sort Given string of tokens of a sorted alphabet, compute the sorted string. V
11
Srand =
-
Cycle Nav Given string of "movement tokens" (+1, -1, STAY) compute the end position of the agent with start position O. The position must be computed modulo the maximum position.
[ =
Srand
M
Even Pairs Given binary string and b tokens, compute whether the number of ab and ba is even. This task can be solved by checking if the first and last token of the string are equal.
V =
Brand 0.5
Majority Given string of tokens, compute the token that occurred most often in the sequence.
V = 64
Srand
-
Majority Count Given string of tokens of an ordered alphabet. Compute the count of the token that occurred most often in the sequence. If the count exceeds the vocab size, the highest vocab token should be outputted.
V
64
Srand
-V-I
Missing Duplicate Given a string of tokens. The string is repeated but one of the tokens is masked in the repetition. Output the token that is masked.
V =1
Smand
|V
Mod Arithmetic (w/o Brackets) Calculate the result modulo the max number - of the arithmetic operations in the context. The maximum number is the vocabulary size minus the number of special tokens (t.-,* $ [PAD]).
V
Srand
|V
Mod Arithmetic (w Brackets) Calculate the result - modulo the maximum number of the arithmetic operations the context. The maximum number is vocabulary size minus the number of special tokens (t,,,* *,),,, [PAD]).
[V]
12
Srand
|V
Odds First An string of tokens t1, t2, t3, t..tn is given. Output all tokens with and odd index (h1, (a,...) then the token with an even index (12, f4...) Apart from that keep the ordering of the initial string.
V
12
Srand
1V1-2
1
Parity Given binary string of a and b tokens, compute if the number of b's is even. If the number is even output a otherwise This is equivalent to sequentially calculating the half-adder sum.
V =
8rand 0.5
Repetition Given a string tokens - repeat it.
V
12 Srand
2
Reverse String Given a string of tokens - repeat it in reverse order. V
12 Srand
|v|-2
Stack Manipulation An initial stack content is given, followed by a sequence of push and pop operations. Compute the stack content after the operations
V
11
8rand
Set Given string of tokens, compute the ordered set of the tokens. Keep the ordering so that tokens that occurred first are also outputted first.
V
128
Frand
-
31

---

Solve Equation Given is an equation with the operators [+.-, ++.,+,,,(,), number, and an unknown variable x. Compute the value of the variable modulo the max number. The maximum number is vocabulary size minus the number of special tokens (+,,,* "... [PAD),
[ACT]]
[ 14 rand -
Bucket Sort
Sequence: 4 8 6 1 1 6
Cycle Nav
Sequence: STAY +1 -1 +1 STAY +1 +1 +1 -1 P3
Even Pairs
Sequence: b
Majority
Sequence: 7 4 3 8 21
Majority Count
Sequence: 7 4 4 1 2
Missing Duplicate
Sequence: 8 5 8 [MIS] 5
Mod Arithmetic (w/o Braces)
Sequence:
0 2 [PAD]
Mod Ari thhmtiei (w Braces)
Sequence: ( ( 2 2 -
Odds First
Sequence: 7 6 [ACT] 2 6 29
Parity:
Sequence: b a
Repetition
Sequence: 4 2 [ACT] 4 6
Reverse String
Sequence: 4
2 [ACT] 6 4
Stack Manipulation
Sequence: ST1 ST1 ST3 POP POP PS3 PS3 (ACT] ST1 ST3 ST3
Set
Sequence: 6
5 5 [ACT] 86354
Solve Equation:
Sequence: ( ((
x) - (1)) 2 [ACT]
Listing I: Examples the formal language tasks. Red tokens are evaluated for loss and accuracy metrics, but are padded for the input. The tokens are illustrated in way that allows easy semantic interpretation for the given task hence, some tokens are represented by multiple characters.
D.1.2 Test of xLSTM's Memory Capacities on Associative Recall Tasks.
We test the memory capacity of xLSTM with the Multi- Multi-Query Associative Recall task proposed by Arora et al. (2023). Figure 11 illustrates the basic task setup.
Why Multi-Query Associative Recall for Memory Tests of L.LM Architectures. Associative Recall (AR), the ability to retrieve specific value (information) associated with a given key (infor- mation), constitutes key capability for L.I to perform well (Poli et al., 2024; Arora et al., 2023; Olsson et al., 2022). Especially its quality of in-context learning seems to be strongly connected to this capability (Olsson et al., 2022). Arora et al. (2023) attribute performance gaps between early non-Transformer and Transformer language models specifically to performance gaps in associative recall. They argue that prior AR evaluations fall short of capturing these differences and propose MQAR, which can show the AR performance differences that translate to performance differences in language modeling performance. Hence, MQAR is especially suitable analyze the memory capacity of LLM. Transformer (e.g. Llama) models can be seen as the gold standard for this task as their memory is exponential in the coding dimension (Ramsauer et al., 2021).
32

---

Experiment Setup. There are two relevant variables that determine different experimental setups. (I) Context Length (CL): Length of the sequence of one sample- - this influences the distances between the key-value definition and the recall. (2) Number Key-Value Pairs (KV): Influences how many key-value pairs the model needs to keep track of. The vocabulary size is always 8192.
In all experiments, we use two blocks (or layers for the pure LSTM) for all models. LSTM (Block) model refers to an architecture where a vanilla LSTM is used instead of self-attention inside Transformer block.
For each task setup, we train each model with 4 different learning rates (batch size > 24: {le-2, 3,15e-3, 4.6e-4, le-4), batch size 24: f1e-3, 2.2e-4, 5e-5, le-5)). The batch size (BS) changes depending the context length (CL) (CL=64/128: BS=512; CL=256: BS=256; CL=756: BS=128; CI==102: BS=96; CI=2048: BS=24). We vary the embedding dimension (Model Dim) between different experiments different numbers of heads used accordingly. For each experiment, generate 100,000 training samples (validation: 3,000 samples) and train for 64 epochs. We apply cosine annealing (min Ir: lc-4 and le-5) with 10% warm-up steps. We use AdamW (Loshchilov & Hutter, 2019) and a weight decay of 0.1 for training.
We conduct three different experiments:
MQAR-Experiment evaluates, in the same fashion as Arora et al. (2023), a vari- ety of models (Llama, Mamba, Mamba (noWT) ie. without weight tying. Reten- tion, Hyena, H3, RWKV-4, RWKV-5, RWKV-6, LSTM, LSTM (Block), xLSTM[0:1], XLSTM[[10] and xLSTT((1:1) on increasing task difficulty by increasing the context length and number of key-value pairs simultaneously. We benchmark three parameter settings: (I
MQAR-Experinees 2 increases the task difficulty notably and goes beyond previous evaluations on this task. We individually scale the context length (CI.=(756, 1024, 2048)) and the key-value pairs (KV=(48, 96, 256)) and evaluate all combinations. This experiment especially probes the memory capacity because the number of key-value pairs is high. To reduce the computational burden only evaluate models that perform flawlessly in Experiment 1 - additionally we evaluate Transformer only in the hardest setting (CL=2048) as sanity check, because no performance decrease is expected.
3 analyzes whether the AR capability learned on a certain context length extrapolates to bigger context lengths, For each KV setting of Experiment we use the models (we select the 3 biggest model dimensions) trained on CL=2048 and evaluate bigger context lengths (CL=(4096, 6144, 8192)).
Extended Results. The result of Experiment can be found in Figure 12. In accordance to the results of Arora et al. (2023). H3, Hyena, RWKV-4 fail to solve the task with smaller model dimension. In contrast, xLSTM[1:1], xLSTMM[1:0, Mamba, RWKV-5 and RWKV-6 are able to solve these settings for all model dimensions. The comparison of 1 with both original LSTM variants indicates that the exponential gating mechanism improves the AR capabilities of the model. However, both fall short because of the reduced memory capacity compared XLSTM[I:1] and XLSTM[1:0].
The results of Experiment 2 are presented in Figure 13. Scaling the context length has low impact on the performance of the models. However, while xLSTM[[:1] and ALSTM[1:0) show no clear decay, both RWKV variants slightly, but consistently lose performance with increasing context lengths. The varying number of key-value pairs, which mainly probes the memory capacity of the non-Transformer models, has more notable impact across all models. RWKV-5 seems outperform RWKV-6. The latter fails to learn the task at all in some KV=256 settings. Overall xLSTM[1:1 is the best-performing TranSrassfrrmer model suggesting that it provides enhanced memory capacity, also in long contexts.
Figure 14 shows the extrapolation results from Experiment 3. For XLSTM[1:1], XLSTM[1:0] and Mamba the model performance does not change in the extrapolation setting. The RWKV models (especially RWKV5) degrade slightly with increasing context length. xLSTM[1:1] performs best, as it maintains its superior performance of Experiment 2.
'The keys are distributed on the "evaluation part" of the sequence given a power-law distribution. This is motivated by similar structures in natural language text.
33

---

Target
Input
KV 4 CL 18
Figure 11: Illustration of the MQAR task. Color pairs represent key-value pairs (keys have darker shade). The first part of the sequence defines the key-value pairs for the respective sample. After that, the keys appear randomly according
power law distribution" 4. Grey tokens in the input sequence represent a zero token. The "target" sequence contains the value after the respective key appearance the rest of the tokens are ignored for the accuracy and loss calculation. The model must predict the value tokens given the respective key.
D.1.3 Test xLSTM's Long Range Capabilities on the Long Range Arena.
We assess the performance of xLSTM across tasks in the Long Range Arena benchmark (Tay et al., 2021), I), examining its ability to effectively handle longer context lengths and diverse data types.
Our experiments Long Range Arena benchmark are composed of five tasks:
Retrieval: The task is to predict if two documents have citation link. The dataset of text documents is derived from the ACL Anthology Network (Radev et al., 2009).
This is of modular arithmetic tasks including brackets and lists of numbers, using the operations MIN, MAX, MEDIAN and SUMMOD (modular sum). A particular example is: [MAX 4 3 [MIN 3 ] 0 [MEDIAN 1 8 9, 2]] 5
ListOps:
Image: This task is based on version of the CIFAR dataset (Krizhevsky, 2009), where images are transformed to a sequence of pixels and this sequence has to be classified into the usual CIFAR classes. We test both gray-scale scale (G- (G-Image) and RGB (RGB (RGB-Image) Image) version this dataset, as Orvicto et al. (2023) uses colored images contrary to the standard setup.
The input for this task is 32x32 gray-scale image, given as pixel sequence, with two dots and several curved lines it. The task is to predict if the two dots are connected by any of the lines (Linsley et al., 2018).
Pathfinder:
We omit the Text classification task (Maas et al., 2011 1), as the language modeling experiments already test this kind of data, and the Pathfinder-X XX version of Pathfinder.
Experiment Setup. The architectures that are tested in this experiment comprise LLama, Mamba, LSTM, RWKV-4, and xLSTM. LSTM (Block) refers to an architecture where a vanilla LSTM is used inside post proprojection block (like Transformer with attention replaced by LSTM). For xLSTM we choose the best performing of XLSTM(0:1] 1] or XLSTM[1:0] on the validation set, specifically the former for the Image tasks and the latter for all other ones.
We use the hyperparameter settings the S5 model (Smith et al., 2022) and Linear Recurrent Unit model (Orvieto et al., 2023), with additional hyperparamter search on learning rates and schedulers for all models. We use two different schedulers: Linear Warm-up Cosine Annealing and Linear Warm-up Cosine Annealing with Restarts. Both learning rate schedulers were evaluated with learning rates of le-3, 6e-4 and le-4. For the second scheduler, the number restarts (R) set 3. The model hyperparameters for each dataset displayed in Table 3.
Results. Table shows the result of experiments on the Long Range Arena benchmark. xLSTM demonstrates consistent strong performance all of the tasks, suggesting that the proposed architec- ture is remarkably efficient in handling different aspects long context problems.
34

---

Llama
RWKV-4
Retention
XLSTM 1]
LSTM (Block)
Mamba
RWKV-5
Hyena
xLSTM|
LSTM
Mamaba (noWT)
RWKV-6
H3
XLSTM[U
Context Length 64
Context Length
128
Contect Length 256
1.0
Transformerr
0.6
0.0
1.0
(*)LSTM Family
Accuray
0.6
Mamba
0.6
0.4
RWV
0.6
Others
0.6
0.0
64
128
256
512
32
64
128
512
32
128
256
512
Dim
Model
Figure 12: Result of EIQAR-Experimenn 1. The columns show different task settings (context length and key-value pairs). The rows group related models for better clarity. The z-axis gives the model size and the a-axis the validation accuracy.
35

---

Liama
Mamba
RWKV-5
RWKV-6
XLSTM|L:0]
XLSTM[L:]]
Vay-V Pairs 48
Key Value Pairs 96
Key- Pairs
1.0
Contect Length 756
0.8
Context Length 1004
0.4
0.2
1.0
Context Length - 2048
Accinoy
0.2
0.0
32
128
32
256
512
Model Dim
Model Dim
Model Dim
Figure 13: Result of MQQAR-Experiment 2. The columns and rows correspond different numbers of key-value pairs and the context length respectivly. The z-axis gives the model size and the y-axis the validation accuracy.
Table 3: Long Range Arena model hyperparameters. These are the model hyperparameters used in each of the Long Range Arena tasks. For each model used the best learning rate and the better of the two learning rate schedulers.
Embedding Dim
Batch Size
Training Steps
Task
#Blocks
Retrieval
6
128
64
100k
ListOps
8
128
32
80k
Pathfinder
6
192
64
500k
G-Image
6
512
64
180k
RGB- -ma-lmaae
6
512
64
180k
36

---

STMMO
STTM Key- 256
Key-Value Pairs
Key-Value
Conteat Longth 4096
0.75
Accuent
0.25
1.00
Contort Length - 6144
0.75
Accuney
0.30
0.25
0.00
1.00
Context Length - 8192
0.75
Accunry
0.50
0.25
0.00
128
512
256
512 128
512
Model Dim
Model Dim
Model Dim
Figure 14: Result of TQQAR-Exerrimenll 3 (Extrapolation). All evaluated models were trained on context length 2048 and the number key-value pairs given by the columns the plot. The rows show the different context lengths used in the evaluation. The r-axis gives the model size and the y-axis the validation accuracy.
Table 4: Long Range Arena test accuracy. Bold highlights the best performing model, underlined the second best. denotes models that fail to outperform random baselines. xL.TTM is the best of XLSTM[1:0] xLSTM[0:1] based validation dataset accuracy.
Retrieval acc t
ListOps acc t
Pathfinder acc f
G-Image acc
RGB-Image acc t
Ranking acc
Random Baseline
0.500
0.100
0.500
0.100
0.100
Llama
0.845
0.379
0.887
0.541
0.629
5.2
Mamba
0.902
0.325
0.992
0.689
0.765
2.2
RWKV-4
0.898
0.389
0.914
0.691
0.757
3.0
LSTM
0.275
0.675
0.718
5.4
LSTM (Block)
0.880
0.495
0.690
0.756
3.4
XLSTM
0.906
0.411
0.919
0.695
0.761
1.6
37

---

D.2 Method Comparison and Ablation Study on SlimPajama (15B)
General Training Procedure. We tokenize our datasets using the HuggingFace GPT-2 tokenizer (Radford et al., 2019; Brown et al., 2020)* and use this tokenizer for all models in this paper. In general, we try to follow Brown et al. (2020) for the general training setup, i.e. we choose context length 2048 and batch sizes 256 or 512 for our models. We use the AdamW (Loshchilov & Hutter, 2019) optimizer with beta parameters (81, (2)= 0.95) and an epsilon parameter of 1e-5, and gradient clipping at gradient norm 1. As learning rate scheduler we use linear warm-up with 750 steps and cosine decay to 10% of the peak learning rate. We apply a weight decay of 0. to all our models and always exclude the token embedding matrix from weight decay. If not specified otherwise, we do not tie the weights of the token embedding and the language model head. Except for gates we do not use biases in our models, e.g. in other línear layers. We do not apply weight decay to biases and LayerNorm weights. For parallelization, we use Py TyTorhh FSDP in SHARD SRADD_GRADOOD mode with mixed precision in bfloat16, where applicable. For small models use WO_SHARD. We keep the weights in float32 and reduce the gradients across GPUs in float32. We use torch. compile to speed up models where applicable, except for Transformer models their training curves did not match the non-compiled versions. For XLSTMM(7:),, we use positions [3, 5, 7, 40, 42, 44) for sLSTMMbased blocks, except for the 125M size, where we use [3, 20J (this is actually a [11:1] ratio). We do not use any positional encoding for our xLSTM models.
Details on Comparison to Other Methods. For the model comparison on 15B training tokens of SlimPajama we train all models with context length 2048 and batch size 256. We use peak learning rate of 1e-3 for all models for comparability. The learning rate decays over 30k training steps. The models are compared after one epoch at training step 28170. As model implementations we use the original repositories' code for Mamba (Gu & Dao, 2023) 6, RWKV-5, RWKV-6 (Peng et al., 2024)". For RWKV-4 we use a cleaned and validated re-implementation based on the original repo and kernels (Peng et al., 2023). In our RWK V-4 implementation we enable weight decay on all parameters except biases, the token embedding weight and all LayerNorm weights. For HGRN (Qin et al., 2023), GLA (Yang al., 2023), HGRN2 (Qin al., 2024) we use the remmmmlemennttionn by the authors of GLA (Yang et al., 2023; Yang & Zhang. 2024)*. For GPT-3 and Llama-like Transformers, we use our own implementations based on PyTorch. Note that for all xLSTMM, Transformers, Mamba and RWKV-4, we use Mixed Precision training with bfloat 16 and weights in float32 precision. Following the general training procedure we use torch. compile for all models, except for models using the flash www--linnaratteet compilation problems and Transformers as for those training curves deviated.
(Yang & Zhang, 2024) library because of
As RWKV-6 performs worse than RWKV-5, we also train a model with peak learning rate 4e-4, as reported in the original repository for 350M parameter models 9. This model reaches perplexity of 16.38, worse than the 15.03 for the standard peak learning rate 1e-3 as reported in Table 1.
Details on Training Precision for Baselines. For models from flash fassh-linear deinnar-aatenttion on and RWKV-5/6 models we found that PyTorch automatic mixed precision training did not work, but casting the model weights to float32 initially with FSDP parameter precision bfloat16 led to a working configuration. In this setting models perform better than in full bfloat16 training, where the weights are casted to bfloat16 initially as well. Full float32 did not work because of the custom kernels.
General Details on Ablation Studies. We follow our general training procedure and train all models with context length 2048, batch size 256 and peak learning rate 1e-3. We report perplexity values on the validation set.
Additional Ablation Study on Matrix Memory. As default block configuration we use the mL.STM the pre up-projection block (see Figure 8) and the STM in the post up-projection block
htttpp:///ithbbb
'https:// httpp/////iiiaab.mmom////aaaalaa
SWKV-v5/train.pyet
38

---

Table 5: Peak learning rates and model dimensions for scaling law plots.
#aramm M
Peak LR (15B)
Peak LR
Model
EmboddingDim
#Blocks
#HeadsHcadDDim
(300B)
RWKV-5
768
12
176.5
3e-3
RWKV-6
768
12
173.6
3e-3
125M
HGRN2
768
12
162.2
3c-3
RWKV4
768
12
169.4
3e-3
6e-4
Llama
768
12
12 64
162.2
3e-3
3c-3
Mamba
768
24
167.8
3e-3
3e-3
XLSTM
768
24
3/384
163.8
3e-3
1.5e-3
RKWV-5
1024
24
455.7
1e-3
350M
RWKV-6
1024
24
441.6
1e-3
HGRN2
1024
24
411.4
1e-3
RWKV4
1024
24
430.5
1e-3
4e-4
Llama
1024
24
16
406.6
1.5e-3
1.5e-3
Mamba
1024
48
423.1
1.5e-3
1.5e-
XLSTM
1024
48
4/512
409.3
1e-3
7.5e-4
RWKV-5
1536
24
947.8
9c-4
760M
RWKV-6
1536
24
907.7
9e-4
HGRN2
1536
24
834.2
9c-4
RWKV4
1536
24
891.0
2e-3
2.5e-4
Llama
1536
24
16/96
834.1
1.25e-3
1.25e-3
Mamba
1536
48
870.5
1.25e-3
1.25e-3
XLSTM
1536
48
4/ 768
840.4
9e-4
6,25e-4
RWKV-5
2048
24
1616.0
9c-4
RWKV-6
2048
24
1537.5
9e-4
L.3
HGRN2
2048
24
1439.4
9c-4
RWKV4
2048
24
1515.2
1e-3
2e-4
Llama
2048
24
16 128
1420.4
le-3
le-3
Mamba
2048
48
1475.3
1e-3
le-3
XLSTM
2048
48
4/ 1024
1422.6
9e-4
5e-4
RWKV-5
2048
24
3194.7
8c-4
RWKV-6
2048
24
3021.9
8e-4
2.7B
HGRN2
2048
24
2795.4
8c-4
RWKV4
2560
32
2984.8
8e-4
Llama
2560
32
/
2779.5
8e-4
Mamba
2560
64
2897.2
8c-4
XLSTM
2560
64
4/ 1280
2788.3
8e-4
19
Figure 15: Method comparison on next token prediction when trained on 15B tokens from SlimPajama. Performance mea- sure in validation perplexity for the best methods each model class (see Table 1) are reported. The performance degradation of XLSTM[7:1] at 2.7B is due to initially slower training conver- gence that leads to especially undertrained model. xLSTM the best method all sizes.
Liama
Mamba
Validation Perplexity
17
RWKV-4
16
XL5TM[7:1]
15
XLSTM(1:0]
14
12
11
15B Tokens
0.2
0.4
1.0
1.4
2.7
Number of Parameters
x10"
39

---

Table 6: Ablation studies. Top: Ablation studies on the new xLSTM components, contributing the strong performance improvement of xLSTM over vanilla LSTM to both the exponential gating and the matrix memory. Bottom: Ablation studies on different gating techniques. We consider an XLSTM[[1:) with sigmoid forget gate and exponential input gate. Bias initialization oo means that the forget gate is set to one, [3. 6] indicates that values are taken equidistant the respective interval, and (O, 0.1) that values are randomly chosen from Gaussian with mean 0 and std 0.1. PPL denotes validation perplexity. The first two lines correspond to models similar to linearized attention, line four to Retention, line five to RWKV-5, and line six to RWKV-6, Dependencies of the gates on the input lead to better performance.
Exponential Gating
Matrix Memory
#Params M
SlimPajama (L5B) ppl
Model
Modification
Vanilla Multi-Layer LSTM
X
x
607.8
2417.86
LSTM
Adding Resnet Backbone
x
506.1 1
35.46
Adding Up-Projection Backbone
X
x
505.9
26.01
XLSTM[O:1] Adding Exponential Gating
V
x
427.3
17.70
XLSTMM[7:1] Adding Matrix Memory
408.4
13.48
Ablation studies on different gating techniques.
Forget Gate
Input Gate
SlimPajama
Learnable Gates
Input Dependent
Leamable Bias
Bias Init
Input Dependent
Learnable Bias
Bias
(15B) ppl
Init
No Gates
X
X
3c
X
x
0
NaN
Gates
X
X
3.
X
x
13.95
Forget Gate
3, 6
X
x
13.58
Input Gate
/
/
A 0.1)
13.69
Forget Gate Bias
X
3, 6
X
x
13.76
Forget Input Gate Bias
X
3. 6
X
13.73
Forget Gate + Input Gate Bins
3.
X
V
0.10..11
13.55
Forget Gate Input Gate
[3.
V
A((0.0.1)
13.43
(see Figure 7). In this experiment study the combination of mLSTM with different block variants using the xI:STM[1:0] architecture. We compare the mI.STM in a post up-projection block (see Figure 7) with ReLU activation function and non-gated feed-forward network to mLSTM in pre up-projection block with and without dimension-we causal convolution. Table 7 shows that the matrix memory benefits from the pre up-projection block structure, and that the convolution within this block is important.
Table 7: Matrix Memory variants. We study different configuration for the matrix memory. Matrix memory in the pre up-projection block performs best and gives XLSTT((1:0]. Notably, it seems that the dimension-wise causal convolution within the pre up-projection block is important.
Embedding Dim
#Params
SlimPajama (1SB) ppl J
Model
Details
#Blocks
Post Up- Pr-orroeetion Block (ReLU22 Pre Up-Projection Block, No Convolution
24
1024
430.4
13.90
XLSTM[1:0]
48
1024
408.8
15.41
Pre Po-Prrjection Block, With Convolution
48
1024
409.3
13.43
Details on new xLSTM Components Ablation Study. In Table (top), show our modifications to the vanilla LSTM that transform the vanilla LSTM into the xLSTM. We start with a large default PyTorch LSTM with 24 layers and 1536 hidden size. Due to a lack of connecconnts and LayerNorms, vanilla LSTMs of this size are not trainable. We then add skip-connections and pre-LayerNorms before each L.TTM layer corresponding to a residual architecture. This enables training for LSTMs at this scale. Replacing every second LSTM layer by non-gated network with GeLU activation function (similar to Vaswani et al.), which corresponds to the post up-projection backbone (see Figure 7), further boosts performance. Adding Exponential Gating to this
40

---

architecture yields the sLSTM as depicted in Figure 7, with another large performance improvement. Finally, adding the best Matrix Memory variant found in Table 7 by replacing some sLSTM blocks with the mLSTM (see Figure 8) gives S|M[7] with the best performance.
Details on Gating Technique Ablation Study. In Table (bottom), we investigate the effect of trainable and input-dependent gates for mLSTM. The results show that, in contrast to other methods (Katharopoulos et al., 2020; Sun et al., 2023; Qin al., 2023; Katsch, 2023; Yang et al., 2023; Qin et al., 2024; Peng et al., 2024), having the gates both learnable and input dependent gives the best results.
Details on Scaling Experiments. We follow our general training procedure (see paragraph above) and train all models, including the 1.3B and 2.7B model sizes, with context length 2048 and batch size 256. We use the peak learning rates from Table 5, For Llama and Mamba we use the learning rates reported by Gu & Dao (2023).
D.3 XLSTM Large Language Models SlimPajama3008
General Training Procedure. We use the same general training procedure as in Section D.2 with peak learning rates from Table For Llama and Mamba we use the learning rates reported by Gu & Dao (2023). All models are trained with context length 2048. The 125M, 350M and 760M models are trained with batch size 256 for 600k training steps, whereas the 1.3B models are trained with batch size 512 for 300k training steps. We keep the same learning rate scheduler across all models.
Additional Scaling Law Plots with Number of FLOPs. In Figure 16 we plot the scaling behavior of our XLSTM variants over the number of training flops and compare to the Llama baseline. We compute the FLOPs for the recurrent (see Section 2) and the parallel (see Section B.3) formulation of the mLSTM. We can see that for the larger models (760M and 1.3B) the recurrent xLSTM FLOP counts are larger than the Llama FLOP counts. The reason for this is that we kept the number of heads constant when increasing the model size (see Table 5) while increasing the head dimension. In contrast Llama increases the number of heads while keeping the head dimension constant.
17
Llama
16
xLSTM[1:0] recurrent
15
xLSTM[1:0] parallel
Validation Perplexity
14
XLSTM[7:1] recurrent
xLSTM[7:1] parallel
13
12
11
10
300B Tokens
10
Number of FLOPs
Figure 16: Scaling laws over number of training FLOPs. We compare the Llama baseline with our xLSTM variants and compute the number of training FLOPs for the recurrent and parallel mode for 300B tokens with context length 2048.
41

---

Details on FLOP computation. For computing the number of FLOPs we follow the procedure from Hoffmann al. (2022). We include all training FLOPs, including those contributed by the embedding matrices. We assume that the backward pass has twice the number of FLOPs of the forward pass, For the forward pass, the number of FLOPS of the mL.TTM and sLSTM for a single sequence can be approximated by:
mLSTM (recurrent): num heads x seq_len x (6 x head_dim dim x head dim + x head dim)
mLSTM (parallel): num heads seq_len seq len (4 x head dim 8)
sLSTM: num_heads seq_len& head_ x (8 x head_dim, 12)
Details on Downstream Evaluation. We use the LM Evaluation Harness from EleutherAI (Sutawika al., 2023) for evaluating the following tasks that measure common sense reasoning: LAMBADA (OpenAl version in LM Evaluation Harness) (Paperno et al., 2016), HellaSwag (Zellers al., 2019), PIQA (Bisk et al., 2020), ARC-challenge, ARC-easy (Clark et al., 2018), WinoGrande (Sakaguchi et al., 2021). This selection of downstream tasks is also used in previous work by Gu & Dao (2023).
Following Gu & Dao (2023), we report accuracy for LAMADA, WinoGrande, PIQA, and ARC-casy, and accuracy normalized by sequence length for HellaSwag and ARC-challenge.
We evaluate all models in full float32, full bfloat16 and bfl oaloat16 Mixed Precision with weights in float32. For each model we select the best value respectively.
Details on PALOMA. We use 16 out of the 18 data sources of the PALOMA dataset (Magnusson et al., 2023). We use C4 (Raffel et al., 2019), MC4-EN (Xue et al., 2021), Wikitext- 103 (Merity al., 2017), Penn'rrebhank (Vadas & Curran, 2011), RedPajama (TogetherComputer, 2023), Fal- con Refinedweb (Refined Web) (Penedo et al., 2023), Dolma v1.5 (Soldaini et al., 2023), M2D2 S2ORC, M2D2 Wikipedia (Reid et al., 2022). C4-100-Domains (C4 Domains) (Chronopoulou et al., (Dolma Subreddits) (Soldaini et al., 2023). Dolma-100-Programmmne Languages (Dolma Coding) (Soldaini et al., 2023; Kocetkov et al., 2022), TwitterAAE (Blodgett et al., 2016; Liang et al., 2023), Manosphere Corpus (Ribeiro et al., 2021), GAB Corpus (Zannettou et al., 20.8)018,.000 4CHAN Corpus (Papasavva et al., 2020). We leave out ThePile (Gao et al., 2021) and ICE (Greenbaum & Nelson, 1996) they are not part of Paloma's 's Huggingface dataset repository 10, detailed description these datasets can be found in Magnusson et al. (2023, Table 2). All models evaluated in bfloat16 Mixed Precision. Results are shown in Table 8.
2022). Nolma--00-Subredae
Results on the data sources TWi Manosphere, GAB and 4CHAN are reported in Table and for each individual dataset the results are given in Section E.
In order evaluate the perplexity values on each data source, we split the text documents into sequences of length 2048, which corresponds to the pre-training context length of all models. For documents longer than 2048 tokens we split each document into non-overlapping input sequences. In this case for the last input sequence, we follow the LM Evaluation Harness and fill up the full 2048 token context window with previous tokens, but compute the perplexity only on the remaining tokens.
We compute the token perplexities per data source in Table as the exponential of the negative liglliiilssoods per domain weighted by the number of tokens per domain in that data source it is defined in Magnusson et al. (2023, Equation 1)
42

---

Table Performance on PALOMA Language Modeling Tasks. Comparison of xLSTM, RWKV44 Llama, and Mamba by the perplexity of next token prediction on the PALOMA language benchmark after training on 300B tokens from SlimPajama. Model sizes are 125M, 250M, 760M, and 1.3B, The second column shows the actual number of parameters. The 571 text domains are grouped into language modeling (next seven columns) and fine-grained domain benchmarks (further columms). The last column shows the average perplexity across all of these tasks. Best model per model size is given bold and the second best is underlined. xLSTM yields the best performance.
Wimmm
Witlirat
Ponn
Refined
M102
Wilipedia
Detra
Deima Codlig
Ancrage
XWKV-4
169.4
26.25
22.33
29.15
38.45
8.99
32.47
17.04
23,56
21.42
22.68
57.08
5.12
23.74
Liama
182
24.64
17.23
23.16
31.55
8.26
29.15
15.10
19.71
20.41
21.45
36.73
3.61
20.92
Manba
301.33
27.75
14.60
19.36
3.77
7.80
27.21
19:04
1.STMM7111
14.13
350M
LJama
Mamba
205.6
423
17.53
18.38
13.05
13.28
16.11
16.41
22.24
21.82
6.34
6.56
22.09
21.04
11.76
11,42
14.83
15.05
14.53
15:25
15.16
15.99
28.30
27.02
3.20
3.12
15.67
15.19
<1. <L.MTM[1:0]
409
17.01
12.55
15.17
22.51
6.20
20.66
11.16
14.44
14.27
14,85
76. 70
3,08
14,88
408.4
16.98
12.68
15.43
21.85
6.23
20.70
11.22
14.62
14. 30
14.85
26.61
3.11
14,88
RWKXV44
8910
15.51
12.76
14.84
21.30
5.91
19.28
10.70
14.27
13.04
13.68
24.22
3.52
14,08
700M
Liama
Manska
834.1
15:75
13.47
19.34
18.33
5.82
19.04
10.35
13.00
12.62
13.05
13.76
24.50
23.94
2,90
13.69
1LSTMIL
12.20
SLSIMIN1
14.22
11.11
12.65
3.55
12:25
2.88
RWKKV44
13.73
10:11
12.
3.25
Llama
1620.4
13.93
10.44
11.74
15.92
5.29
17,03
9.35
11.61
11.53
12:24
22.63
2.74
12.04
Maniaa
1475.3
13.35
10,40
11.76
16.65
5.21
16.50
9.17
11.73
11.18
11:33
21:43
2.83
11.84
<LSTMM[:0)
1422.6
13.13
10.09
11.4]
15.92
5.00
16.25
9.01
11,43
10.95
11,60
21.29
2.73
11.58
1420.L
13.31
10.31
11.32
16.00
5.16
16.48
9.11
11.01
11.10
11.76
21.50
2.75
11.02
Table 9: Perplexity values per domain.
Model
#Parans
Twitter AAB
Manosphere
4CHAN
GAB
RWKV-4
169.4
265.80
39.31
18.48
53.89
123M
Llama
162.2
277.93
32.98
14.03
56.45
Mamba
167.8
258.17
32.14
14.01
$1,58
STM[10]
163.8
244.53
31.45
13.27
$1.00
XLSTM[7:1]
163.7
248.51
30.90
13.45
50.25
RWKV-4
430.5
216.17
30.25
13.82
42.25
350M
Llama
406.6
231.09
25.90
11.49
43.04
Mamba
423.1
202.88
25.24
40.78
XLSTM[L:0]
409.3
200.61
24.58
11.20
39.83
XLSTM[7:1]
408.4
206.25
24.73
11.31
39.86
RWKV-4
891.0
195.27
24.66
12.00
35.73
760M
Llama
834.1
205.50
22.69
10.40
37.68
Mamba
793.2
182.74
22.58
10.47
36.25
XLSTM[]10]
840.4
179.74
21.66
10.11
35.33
XLSTM[7:1]
839.7
180.19
21.78
10.22
34.89
RWKV-4
1515.2
174.87
23.51
11.34
33.18
Llama
1420.4
192,52
20.67
9.67
34.84
Mamba
1475.3
171.38
20.37
32.01
xLSTM[1:0]
1422.6
166.16
19.94
9.64
31.90
XLSTM[7:1]
1420.1
171.36
20.28
9.64
32.17
43

---

E Detailed Results on PALOMA Language Model Evaluation
We report the perplexity values on each of the 571 subdomains of PALOMA in Table 10. Note that the aggregated perplexity values in Table 8 are not macro averages of the values shown in Table 10.
Table 10: PPL Evaluations: For the 1.3B sized models trained on 300B SlimPajama tokens, these are the detailed evaluation results the respective validation datasets.
Dataset
Llama
Mamba
RWKV-4
XLSTM[7:1| XLSTM]1:0)
#Params (M)
1420
1475
1515
1420
1423
9.58
9.72
11.37
9.53
9.55
4chan_ schan_metta_sep__allO0OO0. se__r_ssee_all000000011111
9.95
10.06
11.57
9.91
9.88
4chan meta sep__al-00000000
9.42
9.53
11.00
9.40
9.38
4chan_ @esin_meta_sepe http:mmetaeee_eaaaa___e
9.78
9.93
11.48
9.77
9.77
16.22
15.75
17.10
15.67
15.43
c4
12.72
12.33
13.33
12.30
12.12
04 c4_100oom__val_
8.66
8.54
9.31
8.42
8.33
c4_ c4_100dom__val_222 http://wwww......cccccoorrrrnee.
12.09
11.60
12.49
11.55
11.37
c4_ http://wwww_alllewwooooooooooooo
9.59
9.21
9.83
9.16
9.08
c4 c4_100dom__val_ val_____________________ htwpaaaa__wwwwwaaaaorrrroooom
10.97
10.61
11.31
10.50
10.40
c4_ c4_100dom_val/ hww.://ddwwwwwallmmmmlllll.l
12.42
11.97
12.87
11.85
11.69
c4 144_100dommvaa www.:/6wwwwwww..o...oo nwm.mihh alm.nih.gov gov
7.39
7.31
7.98
7.11
c4 10_10000om__000 val 17_ www.wwwwwwesscooo... co.uk
11.57
11.04
11.84
10.99
10.79
c4 11100000000000 val 18 en.m. wi.ipeddieei
9.28
8.95
9.52
8.89
c4 10_10000ommp00 val_ 19 www.cnet.com
12.61
12.23
13.12
12.09
11.97
c4 34_100dom_val http:aoon_wwllLwwww.ttttntnnsss. com
13.13
12.66
12.68
12.44
c4 1000o000mmmmmmmm val 20 www.telegraph.co co.uk
13.71
13.10
14.28
13.06
12.88
04 c4_100dom__p val_ htl_wwwwtheallatticooom
14.70
14.17
15.54
14.17
13.97
04 c4_100dom__ve _aap:om__aal______ laaaaaal_aaaaaaaaa srr.formmmmmrrmmmrss..
17.77
17.34
19.15
17.22
16.95
c4 c4_110dom_vve ____i__________________ ______________________ www.oreilly.com
13.36
12.99
14.31
13.02
12.88
c4 10110000mmmyi val ___4________ www.washingtonpost.com
12.06
11.58
12.98
11.64
11.41
c4 1000o000mmmmm val 25 www.zdneL.com
13.22
12.86
13.80
12.78
12.61
c4 1100000000mmmmm val 26 www.foxbusiness.com
9.32
9.03
9.58
8.92
8.81
c4 14_100dom__ee val____l_______ _____2__w__ 37_www.reuters.com
10.67
10.13
11.16
10.13
9.97
c4 10100000mmval val__8________ www.wbrimes.co.ooomm
11.36
11.01
11.71
10.89
10.76
c4 100do00 Lacadomm_vll_00 val__9________aeea www.rt.com
13.59
12.96
14.24
12.98
12.74
c4 100dom val 2_en.wikipedia.or
10.75
10.45
11.32
10.32
10.19
c4 100dom 30. www.prweb.com
11.18
10.88
11.92
10.83
10.65
o4_ 14__10domm__ae va__31________ www.weeiiataart.comm
21.78
21.05
22.78
21.00
20.69
c4_ [4__100dom_val_ Baat:om_vaall__wveee _3_____wwwwwcccmmmmmmmmmm
11.49
11.00
11.92
10.90
10.76
c4_ 1__1100dom_vlll 101i:o__val333___3 http:_a____wwwweeoooom
www.bbc.com
9.35
8.91
9,41
8.80
8.70
c4 10.10000mmm000 1a1t:omm_al__3_ggeee ________gtthhcoom
11.57
11.49
12,94
11.40
11.28
c4 110000000000mm val nypost.com
14.31
13.41
15.29
13.62
13.31
c4 10011000omm000 htt.vvav__ttttess_rree
16.49
15.88
17.15
15.98
15.69
c4 1100000000000m Battttom_wal33_wwwwwinrrrrreeee www.instructaales.com
16.75
16.33
17.73
16.28
c4 100dom val 38 www.youtube.com
8.42
8.24
8.83
8.22
8.07
c4 100koomvallll.meemm
8.84
8.49
8.83
8.41
8.32
c4 100domm v0___0_______ www.elsy.com
11.93
11.66
12.66
11.52
11.43
c4 100dom val 41 www.marketwirel.com
7.66
7.47
7.88
7.33
7.27
c4 htta:oom_vaa_42_ssssseooooo
14.23
13.81
14.91
13.68
13.51
c4
11.57
11.16
11.96
11.09
10.95
c4
13.56
12.94
13.57
12.77
12.64
c4 100dom www_:va____wwww.oor
10.59
10.30
11.14
10.19
10.12
c4 100dom val 46 www.fool.com
11.03
10.63
11.35
10.56
10.42
44

---

Dataset
Llama
Mamba
RWKV4
XLSTM[7:1]
XLSTM[[00]
c4 1o__10Moonnva_A
15.80
15.26
16.26
15.10
14.93
c4 100dom_vall caapnm__ll______1111 www.bbe.co.uk
12.55
12.10
13.02
12.00
11.85
100dom
lists.w3.org
18.75
18.24
19.89
18.05
17.84
c4 11_d0000mmmaaa val_4 www.:oawwwww_ww.laaateeeee.mmmm www.latimes.com
11.88
11.46
12.40
11.39
11.24
o4_ o4_100dom_vvl__50
12.44
11.95
12.85
11.90
11.76
o4 1tt.0000wmm.all__sdddddaarrrrooo
11.99
11.29
11.98
11.16
11.00
c4 =4_1100oom_vaa 32t____2_wwwccccccommmm
10.65
10.32
10,99
10.24
10.10
c4 [0110000omvyye val answers.sap.com
23.59
23.09
25.71
22.99
22.55
c4 110000000mm0mm val 54 homestars.com
14.13
13.70
14.51
13.65
13.52
c4 10111000mmmpp0 val__s______ 1aa________________
www.hinnuustatiness.omm
12.13
11.60
12.74
11.60
11.37
c4 11000000mmmmmm val_ val__6_______________ www.referencee.oom
11.57
11.04
11.75
10.92
10.79
c4 1000000mmmmmmmmm val 57 www.city-data.com
18.38
17.94
19.61
17.73
17.62
04 14_1000om__vv Betvodom_val_S50 val__a_l______ 118__mdduumcom
15.50
15.09
16.58
15.18
15.01
04 1__100domm_aal 1ati_____ppppiirinnddiaaraaa
9.74
9.10
9.68
8.88
8.75
04 100domm0vva www.alssswwwttgguaiiinnnmmmm
14.78
14.09
15.47
14.08
13.86
c4 100do00mmmyi val_
val__00l_____ ___ www.csmonitor.com
15.35
14.85
15.92
14.75
14.57
c4 100dom val 61 www.adweek.com
14.55
13.95
15.58
14.09
13.81
c4 100d0000mvvi val
vap__all_____2_ ______22_____ www..
7.69
7.79
8.86
7.68
7.58
c4 10000000mmm. val 63 www.yahoo.com
9.29
8.88
9.71
8.89
8.77
c4 100dom val 64 www.thesun.co.uk
12.18
11.66
12.74
11.59
11.39
c4 101100000mmvaa val 65 www.nydailynews.com
12.15
11.60
12.61
11.56
11.36
c4 11000000mmmmm val www.dailystar.co.o
10.65
10.17
11.03
10.09
9.92
c4 1000o0 67 fineartamerica.com
12.06
11.58
12.29
11.46
11,36
o4_ o4_1000om_val vaa_avll_oooo www.kickstarter.com
13.85
13.58
15.38
13.55
13.38
c4_ e4_100dom_val 2al_ono______________ wwt_______rruuuts.oooomo
9.54
9.13
9,90
9.07
8.92
c4_ 10.1000dmmmvve ____66___________ www.hufffost.com
13.45
13.03
13.96
12.99
12.83
c4_ 10.11000mmmp000 aa___a____aaaaa www.insiderpages.com
13.24
12.84
13.55
12.77
12.64
c4 11110000mmm000 val 71 www.inquisirr.omm
12.12
11.58
12.86
11.71
11.38
c4 11110000mmvvai viv_vall_222 /tt.___iitsddeii..org
18.18
17.81
19.62
17.67
17.30
c4_ wtw..aawww.www.sssttttiisissoom
c4_ 10_110000mvvlla
11.51
11.06
11.91
10.94
10.79
10.29
10.60
9.82
c4_ (00dom_vvll_770 http:///__siimeeeiiippcpiaaoooe
8.25
7.85
8.37
7.78
7.67
c4 100dom0mvvll v6___6_ll_____ le___ddddinecooo
14.75
13.83
15.48
13.92
13.51
c4 100000000mmmm val _7_ www.wwndroideeaaliies.ooo
11.11
10.74
11.43
10.72
10.59
<4 o4_100dom_vall _aa_an___aa_________ 7t..7/_wwwwwirrecom
14.42
13.88
15.14
13.87
13.68
c4 111000000mmvva val_ _a___9 www.bustle.com
12.79
12.33
13.19
12.25
12.09
c4 110000000mmmmmm val_ 1_11______________ srreentsggogle.com
7.59
7.84
9.33
7.72
7.59
c4 c4_100dom_wve val________ 8000-000pmm_....1111111 wpmudev.org
16.86
16.63
18.13
16.50
16.29
<4 110000000mmmmmm val 81 www.librartthing.com
14.36
13.98
15.42
13.91
13.75
c4 e4_100dom__v val 82 mail-aalaaae wat-aaarhhiiepppae
5.67
5.61
6.17
5,56
5,49
c4 sattoom___aa_____shhhoaar_
8.72
8.43
9,03
8.32
8.21
c4 100000oo__va_8___www_aaa__
16.64
15.97
16.99
16.00
15.83
c4 10110000mmvaaa val 885aa_______________ w5t__wwwwwccwwlllccmm
12.34
11.95
12.95
11.90
11.72
c4 11000000mmveee 86
www.shutterstock.com
8.70
8.89
10.75
8.62
8.52
100dom val
14.59
14.24
15.32
14.18
13.98
04_1000om_val _a_______________ wt__wwweeeetbrrteeecmmmm
14.47
13.99
14.89
13.98
13.79
c4_ _4_100dom_val__ 89 http:oonwwau_l_owwssssstttoopppa www.ffastcompaay.ccomm
14.24
13.75
15.52
13.82
13.56
c4 [__1000dmmvveee _aal_____________ 8 www.husinessinsider.com
10.97
10.69
11.35
10.52
10.46
c4 10110000mmm00mm val 90_ www.fistpost..com
11.71
11.24
12.08
11.12
10.96
c4 100dom val 91 www.entrepreneeur.com
13.10
12.68
13.65
12.72
12.54
c4 100dom_va _aa__a_ll___ _0s__2____ea_ www.breitbart.com
13.47
12.67
14.29
12.84
12.56
c4 100dom val 93 techerunchcoom
14.20
13.68
15.18
13.82
13.58
45

---

Dataset
Llama
Mamba
RWKV4
XLSTM[7:1]
XLSTM[[:0]
c4 c4_ 1__10d0dmmvaa www.ndtv.com
14.12
13.28
15.06
13,43
13.12
04 10_d000dmmaaa
10.66
10.26
10.90
10.10
10.00
100dom _allii_vl________aa finance.yahoo.com
9.96
9.55
10.22
9.43
9.34
c4 10000000mmmii 10____7_aa_ wwcchies.lib.state.maa
6.53
6.12
7.09
6.27
5.85
04 [4__00domm_val val______________ www..ww.gsmmrrna.ccommmm
23.21
22.15
24.52
22.10
21.76
c4 110000000m_pmm val 99 www.woonelyplanee.comm
11.33
10.92
12.28
10.84
10.69
c4 100d0m00mmmmo val_ www.forbes.com
13.72
13.31
14.63
13.34
13.13
c4 ci_en_val-000000000
14.34
13.70
13.67
13.46
<4 cd_en_wal-00000000
14.86
14.28
15.51
14.21
14.09
c4 c4_ce__wal-00000000
15.29
14.71
15.95
14.71
14.51
c4 e4_em__o val_en__oll00000003
12.95
12.28
13.32
12.23
12.06
c4 em val.000000
12.56
12.13
13.27
12.05
11.87
c4 c4_en_vall-0000000
12.77
12.35
13.26
12.32
12.18
dolma-v1_5 1____1_______1 val_ books
13.00
12.44
13.64
12.44
12.27
dolma-v1 val. common-crawl
16.86
16.37
18.00
16.35
16.10
9.42
9.56
11.25
9.41
9.29
dolma-v1_5_ 2e_________dd__unnnneaa_a
23.04
21.97
23.84
22.05
21.80
dolma-v1_5_ 3_va____vaaai sall_stack_umpe _ttcckunnfoommm
2.30
2.33
2.53
2.30
2.29
dolma-v1_5_ 5_val__iik wiki
10.86
10.48
11.25
10.41
10.31
dolma 100 proglang val. 00 text
5.61
6.30
6,94
5.67
5.69
dolma_ Baciia_O00__PPPPP0 2l0_pprgllag__ve val__1 markdown
3.16
3.16
3.56
3.15
3.11
dolma_1000_prt 1ro_ooo0__ooan___aaaaaa _aa_alnnn__l___eeeeee______
1.84
1.91
2.23
1.86
1.85
1.75
1.75
1.83
1.73
1.72
dolma_ 100mm__000_proglang___ La__e_aal______aa______
-4_jaaa
1.96
1.99
2.18
1.95
dolma__100_proglang_vaall
2.19
2.25
2.53
2.21
2.19
dolma__10__proglanggaaLLLLLL _6p_thoon
2.35
2.39
2.62
2.36
2.34
Mo0ma__100_proglagg_.a ja____ssc_ittppttt
2.54
2.59
2.83
2.53
2.53
dolma__100_proolaagg_aaall _hmll
1.92
1.94
2.13
1.91
1.91
dolma_ 1admma_10__rogllngg_ao __________l__
2.23
2.28
2.45
2.19
2.24
11000000glagg_yyeeeee val__ 10_yall
2.93
3.01
3.71
2.94
2.92
dolma 100 proglang val 11, go
1.75
1.78
1.97
1.77
1.75
12__tpescciipt
2.17
2.20
2.41
2.18
2.16
Baamma_100_progaaa0 val_ 33ml
2.44
2.50
2.78
2.46
2.48
dolma 100 111111000000000
proglang val. 14 css
2.25
2.25
2.34
2.21
2.20
dolma @alla__100_ppogaatte val__ _aal___________ _5_jupyter-nb
1.57
1.60
1.75
1.58
1.58
dolma__100_prroglagg aaaa _6__usttttt___
1.96
2.01
2.23
1.97
1.96
dolma__10__proglangg_aaLLLL 1asa:ll1t1__tttddassas_aaa
4.01
4.17
4.56
4.10
4.05
dolma__00__proglagg_vaaaa Laaill11__gttttttalllllee
2.84
2.87
3.53
2.86
2.83
dolma__1W__pooglaag_vaaa_______
2.41
2.44
2.70
2.39
2.38
dolma_l10__poogaagg_aallaLL
1.95
1.95
2.10
1.94
1.93
dolma_l00_proglang__v ____1__qlll___
2.18
2.23
2.46
2.17
2.16
dolma p00_lalllopprpggg....om val 22 swift
1.86
1.88
2.04
1.86
dolma dolma_1000_ppp 1ae111000ppppgge val 23 kotlin
2.05
2.07
2.29
2.07
2.04
dolma_100 Baiia__100_prolllag val 2el_224_s00 scala
2.24
2.64
2.25
2.23
dolma 1000aa00000_001100 B0W0opprglang..mmm val 25 sess
2.26
2.27
2.38
2.24
2.24
dolma_1100_ppl
11000_rrgllagge val. 26 tex
4.04
4.21
4.97
4.10
4.04
dolma__10_prrglanng _al__a______ 27_dart
1.79
1.82
2.01
1.80
1.78
dolmm__100_proglanggao _al__s_a__________ kicad
_al_2__kicad
2.57
2.79
3,86
2.68
2.67
Larroollagg_a_____0 val__alll____ ______ellll
3.71
3.74
4.31
3.69
3.63
dolma_1000 10000_prgllan_.... val. ___33_smaall smali
1.38
1.39
1.45
1.38
1.37
dolma 100 proglang val. 31 lua
5.65
6.01
7.18
5.33
5.45
dolma 100__0000000 2000_ppgglan__.mm val. 32, restructuredtext
4.01
4.05
4.66
3.97
3.92
46

---

Dataset
Llama
Mamba
RWKV4
XLSTM[7:1] XLSTM[[:0]
dolma_100
2.57
2.62
3,01
2.59
2.55
Baama___00_proolagg ______iiff
2.87
2.95
3,43
2.89
2.86
val_
3.91
4.16
4,90
4.05
3.98
val_36__jsx
1.83
1.84
1.95
1.83
1.82
dolma__10W_proggagg__aaaa
vat 3___haskellll
2.94
3.07
3.73
3.02
2.95
dolma__1W_proggaagg_aaaLLLL _al_______________ ga__gugpl
2.65
2.88
3.36
2.81
2.77
dolma_l00_proglagg__a___333 ____ppossssrppttttt
19.09
19.52
19.56
18.66
18.64
dolma__10_prrogaaggyyo val 40_groff
6.13
6.32
7.45
6,22
6.21
dolma__10__rroglagg_valll
2.35
2.45
3.17
2.39
2.35
dolma_ dolma__100_proglanggyYY val____________________ fortran
2.32
2.39
2.83
2.35
2.31
dolma__100_proglangge 23_makefile
2.93
3.01
3.51
2.86
2.82
dolma_l10_progglngg_.. |
10.34
11.34
13.24
10.49
10.71
dolma_ Bo0ma_100_prrolange val__________ _l__________________ ppsccal
4.18
4.81
5.49
4.17
4.27
dolma dolma_100_ppaae_ 110_o000000g__...11 val 46 common-lisp
2.56
2.71
3.32
2.62
2.58
dolma_100__prolaaggg Lappggg__________ _____as
2.49
2.73
3.59
2.57
2.53
Jolma_1100pprogange val_ _0____8___ 38_ddll
3.91
4.06
4.69
3.92
3.90
dolma 100 11111100P000000 proglang val 49 _9_julia
3.25
3.36
4.05
3.30
3.26
dolma
@00ma_1000_pp 100 100_llagg val. 111__50eecc edn
1.99
2.10
2.67
2.04
2.03
@00ma_l00__roglangg_ val___aa_________1__11____111111 51_visual-basie
2.42
2.49
2.72
2.37
2.38
10rio_ol0o_prollag___aalllllll 52_powwrshhell
4.08
4.16
4.50
3.86
3.89
dolma_100 1irpaa_000orrggann_oaa val__al______ _3_g-c
2.26
2.66
3,29
2.44
2.37
dolma 1alma_1000___ B000_pogglang_VV val. $4_ocaml
3.06
3.29
4,22
3.19
3.13
dolma 100 proglang 55 java-server-p
2.10
2.11
2,31
2.06
2.09
Bacpaa_000ppooooooo val__a____ _______sooo 56_solidity
4.09
4.41
5.28
4.05
4.10
1ir___olooorrr_________ _al_gg_________________ 57_grrapvizz-ott
2.17
2.48
3.54
2.32
2.29
dolma 100 proglaagg val. 58 less
2.24
2.26
2.33
2.22
2.22
dolma 1atla__100_aa_00 11111100ppppp 100_prollang_vv val__a__l__vv___
1.81
1.81
1.91
1,80
1.79
dolma 100 proglang val 60 asciidoc
5.33
5.50
6.84
5.43
5.34
dolma dolma_1000____ hwo_pproglang_t.o val 61 groovy
2.12
2.15
2.41
2.13
2.11
@o0ma_l10_prrolaatt val_:lang__ll______________2 Jlym
2.26
2.40
3.25
2.31
2.23
dolma__10__proglaag_aaLLLL
2.52
2.56
2.96
2.52
2.48
dolma_100 1aaiaa_1000rrglaag__alaaall 64__htl++ebb
2.10
2.09
2.23
2.08
2.07
dolma_100 1100_proglang_va val___al______ 65 erlang
2.84
2.98
3.87
2.88
2.85
dolma 100 h000_proglan...m val 66 elixir
2.93
2.99
3.58
2.91
2.90
_e_agll
5.35
6.90
10.75
5.64
5.76
dolma_l00__prggaag__Hs "_adduinooo
3.37
3.40
3,81
3.28
3.28
dolma 1aamaa000000000
po0_lorgglagg...mm val. 69, coffeescript
2.80
2.85
3.27
2.80
2.77
1aa_a__1000ppglla___aaaaaal val__0_tooll
7.76
7.62
8.44
7.53
7.58
dolma__100_proglangg_aa____
2.15
2.21
2.56
2.19
2.16
Nolma__100_proglanggaaall _x_________________x
7.80
7.84
9,03
7.88
7.83
@olma_100_proopee val_ __3_smalltalk
9,32
9.61
12.60
9.47
9.20
dolma 100 proglang 74 emake
1.87
1.86
2.02
1.84
1.81
@alima_100pprollaa val_ "5_actionscript
2.45
2.54
2.88
2.46
2.46
holma__00__roolangg.v _at___lll_____e__ _6__lsl
2.40
2.42
2.72
2,36
2.32
dolma 100 proglporglangg..mo val. 77 systemverilog
2.53
2.66
3.17
2.58
2.55
dolma Bailna_100_prglaagg val. haxe
2.74
3.20
2.77
2.76
dddma__t00_proglagg__w _a___9_fl___l
2.89
3.02
3.53
2.93
2.88
Baap:igg_________________0_ 80_max
1.59
1.62
1.80
1.61
1.61
dolma_
1aaal1000prrggggg_.o val _objective-c+++
2.18
2.19
2.40
2.17
2.16
dolma 1000a_0000__00 pwoglorg.gg......... val 82, standard-ml
3.57
4.05
4.79
3.81
3.77
dolma 100 Boll__roglangg._ val 83 dockerfile
4.08
4.17
4.37
4.01
4.05
dolma_100_mpe 1aap110000orogggggg_o val 84, se_emmsss-isp
3.83
3.83
4.44
3.80
3.72
47

---

Dataset
Llama
Mamba
RWKV4
XLSTM[7:1] XLSTM[[10]
dolma_100
1a0ma__100_roglang__aa scheme
2.78
2.86
3,40
2.84
2.77
dolma_lt0_prrolangg__ 66_clojure
3.18
3.30
4.00
3.26
3.17
dolma_l0W_proglang__aaaa Baapiig________h__a__aaaa _h_adddddabnrarrs
1.79
1.79
1.88
1.78
1.78
dolma_l00_proglag___aa _a__arasrrtttyyyyyy
2.30
2.35
2.58
2.29
2.30
dolma__000_prolang__va 89_logos
2.37
2.38
2.98
2.46
2.44
dolma_100) porpgllll..prpggg....om val 90 stata
4.67
5.08
6.85
4.85
4.81
dolma @elma__10_pprogiaaae val _acc
2.42
2.48
2.87
2.44
2.43
dolma_1000 dolma__100_proolagg_YY Bat_t___________________ 92_nimrod
2.75
3.63
2.81
2.77
dolma_l10_pproglagg_.w _at_____ccllllllllee
3.00
3.16
3.95
3.07
3.02
dolma Lolma_|10_prrglannee _____mmll
5.56
5.76
7.21
5.59
5.55
dolma_ 100 haa___1000prpggggooom val_ 1________01
1.79
1.79
1.90
1.77
1.77
dolma 100 B0000porggangg..mm val. 96 protocol-buffer
1.32
1.31
1.38
1.31
1.32
dolma_100_prrolangee _7_r
2.80
2.92
3,66
2.86
2.81
Bolma_110_proglhiai val 98 _8_cythoo
2.34
2.39
2.69
2.36
2.35
vi mediawiki
2.01
2.10
2,48
2.12
2.04
20.25
19.29
20.38
19.28
19.14
22.08
20.70
22.07
20.83
20.61
dolma_ Latiaa_100_uubbedditttsoee 2atpaaittt______________1 A
22.49
21.30
22.89
21.60
21.27
dolma 1aamaa_1000_000 1000_subreddiis__o 03 03_worlddewws
22.57
21,43
22.77
21.50
21.23
dolma subreddits val relationships
18.64
17.80
18.89
17.86
17.67
dolma Laiiaa_100_suuea 1000_sbreediiss__. val_as_______ Saal__o__eelll
relationship_aad
19.40
18.53
19.68
18.63
18.46
@alma_ll00_surrddditt
22.49
21.25
22.51
21.49
21.17
hooma___0___utrccdddds_________0
34.45
32.41
35.13
32.46
32.04
dolma_110_suhreddii ottpr/ddii___aaloooooooooolleeee
22.53
21.30
22.68
21.28
21.10
dolma__100_subrrdddisssaaaa val_oooooooottooooooooooooo
20.20
19.16
20.25
19.20
19.02
18.62
17.65
18.82
17.73
17.64
_changemyviww
20.02
19.10
20.50
19.17
18.99
dolma [000_subreddits__ val_ 12, unpopularopinic
23.39
22.16
23.63
22.32
22.04
dolma 1a0iaa_000_0aa000 [00__surreddits val movies
21.62
20.52
21.79
20.64
20.35
dolmm__l0__sbbrrdddssss__LLL_ Games
22.26
21.15
22.52
21.18
20.87
dolmm__100_surrcdddssssuaal
23.28
21.93
23.60
22.10
21.85
dolma__10__suhrcdddiss_LL__
21.84
20.56
21.82
20.64
20.47
24.45
23.13
24.61
23.15
22.86
dolma 1111100_uuuur subreddits val soccer
23.38
22.12
23.61
22.19
22.03
Lallss____________ val_19__nf
19,86
18.76
20.17
18.81
18.62
haiima__00__ubbredddiss_______ 2e____20ccxpaaiiiiiiiivy
18.35
17.21
18.59
17.32
17.03
dolma__10__suhrcddissssuuuu
23.86
22.53
24.09
22.67
22.54
va_____a_______mmm_m atheism
21.23
20.18
21.43
20.23
20.13
dolma__1W__uurredddis_______ 22kksmne
20.00
19.04
20.11
19.10
18.94
dolma_11M_suhrrdddiss____._ _ideos
22.26
21.24
22.51
21.29
21.04
dolma__00__surredditssss 2at_brrrddddds________e_e
21.13
20.13
21.30
20.09
19.98
dolma__10__surredddisssss 2aaporddddi_______r_rrrreeee ro_drssedbbnaacissiii
22.07
21.08
22.48
21.20
21.02
dolma 100aaa010___saiii 100_a_0100o_uusittts.so.....oo val NoStupidQuesti
19.66
18.59
19.87
18.68
18.52
dolma__100_subrrddinssaaaaa 1ali222_DiinnheGaaaaa
35.27
33.58
36.13
33.78
33.37
ddlma__100_surrcdddssssaaal _nn_nime
23.21
22.04
23.46
22.12
21.77
1olma__10__shrccddissssg _nn_
28.22
26.71
28.78
26.72
26.39
dolma haapa_1000_uuudiissssee val _1__kpolisics
22.35
21.19
22.80
21.31
21.10
dolma 100 subreddits val 32 funny
20.78
19.45
20.70
19.40
19.23
val_33 10__33_eeroppp
21.76
20.59
22.10
20.72
20.52
22.44
21.21
22.44
21.30
21.09
dolma_100__a
17.88
17.02
18.10
17.04
16.94
dolma 100 swwreddddddddssssooooo 366 36_SquaredCrrele
25.87
24.31
25.83
24.34
24.03
48

---

Dataset
Llama
Mamba
RWKV4
XLSTM[7:1] XLSTM[[:0]
Laaiaa_000Ssuureee www_aa/aerddddissaaaaaaaaaaaaa vaaasss___a_3__AWAa
Lo Women
17.72
16.81
17.77
16.85
16.72
val___tt___________________ _____3__eeallavicc
18.66
17.75
18.92
17.74
17.64
dolma 1000_subreddits_. val 39_JSSTNOIIL
24.25
23.16
24.86
23.32
23.02
23.39
22.09
23.52
22.21
21.95
@alma_100_subreee val__1_11_AAAA __AmA
19.83
18.83
19.86
18.71
18.56
dolma @alma_110_suubllaa 11000_sbreddiiss_2 val 42.
31.26
29.25
31.44
29.39
28.82
dolma Batiaa__00Suuudddiiiii val 43 23__eninn
20.15
19.11
20.43
19.30
19.06
dolma__10__uubrcdddiss..att 44 ______om_ooo_o
23.12
21.90
23.44
21.99
21.84
Nolna__100_saarrcdddtsssvva h0t0://aaaaaaaaaaaas44444AA/aaaa AdviccAnimals
22.14
20.96
22.14
20.98
20.79
dolma_ 100ma__00_ssulaa dodma__1W__surredddsssaaaal ohhirooiissaaaaaoaoooccccceeeeee childfree
21.87
20.85
22.13
20.89
20.72
ddlma__10__surrcdddssssstv 477nniteddiiigdoo
23.27
22.00
23.40
22.00
21.85
dolma 100 1100__ubreddis_veee val 48 ffxiv
32.53
30.79
33.33
31.01
30.62
Nolmm__10__surrcdddsssaaal ___dnexxttt
29.67
28.03
30.53
28.26
27.63
dolma__10__surrcdddss_s____.
20.75
19.83
21.14
19.95
19.78
Bolmm__10__shrrdddissssss
19.36
18.39
19.49
18.52
18.33
1aa_a1000_subrrddisssaalaall _a_____ssoiiff
25.28
23.99
25.63
23.94
23.69
dolma 100 subreddits val BabyBumps
20.96
19.82
21.11
19.92
19.76
-iierddddis_v_____AAdiiiieee __l_ic___vvicee
19.17
18.29
19.35
18.38
18.19
dolma 100maa_1000ma0 10111100_S_raaaa 10000_uubeddits_sg val __5_ustralia
23.97
22.51
24.06
22.61
22.40
dolma 100 subreddits val 56 CFB
20.45
19.41
20.92
19.49
19.23
dolma Nolma_100_ssae hat_11000Suuuddisseee val_7, 377offfmcchest
19.63
18.79
19.77
18.93
18.77
25.96
24.49
26.02
24.65
24.39
21.53
20.63
21.70
20.73
20.54
20.44
19.46
20.64
19.51
19.38
@alma_l100ssubrddiii va_g__________mmmemGC
28.82
26.79
28.94
26.69
26.38
dolma 11e111000000diii val_ 62.66.2222 asktransgender
20.72
19.86
21.07
19.83
19.62
@alma_l00_subreddii _al_it___________ _a____ll_l__o___________ DotA2
34.35
32.38
34.74
32.57
32.16
dolma 100 s00_uuudeddits val 64 neoliberal
21.74
20.59
22.26
20.64
20.45
dolma dolma_1000sslll 1000_ubreddits__goe val whowouldwin
29.18
27.81
30.08
27.63
27.30
1olmma_100_surrrddissssooo _ea_:ddddiii____________
18.28
17.52
18.31
17.50
17.41
1odma__10__subrrdddisss__..
22.30
21.18
22.38
21.17
20.99
Baiima_l00ssureeddiit val__vll__aa pathofexile
40.48
38.59
41.43
38.75
38.43
dolma 100 (000_subreddits.. val ______________ "o_PoliticalDiscusss
20.01
18.92
20.16
18.97
18.82
dolma 100 sWW_esubesd val 70 Libertarian
22.97
21.77
23.15
21.87
21.75
24.94
23.66
25.44
23.85
23.55
21.57
20.35
21.48
20.34
20.11
21.12
20.02
21.31
20.09
19.82
dolma_100 haaia_t00_suurrddds__sseeee _ali____l________ 74_dogs
20.13
19.12
20.32
19.20
18.92
dolma 11111000_uaaaaa subreddits val. 75 pemasterrace
23.73
22.49
24.02
22.56
22.21
Colmaa_10_ssubrdddee val__4 val_76______ teenagers
18.37
16.35
16.44
15.56
17.02
dolma__100ssubrdddiiiii val_ 20__?_ssssorriiiinggggg
21.08
20.02
21.19
20.17
19.98
dolmaa_10__suhrcddisssssss _aa_errrddddii_______________ 2_v_eratttthhhhhhh
30.47
28.77
31.13
29.13
28.57
_al_ass____________ _9_teeeivvii
23.97
22.63
24.05
22.75
22,49
dolma 1we11000_uuediitte val buildape
21.55
20.22
21.78
20.29
19,98
dolma 100 1100_subredditt val askscience
17.25
16.39
17.52
16.34
16.11
dolma__10_subrrddiii val htapreddiii__a__________reee 82_programming
23.66
22.61
24.04
22,55
22.24
dolma__1W__surreddissswe ______iiddaaaa2222222
32.98
31.17
33.58
31.39
30.91
@alma_100_surreddiss 84_cars
22.57
21.41
22.73
21.38
21.15
dolma 100__sbrreddiss val 1f_fouullll
23.85
22.65
24.09
22.71
22.49
dolma 100 subreddits val 86 sysadmin
24.23
22.90
24.41
22.96
22.64
dolma 1aina_100_saa0 WWW__subreditsswee val 87_ 17_hockey
21.46
20.26
21.74
20.37
20.20
dolma 100 TWW__sureddiiss.o val 88 india
24.15
22.92
24.42
23.08
22.68
49

---

Dataset
Llama
Mamba
RWKV4
XLSTM[7:1]
XLSTM[[:0]
dolma_ h
19.14
18.26
19.63
18.29
18.12
dolma_ Baaiaaa000ssaaaaaaee 1000ss0redddisss... wat____000000AAAaemm11
DMAcademy
27.77
26.31
28,38
26.41
26,00
dolma 100aa__1000000000000 1110110000_paaa subreddits val dating advice
20.18
19.27
20.42
19.40
19.21
dolma Baaiaa__00ssuureddie subreddirsdss.... val 92 Catholicism
19.11
18.22
19.41
18.17
18.03
dolma_100_subreddiit va___val______aaaa_ 93__r
24.50
23.29
24.74
23.32
23.12
holma_l00_surreddiis__ 10l_______res________
23.56
22.38
23.83
22,41
22.25
dolma_ htt_aaooooo_ussddissoooooooooo val 95 boardgames
22.69
21.48
23.13
21.61
21.38
dolma__1W__surrcddddsssaaal __
22.79
21.53
22.97
21.68
21.53
dolma_llW__uurrddddssssaaal 37__uturolggy
23.55
22.36
23.77
22.37
22.17
21.07
19.89
21.22
20.08
19.83
_aa_ll_________________ weddingplannin,
20.11
19.01
20.33
19.19
18.96
15.92
15.46
17.14
15.37
15.22
18.49
17.91
19.89
17.90
17.71
18.45
17.90
19.69
17.91
17.68
16.75
16.23
17.92
16.16
15.89
16.26
15.66
17.32
15.73
15.41
15.41
14.96
16.56
14.92
14.74
gab val-00000000
33.19
30.55
31.57
30.73
30.32
gab vall0000000000
35.64
32.76
33.96
32.80
32.63
gab__all-Oon wabbvaa-0000000022222
34.38
31.68
32.75
31.80
31.65
gab_aall-oon valvoll00000000
34,86
32.05
33.26
32.20
32,00
gab vall0000000000
36.20
33.35
34.58
33.42
33.23
gab val-00000005
33.46
30.82
31.88
31.06
30.72
gab val_vo00000000066
35.76
32.77
34.26
33.04
32.74
gab valo0000000
35.54
32.60
33.76
32.78
32.41
gab val-00000008
35.11
32.03
33.23
32.25
31.86
gab vall-lll00000009
34.13
31.34
32.36
31.50
31.30
[2orc_s2orr___ val Art
20.07
19.80
21.88
19.78
19.44
m2d2 val Philosophy
14.80
14.82
16.77
14.69
14.47
11.70
11.70
13.18
11.52
11.33
m220__s2or__mmpiii__aaeeetot
11.47
11.49
12.90
11.37
11.15
m2d2_s2orc_nt
12.76
12.73
14.28
12.60
12.45
_2orc_unsplit__al vaci_vll_aaaa val___ttototth.GGA
11.70
11.70
13.18
11.52
11.33
m2d2
s_trc_uoosssss .uspppitttt...... val astro-ph.HIE
11.85
11.77
13.29
11.62
11.46
m2d2_s2orp 2oorc_ussplit__eee aal_i__v_a_ttrpph.l
15.36
15.33
17.16
15.21
14.92
13.08
13.08
14.89
12.86
12.70
m2d2
15.36
15.33
17.16
15.21
14.92
sunsplitvvl__atom-ph
12.74
12.84
14.44
12.75
12.53
m2d2 hidd__s2rc_umppli___a_tttt
13.20
13.29
15.22
13.14
12.97
m2d2_s2ore htt___sooc_umspli_____oooo
11.67
11.78
13.37
11.67
11.50
m2d2 m2d2_s2ore fidl_ssorc_unsplic cataglii__al_commmmaadiin-mmm
12.54
12.67
14.28
12.58
12.38
m2d2 s2orc_unsplit vall lalllalllloaaaaaaaa cond-mat.mes-hee
11.24
11.50
13.19
11.30
11.10
m2d2 s2orc_unspli,
cond-mat.mttl-ae
12.19
12.33
14.09
12.18
11.91
m2d2_s2orc_unspp sttpssorrunusptttaaalallla http:pi__aa_coodmmmttttoooee
11.87
11.96
13.55
11.83
m2d2_s2ore_umspp _tt_:mpppiival__cooddttttuuuu
11.67
11.78
13.37
11.67
m2d2_s2one_unsple saane__usppi__vvlcoooeee wtwnappliwwal_ommdmmatssfffffff soft
12.18
12.23
13.93
12.18
12.02
m2d2 m2d2_s2orc [e02__2orc__usppe wal__conddmaa-ttaae soond-matstat-meehh
12.03
12.14
13.60
12.08
11.89
m2d2 poorc_uspplit... wal.-codd-mattte...
10.39
10.50
11.98
10.41
10.22
m2d2_s2orc.
11.57
11.66
13.13
11.53
11.30
_oog___orc_usplit__. _vl__cond-maa__l
12.54
12.67
14.28
12.58
12.38
m2d2 s2orc_ unsplit val cs.AI
11.71
12.09
14.20
12.01
11.79
m2d2 s2ore. setrs_susssllie cosppunssptaat vasplit_val_cs.AM
13.09
13.36
15.30
13.18
12.99
50

---

Dataset
Llama
Mamba
RWKV4
XLSTM[7:1] XLSTM[[10]
8.45
8.81
10.46
8,70
8.54
m2d2 m222__s2rc_umsspit__all Staiplit_vva___ccCC
13.21
13.31
15.01
13.18
13.02
m202_s20nc_umppiit_vaaccc
8.39
8.68
10.12
8.59
8.47
m=d2__2on__umpiii_vl__.ss
14.66
14.75
16.96
14.70
14.47
m2d2_ medd__sonc_umsplit__alll
14.63
14.86
16.72
14.74
14.56
m2d2 wwtssssusoooittt___ooo_ CV.CV
12.68
12.78
14.38
12.66
12.49
m2d2
16.01
15.93
17.52
15.84
m2d2 m202_s2onc_umppitt____.ss
11.86
12.35
12.27
m2d2 ma2__22nc__mppii_____ccses
13.60
14.02
16.20
13.79
13.56
maa2__2onc_umppi__aaall
14.67
14.83
17.05
14.75
14.50
m2d2 m2dd_s2orc_mp___ stt_s___uussii__aallleo
8.11
8.38
9.84
8.27
8.14
m2d2 phoor_tspplit..mmm val cs
9.63
9.99
11.76
9.88
9.69
m2d2_ _oa__ssore_usspli__ee cs
14.80
14.95
17.00
14.89
14.67
m2d2_s2oree hua___2or_umpplii___a.css
9.51
9.84
11.64
9.74
9.57
16.51
16.43
18.18
16.38
16.21
_uu___socc_umpllt_yy Satpppit_val_csGGG
13.45
13.60
15.53
13.54
13.29
m2d2 s2orc_ val CT
9.25
9.59
11.34
9,49
9.29
m2d2 str___s2oo_____ -2ore_ussplit_ val HC
16.76
16.93
19.08
16.84
16.66
m2d2_s2ore _ts_s__ussspli__eeeee val_
13.30
13.46
15.26
13.31
13.21
m2d2 s2orc_unsss jonpppusspttttt val cs.LG
10.39
10.52
12.14
10.44
10.27
m2d2_s2ore _2or__unsplit val cO
9.75
10.23
12.50
10.03
9.81
m2d2_
11.24
11.65
14.10
11.41
11.19
m2d2 s2ore poorc_ussplittommm val cs.MM
13.12
13.40
15.29
13.25
13.03
13.98
14.14
16.27
14.11
13.89
_ed2_ssorc_unspltt_yo
10.53
10.80
12.52
10.71
10.47
m2d2 s2ore_ _2orc_unsplit.ooo cs.NE cs.NE
13.76
14.00
16.10
13.89
m2d2 aodd__2or__unspitt_vlll
10.00
10.22
11.61
10.04
m242_s2on__mmppii__vl_sss
15.24
15.43
17.62
15.34
15.10
m222_ssonc_umplii_uuul
14.61
14.93
17.35
14.80
14.53
22d2__2orc_umspli___
12.60
12.82
14.71
12.70
12.48
m2d2
PL
15.43
15.74
18.58
15.65
15.40
m2d2_s2ore
s2ore__nspli__val.co.
13.04
13.19
14.95
13.12
12.87
22d2_s2orc_umspltt_vee
11.10
11.42
13.33
11.30
11.10
m2d2 s2orc_unoaaeeee unsplit val cs.SD
13.27
13.42
15.26
13.36
13.13
m2d2_ [idd_s2orc__un -2orc_ussplit_ vasppittval_cc.SBB
17.72
13.47
15.46
13.40
13.21
m2d2_s2orc 22a2_s2on__usspit_vaa_coo
12.03
12.25
14.03
12.19
11.99
m2d2
11.40
11.79
13.51
11.63
11.39
fadd_ssorc_unsplt_vee
8.39
8.68
10.12
8.59
8.47
m2d2
baare_usplii_vee val_econ..MM
11.62
11.76
13.73
11.68
11.41
m2d2 jis__unspit__aa caaplit_val_eoe val_econ.TT
9.75
10.16
11.99
9.99
9.88
m2d2_
9.75
10.16
11.99
9.99
9.88
m2d2_ htt___s2or_unspi______cooo
12.05
12.14
13.88
12.09
11.88
m2d2_s2om aaad___oo__umspl___alll -vncplit_valceess.VV
13.77
13.89
15.71
13.76
13.54
m2d2 (002__2orc__upliit _a
11.29
11.45
12.94
11.28
11.13
m2d2 s2orc_
potpeuusspitt..m val eess 11
13.77
13.89
15.71
13.76
13.54
m2d2 (e___$2ooo__ppll val gr-qc
12.84
12.99
12.84
12.71
m2d2_s2orc_unsppl val_pit____hepeexxx
10.47
10.37
11.61
10.13
m2d2_ secbon_unsplit_vv__eeeee
13.13
13.10
14.57
13.02
m2d2_ 22422_2orccunspple vaciii__vl_heppppl
11.67
11.81
13.38
11.66
11.45
m2d2 s2orc_unspplll
citsplitvvl__eee val hep-hh
11.46
11.49
12.71
11.40
11.24
m2d2. shore_unsplit, vvl math.AC
7.08
7.37
8.71
7.26
7.13
m2d2 s2ore_ chorc_unsplit val math.AG
8.89
9.27
11.05
9.16
8.95
51

---

Dataset
Llama
Mamba
RWKV4
XLSTM[7:1] XLSTM[[:0]
9.35
9.53
10.90
9,41
9.35
m2d2_ _ad___s2or__usslit__o ____uppli__al_mmah.AAl AT
8.57
8.77
10.16
8.72
8.53
m2d2 [edd2_s2ore_iiiee _oop_unsplitt____c val math.CA
9.18
9.49
11.01
9.36
9.30
m2d2_s2oree s2ore_unsplittttttecee Sallpit___ll________ math.
6.99
7.33
8.69
7.21
7.08
m2d2 m2o2__22orcuusplitt.yy
9.78
10.20
12.04
10.12
9.91
m202__2one_umspii_____lmm
7.81
8.07
9.36
7.99
7.87
m2d2 m2d2__2ooc__mspii___alaaaa
7.96
8.18
9.50
8.08
7.98
m2d2 fidd__s2onc_usplit.. ma
7.88
8.12
9.61
8.08
7.96
m2d2 sltss_sussppppce val Math.FA
7.71
7.96
9.35
7.88
7.81
m2d2 m2d2_s2orc_unsslitt.
7.85
8.15
9.57
8.07
m2d2 m202__2onc_umppiit____mmm
6.27
6.56
7.82
6,45
6.38
m2d2 maa2__2onc_mmppii_____lmm
7.39
7.66
9.00
7.51
7.41
m2d2
mea2__sone_umspii____llmm
7.47
9.27
7.62
7.47
m2d2
set_s_2oooo_liiee Ho
14.52
14.70
16.52
14.51
14.31
m2d2_s2onc
_ral__oon__mpplii_____aaat
7.54
7.80
9.14
7.70
7.58
Lidd__socc_umspii__aa
9.84
10.41
12.53
10.13
10.03
m2d2 s___ss2ooo_sspp_ val ma
8.25
8.53
9.99
8.42
8.26
m2d2_
geul__sorc_umsplt_weee math__aaNNN
9.85
10.05
11.66
9.95
9.83
__t_s__oousssi___eee_____ val _ol_math.NT
8.26
8.51
9.92
8.43
8.31
m2d2 s2ore_ cospppusppitt... val math.OA OA
7.21
7.55
9.07
7.47
7.32
-2ore_unsplit_ math.OC
9.70
10.01
11.62
9.85
9.69
fiad__s2rr_uusplityyee _.__uuppit_val__mat.PPP val_math.PR
8.91
9.20
10.58
9.04
8.99
fid2_ssorc_unspll cicippit_val_mattt wal__mathheemmm QA
8.09
8.40
9.93
8,28
8.16
m2d2_ @ed2_s2orc_unge _2orc_unsplit val ma
7.18
7.44
8.75
7.39
7.27
m2d2 s2orc_unsplit val math.T
8.39
8.71
10.33
8.65
8.49
m2d2 sedd2_s2ore____ie s2ore_unsplit, val Math.SG
8.63
8.88
10.36
8.76
8.59
m2d2_s
aidd__2ore__nsplee vattii_____aaahhh
9.39
9.65
11.27
9.52
9.37
7.81
8.07
9.36
7.99
7.87
bavvvll_mmn mal_nlin.AO
11.82
12.01
13.77
11.90
11.75
m2d2 _____$2oc__spppe
m2d2_s2orc____
sis___ussp_______ val nlin.CD
12.73
12.91
14.88
12.87
12.60
m2d2 s2orc popurc_sppptt...mm val nlin.CG
12.43
12.75
14.88
12.61
12.44
m2d2_s2ore _oo__s2rcc_usplit watpurcuuspptttyee val pl
11.29
11.44
12.86
11.39
11.22
9.44
9.64
fea2_s2onc_usplit_aa
12.43
12.75
12.61
13.02
12.94
14.61
12.85
12.63
m2d2 Shtrr_usppli___1m nucl-th
mhhllit
11.65
11.78
13.43
11.68
11.48
13.75
14.01
16.17
13.74
13.58
13.92
14.04
15.91
13.89
13.68
m2d2 s2orc_uunra chorc_unsplit_.. val hwlplphyssssppppee
13.70
13.81
15.54
13.62
13.43
m2d2_s200 [idd_s2orc__un -2ore_unsplit_ harplli__alpphhiisss.ece physics.atm-ol
13.00
13.13
15.11
13.00
12.74
12.74
12.84
14.44
12.75
12.53
m2d2 s2orc_
siinr__uspittyye val physics.bio-pl
13.30
13.42
15.26
13.32
13.08
m2d2 soop__2orr_unspiitt.. cuusspit_va__phyiics.teee physics.chem-ppp
13.20
13.29
15.22
13.14
12.97
m2d2 m2d2__soor_uusplit_yy stt_:ooo__uuppliiiaaappsssseeedd wwtpaaalsssssllsssssssss
11.01
11.27
12.85
11.12
10.94
m2d2 htwp__umpiiivaa__pyyyis..omm
11.23
11.37
12.88
11.26
11.08
m2d2 m2d2_s2one_um fad2_ssorc_unspli..o www.phhhccccsdtaaaaan
13.18
13.33
14,97
13.25
13.00
m2d2 22d22_$2orc_un s2orc_unsplit.tt.ccmccmmmuu val physics.ed-ph
12.21
12.33
13.88
12.18
12.03
m2d2 s2orc_unsplit. val flysics.fuudyn
11.81
11.99
13.73
11.81
11.64
m2d2_ m2d2_s2orc_unsplen http___all__psssgeeeeeees
14.15
14.39
16.76
14.18
m2d2_s2ore phl_psysssssgoppphhhh
14.75
14.86
16.81
14.71
14.57
m2d2_s2ore_uml sanpiiii_wa_ppysccchhttppph
15.57
15.43
16.97
15.40
15.18
m2d2 @dorc_unsplit val physics.in--det
musspii_val_physiccinn-ddd
14.01
14.16
16.14
14.07
13.79
52

---

Dataset
Llama
Mamba
RWKV4
XLSTM[7:1]
XLSTM[[00]
14.34
14.46
16.50
14.29
14.09
m2d2_ [r22__s2onc_unsplt__ ottpa__unppiiiaaappiiisooooioii physicc.ossiiiss
12.74
12.94
14.64
12.80
12.54
mndd__2oom s2orc_unsplit _aalit___l_____ hampplitva__ptyiicspplsss.. physics.plasm-paap.....pp
13.65
13.81
15.77
13.69
13.44
m2d2 [edd2_s2ore_uusphi val Sttpllaaaal_pssssppppppppp physics.pop-ph
13.80
13.67
15.17
13.60
13.41
m2d2 mzo2__22oc_umsppit__ee ptt__/2ooouuuppiiiiiaapaysssssss www.pyyssssssspppbll..
12.79
12.97
14.80
12.83
12.66
m2d2_ http://orr__nniiiiaaaapsssissssp
13.00
13.09
14.77
12.94
12.76
m2d2 m2d2__2onc_umppii___aaaaaa
15.57
15.43
16.97
15.40
15.18
m2d2 m2d2_s22or__u sa12___2or__ussllit..
13.65
13.81
15.77
13.69
13.44
m2d2 s2orc_unspliittttcccccouuuuuus. val q-bio
13.69
13.87
15.75
13.75
13.50
m2d2 m200__sonc_uusppiiiia_gggg
13.28
13.52
15.72
13.41
13.19
m2d2
12.06
12.34
14.21
12.19
m2d2_ maa2__2on__umppiiu__l__bbbbee
13.21
11.40
12.74
11.32
mea___s2nc_umppii_____ppp
11.96
11.95
13.36
11.90
11.70
m2d2 wtt_sssoossssii_veeeee__
13.69
13.87
15.75
13.75
13.50
m2d2 batoor_uusppii_valgggbooo Licsspiit_val__10
14.90
14.94
17.16
14.92
14.73
m2d2_ mrd___2orc_usppiii__a_ooe
12.57
12.71
14.62
12.69
12.41
m2d2_s2orc 22a___2on__mpppii_____gbbo
12.49
12.69
14.44
12.56
12.40
m2d2_
13.68
13.85
15.60
13.75
13.53
13.49
13.53
15.32
13.48
13.33
m2d2 s2ore_ uusppit val q-
13.69
13.87
15.75
13.75
13.50
s2ore_unsplit__e qal_q-tin.C7
11.37
11.61
13.36
11,41
11.28
m2d2 fas22_s2or__uie _2or_uusslitt_ val FC
11.72
11.89
13.77
11.77
11.63
m2d2 s2ore porrppusppltttccmm val GN
13.79
13.91
15.73
13.83
13.61
swwrr__nspllittyyee qal
9.91
10.21
11.92
10.04
9.90
11.00
11.31
13.14
11.14
10.94
m2d2 m2d2_s2ore__nn s2ore__nsplli__ qai_n-fin.PRR
15.87
9.25
10.37
9.20
9.03
m2d2_
11.35
11.49
13.08
11.41
11.22
12.43
12.46
14.18
12.43
12.26
m2d2 sed2_s2orr__uvl
ser___2ooo__sslt__elee varplit_vallggiiini
12.79
13.14
15.32
12.89
12.74
13.79
13.91
15.73
13.83
13.61
m242_,sone__mppiit___.ouuu
11.18
11.44
13.18
11.32
13.37
13.56
15,52
13.42
13.15
(ed2__s2rc__usss
13.07
12.56
14.42
12.46
12.24
m2d2
sta.ME
11.09
11.26
12.91
11.11
10.87
fodd__s2rc_umspli__vee
11.13
11.39
13.29
11.23
11.06
m2d2_ssorr_unsplitvvaa
11.31
11.55
13.28
11.45
11.24
m2d2 _torc_unsplic Sapppitt_vll_a___1 stat 11
13.07
12.56
14.42
12.46
12.24
m2d2_s2ore_unspp
11.57
11.66
13.13
11.53
11.30
12.30
11.90
12.82
11.78
11.66
ins__wikipedia_ussplte Lal___val___utueaantte the
12.13
11.74
12.82
11.63
11.48
aad2_wikipedia_nn cictiii__nspittttv val Culture_and the 14.06
Munspit_val_Culuureeeee
13.86
15.17
13.79
13.57
m2d2, wikipeiaa unsplit Culture and the 12.16
11.80
12.74
11.79
11.55
m2d2_ wil_wwikpiiii_auu uusiiiia___pi___________ val Laa_lall__laaaa Culture and the 11.75
11.25
12.03
11.17
11.03
http.wikppediauuspliii Bapplit_val_Cuuuure_ee
the
10.01
9.63
10.36
9.58
9.54
fadd__wikppedi__unpphhee val vall_auuuureaann the_______________
12.13
11.85
12.83
11.73
11.58
m2d2 m2d2__wkipp cetpipeddaauuspli__._o_ val
Lall__Cltur___ae and the
12.36
12.09
13.05
11.99
m2d2__wikipedia__nsphii val httprauussiiiiaaaaaeeaeeeeeeeeee
11.80
11.46
12.43
11.46
11.30
m2dd2_wikipedia__nppp httpg//uppppittllaeeeeeeeeerrreo General_refercce,
10.52
10.20
10.96
10.12
9.99
m22d2_wkippedaa_ma Lai___nnsp_tvvee val Baaplii_val_Geeera__eeee
11.80
11.46
12.43
11.46
11.30
m2d2_ bidkpwikipedia unsplit val Health and fitnes
10.75
10.47
11.14
10.37
10.30
m2d2. mikipedi unsplit val Health and fitnes
9.64
9.29
9.95
9.27
9.16
m2d2_ heso__wiipedda_aange sictiiia_nsspitteee val Health and fitnes
10.10
9.80
10.43
9.71
9.56
53

---

Dataset
Llama
Mamba
RWKV4
XLSTM[7:1] XLSTM[[:0]
9.14
8.83
9.59
8,63
8.54
m2d2__wiippedaa_mmg val
vaapiit__a_eeahhhhhhhhh Healtheahhat __nd__dd___ fitnes
8.91
8.68
9,40
8.61
8.47
2ad22wikipedid unsplit_ val Health and fitnes
10.75
10.47
11.14
10.37
10,30
m2d2_ wikipedia unsplit val Health and fitnes
12.91
12.49
13.61
12.42
12.28
m2d2_wwiiipedia_unna Baaplii_val_iisttryyoo
13.65
13.29
14.48
13.20
13.00
[ts___iiippedia_ussiittt http__val___ioooaa_____vea m2d2_
11.77
11.44
12.36
11.36
11.26
_anslll___sss____ val History bebbs__nd__eee______
12.78
12.41
13.46
12.37
12.12
m2d2_wiiiipedia_ungae val httpaauuusiiiiaaaasssoooaaaadad
12.36
11.88
12.87
11.79
m2d2__wikipcdia__mppii...wwww _at_aaaaa_aaaaattttttttese
12.43
12.03
11.95
11.81
m2d2__wikipedia__unpphii wwwpppiiiiyalllmuaaanaattiiieess
12.43
12.03
12.98
11.95
11.81
m2d2_
m22d__wikipeedia__uaaee val "llman_acttvviee
12.47
12.05
13.12
12.00
11.82
wikipedaii unsplit val Mathematics and
12.90
12.51
13.79
12.48
12.29
m2d2_wikipediaz val
http:auuuupiiiiaaaaaaaaaaaioooo Mathematics_anm and
8.24
8.26
9.37
8.28
8.06
13.21
12.87
13.90
12.85
12.67
m2d2_wwikppddia_mmppliit Matteaaaaitssaadddt and
12.90
12.51
13.79
12.48
12.29
wtw.:iipppddauuusppiiiii val Natural_and p
9.19
8.22
8.81
7.97
7.96
m2d2_wikkppee unsplit val Natural and phys
10.97
10.70
11.53
10.64
10.51
12d2__ikkppede
unsplit val Natural and phys
11.69
11.36
12.28
11.22
11.05
www.wwipdia_uspppiiiii val Laa__val__aaaa Lat_NNaturaaaan pphys
10.43
10.11
10.95
10.00
9.82
www.:/ipddauuusppiiitt.o Munspit_va__Natural_ann _phys
11.48
11.09
11.93
10.98
10.90
m2d2_ fadd_wikipedia_ni _onaiida__sspi_____ val Philosophy__sne _nnipphhy_________ and_
11.83
11.72
13.04
11.60
11.45
m2d2_ 2ad2_wikipedin unsplit val Philosophy and
12.00
11.61
12.66
11.57
11.43
m2d2 wikipedia unsplit val Philosophy and
10.94
10.61
11.34
10.56
10.42
m2d2_ wikipediae Biciipeda__usplit__o. val Religion and
12.81
12.45
13.44
12.38
12.19
m2d2_wikipedia__nse Displit_vi__Rlliiionn_s and bel
11.11
10.80
11.66
10.71
10.58
m2d2_wikipedia_mmm sttp___usspp________ val Religion_anne and bel
11.46
11.06
11.86
10.95
10.85
mnd2_wikipedia_nn sactiii____s_________ val Religion and bel
12.38
12.03
12.94
11.91
11.79
m2d2 m2d2_wikipedia unsplit val Society_ and soci:
10.53
10.24
11.03
10.16
10.05
m2dd__wiiipedia_ungeeeee Saaplit_va__occct__ooo
10.47
10.16
10.95
10.14
10.04
m2d2_wikippedia_nnge caciiiii__nspi_________ val Lusspli_val_Sociity_ooo Baciily__n___o___
12.48
12.13
13.02
12.07
11.93
madd_wwikipddaauumppitt.wwaa
8.51
8.18
8.66
7.93
7.88
m2d2__wikipediaa_usphhe BawaallTeeehhhho__ae
12.45
12.07
13.00
12.03
11.88
13.62
13.23
14.56
13.18
12.97
13.00
12.72
13.87
12.64
12.43
14.34
13.90
15.20
13.94
13.73
manosphere meta_ seria_se______ wwt__eee_______eeeee_eee____e_ee avfm
19.42
19.27
21.88
19.64
19.18
manosphere meta sep_val val incels
11.26
12.18
21.40
11.51
11.29
manosphere_mett metmaasep_vaallllam_eeeaaeeppaee
24.83
24.27
27.50
24.12
23.80
24.22
23.85
26.52
23.86
23.52
saatierr__mttsss__ve _ill_talk talk
34.59
33.90
37.26
33.90
33,27
manosphere_mme wwt_mmmett___e__eeeeee
sea__see__lllrddiii reddit
20.63
19.78
21.10
19.94
19.58
manosphere meta_ sep_ val rooshy
22.46
22.17
24.78
22.01
21.69
manosphere_meet mwt___ett________________ val the_theeaat attraction -
20.85
20.57
23.17
20.57
20.20
mic__all-00000x
8.35
8.41
10.02
8.23
8.15
mc4. val-00000001
12.17
11.97
13.58
11.74
mc4_ 114_val-00000002
9.96
10.06
11.96
9.86
9.67
mc4_val -00_v00000000003
11.38
11.29
12.77
11.12
mc4_val mc4_val-0000000
11.96
11.64
13.03
11.50
mtb_val
15.92
16.65
19.37
16.00
15.92
redpajama val arxiv
5.15
5.28
5.78
5.12
5.09
redpajama_ vval______ books
12.91
12.71
13.60
12.61
12.50
redpajama_var val_
13.01
12.51
13.55
12.49
12.27
54

---

Dataset
Llama 10.90
Mamba
RWKV4
XLSTM[7:1] XLSTM[
redpajama_ Batiii_vaa__coomoe _al__comooncawll
10.56
11.70
10.52
10.35
redpajama val Las___lll___g_hhaa githuu
1.66
1.66
1.75
1.65
1.64
redpajama val. stackexchange
3.73
3.72
4.03
3.68
3.63
rodpajama_vv saaaaamm__ll_iiiipeeeeee wikipedia
4.64
4.38
4.68
4.35
4.29
twitterAAE, BEMierAAEEEELMM.__ fixed_val_at val_AA
346.98
302.79
310.30
301.65
289.97
twitterAEE HELM fixed val white
118.62
107.34
109.13
107.65
105.13
wikitext 103 val
11.74
11.76
13.73
11.32
55

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See sections 2 and 4.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 5.
Guidelines:
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide proof in the Appendix, which is clearly linked to the main paper. All assumptions are stated. We enumerate and cross-reference all equations.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We state all experimental details in Appendix D. We state which datasets used and provide references. We provide the model code.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The data is publicly available. The code contained in the submission and will be also released publicly.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: See Appendix and attached code.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide error bars only for small scale experiments. For large scale experiments, providing error bars is computationally expensive.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
56

---

Answer: [Yes]
Justification: In Section we state the compute resources used for this project.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : / nnerrps.....p// hc/ppublic/ttiiiccllleeeeeeeeee
Answer: [Yes]
Justification:
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We refer Section A
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: As of now we do release the weights of our pretrained models.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: We provide the model code with this submission.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We rely on openly available datasets and reference them properly.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
57


Reviews:
[{"review_id": "VMTb53V24U", "reviewer": "Reviewer_TAMc", "summary": "The paper introduces xLSTM, an advanced variant of traditional LSTMs, incorporating two extentions aimed at boosting its memory capacity and performance.\nThe first enhancement, termed sLSTM, modifies the standard LSTM by integrating exponential input and forget gates alongside a stabilizing normalizer term, designed to refine gate operations and ensure network stability.\n\nThe second mLSTM involves the incorporation of matrix-form hidden states into sLSTM, reminiscent of architectural advancements observed in models like the GLA, RetNet, and RWKV5/6. \nThis approach significantly augments the memory capacity of the LSTM units, aligning with recent trends in RNN design that prioritize enhanced memory retention.\n\nThe xLSTM architecture is constructed through a residual stacking of xLSTM token mixing layers coupled with Linear layers, mirroring the design principles employed in architectures such as Llama. \nThis configuration facilitates efficient information propagation and processing within the network.\n\nThe authors successfully scaled xLSTMs to 1B trained on hundreds of billions of tokens, from which they found xLSTMs perform better than exising Transformer, SSMs and other linear attention archs. \nAs for associative recall abilities, xLSTMs exibits better performance evaluated on MQAR tasks.\nThe paper also includes an exploration of scaling laws, revealing that xLSTMs maintain favorable performance characteristics when scaled to larger model sizes.", "questions": "If possible, I would like to known the comparisons between xLSTMs and GLA", "limitations": "The proposed exponential gating makes xLSTM hard to be parallelized, hindering its abilities to scale to larger sizes in compared to other exisiting architectures.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes a new RNN variant, denoted as xLSTM, making two extensions, i.e., sLSTM and mLSTM, to the classical LSTM, which has potentials to rejuvenate the classical RNNs in the LLM era.\nExperiements on 340M / 1B sized models show that xLSTMs perform very well on language modeling tasks.\nAnalysis indicates that the xLSTM variant excels in retrieval-intensive scenarios, a critical advancement given the acknowledged limitations in memory capacity that have historically constrained RNNs. I look forward to seeing the performence of xLSTMs scaled to larger sizes.", "weaknesses": "The overall designs of mLSTM are not that novel. \nNotably, architectures such as the RetNet and GLA has successfully applied 2-d matrix-formed hidden states into RNN architectures that greatly enlarge the memory capabilities.\n\nxLSTMs additionally propose a exponentials gating, which, as claimed by the authors, can enable the abilities to solve state tracking problems. The authors verify the effectiveness of some synthetic tasks in Appendix D.1. \nHowever, I believe this design can greatly hurt the parallelizability of xLSTMs. \nTo enable stable training, the authors necessitate maintaining $i$ and $f$ in log-space first, at the scarifice of hardware optimizations, making it hard to benefit from tensor-core accelerations, a feature that is pivotal for achieving high computational efficiency. \nThis partially explains why their kernel impls are much slower than flash-attention.\nMoreover, the input and forget has to be kept in scalars, which I believe is inferior to the more fine-grained gating of GLA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review which helps to improve our work! \n\nIndeed the reviewer is right, GLA and Retention are similar to mLSTM in that sense as they both have a matrix cell state, too. However, we highlight that neither Retention nor GLA have an (input-dependent) input gate and Retention has no input-dependent gates at all. mLSTM has an input-dependent exponential input gate and an input-dependent sigmoid forget gate. We would like to point the reviewer to Table 6 in the Appendix, where we carefully ablate the design decisions of the different gating mechanisms and also relate to the other models like Retention. We find that our Exponential Gating with all gates trainable performs the best and introduce Exponential Gating as a gating mechanism that is applicable to matrix memory cells (mLSTM) as well as scalar memory cells (sLSTM).\n\nFor a direct comparison to GLA in terms of performance in Language Modeling measured in Perplexity (PPL) on the validation set, we refer to Table 1 in the main paper. There, we show that our xLSTM variants xLSTM[1:0] (i.e. mLSTM only) and xLSTM[7:1] (i.e. ratio mLSTM:sLSTM blocks = 7:1) achieve a PPL of 13.43 and 13.48 and significantly outperform RetNet (16.23 PPL) and GLA (16.15 PPL). \nAs noted by the reviewer we keep the per-head input and forget gates for the mLSTM as scalars and then broadcast these scalars along the head dimensions. We do not think that this decreases performance compared to the dimension-wise gating of GLA (see Table 1 and Table 6 in our paper). This design decision has also been made by the authors of Mamba 2, when moving from Mamba 1 to Mamba 2 (see[1] and Table 1 in [2]). Additionally, having the gates as scalars simplifies the parallel form even though we maintain the forget gates in logspace, since we can first accumulate across the head dimension and then multiply by the gates (compare e.g. to equation (4) in [2]). Apart from that, the chunkwise formulation of Linear Attention of GLA [2], is applicable to mLSTM, too. \n\nWe believe that in our paper we did not sufficiently separate exponential gating and memory mixing, which are independent architecture characteristics.  Exponential gating alone does not hinder parallelization. When we use  memory mixing as it is done in the sLSTM, we cannot parallelize the computation. In this case the reason for the non parallelizability is memory mixing, i.e. the dependence of the gates on the previous hidden states via recurrent weights (see r_{i,f,z,o} and R_{i,f,z,o} in equations (11-14) and (34-37), instead of exponential gating. \nOur formal language experiments indicate that exponential gating in combination with memory memory mixing can solve the Parity task, which is one simple instance of a problem that requires state tracking (see Table 8, rows: LSTM, xLSTM[0:1] (i.e. sLSTM only), xLSTM[1:1]).\nSince the mLSTM alone (xLSTM[1:0]), which has exponential gating but no memory mixing, cannot solve the task, and the original LSTM, which has memory mixing but no exponential gating) can solve it, we think that memory mixing is crucial for the state tracking capability.\nWe are sorry that we did not state this clearly in the main paper and will do so in a potential camera ready version. \n\nWhen we apply Exponential Gating to the original LSTM, we obtain the sLSTM (i.e. xLSTM[0:1]), which improves performance on language modeling by almost 9 PPL points (difference between line 3 and 4 in upper part of Table 6). Note that both original LSTM and sLSTM have memory mixing and are not parallelizable. \n\nIn contrast to the sLSTM, the mLSTM is fully parallelizable analogous to Transformers with Self-Attention. We outline the parallel formulation (forward and backward pass) of the mLSTM with Exponential Gating in Section B.3 in the Appendix.\n\nDespite the non-parallelizability of the sLSTM,  we propose a more efficient variant of memory mixing compared to the original LSTM and use optimized CUDA kernels for the sLSTM, which is less than two times slower than the parallel mLSTM implementation.\n\nWe hope that we could clarify your concerns and kindly ask you to raise your score if you find them addressed properly. \n\n[1] Dao, Tri, and Albert Gu. \"Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality.\" arXiv preprint arXiv:2405.21060 (2024).\n\n[2] Yang, Songlin, et al. \"Gated linear attention transformers with hardware-efficient training.\" arXiv preprint arXiv:2312.06635 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eFA3ZmBa8j", "reviewer": "Reviewer_WcbD", "summary": "Proposes some extension to LSTM. Specifically:\n\n* exponential gating with appropriate normalization and stabilization techniques\n\n* sLSTM with a scalar memory, a scalar update, and new memory mixing. This still needs to be calculated sequentially.\n\n* mLSTM that is fully parallelizable with a matrix memory (instead of just a vector) and a covariance update rule\n\n* putting sLSTM & mLSTM together into a residual block: the xLSTM block\n\n* Extended LSTM (xLSTM) blocks are residually stacked into xLSTM architecture\n\nTested for language modeling.\n\nCompared to Transformers and State Space Models.", "questions": "In all the formulas, it's not immediately clear whether e.g. h_t, c_t etc are scalar values or vectors. I assume those are vectors. It would be good to write that explicitly (sth like c_t, h_t, ... \\in \\R^D) or so.\n\nHowever, then it says that w_z, w_i, w_f, w_o are also vectors (eq 4-6). This is weird. That should be matrices, or not? Also the r_z, r_i, r_f, r_o, they are just called \"weights\" (p.3 line 94), but that should be matrices as well, specifically all in \\R^{D \\times D}. It would help to state that explicitly. The notation is also a bit uncommon. Most common is to use capital letters for matrices.\n\nOr is h_t here really a scalar, not a vector, e.g. in eq 2? But then, in eq 4-7, the multiplication with r is wrong? Or are those r also scalars? But this is not a normal LSTM then, because it would mix also the other dimensions.\n\nI see some different definition in B.1. Is this just now a different notation, specifically in vector notation? Or is this really an alternative different model? As this says \"vector notation\", that really means that the initial equations (eq 1-7 and more) are all on scalars? Specifically, for example, eq 27 does not fit together with eq 4. If r is a vector in eq 4, then it just multiplies with a scalar h_{t-1} in eq 4, which is wrong.\n\nOr later in Sec 2.1, it then says \"In later formulations, multiple memory cells were combined in a vector, which allows the usage of recurrent weight matrices\". So actually only this aspect really leads to the vanilla LSTM. It means the presented formula in Sec 2.1 are really not the same as B.1. And also the Sec 2.2 formula are also not what you actually use, because then later you also have matrices R.\n\nI think this is all confusing. Why present the scalar variants at all, when they are never used like this?\n\n\nNow, for sLSTM, it was said that sLSTM has scalar memory and a scalar update. What does this mean? What is scalar about it? In what way is it different from the normal LSTM?\n\nI see in appendix B.2 that \"The matrices Rz , Ri , Rf , Ro are block-diagonal\". Is this specific only for the sLSTM? For the vanilla LSTM, you have fully dense matrices, right? So is this actually the main difference to the vanilla LSTM? Block-diagonal refers to \"scalar\"? So it's not really scalar?\n\n\n\n\nWhat is the motivation of exponential gates? To be able to open the gate and even amplify the input (gate > 1)? But why exp, why not relu or some other function which does not grow so fast?\n\nsLSTM Stabilized Version only in appendix. Why? Is this not so important?\n\nsLSTM normalization (n_t), how important?\n\n\n\n\nSec 2.2 \"we introduce a normalizer state that sums up the product of input gate times all future forget gates.\" - I think some word is missing.\n\n\nSec 2.3 \"we increase the LSTM memory cell from a scalar c ∈ R to a matrix C ∈ Rd×d\" - this is misleading, or I don't understand it. Actually you increase the LSTM memory cell from a vector to a matrix, not from a scalar to a matrix, or not? Or do you really have a different matrix C for every dimension, i.e. you actually have a 3D tensor as memory when there are multiple cells? Specifically, in eq 16-24, the i_t, f_t, h_t, q_t, etc, are those scalars or vectors now? In eq 16, when i_t/f_t are scalars, that means when you now have multiple cells, that the C_t becomes a 3D tensor?\n\n\nSec 2.4: xLSTM blocks and xLSTM architecture: I think it's bad to refer to the appendix for figures. I think those are crucial aspects about the whole model, which should be in the main text. Also, there should be some exact formulas to define the model, not just figures.\n\n\nSec 2.4 xLSTM architecture: \"LayerNorm ... See last column in Figure 5.\" - when I look at the last column in Fig 5, I only see some gray blocks. I don't see LayerNorm there.\n\n\nSec 2.4 xLSTM architecture: The only real \"definition\" of the model is Figure 5? In Figure 5, when it says xLSTM blocks, that means either a sLSTM block (like Figure 6) or a mLSTM block (like Figure 7)? So the light gray blocks are mLSTM, and the dark gray blocks are sLSTM in the last column in Fig 5?\n\nThe sLSTM and mLSTM blocks (Fig 6, Fig 7) have many further aspects which are not really discussed, like the group norm, etc. How did you end up with this specific architecture of the blocks? Have you tested any variations on that?\n\nSec 2.4: From Fig 5, it seems that the amount of sLSTM and mLSTM blocks is not the same? This is not really discussed or even mentioned at all? It seems to be also a crucial aspect of the whole xLSTM model. Only later in Sec 4, I read \"For all experiments, we use the notation xLSTM[a:b] for the ratio a/b of mLSTM-based versus sLSTM-based xLSTM blocks.\".\n\n\n\nFigure 4, scaling laws: I don't think it is reasonable to have the number of parameters on the x axis. More reasonable is to put e.g. the inference time at the x axis.\n\n\n\nRelation to multiplicative LSTM (https://arxiv.org/abs/1609.07959)?\n\nRelation to associative LSTM (https://arxiv.org/abs/1602.03032)?\n\nStudy on activation functions? Or only tanh?", "limitations": "-", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Scales up LSTM models.\n\nProposes extensions to the standard LSTM model which perform really well.\n\nLots of experiments. (Although most is only in the appendix...)\n\nOpen code.", "weaknesses": "Some of the notation is confusing. Specifically, I'm never really sure whether it's about scalars or vectors. I also think there are some errors regarding this. I don't understand why to write it at all in scalar notation.\n\nThere is never really an exact definition of the whole model. We can only infer it from the figures. (Or the code.)\n\nThe scaling laws comparison is a bit flawed: I don't think you can just compare the number of parameters across different architectures. More reasonable would be to compare inference time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review, which helps to improve our paper!\n\nWe are sorry that the scalar notation for LSTM and sLSTM (eqs. 2-14) caused confusion. This notation was chosen to reflect the original LSTM idea of a single cell and to make the distinction to matrix memory cells more pronounced. The scalar notation assumes that an LSTM cell is defined by a scalar input and forget gate. The vectorized version (where we stack multiple cells in a vector) with cell interactions (i.e. memory mixing, eqs. 25-37) is probably more familiar from recent literature. Some of our contributions, namely the block-diagonal recurrent matrix R for sLSTM also relies on the vectorized part essentially. We checked all notations again and they all should be correct and consistent. For all our formulas we use the convention:\n\n- non-bold, lower case letters represent scalars\n- bold, lower case letters represent vectors\n- bold, capitalized letters represent matrices\n\nAccording to this convention the difference between LSTM and sLSTM formulation in equations (2-14) and equations (25-37) is that (25-37) essentially stack multiple memory cells / hidden states into a vector.\n\nIn short, the main difference between the original LSTM and the sLSTM are: sLSTM has Exponential Gating vs. LSTM has sigmoid gating. sLSTM has block diagonal recurrent weights R vs. LSTM has dense weights. (However one could also apply block diagonal recurrent weights R to the original LSTM).\n\nThe motivation for the exponential gate is the similarity to a running softmax and the fact that no matter how large an input is amplified, a later input can still surpass it in its weight by an additive increment in the input gate preactivation. This way it is always possible to revise a previous storage decision. The stabilization is essential for both variants but was moved to the appendix for space reasons.\nRegarding the cell expansion from scalar to matrix, we assume one cell to be defined by one scalar input and forget gate. In this sense it is an expansion from scalar to matrix. There are different heads for mLSTM in parallel, so effectively this is a third dimension orthogonal to the matrix state.\n\nRegarding the complete model figures, we want to emphasize their size for readability and the limited space in the main paper. The specific positions of sLSTM vs mLSTM blocks influence the training performance, and the models shown performed well. The shown ratios performed well on the specific tasks, especially for language modeling there seems to be an emphasis on memory rather than the state tracking capabilities of sLSTM. The tanh as activation function for sLSTM enables long context training stability, but others were not tested.\n\nRelation to multiplicative LSTM: Multiplicative LSTMs modify the memory-mixing part of LSTMs to an input dependent recurrent matrix (see Eqn. 8). Their gating is equivalent to vanilla LSTMs and the architecture is not sequence-parallelizable.\nRelation to associative LSTM: Associative LSTM act on complex numbers and also use some kind of state expansion by using multiple \"copies\" with different (non-learnable) random permutations, whereas mLSTM uses learnable projections for key and value vectors that form the cell state update. They do not have a parallelizable version and use vanilla LSTM gating.\n\nOur scaling laws (with the number of parameters on the x-axis) show how effective the model parameters are used. We agree, the inference time is very important. Therefore, in additional experiments, we have measured the inference time of the models, which we present in the attached figures. Please find these figures in the supplementary one-page PDF attached to our general response.\nThe inference time behavior for other model sizes is similar. \n\nAlso they do not include optimized kernels for mLSTM.\n\nWe address the following of your points directly below: \n> “Sec 2.4 xLSTM architecture: \"LayerNorm ... See last column in Figure 5.\" - when I look at the last column in Fig 5, I only see some gray blocks. I don't see LayerNorm there.”\n\nIndeed, this is a typo. We mean the last two columns. In the blocks on the left you see the pre-layernorm architecture of the transformer and skip connections. The last column shows the final stacking. Thank you for reading the paper so carefully.\n\n> “Sec 2.4 xLSTM architecture: The only real \"definition\" of the model is Figure 5? In Figure 5, when it says xLSTM blocks, that means either a sLSTM block (like Figure 6) or a mLSTM block (like Figure 7)? So the light gray blocks are mLSTM, and the dark gray blocks are sLSTM in the last column in Fig 5?”\n\nCorrect. Light gray = mLSTM, dark gray = sLSTM.\n\n> “Sec 2.4: From Fig 5, it seems that the amount of sLSTM and mLSTM blocks is not the same? This is not really discussed or even mentioned at all? It seems to be also a crucial aspect of the whole xLSTM model. Only later in Sec 4, I read \"For all experiments, we use the notation xLSTM[a:b] for the ratio a/b of mLSTM-based versus sLSTM-based xLSTM blocks.\".”\n\nWe view the block ratio as a hyperparameter that needs to be tuned for the respective task. In the experiment section we detail which ratio we used for each experiment. \n\n> “The sLSTM and mLSTM blocks (Fig 6, Fig 7) have many further aspects which are not really discussed, like the group norm, etc. How did you end up with this specific architecture of the blocks? Have you tested any variations on that?”\n\nOur Pre-Up Projection Block (Fig. 7) was inspired by previous work like Mamba, H3 or Retention. The Post-Up Projection Block was inspired by Transformer architectures like Llama or GPT. We refer to these publications for details.\n\nWe hope that we could answer your questions and clarify your concerns. If you find them addressed properly we kindly ask you to raise your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "njTwdhuO1E", "reviewer": "Reviewer_ZvGL", "summary": "This paper introduces the Extended Long Short-Term Memory (xLSTM), which enhances traditional LSTMs with exponential gating and new matrix memory. These improvements address LSTM limitations and enhance their memory storage capability, as well as the ability to revise storage decisions and be more parallelizable. The authors test the efficacy of their method with a set of very exhaustive experiments, which confirm that xLSTMs are competitive with both SSMs and Transformer-based architectures.", "questions": "For my interest, I would like the authors to clarify a few things:\n\n1. Structure of Memory Mixing Matrices: As far as I can tell, the idea of matrix-based updates has been used for a while now. I do appreciate the introduction of different “heads” through a block-diagonal structure. In some sense, this approach could be viewed as restrictive, as it imposes a specific form on the update matrix. I wonder if there might be any benefit to considering a more general structure (which could be viewed as a sort of graph) that could be learned from the downstream task loss? \n\n2. State Normalization: The idea of normalizing the state to avoid overflow using the additional state $m_t$ is clever. However, at some point I pressume $i_t$ has to be calculated and stored as well. Could this still lead to some overflow issues? What did you observe in practice?\n\n3. Matrix Memory: I find this idea very intuitive. While the authors have focused on using this for the cell state of the LSTM in this context, I wondered if they had tested the same idea for other (even vanilla) RNNs. Should similar improvements in memory performance be expected intuitively?\n\n4. Overall Architectural Decisions: What was the thought process behind designing the whole block structure (e.g., 1D convolutional layers, etc.) shown in the appendices? Was this inspired by previous high-performing architectures in sequence modeling (i.e. previous high-performing sequence modeling architectures)? Is this something that the authors tuned significantly, or is there potential for even better performance?\n\n5. Extrapolation: Do the results in Figure 3 suggest that xLSTMs could train on shorter context lengths while retaining the ability to extrapolate to longer sequences? Note that this is just a question to get the authors' opinion; I am not suggesting that they should run experiments along these lines, as the cost would be too high.\n\n6. Other Applications: This paper has focused on text data as well as other synthetic tasks, but I was wondering if the authors would expect it to work well for other data types (e.g., temporal or image data)?", "limitations": "Yes, the limitations of the proposed method are mentioned in the paper in a dedicated section.", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I believe this paper is exceptionally strong and represents a highly innovative contribution to the field of sequence modeling. It is compelling to see that traditional RNN-based architectures, with some modifications, can outperform Transformer-based architectures, which aligns with some of the recent trends in this field [1,2]. I hope this direction gains traction within the community. This paper also reinforces the notion that old ideas, with slight adjustments, can effectively compete with the most popular models today. Time will tell if these architectures will become a viable alternative to their attention-based counterparts.\n\nHere are some of the key strengths of the paper:\n\n1. An innovative approach to replacing input gates in LSTMs to improve storage decisions.\n2. An intuitive methodology for storing and retrieving information in memory in matrix form.\n3. Linear computation and constant memory complexity with respect to sequence length.\n4. Extremely exhaustive evaluation in both language-based and synthetic tasks.\n\n[1] Gu, Albert, Karan Goel, and Christopher Ré. \"Efficiently modeling long sequences with structured state spaces.\" arXiv preprint arXiv:2111.00396 (2021).\n\n[2] Orvieto, Antonio, et al. \"Resurrecting recurrent neural networks for long sequences.\" International Conference on Machine Learning. PMLR, 2023.", "weaknesses": "I think the paper has been executed almost flawlessly, so I will forgo this section and will simply add some questions in the section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the excellent score and the follow up questions:\n\n1. It is possible to use a different structure to make the recurrent matrix more parameter efficient. Because of the similarity to Transformer heads and hardware-optimal training on GPUs the block-diagonal form was taken.\n\n2. We did not see any overflow issues for mLSTM or sLSTM. However as the normalizer is aggregating it might be limited by numerical ranges at some point if forget gates stay too close to one. The normalizer state is bounded from below by one for both sLSTM and mLSTM, so the division is numerically stable.\n\n3. In principle a matrix memory should enhance other (vanilla) RNNs as well. A purely recurrent version (as for vanilla tanh RNNs) will be slow as it cannot be computed efficiently on GPUs. This is why it was not tested in our work here.\n\n4. Architecture decisions: This was inspired by previous work on State Space models (H3, Mamba). The convolution turns out to be important and the convolution window size can be tuned further. There is still a lot of architectural search space to explore.\n\n5. Extrapolation: The results in Figure 3 imply this capability, such that shorter context lengths can be used for training, while having larger context during inference. However there are fundamental limitations for retrieval-focused tasks like the phone book task or the needle in the haystack tasks due to the fixed size hidden state. \n\n6. As the model performs well on the tested sequence modeling tasks, we expect it to perform well on time series too. For vision, we expect good results as well, although the sequential inductive bias in our model seems unintuitive.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their comments and constructive feedback. 
In a potential camera ready version of our paper we addressed all your comments and feedback, which considerably improved our paper. 

We thank the reviewers WcbD & ZvGL for appreciating our extensive experiments that we conducted over the last year.

We answer all questions for each reviewer directly below their reviews. 

Reviewer WcbD proposed to measure the inference (wall-clock) time and compare xLSTM to the other baselines. We agree that this is a very interesting experiment.

Therefore, we have measured the time for text generation for our Transformer (Llama architecture), Mamba and RWKV4  baselines and our two xLSTM variants xLSTM[1:0] and xLSTM[7:1]. We compare the generation time for the 1.3B sized models with a context prefill of 16 tokens, and generation speeds for 64 tokens at varying context pre-fill. The results are shown in the attached PDF. 

Due to the recurrent nature and the fixed state size of Mamba, RWKV4 and xLSTM, these models have a constant generation time per token, independent of long pre-fill contexts. Remarkably, xLSTM[1:0] with torch.compile and no custom GPU kernels is on par with a Huggingface Llama implementation in generation speed for small contexts and greatly surpasses it for long contexts.

Meta-review: This paper proposes the so-called extended long short-term memory (xLSTM) model to include numerous enhanced features such as exponential gating and modified memory structures in the traditional LSTM framework.  These enhanced features can effectively mitigate some of the well-known limitations of LSTM (e.g. poor at revising storage decision, limited storage capacity and lack of parallelizability).  The authors conducted extensive experiments to compare the proposed xLSTM with state-of-the-art Transformers and state space models in both performance and scaling and demonstrated its effectiveness.  Overall, this is an interesting paper and may have good impact to the machine learning community.  Concerns raised by reviewers are cleared in the rebuttal and following discussions.  The authors may consider restructuring the paper a bit as some of the important contents are put into the appendix.
Decision: Accept (spotlight)