xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
Authors: Maximilian Beck
Kajetan Schweighofer
Sebastian Böck
Sebastian Lehner
Sepp Hochreiter
Venue: ICLR 2026 Poster (2026)
Keywords: xLSTM
Transformers
Scaling Laws
Sequence Modeling
TFLA
Linear Attention
Inference
Primary area: foundation or frontier models, including LLMs
TL;DR: Scaling laws for linear time-complexity xLSTM model, comparing against Transformers for both training and inference.

Abstract: Scaling laws play a central role in the success of 
Large Language Models (LLMs), enabling the prediction of 
model performance relative to compute budgets prior to training.
While Transformers have been the dominant architecture, 
recent alternatives such as xLSTM offer linear complexity
with respect to context length while remaining competitive in the billion-parameter regime.
We conduct a comparative investigation on the scaling behavior of Transformers and xLSTM along the following lines, providing insights to guide future model design and deployment.
First, we study the scaling behavior for xLSTM in compute-optimal and over-training regimes using both IsoFLOP and parametric fit approaches on a wide range of model sizes (80M-7B) and number of training tokens (2B-2T).
Second, we examine the dependence of optimal model sizes on context length, a pivotal aspect that was largely ignored in previous work. 
Finally, we analyze inference-time scaling characteristics.
Our findings reveal that in typical LLM training and inference scenarios, 
xLSTM scales favorably compared to Transformers.
Notably, xLSTM models consistently Pareto-dominate Transformer models, delivering lower cross-entropy loss for the same compute budget.

Published as a conference paper at ICLR 2026
XLSTM SCALING LAWS: COMPETITIVE PERFORMANCE WITH LINEAR TIME-COMPLEXITY
Maximilian Beck 1.2
Kajetan Schweighofer
Sebastian Böck? 2
Sebastian Lehner
Sepp Hochreiter 1,2
ELLIS Unit Linz, Institute for Machine Learning, JKU Linz, Austria 2 NXAI GmbH, Linz, Austria
(beck, schweighofer, s slehner) @m1 jiku.at
ABSTRACT
Scaling laws play a central role in the success of Large Language Models (LLMs), enabling the prediction of model performance relative to compute budgets prior training. While Transformers have been the dominant architecture, recent al- termatives such as xLSTM offer linear complexity with respect to context length while remaining competitive in the billion- billion-parannnnn parameter regime. We conduct a com- parative investigation on the scaling behavior of Transformers and xLSTM along the following lines, providing insights to guide future model design and deploy- ment. First, study the scaling behavior for xLSTM in compute-optimal and over-training regimes using both IsoFLOP and parametric fit approaches on a wide range of model sizes (80M-7B) and number of training tokens (2B-2T). Second, we examine the dependence of optimal model sizes on context length, pivotal aspect that was largely ignored in previous work. Finally, we analyze inference-time scaling characteristics. Our findings reveal that in typical LLM training and inference scenarios, xLSTM scales favorably compared to Transform- ers. Notably. xLSTM models consistently Pareto- dominnt Transformer models, delivering lower cross-entropy loss for the same compute budget.
INTRODUCTION
Scaling up models sizes and training data sets enables the recently observed rapidly advancing capabilities of Large Language Models (LLMs). As a result the computational expenses associated to training and inference of state-of-the-ar LLMs results are dramatically growing. The goal of predicting the achievable performance with specified architecture and computational resources resulted in the recent exploration in LLM scaling laws, i.e. the quantitative relationships between LLM performance metrics and the corresponding computational resources. The works of Kaplan et al. (2020); Hoffmann et al. (2022) showed that these scaling laws take the form of power laws which hold over several orders of magnitude terms of model sizes and number of pre-training tokens. These insights provided practical guidance in the design of recent frontier models (Achiam et al., 2023; Grattafiori et al., 2024; DeepSeck-Al 2024a).
Recent works (Sardana et al., 2024; Gadre et al., 2024) rightfully argue that these scaling laws are nevertheless limited by their neglect of inference costs. Consequently, these works focus on performance investigations more tokens than would be optimal in terms of pre-taining compute, Importantly, these works and subsequent ones focus Transformer architectures (Vaswani al., 2017). In these architectures, the attention mechanism inflicts computational costs during training and inference that are quadratic in terms of context length. Besides the associated economic and ecological costs, this quadratic scaling is prohibitive for a large range of application areas in which models are deployed on de- vices with limitations on available memory, energy, or allowable TFTT. Even on GPUs that are dedicated to LLMs this scaling property of Transformers represents limitation in task that require
models that trained in the so-called over-training regime, i.e.
'The code and data reproduce analyzes and figures is available at: https: / / thub. com/ NX- /x1 stm_sca ming__a laws

---

Published as a conference paper at ICLR 2026
3.4
Model Sizes
3.2
XLSTM
3.2
160M
Transformer
400M
3.0
3.0
830M
Validation 1io
1.4B
2.8
2.8
2.70
78
2.6
Better
Empirical
Transformer
2.4
2.4
Fits
2.2
Cheaper
2.2
Transformer
XLSTM
103>
1030
16th
10
2023
1035
10
1021
10
1023
Compute (FLOPs) (log scale)
Compute (FLOPs) (log scale)
Figure XLSTM scaling laws: Validation loss over training compute. Left: xLSTM pareto- dominant over dense multi-head Transformers in terms of loss. For a fixed FLOP budget, xLSTM models are better. For a fixed validation loss, xLSTM models require less FL.OPs. Right: Parametric fit of the loss surface L(N, D) as function of model size N and dataset size D.
very long contexts, like reasoning (Muennighoff et al., 2025). Consequently, the development of LLM architectures that mitigate the attention mechanism is an active area of research (Gu & Dao, 2024; Beck et al., 2024; Lieber et al., 2024). While these architectures were demonstrated to be scalable into the billion parammeee regime (Zuo al., 2024; Beck al., 2025b), there is so far no systematic comparison between linear complexity LLM architectures, i.e. LLMs that scale linearly computational costs with respect to context lengths, and transformer-based LLMs with quadratic complexity.
This work presents systematic comparison of the scaling laws of performance-optimizzd XLSTM architectures (Beck et al., 2025b;a) and dense multi-head self-attention Transformer architectures (Touvron et al., 2023). Our investigations of XLSTM and Transformer models are guided by the following research questions:
Training: Which architecture can be trained more efficiently terms of computational resources and how do they scale in the practically relevant overtaining regime?
Context length: How does the striking difference between xLSTM and Transformers- linear versus quadratic context length dependency-imeen and inference performances?
scaling laws and the resulting pre-training
Inference: How does the inference speed in terms of time to first token (prefill) and step time (generation) scale for xLSTM and Transformer under different context lengths and model sizes?
Our investigation shows, that xLSTM models Pareto-dominate Transformer models in the com- pute-loss trade-off (Fig. 1), enabling models that are both better and cheaper. We find that, for a given training compute budget, compute-optimal xLSTM models are larger (Fig. 4), i.e. have more parameters, than compute-optimal Transformer models. During inference, xLSTMs are faster than same-sized Transformers (Fig. 6), and their performance advantage grows with context length due to Transformers' quadratic time complexity.
PRELIMINARIES
We begin with background on scaling laws and definition of the training regimes considered in this work (Sec. 2..). We next present approaches for scaling law fitting used this study (Sec. 2.2).
2.1 BACKGROUND ON SCALING LAWS
Scaling laws for large language models predict the cross-entropy loss as a function of the compute C used for model training in FLOPs. The compute C' for training increases with larger model size measured in number of model parameters N and larger dataset size in number of training tokens D. how the total compute budget
Hence, we assume C' is function of N and D. Depending distributed between increasing the model size and enlarging the dataset, training is typically characterized either being in compute-optimal in over-training regime.
2

---

Published as a conference paper at ICLR 2026
Compute-optimal training. Hoffmann et al. (2022) establish the notion of compute-optingt training, which refers to the optimal choice of and for given compute budget H according to the constrained optimization problem:
argmin N (H), a.. C(N,D)=H L(N.D).
(I)
The optimal N* and D can be obtained by sweeping over N. D for each compute budget. Hoffmann al. (2022) find that for increasing computation budgets, N and D scale roughly proportionally. Assuming this proportionality, there exists a compute-optimal token per parameter ratio M D* D"/N" for a fixed model class and training distribution.
The compute-optinal allocation D*, N' only accounts for compute costs during training. However, during inference larger models incur a higher inference compute cost. Taking this into account, Sardana et al. (2024) argue that, once inference costs are considered, it can be preferable to train smaller models on larger datasets. The resulting values for D and N, with a higher than compute-optingg token per parameter ratios M > M is generally referred to over-training regime (Gadre et al., 2024).
Over-training.
Calculating compute costs. Previous works transformer scaling laws commonly approximate compute costs with C(NN, D) = 6ND FLOPs (Kaplan et al., 2020; Hoffmann et al., 2022; Gadre al., 2024; Sardana al., 2024). This approximation ignores the FLOPs associated the atten- tion mechanism and covers only the feed-forward network contributions. Recently, several works (DeepSeek-AL) 2024a; Busbridge al., 2025; Li et al., 2025) pointed out that this approximation is not justified for sufficiently large context lengths and models. For the purpose of this work, this ap- proximation is even less suitable since neglects entirely the difference between linear and quadratic time-complexity models. Hence, we adopt more precise calculation of C(N, D) as provided in Appendix B.3 that accurately captures the differences in computational complexity between model classes.
2.2 FITTING SCALING LAWS
Scaling laws obtained by fitting the dependence of the model's training or validation loss on the model size and the number of training tokens with power laws. Two commonly used procedures for extracting parametric scaling laws for the loss L, depending N and/or D are the parametric fir approach and the IsoFLOP approach, which are introduced in Hoffmann et al. (2022) as the third and second approach, respectively.
Parametric fit approach. Assuming that the loss follows a power law in model parameters and training tokens D, the parametric fit approach estimates the observed cross-entropy loss as:
L(N, = E (AN +BD D" 9y7
where E, A, B, B, and are task-specific positive parameters. The constant term E accounts for an irreducible loss component, while the second term captures the model-specific predictive performance. While Hoffmann et al. (2022) set 1, we follow the practice from Busbridge et al. (2025) and treat y as fit parameter.
A robust estimation of the scaling parameters for (2) requires data from diverse training strategies, including non-compute optimal token-to-pe to-parameter ratios. Therefore, Hoffmann et (2022) include data from two training strategies: (i) The number of training tokens is varied for fixed set of models. (ii) Model size and training tokens are both varied subject to total compute constraint.
IsoFLOP approach. For the IsoFL.P approach a set of compute budgets H is defined and for each budget the values of N and are varied such that the constraint C(N, D) = H is fulfilled. Following Hoffmaan et al. (2022), second-order polynomial is fitted each of the resulting IsoFLOP profiles. The minimum of each fit corresponds to the loss-optimal number of model parameters N' (HH) and training tokens D" (H) for the given compute budget H. In order to predict these quantities, individual power laws of the forms
and D' B
where we fit the exponents a, b and coefficients A', B' from the data.
3

---

Published as a conference paper at ICLR 2026
Transformer (292 Runs)
XLSTM (348 Runs)
108
108
Training Compute (FLOPS)
Model Parameters
48
Model Parameters
4B
-10
IsoF
ctx=2048
102
ctx-81
ctx 16384
400M
400M
10
Token/Param ctx-8192
10th
100M
100M
48 10B 30B 1008
4008 1T 2T
2B 108 30B 1008
4008 1T 2T
Training Tokens
Training Tokens
Figure 2: Dataset of training runs for our scaling law study. The dataset contains training runs for the XLSTM and the Transformer architecture, with two configurations each: IsoFLOP and Token/Param.
TRAINING SCALING BEHAVIOR
In this section, we conduct comparative study of the scaling behavior of XLSTM and Transformer models along multiple axes. First, explore the pareto frontier of performance in terms of loss and training compute in Section 3.2. Second, we study the scaling in the over-training regime with large token parameter ratios in Section 3.3. Finally, we determine the compute-optimal model and dataset sizes in Section 3.4 and their dependence on the context length in Section 3.5. We begin with the introduction of our experimental setup in Section 3.1.
3.1 EXPERIMENTAL SETUP
To systematically study scaling behavior, we collect a large dataset of training runs across two model classes (Transformer and xLSTM) and multiple training configurations. The following describes the architectures, training recipe, and dataset of training runs used in our scaling law study.
Model architectures: Transformer and xLSTM. Following previous scaling law studies (Porian et al., 2024; Gadre et al., 2024), we use the dense multi-head attention decoder-only Llama-2 architecture (Touvron et al., 2023) for our Transformer models. For the xLSTM models, we consider the architecture of the recently proposed xLSTM 7B model (Beck et al., 2025b). The XLSTM-7B architecture built entirely on ML.TMM cells with parallel training mode applied within the model's embedding dimension. Similar to the Transformer, it alternates STS layers with position-wise feedforward MLP layers. The crucial distinction between the two architectures lies in the sequence- mixing mechanism: self-attention with quadratic time-complexity in Transformer versus recurrent mLSTM dynamics with linear time-complextty in xLSTM.
Training recipe and data. For both model classes we the same training recipe derived from the XLSTM 7B training recipe (Beck et al., 2025b). The recipe uses the Adam W optimizer (81 = 0.99, 82 = 0.95. e 10 *), weight decay 0.1 and gradient clipping norm 0.5. The learning rate scheduler has three stages, linear warm-up, cosine decay to 10% of the peak learning rate, and linear cool-down. For varying compute budgets,
scale the steps in the second stage while the first and third remain fixed. Further details are given in Appendix A. The overall number of training steps is determined by the FLOP budget or token-to-parametee ratio of the specific experiment. As training dataset, we use BAMMM---SLLEEE, a collection of high-quality filtered web documents (Li et al., 2024). tokenized with the GPT-NeoX tokenizer (Black al., 2022) into sequences of length 8192, unless specified otherwise. We use grain² to prepare batches with sequence packing, particularly first-fit packing, which avoids splitting, but adds padding tokens.
Dataset of training runs. Using the above defined architecture and training recipe, we produce large dataset of training runs for our scaling law study totaling 640 individual runs (292 for Llama, 348 for xLSTM). The dataset contains model sizes ranging from 80M to 7B parameters trained with compute budgets ranging from 2.8 x 10 to 8.5 10 FLOPs on 2B to 2T tokens. This amounts to total compute budget spent for this dataset of 3.2 20 FLOPs. Our dataset is divided in into runs from two different training configurations: IsoFLOP and Token/Param. For the IsoFLOP configuration, vary model parameters and training tokens subject to fixed compute budgets for
http://oooogge.ggaiaaeeeetheeeee FirsFFinaackreerDattsstt))
4

---

Published as a conference paper at ICLR 2026
Transformr
XLSTM
3.4
3.4
3.2
8
Token/Param Ratio
Model Size
3.0
3.0
160M
44
400M
2.6
2.6
110
830M
Val dondooionn
2.4
Vali Valicaation
2.4
220
+
1.48
2.2
2.2
1100
7B
10th
10
10 10
10th
10
10
10a
2200
Compute (FLOPs) (log scale)
Compute (FLOPs) (lg scale)
Figure 3: Power law fits to loss over training compute with increasing loken-to-paramen
(Token/Param) ratios M. We fit power laws of the form in E(C) À C and observe that similar Transformer the exponents 17 of xLSTM remain constant even for large M, indicated by the parallel lines in the log-log plot.
three different context lengths. In the Token/Param configuration, vary the number of training tokens for set of fixed model sizes. We show our dataset as {N, D. C} points in Figure 2. XLSTM's linear scaling preserves training tokens with longer contexts (overlapping IsoFLOP points), whereas Transformer's 's quadratic scaling reduces them.
3.2 LOSS vs. COMPUTE: XLSTM IS
We begin our study with the question: Given fixed training compute budget, which model architecture performs better (in terms of cross-entropy loss)? To answer this ques- tion, we define a grid of model and dataset sizes with pre-defined token-to-parameter ratios of [22, 44, 110, 220, 550, 1100, 2200] and train Transformer and xL.STM models for each point in the grid. This forms the Token/Param subset in our dataset of training runs (see Sec. 3.1). We then use our FLOP calculations in Appendix B.3 and plot validation loss over FLOPs in log-log plot in Figure 1.
Pareto-frontier. In Figure 1 (left), we visualize the Pareto frontier by connecting the data points for XLSTM and Transformer. We find that xLSTM is strictly dominant over Transformers across the almost five orders of magnitude of compute encompassed by our data. In other words, for fixed FLOOP budget, xI STM models are better and for a fixed validation loss, they require less FLOPs.
Parametric loss surface fit. In Figure 1 (right), we fit parametric loss surface İ(N, D) to our Token/Param data. find that our of the loss surface provides reliable description of performance of Transformer and xLSTM models for given size even far in the over-training regime, i.c. far right to the pareto front. Following the practice of Busbridge et al. (2025), we find that including the parameter "y in the model of İ(N, D) improves the fit quality (see Fig. 8 in the Appendix). We provide additional details on parametric fits in Appendix A.2.
3.3 XLSTM IN THE OVERTRAINING REGIME: CONSISTENT POWER LAW EXPONENTS
Our parametric İ(N, D) fit predicts, that model quality in terms of loss improves when N D is increased. Hoffmann et al. (2022) have found that for Transformers, the optimal token-to-paramett ratio M D* /N* that yields the minimal loss under compute constraint is approximately 22. However, training runs with this ratio yield rather large models that are expensive and slow during inference (Sardana et al., 2024). Consequently, it is common practice train smaller models in an overtraining regime, i.e., with ooken-to-paramedt thus of practical importance to demonstrate that the loss of new model architectures continues to improve with increasing amounts of data.
ratios far exceeding the compute-optimal M
Power-law w exponents in over-training, Gadre al. (2024) have found that Transformers scale reliably in this over-training regime, indicated by constant exponents n, when fitting a power law of the form i(C) A C-" for different fixed token-per per--paaeeter ratios M. Therefore, we perform similar analysis and fit power laws i(C) to our Token/Param training runs. In Figure 3 and Tab. we find that similar to Transformer the exponents of xLSTM remain constant even for large M. indicated by the parallel lines in the log-log plot. This observation is relevant because it implies that small, inference-optimized xLSTM models can be trained on large datasets while still achieving consistent improvements in loss.
5

---

Published as a conference paper at ICLR 2026
1008
XLSTM
am0.547,A' A0
Compute
3.4
408
a=0.575. A0000002
6e+18
Model Parameters
1e+19
3.2
Validation
1e+20
3.0
6e+20
Training Runs
2.8
18
Transfomer
400M
2.6
FLOP Optima
100M
Transformer
100M
400M 1B
4B
10B
10
1040
10a1
10a2
102s
Model Parameters
Compute (FLOPS)
Figure 4: Varying model size and tokens with a fixed compute budget (IsoFLOP). Left: IsoFLOP profiles for varying number of model parameters with marker at the minimum N of the fitted polynomial. Right: Power-law fit (* = A' He for the compute optimal number of model parameters. Our setup reproduces the power-law exponent a for Transformers established in Porian et al. (2024). The compute-optimal model size of xLSTMs is larger than for Transformers.
3.4 COMPUTE-OPTIMAL XLSTM MODELS ARE LARGER
In this section, we aim to determine the compute-optimal model size N and dataset size D for the xL.TTM and Transformer models. However, so far, we have performed our scaling analyses on training configurations with preset model sizes and set of token-per-paunee per-parameter ratios M, which do not allow us to determine N and D directly. Therefore, for this analysis, we use the IsoPLOP training configuration, where we vary the number of model parameters and training tokens subject to a set fixed compute budgets H. For each compute budget, we plot the loss over the model parameters N and number of training tokens D and fit second-order polynomials to determine the optimal N*()) and D* (H) for each compute budget H. Using these optima, we then fit power laws as described in Section 2.2 to obtain the functional forms for N (H) and f) (H) (see Eq. (3)).
Compute-optimal model size. Figure (left) we show the IsoFLOP profiles for variable model size and (right) the corresponding power-law fits for the optimal model size for xLSTM and Transformer. Our results show that for given compute budget, xLSTM consistently attains a lower validation loss than Transformer, which is in line with the findings in Section 3.2. Moreover, we find that for a given compute budget, the corresponding computc-optine xLSTM models have more parameters than the corresponding Transformer models; Figure 4 (left and right). Note that power-law exponent for the Transformer matches the one found by Porian et al. (2024); see App. A.4 for details.
Compute-optimas dataset size. Analogous results are shown in Figure 9 in the appendix for the number of training tokens of compute- optimal models. We find that compute-optimal xLSTM and Transformer models are trained on a similar number of training tokens Đn (H). In Appendix E, we show the estimated optimal training FLOOPs and training tokens for various model sizes.
Universality of the relation between compute-optimal performance and model size. The compute- optimal models in Figure 4 (left) fall close to a single shared line for the Transformer and xLSTM models. This suggests that for compute-optimal models, there is a universal relationship between performance and model size for xLSTM and Transformer models. From this perspective, the fact that compute-optimal xLSTM models are larger for a given compute budget can be regarded as heuristic expl explanation for the superior performance of XTM. The reason why xL.STMs can be larger is the reduced computational complexity of their recurrent sequence-mixing operation compared to the self-attente attention operation Transformers. As this main operation is cheaper, more compute can be allocated to the rest of the model, e.g. increased number of layers or embedding dimension.
3.5 COMPUTE-OPTIMAL XLSTM MODEL SIZE REMAINS STABLE ACROSS CONTEXT LENGTIIS
The main difference between the model architectures in this study is their scaling in FLOPs with context length: Transformers scale guadratically, due to the self-attention, while xLSTMs scale linearly. This implics that, in Transformers, an increasing fraction compute is devoted to attention sequence length grows, whereas in xI. SLMS the recurrent updates consume only modest portion

---

Published as a conference paper at ICLR 2026
Compute
Training Runs
PLOP Optima Tranafomer XLSTM
6e+18
Transformer
Transformer (8192) --0.577.A''0000
Transforerr 554, A'00,077
XLSTM (8192) am0.541, A 016
SL.TM (2048) 556,
1e+19
3es19
XISTM
Transformer 1163384] am0.569, Almo..03
xLSTM (163884) 0.5.6. 0.005
Context Length 2048
Context Length B1
Context L Length 16384
Motrel Peremeters
3.5
3.5
3.5
3.4
3.3
3.2
400M
9.1
3.1
3.1
3.0
3.0
8.0
300M
RU8
2.9
2.9
2.9
2.8
2.8
0.00M
100M
400M
100M
400M
100M
400M
18
10
10a
Model Parameters
Model Parameters
Model Parameters
Compute (FLOPs)
Figure 5: Left: IsoFLOP curves as function of model parameters at 3 different context lengths. Right: Plot of the power-law fits for the compute optimal number of parameters dependent on the compute budget N (H). Colors indicate compute budget and marker types indicate the model types. The compute optimal model size for Transformers gets smaller for larger context lengths, while the compute optimal model size for xLSTM remains similar across context lengths.
of the total compute. In this section, we investigate, therefore, the impact of the context length on compute-optimal model and dataset sizes. We add experiments with context lengths 2048 and 16384 in the IsoFLOP training configuration and then fit the power-laws to each context length for both models, analogously Section 3.4. We note that the losses are not directly comparable across different context lengths since we use sequence packing for the construction of our training and validation datasets. Hence, for larger context lengths, longer documents can be packed into batch, effectively changing the data distribution.
Context length & compute-optimality. In Figure we show the IsoFLOP profiles for varying model sizes and three different context lengths and compute budgets, including their power-law fits N (H) in the rightmost plot. We observe that with increasing context lengths the compute-optimal model size of Transformers drops significantly, while for xLSTM it drops only mildly. These results suggest that for Transformers, growing fraction of compute is consumed by attention operations as sequence length increases, whereas in xLSSMMs most FLOPS remain allocated to depth and hidden dimensions. In Figure 10 in Appendix A.5 we show the corresponding IsoFLOP profiles and power-law fits De (H) for the optimal number of training tokens. We observe similar trends for the model size: The compute-optingg number of training tokens decreases markedly with larger context length for Transformer models and for xLSTM it slightly increases.
INFERENCE SCALING BEHAVIOR
The scaling laws analysis in Section 3 is motivated by the goal of the optimal design of pre-training runs for LLMs. However, these considerations neglect inference efficiency. When deploying LLMs at large scale, inference costs and performance critical aspects. Hence Pope et al. (2023) investigate the inference efficiency of transformer-bast
LLMs in terms of three criteria: compute, latency, and throughput. More recently Sardana et al. (2024) provided scaling law analysis of Transformers that extend the pre-training compute optimality consideration (Eq. (I)) to also account for inference compute. This work presents even more comprehensive analysis in terms of the attainable latency, iee, time to first token, and the step time during generation. We complement our empirical findings with a quantitative model of a lower bound on time to first token and step time, using the detailed calculation of FLOPs (App. B.3) and MemOps (App. B.4) for both model architectures.
Inference stages. Typically, large-scale L.MM inference is split into the prefill and the generation stage (Austin et al., 2025; Pope et al., 2023; DeepSeek-AI, 2024b). In the prefill stage the LLMS process the prompt, compute the logits for the first token to be generated, and store the intermediate internal representations of the prompt, i.e. the KV cache for Transformer models or the mLSTM cell states for xLSTM. In the generation stage a token is sampled according to the logits and then the internal representations of the previous tokens in the context window are updated to account for the new token. The generation procedure is repeated for a certain budget or until the end-of- -eeuuenee token is sampled. In the following, investigate the prefill and generation performances separately.
7

---

Published as a conference paper at ICLR 2026
202
Taanformer 160M
Time to First Taken (ms)
Step Time (maistop)
Baasfoome 830M
10a
104
Transtomer 140
Tarsdomme 2.70
-- Transtomer 78
XLSTH 16ON
10:
XLTHH B30N
XLSTH 1.46
10a
100
XLTTM 2:78
XLSTH 78
10a
10
10"
10 10" 10
10'
10a
Prefill Length
Prefill Length
Figure 6: Scaling of TTFT (left) and step time (right) as a function of prefill length (1-16k) for different model sizes, with a batchsize of one.
Inference runtime metrics. For the prefill stage, the key performance metric is the time to first token (TTFT). Prefill speed is primarily determined by how well the model can maintain a low TTFT while handling large batch sizes and long input sequences. During the generation stage, the key performance metric is the step time, i.e. how long takes to obtain the next token given the current (potentially batched) sequence. For Transformers, the quadratic complexity of the attention mechanism with respect to the prefill length (App. B.3.3) implies that TTFT is expected to scale quadratically in terms of the prefill length. In terms of step time we expect linear scaling with respect to the prefill length, as each decoding step involves attention over the entire KV cache. For xLSTMs, in contrast, we expect linear scaling of TTFT and step time that is independent of the prefill length.
4.1 EMPIRICAL INFERENCE RUNTIMES
We consider the same model architectures as in the Token/Parum configuration (see Tab. 19 and 20). We utilize the implementation of xLSTM and Transformers models available through the trans formers library (Wolf et al., 2020) and optimize runtimes using torch. compil le and torch, cuda londd.grapee graph. The TTFT is measured as the time needed for generating a single token under given batch size and prefill length (i.e., the context length). The step time is measured by generating sequence of 100 tokens, subtracting the TTFT and dividing by the sequence length. We measure the average TTFT and step time over four repetitions after two warm-up iterations.
Figure presents TTFT (left) and step time (right) measurements for both architectures at matched model sizes as a function of prefill length (1-16k). At short prefills, the two model classes exhibit comparable TTFTs, while at longer prefills xLSTMs consistently achieve lower values. For 16k prefill, xLSTM has 30-50% lower TIFT for the same model size. This difference reflects the expected scaling: quadratically for Transformers and linearly for xLSTMs. A similar trend is observed for the step time. At small prefills, both architectures perform comparably. As the prefill length increases, the Transformer step time degrades due to the rising cost of attention over longer KV caches. In contrast, xLSTM step time is independent of prefill length, resulting in consistently higher throughput across all evaluated model sizes and prefill lengths. For 16k prefill, the largest xLSTM has lower step time than the smallest Transformer we considered. In summary, when matched in model size, XLSTMs outperform Transformer models all inference speed metrics considered.
4.2 MODELING INFERENCE RUNTIMES
In our analysis, the inference processes are characterized by the associated number of floating point operations FLoPs and the number of memory operations Bytes measured in bytes that are read or written. We provide calculations of these two quantities for xLSTTM and for Transformers in Appendix B. Importantly, these calculations capture the difference between xLSTM and Transformers in the dependence of FLOPsalgo and Bytesmee.alg on the context length T. Based on these calculated quantities, model the runtimes associated with the floating point and memory operations as:
FLOPsalgo
Bytes Ban
T FLLOPs.dgg
+ E,
T mem,algo
+
where a etr is the effective rate of FLOPs/s, B is the effective rate of Bytes/s, and is constant overhead when running the inference processes on the GPU. Depending on the model type, model size, prefill length, batch size and inference stage (prefill or generate), either 7 FLoPsalgo memaaggo is
8

---

Published as a conference paper at ICLR 2026
the dominant contributor to the runtime. We outline in Appendix C.I how this is determined based on the roofline model. Using empirical runtime measurements, we then fit one of the two models depending on which one is expected to yield the dominant runtime contribution. Each fit corresponds to a specific model type, size, and inference stage, and is evaluated over varying batch sizes and prefill lengths. As evidenced by the fits empirical TTFT (App. C.2) and step time measurements (App. C.3), our model provides an accurate description of the observed inference runtimes for both architectures and explains the empirically observed runtimes in Figure 6.
5 RELATED WORK
Modeling scaling behavior with parameters and data. The empirical scaling behavior of Deep Learning models w.r.t the size of their model parameters and training data has been actively researched (Hestness et al., 2017; Rosenfeld et al., 2020; Henighan et al., 2020; Alabdulmohsin et al., 2022; Caballero al., 2023). Such scaling laws have been demonstrated across many tasks and data modalities (Tan & Le, 2019; Ghorbani et al., 2022; Zhai et al., 2022; Abnar et al., 2022; Ardalani al., 2022: Gao et al.. 2023) However, beginning with Kaplan al. (2020) and Hoffmann al. (2022), the main objective has been guidance on how to optimally scale Large Language Models with Transformers. Follow-up work investigated the data constrained setting (Muennighoff et al., 2023). the effect of data pruning (Sorscher et al., 2022), extreme token per parameter ratios (Gadre et al., 2024). Furthermore, replication efforts regarding the scaling laws established Kaplan et al. (2020) and Hoffmann et al. (2022) have been performed in order to reconcile their findings (Besiroglu et al., 2024; Pearce & Song, 2024; Porian et al., 2024). Critical practical considerations such as specific architectures and hyperparameters on the resulting scaling laws have been investigated (McLeish et al., 2025). The recent survey Li et al. (2025) gives a comprehensive overview and give practical guidelines in establishing scaling laws. Scaling laws have also been investigated theoretically, providing justification for the functional forms used in practice (Amari et al., 1992; Amari, 1993; Seung al., 1992; Amari & Murata, 1993; Cortes et al., 1993; Yarotsky, 2018; Liang et al., 2020; Sharma & Kaplan, 2022; Hutter, 2021; ; Bahri et al., 2024).
Incorporating inference characteristics into scaling laws. Multiple studies seek include infer- ence characteristics such as the foirsttofistt (latency) and the time-per-token (throughput) into their considerations on model scaling. Sardana al. (2024) propose to incorporate inference costs into scaling laws for an expected inference compute demand. Gadre et al. (2024) investigate scaling laws in training regimes with high token/paramettr ratios, much higher than "Chinchilla-optimal" which incurs higher inference speeds due to smaller models. Bian et al. (2025) devise inference- scaling laws, focusing on obtaining the most inference efficient model for a certain performance. Paliotta et al. (2025) show, that under fixed time budget during inference, distilling Transformers into linear complexiy Mamba models leads to higher performance on reasoning tasks, as their faster inference speeds allow for better scaling with inference compute.
Other scaling behaviors. Beyond scaling behavior with model parameters and training data, other scaling behaviors have been investigated. Hernandez et al. (2021) considers scaling laws for transfer learning. Clark et al. (2022) and Abnar et al. (2025) investigate scaling laws for routed language models, such as the widely considered -fxture-of-Experti method (Shazeer et al., 2017). Scaling inference compute is a major consideration for L.M reasoning models (OpenAI, 2024). For example Snell et al. (2025); Brown et al. (2024); Muennighoff et al. (2025) demonstrated such scaling behavior with additional inference tokens. Kumar et al. (2025) devise precision-aware scaling laws, investi- gating the tradeoff's between precision, parameters and data. Tao al. (2024) suggest the vocabulary size as additional parameter when scaling language models. Busbridge et al. (2025) investigate scaling laws for distilled models based on the compute budget allocation between teacher and student. Zhao et al. (2025) reconcile the smooth improvements predicted by scaling laws with the reported sudden emergent capabilities of L.MMs at scale through distributional scaling laws. Chen et al. (2025) introduce parallel scaling laws, where compute is scaled by using single set of model parameters in parallel with different learnable input transformations and output aggregation. Related to our work, Xiong et al. (2024) and Shi et al. (2025) investigate the scaling behavior of transformer models w.r.t. their context length. Springer et al. (2025) show that overtrained models are harder to fine-tune.
Closest to work are Shen et al. (2024) and Poli et al. (2024). Shen et al. (2024) demonstrate scaling behavior of their considered linear complcompy architectures that is on par with Transformers. Poli et al. (2024) shows, that hybrids between linear time-complexity and transformer models can

---

Published as a conference paper at ICLR 2026
improve upon Transformers. Contrary, work shows that the xLSTM linear time-complexity architecture outscales Transformers for language modeling.
6 LIMITATIONS AND FUTURE WORK
The main focus of this work is comparative study of the training scaling behavior of Transformer and xLSTM architectures in terms of cross-entropy loss. We not consider the impact of different training data distributions, nor do we investigate scaling behavior other downstream tasks; instead, we build on the findings of related work on these aspects (Sardana et al., 2024; Gadre et al., 2024; Porian et al., 2024). Similarly, our empirical inference runtime scaling is designed capture the fundamental differences in computational complexity with respect to sequence length between Transformers and xLSTM. Therefore, adopt fair and controlled comparative setup, focusing single-GPU experiments rather than exhaustive inference optimizations.
hybrid architectures combining attention and XLSTM, explore diverse data distributions, include additional downstream and long-context tasks, and investigate inference runtimes in production scale multi-GPU regimes to provide further insights into efficient sequence modeling.
Future work could extend the scaling comparisons Mixture-of-
7
CONCLUSION
Our study provides systematic comparison of scaling behaviors between xLSTM and Transformer architectures. We show that xLSTMs are Pareto-dominant in training loss versus compute, maintain consistent power-law exponents in the overtraining regime, and scale more efficiently with context length due to their linear complexity. While our results suggest universal relationship between performance and model size that applies both compute-optimal Transformers and xLSTM models, we find that compute-optimal XLSTM models are larger than their Transformer counterparts and that the compute-optimal model size of xLSTMs robust to variations in context length. During inference, XLSTM models achieve lower time to first tokens and generation step times than Transformer models of the same size. These results are well explained by our runtime model, which is grounded in theoretical FLOOP and memory operation calculations and shows close agreement with the empirical data. Throughout all experiments, we find that the advantages xLSTM grow with context length, both for training and inference characteristics, positioning xLSTM promising and scalable architecture for future language models.
REPRODUCIBILITY STATEMENT
We release the code to reproduce our experiments, the datasets of training runs as well as results for inference publicly upon acceptance to facilitate future research in this direction. The datasets of training runs have been obtained using the publicly available xL.TMM 7B training repository (https: / //github. MIm/NX-AI// A.IIxxxsti xlstm- mmmjjxj) using the model configurations stated in Appendix D. Inference results have been obtained using the publicly available benchmarking pipeline for efficient more specifically, the
XLSTM kernels (https: / /gitt hub. com/NX AIImII/m.. model benchmarks, not those for individual kernels.
REFERENCES
Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the Limits of Large Scale Pre-training. In International Conference Learning Representations (ICLR), 2022.
Samira Abnar, Harshay Shah, Dan Busbridge, Alaaeldin El-Nouby, Joshua M. Susskind, and Vimal Thilak. Parameters FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of ceature-offExpeeeee Experts Language Models. International Conference on Machine Learning (ICML), 2025.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 Technical Report. ArXiv,
2023.
10

---

Published as a conference paper at ICLR 2026
Joshua Ainslie, James Lee-Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training Generalized multi-Query Transformer Models from Multi-Head Check- points. In Conference on Empirical Methods in Natural Language Processing (EMNLP). Associa- tion for Computational Linguisties, 2023.
Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting Neural Scaling Laws in Language and Vision. In Advances Neural Information Processing Systems (NeurIPS), 2022.
Shun-ichi Amari. A universal theorem on learning curves. Neural Networks, 1993.
Shun-ichi Amari and Noboru Murata. Statistical Theory of Learning Curves under Entropic Loss Criterion. Neural Computation, 1993.
Shun-ichi Amari, Naotake Fujita, and Shigeru Shinomoto. Four Types of Learning Curves. Neural Computation, 1992.
Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, and Adnan Aziz. Under- standing Scaling Laws for Recommendation Models. ArXiv, 2208.08489, 2022.
Jacob Austin, Sholto Douglas, Roy Frostig, Anselm Levskaya, Charlic Chen, Sharad Vikram, Federico Lebron, Peter Choy, Vinay Ramasesh, Albert Webson, and Reiner Pope. How Scale Your Model. Online, 2025.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 2024.
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xLSTM: Extended Long Short-Term Memory. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Maximilian Beck, Korbinian Pöppel, Phillip Lippe, and Sepp Hochreiter. Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels. In Advances in Neural Information Processing Systems (NeurIPS), 2025a. URL ht tuttps::0 :ttptt::://ppeeeeee http///ppeneeeeeeeee/e net/ forum 2dd-b6H6446614
Maximilian Beck, Korbinian Pöppel. Phillip Lippe, Richard Kurle, Patrick M. Blies, Günter Klam- bauer, Sebastian Böck, and Sepp Hochreiter. xL.TTM 7B: A Recurrent LLM for Fast and Efficient Inference. ArXiv, 2503. 2025b.
Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla Scaling: A replication attempt. ArXiv, 2404.10102. 2024.
Song Bian, Minghao Yan, and Shivaram Venkataraman. Scaling Inference-Efffiienn Language Models. ArXiv, 2501. 18107, 2025.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An Open-Source Autoregressive Language Model. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models, 2022.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V. Le, Christopher Ré, and Azalia Mirhoseini. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. ArXiv, 2407.21783 2024.
Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distillation Scaling Laws. ArXiv, 2502.08606, 2025.
Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken Neural Scaling Laws. In International Conference on Learning Representations (ICLR), 2023.
Mouxiang Chen, Binyuan Hui. Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Jianling Sun, Junyang Lin, and Zhongxin Liu. Parallel Scaling Law for Language Models. ArXiv, 2505. 10475, 2025.
11

---

Published as a conference paper at ICLR 2026
Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc. Blake Hechtman. Trevor Cai, Sebastian Borgeaud, George Bm Van Den Driess- che. Eliza Rutherford, Tom Hennigan, Matthew J Johnson, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Marc' Aurelio Ranzato, Jack Rae, Erich Elsen, Koray Kavukcuoglu, and Karen Simonyan. Unified Scaling Laws for Routed Language Models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research. PMLR, 2022.
Corinna Cortes, L. D. Jackel, Sara Solla, Vladimir Vapnik, and John Denker. Learning Curves: Asymptotic Values and Rate of Convergence. In Advances in Neural Information Processing Systems (NeurIPS), 1993.
Tri Dao. FlashAttention-2:: Faster Attention with Better Parallelism and Work Partitioning. In International Conference on Learning Representations (ICLR), 2024.
DeepSeek-AL. DeepSeek- V2: A Strong, Economical, and Efficient Mixture-ou of--exerts Language Model. ArXiv, $405504434, 2024a.
DeepSeek-AI. DeepSeek-V V3 Technical Report. ArXiv, 2412.19437, 2024b.
Samir Yitzhak Gadre, Georgios Smymis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Alexandros G. Dimakis, Gabriel Ilharco, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt. Language models scale reliably with over-training and on downstream tasks. ArXiv, 2400.08540, 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization. In International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research. PMLR, 2023.
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling Laws for Neural Machine Translation. In International Conference on Learning Representations (ICLR), 2022.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and et al. The Llama 3 Herd of Models. ArXiv, 2407.21783 2024.
Albert Gu and Tri Dao. Mamba: Linear-Tine Time Sequence Modeling with Selective State Spaces. In International Conference on Learning Representations (ICLR), 2024.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy. Benjamin Mann, Alec Rad- ford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling Laws for Autoregressive Generative Modeling. ArXiv, 2010. 2020.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling Laws for Transfer. ArXiv, 2102.01293, 2021.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep Learning Scaling is Predictable, Empirically. ArXiv, 1712.00409, 2017.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training Compute-Optimal Large Language Models. ArXiv, 2203. 2022.
Mareus Hutter. Learning Curve Theory. ArXiv, 2102.04074, 2021.
12

---

Published as a conference paper at ICLR 2026
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. ArXiv, 0001.08361, 2020.
Tanishq Kumar, Zachary Ankner, Benjamin F. Spector, Blake Bordelon, Niklas Muennighoff, Man- sheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling Laws for Precision. In International Conference on Learning Representations (ICLR), 2025.
Jeffrey Li, Alex Fang, Georgios Smymiss, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng- Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco. Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang. Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby. Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. DataComp-LM: In search of the next generation of training sets for language models. ArXiv, 2406.11794, 2024.
Margaret Li, Sneha Kudugunta, and Luke Zettlemoyer. (MitiFitting: A Survey of Scaling Laws. In International Conference Learning Representations (ICLR), 2025.
Tengyuan Liang, Alexander Rakhlin, and Xiyu Zhai. On the Multiple Descent Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels. In Proceedings of Thirty Third Conference on Learning Theory. PMLR, 2020.
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwards Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Frez Shwartz, Mor Zusman, and Yoav Shoham. Jamba: A Hybrid Transformer-Mambe Language Model. ArXiv, 19887, 2024.
Sean McLeish, John Kirchenbauer, David Yu Miller, Siddharth Singh, Abhinav Bhatele, Micah Goldblum, Ashwinee Panda, and Tom Goldstein. Gemstones: A Model Suite for Multi-Faceted Scaling Laws. ArXiv, 2502.06857, 2025.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus. Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling Cata-Coontr Language Models. In Advances in Neural Information Processing Systems (NeurIP 2023.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. ArXiv, 2501.19399 2025.
OpenAl. Learning to reason with LLMs. 2024. URL https::/ con/deex// Learning-to-resson: witassonwwtt-hlmee/.
Daniele Paliotta, Junxiong Wang, Matteo Pagliardini, Kevin Y. Li, Aviv Bick, J. Zico Kolter, Albert Gu, François Fleuret, and Tri Dao. Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasoners. ArXiv, 2502.20339, 2025.
Tim Pearce and Jinyeop Song. Reconciling Kaplan and Chinchilla Scaling Laws. Transactions on Machine Learning Research. 2024.
Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Bjöm Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, Ce Zhang, and Stefano Massaroli. Mechanistic Design and Scaling of Hybrid Architectures. ArXiv, 2403. 17844, 2024.
Reiner Pope. Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently Scaling Transformer Inference. In Conference on Machine Learning and Systems (MISys), 2023.
13

---

Published as a conference paper at ICLR 2026
Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving Dis- crepancies in Compute-Optinat Scaling of Language Models. In Advances Neural Information Processing Systems (NeurIPS), 2024.
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A Constructive Pre- diction of the Generalization Error Across Scales. In International Conference on Learning Representations (ICLR), 2020.
Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond Chinchilla-optimal: accounting for inference in language model scaling laws. In International Conference on Machine Learning (ICML), 2024.
H. S. Seung, H. Sompolinsky, and N. Tishby. Statistical mechanics of learning from examples. Phys. Rev. A, 1992.
Utkarsh Sharma and Jared Kaplan. Scaling Laws from the Data Manifold Dimension. Journal of Machine Learning Research (JMLR), 2022.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously Large Neural Networks: The Sparsely-Gater Mixture-of-Experts Layer. In International Conference on Learning Representations (ICLR), 2017.
Xuyang Shen, Dong Li, Ruitao Leng, Zhen Qin, Weigao Sun, and Yiran Zhong. Scaling Laws for Linear Complexity Language Models. ArXiv, 2406. 16690, 2024.
Jingzhe Shi, Qinwei Ma, Hongyi Liu, Hang Zhao, Jeng-Neng Hwang, and Lei Li. Explaining Context Length Scaling and Bounds for Language Models. ArXiv, 2502.01481, 2025.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Parameters for Reasoning, In International Conference on Learning Representations (ICLR), 2025.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Jacob Mitchell Springer, Sachin Goyal, Kaiyue Wen, Tanishq Kumar, Xiang Yue, Sadhika Malladi, Graham Neubig, and Aditi Raghunathan. Overtrained Language Models Are Harder to Fine- Tune. In International Conference on Machine Learning (ICML), 2025.
Mingxing Tan and Quoe Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research. PMLR, 2019.
Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama Open Foundation and Fine- Tuned Chat Models. ArXiv, 2307.09288. 2023.
14

---

Published as a conference paper at ICLR 2026
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Samuel Williams, Andrew Waterman, and David Patterson. Roofline: an insightful visual performance model for multicore architectures. Commun. ACM, 2009.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick VOn Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-Ar Natural Language Processing. In Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2020.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajiwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang. Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale. Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective Long-Context Scaling of Foundation Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), 2024.
Dmitry Yarotsky. Optimal approximation of continuous functions by very deep ReLU networks. In Proceedings of the 31st Conference On Learning Theory. PMLR, 2018.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling Vision Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Rosie Zhao, Tian Qin, David Alvarez- Sham Kakade, and Naomi Saphra. Distributional Scaling Laws for Emergent Capabilities. ArXiv, 2502.17356 2025.
Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem, Ilyas Chahed, Younes Belkada, Guillaume Kunsch, and Hakim Hacid. Falcon Mamba: The First Competitive Attention-free 7B Language Model. ArXiv, 2410.05355, 2024.
15

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
1
Introduction
2
Preliminaries
2.1
Background on Scaling Laws
2.2
Fitting Scaling Laws
3
Training Scaling Behavior
3.1
Experimental Setup
3.2
Loss Compute: xLSTM is Pareto-Dominant
3.3
XLSTM in the Overtraining Regime: Consistent Power Law Exponents
3.4
Compute-Optimal xL.TTM Models are Larger
3.5
Compute-optimal xLSTM model size remains stable across Context Lengths....
4
Inference Scaling Behavior
4.1
Empirical Inference Runtimes
4.2
Modeling Inference Runtimes
5
Related Work
6 Limitations and Future Work
10
7 Conclusion
10
A Extended Training Scaling Behavior
18
A.I Details on the Experimental Setup
18
A.2 Details on the Parametric Loss Surface Fit
18
A.3 Power-Law Exponents in Over-Training
20
A.4 Additional Results: IsoFLOP Approach
20
A.5 Additional Results: IsoFLOP Approach for Different Context Lengths
Accounting:
Parameters, Cache Sizes, FLOPs, Memory Operations
23
B.1 Parameter Counts
24
B.I.1
mLSTM Params
24
B.1.2
Transformer Params
24
B.2
Memory State and KV-Cache Size
24
B.3
FLOP Counts
25
B.3.1 mLSTM Cell FLOPs
25
B.3.2
mLSTM Model FLOPs
26
16

---

Published as a conference paper at ICLR 2026
B.3.3
Self-Attention FLOPs
27
B.3.4
Transformer Model FLOPs
28
B.4
Memory Operation Counts
29
B.4.1
mLSTM Cell MemOps
29
B.4.2
mLSTM Model MemOps
29
B.4.3
Self-Attention MemOps
30
B.4.4
Transformer Model MemOps
31
Modeling Inference Characteristics
32
C.1 Background: Theoretical Runtime
................................
32
C,2 Prefill Stage: Time First Token
................................
33
C.3 Generation Stage: Step Time
34
Model Configurations
36
D.1 Model Sizes and Hyperparameters in Token/Param Configuration
36
D.2 Model Sizes and Hyperparameters in IsoFLOP Configuration
37
Compute Optimal Parameter, Token and FLOP Count Estimates
39
E.1 Compute Optimal Configurations for Context Length 8192
39
E.2 Compute Optimal Configurations for Varying Context Lengths
40
17

---

Published as a conference paper at ICLR 2026
A EXTENDED TRAINING SCALING BEHAVIOR
A.1 DETAILS ON THE EXPERIMENTAL SETUP
We provide additional details our experiments, that we conducted cluster of NVIDIA H100 GPUs.
Model Configurations. Appendix we provide a list of model architecture configurations for all Transformer and XLSTM models used in our scaling law study in Token/Param (App. D.I) and IsoFLOP (App. D.2) training setups.
General Hyperparameters. We use the AdamW optimizer with 31 = 0.99, 52 = 0.95, e = 10 weight decay 0.1 and gradient clipping norm 0.5. Our learning rate schedule comprises three stages: A linear warm-up of 750 training steps, a cosine decay to 10% of the peak learning rate and final linear cool-down of 1000 training steps. While we keep the steps for warm-up and cool-down constant, match length of our learning rate decay the token budget, which is either determined by specific to-aarmeteerr ratio or a compute budget for a given model size (see Sec. 3.1). Unless specified otherwise, we use context length of 8192 for our scaling law study.
Hyperparameters for Token/Param setup. We specify our batch sizes and learning rates for token-to-parameter experiments in the overtraining regime with large token-to- parameter ratios for xLSTM and Transformer models in Tab. 19 and 20, respectively. For larger models we decrease the learning rate and use larger batch sizes. We find that for very large token-to-parameter ratios the performance in terms of validation loss becomes less sensitive to the choice of learning rate.
batch size of 1M tokens for all but the largest compute budget of 6e+20 FLOPs, where we double the batch size to 2M tokens, as the training runs would become prohibitively long (see Tab. 1). In contrast to the Token/Param experiments, we do not increase the batch size with model size, since we found that this leads to loss offsets the isoflop profiles (see Fig. left). Instead, we keep the batch size constant for each compute budget, regardless of the model size. We validate this choice by repeating the experiments for the isoflop profile with compute budget le+20 with batch size of 1M and 2M tokens. We find that the larger batch size yields higher validation loss due to fewer training steps, but does not have a major impact on the optimal number of parameters N for this compute budget (see Fig. 7, right). Starting from the Token/Param learning rates, we tune the learning rates for selected model sizes, and use the best learning rates for models of similar size.
Hyperparameters for IsoFLOP setup. For our IsoFLOP experiments
Table 1: Batch sizes used for the IsoFLOP training setup at context length T 8192. For the other context lengths T we adjust B such that batch size in number of tokens B T remains constant.
IsoFLOP
B (seqs)
B x T (tokens)
6c+18
128
1,048,576
le+19
128
1,048,576
3e+19
128
1,048,576
le+20
128
1,048,576
6e+20
256
2,097,152
A.2 DETAILS ON THE PARAMETRIC LOSs SURFACE FIT
For the parametric loss surface fit İ(N, D) in Figure we follow the procedure outlined in Bus- bridge et al. (2025, App. F.I). We fit the coefficients { {E, A, B, a, 7 for the parametric func- tion of the loss surface L(N, D) in (2) with different values for the Huber 8. Similar to Bus- bridge et al. (2025), we observe that including 7, significantly improves the quality of our fits (see Fig. 8, We use the the Token/Param training configurations for Transformer (31 samples) and xLSTM (35 samples) from our dataset of training runs and fit over a grid of L-BFGS-B B initializations given by: log A {0.0, 5.0. 10.0, 15.0. 20.0), log B E {0.0, 5.0. 10.0, 15.0, 20.0). log E E { {-1.0, -0.5, 0.O, 0.5, 1.0), a E {0.0, 0.2, 0.5, 1.0), 3 E (0.0,0.2, 0.5, 1.0] and e 00.0, 0.5, 1.0, 1.5).
18

---

Published as a conference paper at ICLR 2026
3.2
3.2
3.1
3e+119
1e++20
3.0
Validation
Validation
Batch Size
2.9
2.9
4.
128
2.8
2.8
x
256
2.7
2.7
2.6
100M
400M 1B
2B
4B
100M
400M 1B
2B
4B
Model Parameters
Model Parameters
Figure 7: Impact of the batch size on IsoFLOP profiles. Left:
IsoFLOP curves with large batch size and different learning rates for large models. Varying the batch size for different model sizes, leads to offsets in the IsoFLOP profile, which are more pronounced for smaller compute budgets. Right: IsoFLOP profile for compute budget 1e+20 with different batch sizes. The larger batch size leads to larger loss, but similar optimal model size.
Tab. 2, we report coefficients that achieve the lowest MSE on the data out all initializations for different Huber 5. We find that the optimal fit parameters are sensitive to the choice of 6. For > 0.1 the optimal values for the fit parameters did not change in the digits shown in Tab. 2.
Table 2: Optimal fit parameters for the loss surface İ(N, D) model from equation (2) for Transformer and XLSTM models for different Huber 6. In Figure I we plot the fit for = 10
Huber 5
log A
log B
log E
a
8
10-5
12.96
14.35
0.05
0.58
0.55
0.28
Transformer
10
11.99
13.35
0.01
0.53
0.51
0.29
10
14.45
16.33
0.09
0.64
0.63
0.25
10-5 5
16.13
17.10
0.07
0.71
0.66
0.24
XLSTM
10-3
16.22
17.31
0.11
0.73
0.67
0.24
> 10
15.46
16.53
0.18
0.71
0.65
0.26
Model Sizes
160M
3.2
3.2
Varidation Loss (eg scaKe
3.0
3.0
1.48
2.78
Points
Thansformer Data Points
xLSTM Data hoints
3.4
2.4
LAN, B) Fit Function
Chinchila
Divillation Scaling Law (y fitted!
10
100
200
500
2000
Token Param Ratio Blog scale)
Taken Param Ratio (log scale)
Figure 8: Comparison between the parametric fit with (Hoffmann al., 2022) and 7 as free parameter (Busbridge et al., 2025). Including as fit parameter improves the fit quality.
19

---

Published as a conference paper at ICLR 2026
A.3 PAW EXPONENTS IN
In Tab. 3 we report the power-law exponents for different token-to-parameter ratios.
Table 3: Power-law exponents 1/ for increasing token-to-parameter ratios M.
M
Transformer
xLSTM
22
0.050
0.047
44
0.048
0.046
110
0.047
0.046
220
0.048
0.047
550
0.049
0.047
1100
0.047
A.4 ADDITIONAL. RESULTS: ISOFLOP APPROACH
Comparison of our scaling law to Porian al. (2024). In order to validate our scaling law framework, we compare our power-law fits for the optimal model size from Fig. 4 with the results from Porian al. (2024). Porian al. (2024) investigate and resolve the discrepancies in scaling laws between the influential works by Kaplan et al. (2020) and Hoffmann et al. (2022). We find that our power-law coefficient Gours 0.575 is very close to the coefficient reported in Figure Id) from Porian et al. (2024) with dpoorian,d 0.571 and even falls well into their confidence interval of (0.56, 0.59), despite the well-documented reproducibility challenges in scaling laws (Porian et al., 2024; et al., 2025; McLeish al., 2025). Porian et al. (2024) report that for their aporian.d they match their learning rate cosine decay schedule to each token budget a practice that we follow our experimental setup (see App. A. This agreement validates
framework and affirms its credibility. As the final step, to fully match the coefficients reported by Hoffmann et al. (2022), Porian et al. (2024) report that is necessary tune learning rate, batch size and AdamW W 32 parameter individually for each model size. However, in our case this would require considerably more compute resources due to our much larger compute budgets (6e+18- 6e+20), and hence larger model sizes used for our scaling law study.
Compute-optimal dataset size. In the main paper (Sec. 3.4, Fig. 4), we presented results for the compute-optin model size. In Fig. 9 we present results w.r.. the number of training tokens. We observe that compute-optimal XSSSM and Transformers are trained on similar number of tokens.
4008
b=0.417, 7'-77.48
Compute
3.4
2008
Transformer b- 0-00,44, 8'-58.446 460
6e 61818
Validation Lose
Training Tokens
1e+19
3.2
1008
3e+19
1e+20
3.0
408
6e+20
Training Runs --- Transformer
2.8
208
108
XLSTM
FLOP Optima
4B
Transformer
48
100B200B
101b
1020
10
1022
1023
XLSTM
Training Tokens
Compute (FLOPs)
Figure 9: Left: IsoFLOP curves for varying number of training tokens with marker the minimum of the fit. Right: Plot of the power-law for the compute optimal number of training tokens (C) Colors indicate compute budget and marker types indicate the model types.
20

---

Published as a conference paper at ICLR 2026
A.5 ADDITIONAL RESULTS: ISOFLOP APPROACH FOR DIFFERENT CONTEXT LENGTHS
Complementary to the IsoFLOP results in Sec 3.5, where we showed scaling behavior w.r.t. the model parameters, we also show the scaling behavior w.r.t. the dataset size. The results are provided in Figure 10, showing that for xLSTM it slightly increases with context length, whereas for Transformer substantially decreases. This is caused by the quadratic cost of the attention mechanism that becomes dominant at larger context lenghts, causing substantial compute that shifts compute-optimat models towards smaller models that are trained on less tokens. For all considered context lenghts, is favorable to train an xLSTM model compared to a Transformer model under the same compute budget. The longer the training context length, the more favorable is to train an xLSTM compared to Transformer.
Compute
Training Runs
FLOP Optima
6w+18
Transfomerr
Transfamer
Transfomer (2008) b-0.423, 87-74
Transformer (8199) b=o0 A20. 0' 962
XLSTM (8392) b-0 429, (1's 803
XLSTM (2048) b- 8'-140.61 611
1e+19
XLSTM
XLSTM
Transfomer (16384) 0-0.433, 8'-34.478
XLSTM (266364) b-0.385, 8'-336.096
3e+19
Context Length 2048
Context Length 0192
Context Length 16384
3.5
3.5
3.5
Optimal Training Tokens
220
3.4
3.4
3.4
3.3
109
3.0
3.0
3.0
2.9
2.9
2.9
2.8
2.8
2.0
10B 208 Tokens
2B
108
408
408
109
10
Training Tokens
Compute (FLOPs)
Figure 10: IsoFLOP curves for xLSTM and Transformer for different context lengts and varying number of training tokens.
By rearranging the data obtained from the IsoFLOP approach under different context lengths, one can also fit scaling laws for the context length. This is done equivalently to scaling laws for the model parameters and number of training tokens (Eq. (3)). Figure 11 shows the results w.r.t the number of model parameters and Figure 12 shows the results w.r.t. the number of training tokens. The obtained scaling laws mirror the findings from before. Compute-optimal xLSTM models have more or less constant model size and use slightly more tokens w.r.t. the context length. Compute-optimal Transformer models are becoming smaller and use less training tokens w.r.t. the context length.
Context Length Training Runs
FLOP Optima
Transformar 18) 162, 16
(6a1-18) 18) am.0.043 A'- 3 KLSTM (1e+19) a.00.029 4333.915
Transfomerr
Transtomerr XLSTM
Transformer (3++19) a-10.246, 8
8192
XLSTM
Transformer (3e (e+19) am-0.145, A-1133e1
XLSTM (3e+ 19) am.0.035 Al- 7.5e48
16384
Compute 6e+18
Compute le+19
Compute 3e+19
Optimal Model Perameeeess
18
3.5
3.5
3.4
3.4
3.4
600M
3.4
3.3
#00M
3.3
3.2
3.1
3.2
3.0
200M
3.2
3.2
3.1
3.0
1.00MM
100M
100M
400M
18
400M
Madel
Model Parameters
Model Parameters
Context Length
Figure 11: Left: IsoFLOP curves for XLSTM and Llama function of model parameters at 3 different compute budgets. Right: Plot of the pawer-law fits for the compute optimal number of parameters dependent on the context length No (T)). Colors indicate context length and marker types indicate the model types.
21

---

Published as a conference paper at ICLR 2026
Context Length Training Optima
Transformer (6e118) 0m-0.148, 8l-20e410
XLSTM (6e 6m0.047, 00-00047.8'-3.666 8'
Transfomeer XLSTM
Transfomer
d'm5.0=+0
Transformer (3e+19)) 6m.0.145, 8i-38e+10
(Be+199) bm0.037, 8.6..e
16384
Compute 6e+118
Compute 1e+19
Compute 3e+19
3.5
3.4
Optimal Training Tokeess
7.88
11
3.4
3.3
3.8
3.3
108
3.2
9.2
3.0
8
3.1
28
48
108
208
40B
2B
48
10B 208
408
2B
4B
108 20B
408
102
10
Training Takens
Training Takens
Training Takens
Context Length
Figure 12: Left: IsoFLOP curves for xLSTM and Llama as a function of training token at 3 different compute budgets. Right: Plot of the law fits for the compute optimal number of parameters dependent on the context length N (T"). Colors indicate context length and marker types indicate the model types.
22

---

Published as a conference paper at ICLR 2026
B ACCOUNTING: PARAMETERS, CACHE SIZES, FLOPS, MEMORY OPERATIONS
In this section, we count number of parameters (App. B.1), memory state KV cache size B.2, FLOPs (App. B.3), and memory operations (App. B.4) for mL.TM models based on the architecture of xLSTM 7B (Beck et al., 2025b) and Transformer models with Self-Attention based on the Llama 3 architecture (Grattafiori et al., 2024).
We the notation defined in Tab. 4.
We start with counting the number of memory operations and FLOPs for matrix multiplication, which a very common operation in neural networks. A linear layer with input X and output Y and weight matrix W can be written as
Y (B (Bxda))
X
W
(5)
(Bxda) (du X
This linear layer has 2Bd dour FLOPs:
FLOP 2Bdin
(6)
In order to compute the output Y, we need to read the input X and the weights W and write the output Y. This yields
ByteS Bdii + dlaa) x bytesxy + doa x bytesw
memory operations in loaded and stored bytes. We will use these counts throughout the remainder of this section.
Table 4: Notation for FLOP and Memory Operation Counts.
Symbol
Description
B
Batch size
T. (Tp T
Sequence length, (prefill length, generation length)
S
Query sequence length (only for Self- liff-Attention) Chunk size
L
dnv
Head dimension for values and hidden states
dg..
Head dimension for queries and keys
dmodel
Model / Embedding dimension
da
Feedforward dimension
Pu
Feedforward projection factor
Pyk:
Query key projection factor
Thead(,))
Number of (query) heads
Number of key and value heads
Tebank
Number of chunks
Deocab
Vocabulary size
Maayer
Number of layers
FoP
FLOPs for the operation OP (e.g. exp)
F
Factor that accounts for causality, typically 0.5
bytesx
Number of bytes used for each element in tensor
23

---

Published as a conference paper at ICLR 2026
B.1 PARAMETER COUNTS
We count the number of parameters of mLSTM models (B.1.1) and Transformer models (B.1.2). We include embedding and normalization layer parameters in our parameter counts.
B.1.1 MLSTM PARAMS
For the mLSTM models, the optmized xLSTM architecture from Beck et al. (2025b) and count the parameters in Tab.
Table 5: Parameter counts for the mLSTM Model.
Parameters
Embeddings:
To doodel
mLSTM (single layer)
PreNorm:
dmodel
QKV:
d Whead (2dgk + dhe)
Inpute & Forget Gates:
mo 7e + 2m
Output Gate:
mmodel [ dny
Output Norm:
Thead
Output Projection:
dmodel Thend Cnv
Total mLSTM layer N model head (2dk + dny + 2) + 20 Pooned 2nhead + 2d
Feedforward (single layer)
PreNorm:
dmodel mdo
MLPs:
Total Feedforward Nillayer:
mdd + dmodel
Output Norm:
dmodel
Unembedding:
dmodel Poyacah
Total mLSTM model NULSTM:
Maaye (N (N Maye
N Niylyyer) 2dmoddl dnodee
B.1.2
TRANSFORMER PARAMS
For the Transformer models, assume the Llama architecture with Grouped Query Attention from Grattafiori et al. (2024) and count the parameters Tab. 6.
B.2 MEMORY STATE AND KV-CACHE SIZE
In Tab. 7 list the memory state and KV cache sizes for the mLSTM and Transformer model archi- tectures. We compare the mLSTM with standard Multi-Head Attention (MHA) (Vaswani et al., 2017), Grouped-Quee Attention (GQA) (Ainslie al., 2023) and Multi-Hee Head Latent Attention (DeepSeek- AI, 2024a).
In contrast to the KV caches of the attention variants, the mL.STM has a fixed size memory state that does not depend on the sequence length T.
We compare the size of the memory state and KV cache sizes in number of elements. To obtain the number of bytes, we multiply by number of bytes per element bytesx.
24

---

Published as a conference paper at ICLR 2026
Table 6: Parameter counts for the Transformer Self-Attention Model.
Parameters
Embeddings:
Avocab
Self-Attntion (single layer)
PreNorm:
dmodel
QKV:
Amodal Thkad.. + (dgk + dne )Theadk)))
Output Projection:
1imode//nedd... thav
Total Attention layer Auulayrr
dmodel Theadg
dye Whendkv
dhe Theaddvv + cl model + d
Feedforward (single layer)
PreNorm:
dmodell
MLPs:
3dmode de
Total Feedforward Nitlayer:
mdda dit + dmodel
Output Norm:
dmodel
Unembedding:
dmodel Toyach
Total Transformer model NAu:
Player
N NAtllayer
Milayer) + Wedcc + dmnndel
Table 7: Memory State and KV-Cache Sizes for different Sequence-Mix operations. All terms denote the number of elements.
Sequence Mix Operation
Memory Size in #Elements
Multi-Hear Attention (MHA):
Treead dny T
Grouped- -uteeey--uuuuuuuu Attention (GQA):
2thead dave? T dgv
Multi-Head Latend Attention (MLA):
SLSTM:
Thenlg (d dup + dup +
B.3 FLOP COUNTS
In this section, we count the FLOPs for the mLSTM and the Transformer model architecture. For each model architecture we count the sequence length dependent FLOPs for the sequence mix layer first, i.e. the mLSTM cell (B.3.1) and the Self-Attenti layer (B.3.3), and then combine them with the FLOPs of the other layers in the model architecture to obtain the total FLOPs for the mLSTM (B.3.2) and the Transformer model (B.3.4).
We do not drop subleading terms and set also count FLOPs for all operations equally, i.e. For We also count the FLOPs for the normalization layers with Fnom (we assume the factor of because we have mean, variance and division operations). The skip connection FLOPs are counted with Fskip
or if neglected with Faip 0. Following our training configuration, chunkwise-paree parallel formulation with chunk size 64 and Feausal 0.5 for the FLOP counts and scaling laws in the main text.
the
B.3.1 MLSTM CELL FLOPS
The mL.TTM is linear RNN with gating and can be computed either with recurrent, fully parallel or a chunkwise-paral formulation (Beck et al., 2025a). Each of these formulations has a different FLOP and memory operation count. For training and for prefill in inference the mLSTM relies on the mhunkwise-paralle formulation, which parallelizes the computation over the input sequence and therefore fully utilize modern hardware. For generation, the mLSTM uses the recurrent formulation, which uses constant compute and memory per generation step (i.e. compute and memory requirements are independent of the sequence length).
In this section, we count the number of FLOPs for both the chunkwise paraallel and the recurrent formulation of the mL.TTM cell.
25

---

Published as a conference paper at ICLR 2026
Formulation (Tab. 8, Eq. 8). We list the FLOP counts for the individual terms of the chunkwise laannkll---paaalllllll mLSTM formulation for a single head and a single chunk in Tab.
To obtain the total FLOPs for a full sequence of length T, we multiply these counts by the number of (query) heads bead and chunks Rechnnk = T/L. This yields
FLOP SmLMMsw Theadd X
L' (2(dgx dnv) +8 + TL
2T Feauall T (Adea 6dge + 4dhr 13)
(8)
T (2d 2dgx 5)
Table 8: FLOP counts for the chunkwise-parally mLSTM formulation for mLSTM. All terms denote the FLOP count per head and chunk.
FLOPs
Exact
Simplified (For 1)
Recurrent computation of the inter chunk states
Gates:
LLLL+1]
1 Fuz) + Fioe Fos
0.51? + 6.5T +
+ M1 Fmp + Flg
Numerator:
2d 2.dgeh + Ldg
2dqrhh + 2L Ldg 2dg 2Ld
Denominator:
2d 2Ld
Parallel computation of the intra chunk outputs
Cumulative Forget Gates:
LLL L(Fuee Fag)) (LP(3 Fes Fmna) L(1 Fmm)))
0.5L 2.5L
Gate Matrix:
Fenual
Fonsa (5L2 1 2L)
Intra Outputs:
F Fomual (212 thn) 322)
Fanut (21 du.) 31S)
Parallel computation of the inter chunk outputs
Inter Outputs:
Lh 3Ldlg
2d 3Ldgs
Combination of ister and intra clank enttputs
Outpul Combination:
2i L(1 Funa
Pas Fag)
2Lhhr 4L
L2 Fomm (2(dg + du.) + 8) + 122 2LFon
Total:
+L (Adq 6dg.
183 + (2dgehe: 2dg +
Recurrent Formulation (Tab. 9, Eq. 9). We list the FLOP counts for the individual terms of the recurrent mLSTM formulation for single head and single time step in Tab. 9.
To obtain the total counts for one generation step, we multiply by the number of heads Thead This yields
FLOPSsm..TMM Yihead X (6dg + 7dqp + dny + 12).
Table 9: FLOP counts for the recurrent mLSTM formulation for mLSTM. All terms denote the FLOP count for single timestep per head.
FLOPs
Exact
Simplified (For
Gates:
4 2Fesp + Fing + Paig + Fmax 4eggthhy
Memory Cell Update:
4elx 6el,t + dor +
Denominator & Scale:
6dgt + dhe +1+ Fsts Fmax 2dh + dgk
Output:
2d dag + dap.
Total:
Gdgthhv + 7dg + dnw +
B.3.2 MLSTM MODEL FLOPS
The number of FLOPs for the backbone is identical for training, prefill and generation as the operations (embeddings, linear layers and layernorms) do not depend on the sequence length. Therefore, we count the FLOPs per token for the mLSTM backbone. To obtain the total FLOPs for the specific setting we have to use the respective expression for the mLSTM cell FLOPs from Appendix B.3.1.
mLSTM Backbone (Tab. 10). We count the FLOPs for the mLSTM backbone for single token in Tab. 10 and leave the mLSTM cell FLOPs unspecified. The number of tokens for one batch of sequences is BT.
26

---

Published as a conference paper at ICLR 2026
Table 10: FLOP counts for the mLSTM backbone. All terms denote the FLOP count per token, i.e. to obtain the FLOPs for one batch we multiply by BT tokens.
FLOPs
Embeddings:
mLSTM (single layer)
PreNorm & Skip:
denondel Fssip + Fmorn) )
QKV:
mndd (2dgt clav)
Inpute & Forget Gates:
2mmdel/Vheen + 21hhend
mLSTM Cell:
FLOPSmI STM
Output Gate:
2d Plead day + -hhead Fsig Thead Chy Fnom
Output Norm:
Output Projection:
2dmaeeel Phent
Total MLM layer FL.PPS
Feedforward (single layer)
PreNorm & Skip:
Aenoteed ( (Faip + Fmomn) 6c dr(1 + Fswish)
MLPs:
Activations:
Total Feedforward FLOPSfr,hyer'
Output Norm:
denoded E Foom mm Pevecah
Unembedding:
Total mLSTM model FLOPS FLOPSSn.TTMMMmed mTM.m
B.3.3 SELF-ATTENTION FLOPS
We count the FLOPs for single Self-Attention head during training prefill and generation in Tab. 11. We denote the number of keys and values in the sequence as T, and the number of queries S. During prefill we have S T, since the input sequence is processed in parallel and during autoregressive generation we have S = 1, since we generate one token at time. We typically use F softmax 5 and l'cusall 0.5 following Busbridge al. (2025) as FLOP factor for softmax (sm).
Table 1I: FLOP counts for Self-Attention All terms denote the FLOP count per (query) head.
FLOPS
Generic
Prefill (S T)
Generate (5 1)
Attention computation
Logits:
2.T Fameal ST FT x Fenual
2T x Fensal T Fastmms Fannal
dg F Fensil
Attention:
TF x Famal
Hiddens/Outputs:
25T day x Fancil
21 dhe x Fenual
21 th X Fonssl
Total:
2ST F Fanaal d4 dr 0.5F... 2T Famaal (dg the + 0.5F... 2T Fa (14 + dne + 0.5F...
Aeff-tttetionn in Training (forward only) and Prefill (Eq. 10). To obtain the FLOPs for all heads for full sequence T or Tp. we multiply by the number of (query) heads Pheada and the number of tokens T. This yields
2Fcmsal T Whendy 2 (dgt
+ due + 0.57Fm)
(10)
Self-Attentinn FLOPs in Generation (Eq. 16). During generation the number of FLOPs per token is dependent on the number of previous tokens T = Tp + +lg where Tp is the number of prefill tokens and lg is the number of generated tokens far. We denote the number of total tokens be generated as T. To obtain the FLOP counts for the f--th generated token, we need to multiply the FLOPs for the Self-Attentin layer by the number of (query) heads Thealg: We obtain the FLOPs for the tg -th generated token as
FLOPSs Aut.en step (1y) 2Femssa CaussalThead.g (dgp + dne 0.5F (Tp 1g)
(11)
27

---

Published as a conference paper at ICLR 2026
With 2F causal bead. (dg + day + 0.5 compute the total FLOPs for Tg generated tokens as the sum of FLOPs for each generated token as
T
FLOPs, AlLgen-sey
FLOPs FLOPSA st
(12)
(aTp + alg)
(13)
1=1
T
aTpTT +
fg
(14)
4=1
1
aTT g
マタ
oTl (Tx
+ 1).
(15)
As result we obtain the total FLOPs with prefill prompt length Tp and total number of generated tokens Tg as
FLOPSAnr.ee---ee 2F cacusal Vhhaad. (dak + dne + 0.5lfmm (T + TeT +1((T++1))
(16)
B.3.4 TRANSFORMER MODEL FLOPS
Similar to the mLSTM backbone in Appendix B.3.2, the number of FLOPS for the Transformer backbone identical for training, prefill and generation the operations (embeddings, linear layers and layernorms) do not depend on the sequence length. Therefore, we count the FLOPs per token for the Transformer backbone. obtain the total FLOPS for the specific setting we have use the respective expression for the Self-Attention layer FLOPs from Appendix B.3.3.
Transformer Backbone (Tab. 12). We count the FLOPs for the Transformer backbone for single token in Tab. 12 and leave the Self- Self-Attention FL.OPs unspecified. The number of tokens for one batch of sequences is BT.
Table 12: FLOP counts for the Transformer backbone. All terms denote the FLOP count per token, i.e. to obtain the FLOPs for one batch we multiply by BT tokens.
FLOPs
Embeddings:
Attention (single layer) PreNorm & Skip:
Smodel (Fsipp + FFoon)
QKV:
2dmoodel ( (ds Theud. + dik Thead.y + Aiyeuk) FLOPSAU
Attention:
Output Projection:
Thead, Chv
Total Attention layer FLOPSAA.yyrr:
Feedforward (single layer)
PreNorm & Skip:
dmodee (Faiip + F Foomn) Gdmondeldi de (1 Fswish)
MLPs:
Activations:
Total Feedforward FLOPSt,
Output Norm:
duwdiel Froem 2d 2dmodeeDlocen Pevocab
Unembedding:
Total Transformer model FLOPs, Aummedel
28

---

Published as a conference paper at ICLR 2026
B.4 MEMORY OPERATION COUNTS
In this section, we count the memory operations for the mLSTM and the Transformer model architec- ture. We follow the same outline as for the FLOP counts in Appendix B.3 and first count the memory operations for the mLSTM cell (B.4.1) and the Self-Attention layer (B.4.3), and then combine them with the memory operations of the other layers in the model backbone to obtain the total memory operations for the mLSTM (B.4.2) and the Transformer model (B. (B.4.4). We model weight MemOps as one-time streaming cost (perfect on-chip reuse), i.e., independent of the number of token in the batch BT. This is reasonable with persistent/fused kernels and per-rank weight matrices that fit in on-chip cache. Depending on the exact experimental configuration, this assumption might not hold as we observe when modeling the step time through MemOps in Section C.3.
We include the memory operation count for the normalization layers, but can neglect them by setting bytes O and aet 0.
B.4.1 MLSTM CELL MEMOPS
Similar to the FLOP counts in Appendix B.3.1, we count the memory operations for the mLSTM cell for both the chunkwise-pa paraplallll and the recurrent formulation.
Chunkwise-Parre Parallel Formulation (Tab. 13, Eq. 17). The implementation of the chunkwise -arrallel mLSTM formulation consists of two kernels (Beck et al., 2025a). We count the memory operations for the loading and storing of the inputs and outputs of each kernel for single chunk and head in Tab. 13.
By multiplying with the number of heads Thhaad and the number of chunks Nehunk = T/L, we obtain the total memory operation counts for the chunkwise-paralle mLSTM formulation as
T Theead L
Bytes
(4L x bytes, + 3L (chv + dgk) bytes
(17)
+ 2Mpead (L + dny + dyk + 1) bytes
Table 13:
Memory operation counts for the chunkwise-paralle mLSTM formulation, All terms denote the memory operation count per head and chunk.
Bytes
Inter-chunk Recurrent Kernel
Load:
L(dgx + day) x bytes + 21. bytes,
Store:
(dgethe + duk + 1) bytes
Intra-chunk Parallel Kernel
L(2dg + dhr) x bytes + 2L x bytes, +(dydhy + du + x hytes
Load:
Store:
Leho x bytes + 2L x bytes
4L x bytes,
Total:
+3T (dhe + dqe) x bytes do.
+2 (L + dm da + 1) x bytes
Recurrent Formulation (Tab. 14, Eq. 18). During text generation we use the recurrent formulation. which loads the previous memory state and the current input and stores the output and the next memory state. We obtain the total memory operation counts for the recurrent mLSTM formulation by multiplying the counts in Tab. 14 with the number of heads Thead:
Bytes = Thead (2 x bytes, + 2(dhv + dg)) x bytes 2dho x bytescma) (18)
B.4.2 MLSTM MODEL. MEMOPS
The memory operations of each layer of the backone (excluding the mLSTM cell) consist of the input and output activations as well as the parameters. The inputs and outputs depend on the number of tokens BT in the batch, whereas the parameters are independent of the number of tokens.
29

---

Published as a conference paper at ICLR 2026
Table 14: Memory operation counts for the recurrent mLSTM formulation. All terms denote the memory operation count for single timestep per head. We assume the states are materialized at every timestep.
Bytes
(2da + dnv) x bytes gev 2 bytes, + dday + doe
Load:
bytescimm
Store:
dne byr + (doxdhr + dgk + 1) x bytescmm
2 x bytesy + 2(ch + dat) x byles +2dnu x bylescmn
Total:
The total memory operations for each layer are the sum of the memory operations for the input and output activations and the parameters and are given in Tab. 15. By default, we assume that all weights are stored in the same precision and use the same number of bytes bytesw for all weights.
Table 15: Memory Operation counts for the mLSTM Model.
Memory Ops bytes
Input & Output Activations
Weights
Embeddings:
BI Deschhhmeddd bytesw...
mLSTM (single layer)
PreNorm:
moddl x bytesyatrom
Gmodel byte8 Poedta (2dgg. + doe) x bytes (2clguote) Thead 2mhead) x byteswa
QKV:
131 (dmastd + Daeaal (2dg. + due)) x bytesgu
Inpute & Forget Gates:
(dmndel + rhhead) x bytesu
mLSTM Cell:
Bytes
Output Gate:
BT (d (made Theaathe bytes
AnnaaPaeal dew bytesw. Phead bytes
Output Norm:
B1 Phead yetos
Output Projection:
BI'(d (dmndd Theathe) bytes
denttaheendth bytes,,
Total mL layer aasmimes
Feedforwarl (single layer)
PreNorm:
dmedee X byle% 3BT (dmatee da) bytes
Omedee byleSw.. de
MLPs:
Total Feedforward Bytesgnuye
Output Norm:
BT dmedal x bytcs aatorma
dmodel X byte8 Toooee/Toocab X bytes
Unembedding:
BT (dmmdel Throcab) bytes
Total mLSTM model NaaSTA:
B.4.3 SELF-ATTENTION MEMOPS
Similar to the FLOP counts in Appendix B.3.3, we count the memory operations for single Self-Autention head during training or prefill and generation.
These two cases have very different memory operation counts, as during training and prefill we need to load the full sequence of tokens only once, whereas during autoregressive generation we have to load all previous tokens Tр + tg (i.c. the whole KV cace) for each generated token.
We consider FlashAttention implementations for the Self-Attention operation (Dao, 2024), where the Attention logits are not materialized in HBM. Therefore, we only count the memory operations for loading the query, key and value inputs and the output of Self-Attention in Tab. 16.
Self-Attention in Training and Prefill (Eq. 19). During training and prefill need to load the full sequence or Tp tokens only once. The total memory operation counts are given by
Bytes Art.train-pref (d + dnw) (1head, Thead.kv) bytes
(19)
Self-Attention in Generation (Eq. 21). Similar to the FLOP counts in Appendix B.3.3, also the memory operation counts for the Self-Attention layer during generation depend the number of previous tokens = Tp + ty. where Tp is the number of prefill tokens and tg is the number of generated tokens far.
30

---

Published as a conference paper at ICLR 2026
Table 16: Memory operation counts for FlashAttention. For training and prefill S, while for generation $ = 1.
Bytes
Generic
Load:
(Sdyk Abendq + T(dyk + dny (inaddk) bytesqkv
Store:
Sdhr Ahead, byteSqkv
Total:
(S(dkk + dne Minead. + T(dqe day ) Mbeddky)
bytes
The number of memory operations for the fg-th generated token is given by
Bytes gue (4y) ((dyk + dhe M (Tp + (gg( (
bytes ykv
(20)
compute the total number of memory operations for Tg generated tokens by summing up the per-step memory operations
Similar equations (12)-(15),
Bytes sel Alteeen. = bytes x (dg + dnv Miedd.
(21)
+ (IoTg + 1 (Tg + 1)) (dgk + (hv)rkkakky
2
B.4.4 TRANSFORMER MODEL MEMOPS
Similar to the mLSTM backbone in Appendix B.4.2, the number of memory operations for the Transformer backbone (excluding the Self-Attention layer) consist of the input and output activations as well as the parameters. The memory operations for input and output activations depend on the number of tokens BT in the batch, whereas the parameters independent of the number of tokens.
The total memory operations for each layer are the sum of the memory operations for the input and output activations and the parameters and given in Tab. 17. By default, assume that all weights are stored in the same precision and use the same number of bytes bytes, for all weights.
Table 17: Memory Operation counts for the Transformer Model.
Memory Ops in bytes
Inpat & Output Activations
Weights
Embeddings:
HT Poode x hyteser,
Attention (single loyer)
PreNorm:
BTdmas bytesyszom
dreeds X bytesyy....
QKV:
BT (dundo Thpess (2dq + dye]] x byles downal (dgp Mhredd + (dig. the (ohnsstio)) x bytes
Attention:
Bytos xy
Output Projection:
BT (dmatt tho) bytes
dnvott the bytes.....
Total Attention layer antoe
Feedforward (single layer)
PreNorm:
BT dandks bytes 3BT (dumeaa (BTol
drecks X bytesmy.... 3d
MLPs:
bytes
Total Feedforward Bylesgawe"
Output Norm:
BT dandd X bytes (didu + Troceas) bytes
descas X bytes
Unembedding:
Paode bytes
Total Transformer model Naii
31

---

Published as a conference paper at ICLR 2026
C MODELING INFERENCE CHARACTERISTICS
In this section, we create a model of the theoretical runtimes of operations the xLSTM and Transformer model architectures to model their inference characteristics (TTFT and step time). This theoretical model is based on the FLOP and the memory operation counts in Appendix B.
This theoretical model of inference characteristies has two purposes: First, it allows to investigate the theoretical differences in maximal inference speed between xLSTM and Transformer architectures and explain the empirically observed behavior. Second, based TTFT and step time measurements for specific architecture configurations, it allows to predict the theoretical inference speed for other (possibly larger) configurations and take this into account for selecting the optimal architecture configuration based on our scaling laws. This is important if there are certain requirements on maximal TTFTs step times for a particular case. With this theoretical model, easily possible to determine model configurations which satisfy those conditions.
C.I BACKGROUND: THEORETICAL RUNTIME
In order to estimate the total theoretical runtime of workloads on GPUs or TPUs, we can break down the runtime into three components (Austin et al., 2025, Part l):
Compute time TFLOPS: The time it takes to perform the FLOPSs of the workload on the GPU(s).
Memory time 7 mem: The time for memory loads and stores from and to GPU memory during a workload.
Communication time comm: The time for communicating or transferring data (e.g. inter- mediate results) between multiple GPUs taking part in workload.
Given the number of floating point operations FLOP - the number of bytes mem.agoo that must be loaded and stored, and the number of bytes Bytes commaaloo that must be communicated between GPUs, we can compute the individual runtimes as
FLOPo
meteSao
ByteS.omm.alee commaalgo
TFLOPs.aggo
mmemaalgo
and
T comm.algo
(22)
ace
B acc
? Bytes
where ace. 3 and Bytes are the accelerator specific compute speed in FLOPs/s, the accelerator memory bandwidth in Bytes/s and the accelerator communication bandwidth in Bytes/s, respectively.
For accelerator speed ace, accelerator memory bandwidth aces and accelerator communication bandwidth 7 Bytes, we use the hardware specifications of NVIDIA V100², A100 H100 and B200 GPUs, which summarize in Tab. 18.
Table 18: Hardware Accelerator Specification for NVIDIA GPUs used in this analysis. Values without sparsity. If only the value with sparsity is known, we divide by 2.
Communication Bandwidth
bfloat16 [FLOP//s]
Memory Bandwidth [Byte/s]
Arithmetic Intensity [FLOPs/byte]
GPU
Year
[Byte/s]
V100 SXM2
2017
120e12
0.9e12
133
0.3e12
A100 SXM
2020
312e12
2.039e12
161
0.6e
H100 SXM
2022
989e12
3.35e12
295
0.9e12
B200 HGX
2025
2250e12
7.7e12
292
1.8e
If there is no overlap between computation and memory or communication operations, or other words if we cannot load, store or communicate data while the GPU is doing FLOPs, the total runtime
whttps::/www.ntt
32

---

Published as a conference paper at ICLR 2026
is the sum of the two, i.e.
Talgo.upeer = aLooo + 7mem/comm.ago-
(23)
If the computation and memory or communication operations can be overlapped (i.e. happen in parallel), the total runtime is the maximum of the two, i.e.
Jolee = max (Looo 7mem//com..aggu)
(24)
This means the runtime is lower bounded by the maximum of the two and upper bounded by their sum (Austin et al., 2025, Part 1).
Roofline model. helpful model for determining whether runtime is bounded by computation (compute-bound) or by memory/bandwidth (memory- boundd) is the rooffine model (Williams et al., 2009), see Figure 13 for an illustration. The roofline relates the attainable FL.Oss/s with the arithmetic intensity Talgo of the operation performed on the GPU which is given by
FLOPSalgo
algo
(25)
Byte
Thus, the arithmetic intensity is the FLOPs per byte for given operation. When the arithmetic intensity of operations increases, the attainable FLOPs/s increase linearly- operations are essentially memory-bounds the GPU has to wait for bytes to arrive perform caleulations. In this setting, the runtime is effectively given by Tmemu/com.algo-
Upon reaching the arithmetic intensity of the accelerator Iace (see Tab. 18 for specifications for common GPU types), the "roofline" is reached and operations are essentially compute bound; the GPU still performs calculations while the next inputs are ready. In this setting, the runtime is effectively given by TFLOPs.ago-
Memory Bound Compute Bound
Realized FLOPs/s
Inference stages. As outlined in Section 4, inference with LLMs typically split into two stages, prefill and generation.
For the prefill stage, the TTFT is the key performance metric which is the runtime of the LLM in processing an input sequence if certain prefill length, building up caches (Transformer) / memory cells (XLSTM) and generating the first token. Following Austin et al. (2025, Part 7), we assume that even at relatively low prefill lengths of 256, inference is dominated by large matrix multiplications for both Transformers and xLSTM and therefore consider the prefill stage the be compute bound. While this might not perfectly model very small prefill lengths, those are generally dominated by constant overheads.
Arithmetic Intensity (FLOPs/Byte)
Figure 13:
Roofline model.
For the generation stage, step time is the key performance metric which is the runtime of the LLM in generating a new token after having processed the the whole input sequence up to the last token. This means that during a forward pass, only tiny amount of compute is necessary to account for this new token. However, for Transformers is necessary to load from the KV cache, which is a very handwidth-intensin operations, followed by streaming weights and storing and loading activations for both architectures. Consequently, arithmetic intensities during generation are generally rather low (see also Austin al., 2025, Part 7). We thus assume that during the generation stage, both Transformers and xLSTM are memory bound.
C.2 PREFILL STAGE: TIME To FIRST TOKEN
As we assume to be compute bound during prefill, we model the runtime of the prefill stage which corresponds to the TTFT as (c.f. Eq. (4)):
FLOPs
T aLoos
+ 6.
(26)
ar ell
FLOP Salgo can be calculated analytically given the FLOPs calculations provided in Appendix B.3, a and need to be fitted using the measured data. Exemplarily, we show the runtimes fitted for the measured TTFT in Figure 14 (Transfomer) and Figure 15 (xLSTM) for different model sizes. We fit a en and e per model configuration on TTFTs obtained under various combinations of batch sizes and prefill lengths. Our fits show excellent agreement between the predictions from quantitative
33

---

Published as a conference paper at ICLR 2026
Runtime by Batch Size
Predicted (dotted)
Time to First Token (ms)
10a
Measured (solid)
2
10a
- 32
- 6-1
128
256
512
768
10a
10c
Prefill Length
Figure 14: Time to first token, measured and fitted, for a Transformer model as a function of prefill for different batch sizes.
Runtime by Batch Size
Predicted (dotted)
Time to First Token (ms)
Measured (solid)
10a
2
101
- 6-1
-
128
512
768
10a Prefill Length
10c
Figure 15: Time to first token, measured and fitted, for a 400M xL.STM model as function of prefill for different batch sizes.
runtime model and the measured data. In Figure 16 we further show the quotient of the fitted a af and the hardware parameter ace for all model sizes. If eft/a acc the hardware would be perfectly utilized according to our model. We see that for both Transformers and xLSTM, the quotient increases, thus larger models utilize the hardware better. Furthermore, both models show relatively similar trends and magnitudes, indicating that the empirical measurement setup allowed for fair comparison.
C.3 GENERATION STAGE: STEP TIME
As assume be memory-bound during generation stage, model the runtime of the generation stage which corresponds to the step time as (c.f. Eq. 4):
Bytes
memalgo
mem,algo
B Beff
+ E.
(27)
Bytes memaago be calculated analytically given the MemOps calculations provided in Appendix B.4, eff and need to be fitted using the measured data. Furthermore, found that the fit quality for Transformer further improved by fitting another constant that scales with the batch size. Exemplarily, show the runtimes fitted for the measured step times in Figure 17 (Transformer) and Figure 18 (xLSTM) for different model sizes. Again, we find a very good agreement between the predictions from quantitative runtime model and the measured data.
34

---

Published as a conference paper at ICLR 2026
0.40
Transformer
XLSTM
0.35
0.30
aeff
0.25
0.20
100M
10B
Model Parameters
Figure 16: Comparing the fitted Oeff to the accelerator Oace (989e12 for a H100 see Tab. 18). With our experimental setup, we attain similar effective FLOPs for both the Transformer and xLSTM. As expected, the accelerator is better utilized by larger models.
Step Time by Batch
10?
Predicted (dotted)
Measured (solid)
Step Time (ms/step)
32
256
384
10a
10a
10a
Prefill Length
Figure 17: Step time, measured and fitted, for 7B Transformer model as function of prefill for different batch sizes.
Step Time Batch Size
Predicted (dotted)
Measured (solid)
Step Time (ms/step)
16
64
128
256
1xx100 10"
384
768
x
2
10a
10
Prefill Length
Figure 18: Step time, measured and fitted, for a 400M xLSTM model as function prefill for different batch sizes.
35

---

Published as a conference paper at ICLR 2026
D MODEL CONFIGURATIONS
In this section, we list the model hyperparameters and sizes of all training runs Token/Param (Sec. D.1) and IsoFLOP (Sec. D.2) of the dataset for our scaling law study.
D.1 MODEL SIZES AND HYPPARRAMEEERR TOKEN/PARAM CONFIGURATION
Table 19:
List of hyperparameters for xLSTM models trained with the Token/Param configuration
with context length T 8192.
#Params (M)
dmodel
dr
dyk
day
Theads
Tlaayer
(seqs)
LR
164
768
2112
64
128
128
3e-3
406
1024
2752
128
256
4
24
128
3e-3,
le-3
841
1536
4160
192
384
24
256
le-3, 8c-4
1420
2048
5504
256
512
4
24
256
8c-4, 7c-4
2780
2560
6848
256
512
32
512
7e-4
6865
4096
10944
256
512
8
32
256,
512
5e-4, 4e-4
Table 20: List of hyperparameters for Transformer models trained with the Token/Param
configuration with context length T 8192.
#Params (M)
dmnodel
df
dnv
Taayer
B (seqs)
LR
162
768
2048
64
12
12
128
3e-3,
1e-3
406
1024
2752
24
128
3e-3, le-3
834
1536
4096
96
16
24
256
le-3
1420
2048
5504
128
24
256
8e-4
2779
2560
6848
80
32
32
512
7
6863
4096
10944
128
32
256,
512
5e-4
36

---

Published as a conference paper at ICLR 2026
D.2 MODEL SIZES AND HYPERPARAMETERS IN ISOFLOP CONFIGURATION
Table 21: List of hyperparameters for xLSTM models trained with the IsoFLOP configuration.
#Params (M)
dwodel
dr
dgp
dhe
Pheads
Payer
1408
64
128
90
512
1408
64
128
4
12
96
512
1408
64
128
4
14
102
512
1408
64
128
4
16
114
640
1728
64
128
5
10
123
640
1728
64
128
5
12
128
640
1728
64
128
13
133
640
1728
64
128
14
143
640
1728
64
128
5
16
164
768
2112
64
128
12
185
768
2112
64
128
6
15
207
896
2432
64
128
12
207
768
2112
64
128
6
18
236
896
2432
64
128
265
896
2432
64
128
18
295
2432
64
128
21
324
2432
64
128
24
330
1024
2752
256
353
896
2432
64
128
27
368
1024
2752
256
21
406
1024
2752
128
256
24
444
1024
2752
128
256
27
482
1024
2752
128
256
4
30
503
1152
3136
64
128
9
24
552
1152
3136
64
128
27
601
1152
3136
64
128
30
604
1280
3456
128
256
24
664
1280
3456
128
256
27
715
1408
3776
64
128
24
724
1280
3456
128
256
30
787
1408
3776
64
128
27
841
1536
4160
128
256
24
859
1408
3776
64
128
30
1536
4160
256
27
1013
1536
4160
256
1108
1792
4800
256
24
1224
1792
4800
256
27
1340
1792
4800
256
30
1421
2048
5504
128
256
8
24
1573
2048
5504
128
256
8
27
1772
2304
6208
128
256
9
24
1876
2048
5504
128
256
8
33
1964
2304
6208
128
256
9
27
2028
2048
5504
128
256
8
36
2157
2304
6208
128
256
30
2350
2304
6208
128
256
33
2781
2560)
6848
128
256
10
32
3017
2560)
6848
128
256
10
35
3150
2816
7552
128
256
30
3254
2560
6848
128
256
38
3342
2816
7552
128
256
32
3533
2816
7552
128
256
34
3724
2816
7552
256
36
3726
3072
8256
128
256
30
3954
3072
8256
256
32
4410
3072
8256
256
12
36
4597
3328
$896
256
13
32
51330
3328
8896
256
13
36
5311
3584
9600
128
256
14
32
5930
3584
9600
128
256
14
36
6464
4096
10944
128
256
16
30
6867
4096
10944
128
256
16
32
37

---

Published as a conference paper at ICLR 2026
Table 22: List of hyperparameters for Transformer models trained with the IsoFL.OP configura- tion.
#Params (M)
dmodel
dff
dy
Thhcads
flayyrr
83
512
1408
64
10
90
512
1408
64
12
96
512
1408
64
14
102
512
1408
64
113
640
1728
64
10
10
128
640
1728
I0
13
133
640
1728
64
10
14
143
640
1728
64
10
162
768
2048
64
12
12
183
768
2048
64
12
15
204
768
2048
64
12
18
207
896
2432
64
14
12
236
896
2432
64
14
15
265
896
2432
64
14
18
294
896
2432
64
14
21
324
896
2432
64
14
24
330
1024
2752
64
16
368
1024
2752
64
16
21
406
1024
2752
64
16
24
444
1024
2752
64
16
27
482
1024
2752
16
30
498
1152
3072
128
24
545
1152
3072
128
27
593
1152
3072
128
30
604
1280
3456
128
10
24
664
1280
3456
128
10
27
714
1408
3776
128
11
24
723
1280
3456
128
10
30
786
1408
3776
128
11
27
834
1536
4096
128
12
24
858
1408
3776
128
11
30
919
1536
4096
128
12
27
1003
1536
4096
128
12
30
1107
1792
4800
128
14
24
1223
1792
4800
128
14
27
1339
1792
4800
128
14
30
1420
2048
5504
128
16
24
1572
2048
5504
128
16
27
1723
2048
5504
128
16
30
1760
2304
6144
128
18
24
1951
2304
6144
128
18
27
2142
2304
6144
128
18
30
2334
2304
6144
128
18
33
38

---

Published as a conference paper at ICLR 2026
E COMPUTE OPTIMAL PARAMETER, TOKEN AND FLOP COUNT ESTIMATES
In this section, we determine compute-optimal training setups for various model sizes based on the scaling laws derived from our IsoFLOP approach in sections 3.4 and 3.5. In Section E.1, we show the configurations for our power laws obtained for a context length of 8192 (see Figures 4 and 9), while Section E.2 we present the compute optimal configurations obtained from our power law fits for varying context lengths (see Figures 5 and 10). The power law fits for Section E.2 contain fewer IsoFL.OP profiles than the fits for Section E.
We construct these tables by first choosing range of model sizes, then identifying the optimal compute budget associated with each size (for example, from Figures or 5), and finally inferring the corresponding optimal number of training tokens, such as from Figures 9 or 10.
Across all tables in sections E.1 and E.2, we observe that Transformer models have higher compute- optimal loken-to-paramett ratio than xLSTM models.
Moreover, contrast to the Chinchilla scaling laws, which find that the optimal ratio is constant at around 22 across model sizes (Hoffmann et al., 2022, Table 3), our compute optimal token-to- to-parameter ratio decreases for larger models. This difference arises primarily from 0.575, 6 0.424 vs. (Hoffmann al., 2022, 0.51(0.483, (0.483, 529)). Porian et al. (2024) have investigated these
the distinct exponents in the scaling laws (Ours:
Table 2): a = (0.42,,, 0.534), , b 0.51 discrepancies and found the root cause be in the learning rate decay for the training runs in the IsoFLOP config configurations (see also Appendix A.3). They found exponents comparable to those in our work and were able reproduce the Chinchilla scaling law exponents by using fixed learning rate across all IsoFLOP training runs.
E.1 COMPUTE OPTIMAL CONFIGURATIONS FOR CONTEXT LENGTH 8192
Table 23: Estimated optimal training FLOPs, Tokens, and Token/Param Ratio for varying model sizes from IsoFLOP power-law fits for Transformer and xLSTM models trained with context length 8192. The table obtained from Figures and 9.
Transformer
xLSTM
#FLOPs
#Tokens
Token/ Param Ratio
#FLOPs
# Tokenss
Token/
A' 0.0023
B' 58.5
A' 0.012
B' 77.7
Param Ratio
0 0.575
b 0.424
0.547
b 0.417
#Params
100M
3.24e18
4.17B
41.7
1.33e18
2.83B
28.3
400M
3.61e19
11.6B
29.0
1.68e19
8.15B
20.4
IB
1.78e20
22.8B
22.8
8.97e19
16.4B
16.4
2
5,94e20
38.1B
19.0
3.18e20
27.8B
13.9
4B
1.98e21
63.5B
15.9
1.13e21
47.1B
11.8
6.62e21
106B
13.2
4.01e21
79.9B
10.0
10B
9.76e21
125B
12.5
6.03e21
94.8B
9.5
14B
1.75e22
160B
11.4
1.11e22
122B
8.7
32B
7.38e22
295B
9.2
5,05e22
230B
7.2
67B
2.67e23
508B
7.6
1.95e23
404B
6.0
175B
1.42e24
1.03T
5.9
1.13e24
840B
4.8
39

---

Published as a conference paper at ICLR 2026
E.2
COMPUTE OPTIMAL CONHIGURATIONS FOR VARYING CONTEXT LENGTHS
Table 24: Estimated optimal training FLOPs, Tokens, and Token/Param Ratio across context lengths from IsoFLOP context- sontext-specific power-law fits for Transformer models. The table obtained from Figures 5 and 10.
length:
2048
8192
16384
NFLOPs
#Tokens
Token/ Ratio
#FLOPs
#Tokens
Token/ Ratio
#FLOPs
#Tokens
Token/
0.0069
4'
B'
0.0025
B'
Param
0.553
= 0.423
0.577
0.422
0.569
0.432
Ratio
#Params
100M
2.27e18
4.24B
42.4
3.26e18
4.19B
41.9
4.19c18
3,91B
39.1
400M
2.78c19
12.2B
30.5
3.59.19
11.5B
28.8
4.78c19
11.2B
28.0
IB
1.46200
24.6B
24.6
1.76c20
22.58
22.5
2.39c20
22.5B
22.5
2B
5.09620
41.7B
20.8
5.84e20
37.4B
18.7
8.07e20
38.1B
19.0
4B
1.78211
70.8B
17.7
1.94221
62.1B
15.5
2.73c21
64.5B
16.1
6.23c21
1208
6.44211
1038
12.9
9.21c21
1098
13.6
10B
9.32e21
143B
9,48e21
121B
12.1
129B
12.9
14B
1.71e22
184B
13.1
1.7e22
155B
11.1
2.46e22
167B
11.9
32B
7.62e22
346B
10.8
7.11e22
284B
8.9
1.05e23
313B
9.8
67B
2.9e23
609B
9.1
2,56e23
487B
7.3
3.85e23
549B
8.2
175B
1.6c244
1.27T
7.3
1.35c24
982B
5.6
2.08e24
1.14T
6.5
Table 25: Estimated optimal training FLOPs, Tokens, and Token/Param Ratio across context lengths from IsoFLOP context-specific pawer-law fits for xLSTM models. The table is obtained from Figures and 10.
length:
2048
8192
16384
#FLOPs
#Tokens
Token/ Param Ratio
#FLOPs
#Tokens
Token/ Param Ratio
#FLOPs
#Tokens
Token/
A' 0.0086
8' = 141
4' 0.0161
B' = 46.8
A' 0.005
B' 336
Param
0.555
= 0.403
0.541
0.429
0.566
0.385
Ratio
#Params
100M
1.32e18
2.83B
28.3
1.3e18
2.74B
27.4
1.58e18
3.44B
34.4
400M
1.6e19
7.73B
19.3
1.69e19
8.22B
20.6
1.83e19
8.85B
22.1
IB
8.32e19
15B
15.0
9.21e19
17B
17.0
9.23e19
16.5B
16.5
2B
2.9c20
24.9B
12.4
3.32c20
29.5B
14.8
3,14e20
26.5B
13.2
4B
1.01e21
41.1B
10.3
1.2c21
51B
12.8
1.07e21
42.4B
10.6
8B
3.51e21
68B
8.5
4.31e21
88.4B
11.0
3.64e21
68B
8.5
10B
5.25e21
79.9B
8.0
6.52e21
106B
10.6
S.39e21
79.
9.62e21
1.21e22
138B
9,77e21
32B
4.26e22
186B
5.6c22
266B
8.3
4,21e22
175B
67B
1.61e23
318B
4.7
2.2e23
477B
7.1
1.55e23
289B
4.3
175B
9.07e23
638B
3.6
1.3e24
1.02T
5.8
8.47e23
555B
3.2
40


Reviews:
[{"review_id": "kQqC4BukhF", "reviewer": "Reviewer_R1D3", "summary": "This paper systematically compares the scaling of xLSTM and Transformers. Using extensive controlled runs (80M–7B parameters; 2B–2T tokens) and two protocols (IsoFLOP budgeting and parametric fitting), it examines: (i) scaling in compute-optimal and over-training regimes, (ii) how compute-optimal model size depends on context length, and (iii) inference scaling in algorithmic complexity, latency, and throughput. The key finding is that xLSTM consistently scales better than Transformers, and the advantage widens as training or inference contexts grow longer.", "questions": "1.\tIt would be good to provide cross-dataset and cross-tokenizer results to test whether the reported margins persist?\n\n2.\tHow do you normalize FLOPs/MemOps across architectures (kernel fusion, activation checkpointing, flash-attention)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.  Provides a large, controlled scaling study comparing xLSTM and Transformers across budgets and context lengths. Combines IsoFLOP budgeting with parametric fitting to bridge compute-optimal and over-training regimes.\n\n2.  Extensive, well-structured experimental sweeps (80M–7B; 2B–2T) with clear loss–compute Pareto analyses.\n\n3.  Coherent problem framing and storyline from scaling laws → compute-optimal sizing → inference scaling. Figures generally align with claims and support the narrative.\n\n4. Offers compute-aware guidance on architecture choice, model size, and context length. Helps quantify when linear-time sequence processing becomes advantageous, informing system design and resource allocation decisions.", "weaknesses": "1.  Several key definitions/configs are only provided in the appendix; Moving essential config details into the main text, unifying definitions/notation would significantly improve the presentation of the paper.\n\n2. Results seem to be on a specific data mix, tokenizer, and recipe. Adding cross-dataset, cross-tokenizer, and recipe sensitivity studies would make the paper stronger. \n\n3. Some figures are too crowded. Several plots are visually dense, making it hard to extract key trends.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AyvX6rNSuX", "reviewer": "Reviewer_LrJB", "summary": "This paper presents a systematic empirical comparison of scaling behaviors between xLSTM and Transformer architectures for large language models. The study spans three dimensions: (1) training efficiency under compute-optimal and over-training regimes across 80M-7B parameters and 2B-2T tokens, (2) the relationship between optimal model size and context length, and (3) inference-time characteristics (TTFT and step time). The authors claim xLSTM is Pareto-dominant in training loss vs. compute, requires larger compute-optimal models, and shows widening advantages with increasing context length. This is a technically solid, large-scale empirical study that provides valuable evidence for xLSTM's scaling properties. The methodological innovations (accurate FLOP accounting, context-length scaling analysis) are notable contributions.", "questions": "1. L024: Quantify \"advantage widens\", what is the relative improvement at 2K vs. 16K contexts?\n3. L313: If losses aren't comparable across contexts, how should we interpret the intersecting curves in Figure 5 (right)?\n4. L399: \"Largest xLSTM has lower step time than smallest Transformer\", is this specific to 16K prefill, or does it hold at shorter contexts (e.g., 512)?\n5. It would be better to include at least one long-context benchmark evaluation (e.g., LongBench) to more convincingly demonstrate xLSTM’s effectiveness in handling extended context scenarios.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Comprehensive Experimental Scale：The study demonstrates an impressive experimental scale, encompassing 672 training runs with a total compute of 3.2×10²³ FLOPs. It systematically explores both the compute-optimal and over-training regimes—a distinction of significant practical relevance—while maintaining a fair and consistent comparison framework through unified training recipes, shared data (DCLM-BASELINE), and identical hyperparameter schedules.\n\n2. Clear Empirical Findings: The paper presents clear and consistent empirical findings. xLSTM demonstrates Pareto dominance over Transformers, achieving lower loss at fixed compute across five orders of magnitude. It also exhibits remarkable over-training stability, with constant power-law exponents observed up to M=2200, confirming its reliability for inference-optimized training. Furthermore, xLSTM maintains strong context length robustness, as its optimal model size remains stable even when context length increases, whereas the Transformer's performance degrades notably.", "weaknesses": "1. **No Downstream Task Evaluation**. The entire paper focuses exclusively on pretraining cross-entropy loss without any downstream benchmarks (e.g., MMLU, HumanEval, common sense reasoning, summarization).\n\n2. The context length experiments appear confounded, as different context lengths are trained on non-identical data distributions without clarification on how this issue is handled—for instance, whether through re-chunking, padding, or different data splits. Moreover, the y-axes in Figure 5 are not directly comparable across context lengths, and it remains unclear how the “tokens D” are counted when context size varies (i.e., whether they represent effective or nominal tokens). Clarifying the data preprocessing process or adding a fixed-dataset ablation where only context packing changes would strengthen the validity of these results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4qvBLt2q5e", "reviewer": "Reviewer_dz4g", "summary": "The authors study scaling behavior of a recently proposed sequence modeling architecture, finding it does well on many desirable axes compared to Transformer baselines.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Study relevant regimes, incl. overtraining/inference time compute which are rarely paid attention to in scaling work even though they are principle considerations when training/serving modern frontier models. \n- Science of scaling is cleanly and thoroughly done. Reproduction of important past work (power-law exponents from Chinchilla) is reassuring as a sanity check. \n- Lots of large-scale empirics and careful parametric fits, very valuable contribution.", "weaknesses": "- Transformer baseline is weak and out of date (Llama-2 architecture in late 2025). There are a number of Transformer++ improvements that have appeared since Llama-2, all else fixed, that make big improvements to performance at scale (eg. RoPE, GQA, addressing attention sinks, no biases on linear layers, etc etc). \n- I'm not convinced xLSTM actually outscale Transformers, and in fact would be willing to bet they do worse at frontier compute regimes. We can see the advantage decreasing with compute in Fig1(left), and we can see the faint Transformer curves overtaking the xLSTM ones on Fig1(right). This looks exactly like Fig4 of the Mamba [1] paper where Mamba was touted as \"scaling favorably\" but the gap shrunk slightly with compute in their plots. When things were scaled up, of course Transformers did much better than Mamba. I expect a similar thing to hold here (notice how the plot is missing Transformer at 1e23 where xLSTM begins to plateau, I would be very interested in seeing that last Transformer data point). Nonetheless the thorough and expansive empirics here are valuable, and this is how progress in architecture design is made, so this is a valuable paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xQgPjOqE8X", "reviewer": "Reviewer_H1t8", "summary": "This paper points out that the authors systematically compared the scaling laws of the performance-optimized xLSTM architecture with those of the dense multi-head self-attention Transformer architecture. The authors conducted comparative analyses from perspectives such as Training, Context length, and Inference.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well-written, with clear logic and concise readability.\n\n2. It conducts extensive experiments, including comparative experiments on models of different types and scales.\n\n3. The authors provide a wide range of comprehensive evaluation metrics.", "weaknesses": "1. First, this paper lacks sufficient novelty. Models based on the xLSTM architecture have already demonstrated significant advantages in areas such as inference in previous studies. However, this paper only conducts extended comparative experiments based on this existing foundation, which casts doubt on its innovativeness.\n\n2. Second, compared to models with quadratic complexity, the design of LLM architectures based on linear complexity is inherently intended to reduce computational costs and achieve more significant benefits during training and inference—and this is a well-known fact. The paper conducts comparative analyses from perspectives including Training, Context length, and Inference, but these experiments are carried out under the premise that such performance characteristics are already known, resulting in insufficient contributions from the paper.\n\n3. Finally, to fully compare the performance of models with linear complexity and quadratic complexity, limiting the scope solely to the xLSTM architecture is flawed. It is necessary to include a broader range of models and conduct more comprehensive result comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)