MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
Authors: Johannes von Oswald
Nino Scherrer
Seijin Kobayashi
Luca Versari
Songlin Yang
Maximilian Schlegel
Kaitlin Maile
Yanick Schimpf
Oliver Sieberling
Alexander Meulemans
Guillaume Lajoie
Rif A. Saurous
Charlotte Frenkel
Razvan Pascanu
Blaise Aguera y Arcas
Joao Sacramento
Venue: ICLR 2026 Poster (2026)
Keywords: Sequence modeling
test-time training
RNN transformer alternatives
Primary area: foundation or frontier models, including LLMs

Abstract: Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention. Although widely adopted, transformers require scaling memory and compute linearly during inference. A recent stream of work linearized the softmax operation, resulting in powerful recurrent neural network (RNN) models with constant memory and compute costs such as DeltaNet, Mamba or xLSTM. These models can be unified by noting that their recurrent layer dynamics can all be derived from an in-context regression objective, approximately optimized through an online learning rule. Here, we join this line of work and introduce a numerically stable, chunkwise parallelizable version of the recently proposed Mesa layer (von Oswald et al., 2024), which could only run sequentially in time and was therefore not scalable. This layer again stems from an in-context loss, but which is now minimized to optimality at every time point using a fast conjugate gradient solver. Through an extensive suite of experiments study up to the billion-parameter scale, we show that optimal test-time training enables reaching lower language modeling perplexity and higher downstream benchmark performance than previous RNNs, especially on tasks requiring long context understanding. This performance gain comes at the cost of additional flops spent during inference time. Our results are therefore intriguingly related to recent trends of increasing test-time compute to improve performance -- here by spending compute to solve sequential optimization problems within the neural network itself.

Published as a conference paper at ICLR 2026
MESANET: SEQUENCE MODELING BY LOCALLY OPTIMAL TEST-TIME TRAINING
Johannes Oswald Nino Scherrer'
Seijin Kobayashi', Luca Versari', Songlin Yang", Maximilian Schlegel', Kaitlin Maile', Yanick Schimpf', Oliver Sieberling", Alexander Meulemans', Rif A. Saurous', Guillaume Lajoie', Charlotte Frenkel', Razvan Pascanu", Blaise Agüera y Arcas' and João Sacramento'
1 Google, Paradigms of Intelligence Team, Google Deep.indd MIT CSAIL
ABSTRACT
Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention. Although widely adopted, transformers require scaling memory and compute linearly during inference. A recent stream of work linearized the softmax operation, resulting in powerful recurrent neural network (RNN) models with constant memory and compute costs such as DeltaNet, Mamba or XLSTM. These models can be unified by noting that their recurrent layer dynamics can all be derived from an in-context regression objective, approximately optimized through an online learning rule. Here, we join this line of work and introduce a numerically stable, chunkwise parallelizable version of the recently proposed Mesa layer (von Oswald et al., 2024), which could only run sequentially in time and was therefore not scalable. This layer again stems from an in-context loss, but which is now minimized to optimality every time point using fast conjugate gradient solver. Through an extensive suite of experiments study up to the Dillion-paramed scale, show that optimal test-time training enables reaching lower language modeling perplexity and higher downstream benchmark performance than previous RNNs, especially tasks requiring long context understanding. This performance gain comes at the cost of additional flops spent during inference time. Our results are therefore intriguingly related to recent trends of increasing test-time compute improve performance here by spending compute to solve sequential optimization problems within the neural network itself.
INTRODUCTION
While Transformers dominate sequence modeling, their per-token computational and memory require- ments scale linearly with sequence length during inference. This limitation motivates the development of efficient recurrent neural networks (RNNs) with constant complexity, particularly for autoregres- sive tasks like language modeling. Recent progress has focused on fast weight programming layers, which process a given sequence by representing and learning a linear model in their activations (Schmidhuber, 1992; Schlag et al., 202 Yang et al., 2024c; Dao & Gu, 2024). Such "fast weights' undergo one learning step whenever the input sequence advances, following simple Hebbian (Hebb, 1949) or error- aroo-correcting (delta) rules (Widrow & Hoff, 1960). Both rules correspond to gradient descent on suitable quadratic loss function, measured on the latest input.
Here, we take this concept one step further, and design an optimal fast weight programming layer. Following previous related work, we consider linear fast weight models, and measure how well a given context is modeled using quadratic loss. However, instead of gradually learning through gradient descent, we design a layer that always responds with the optimal fast weights, which achieve minimum loss on all data seen so far. This allows retaining past information while adapting new evidence quickly as a sequence unfolds. Our work builds off the recent recurrent Mesa layer (von Oswald et al., 2024), proposing a version of this layer that is parallelizable leveraging matrix multiplication accelerators, numerically stable, and that allows for context- depeenett forgetting. Moreover, the layer dynamically adapts its computational cost test time the sequence hand. This is because the

---

Published as a conference paper at ICLR 2026
Linear
Lineer
RMSNomm
Linear
RMMNNrm
Mesa Rule
(B) Gated MLP block
Sequence mixing black
Cony
Cony
Conv
RMSNom
MHH. MambaZ, DetaWet or Mosa
Linar
Linear
Linsar
(A) Residual block
Recurrant
Figure 1: Model Architecture of the MesaNet. (A) We adopt the widespread decoder-only transformer architecture (Touvron et al., 2023) stacking N residual blocks of channel mixing (B) and sequence mixing (C) components. (B) Channel mixing a vanilla SwiGLU MLP. (C) Sequence mixing is performed by Mesa layer. From its inputs, it generates keys, queries and values as well as input and forget strengths. These are then processed according to the Mesa Rule (Fquation 7). We compare the MesaNet to models which share the exact architecture and only change the sequence mixing rule multi-head -etddattentionn (MHA), XLSTM, Mamba2 or (Gated) DeltaNet.
layer introduced here explicitly invokes an external solver, for which the number of iterations required to reach a given stopping criterion differs across sequences. We summarize our contributions below:
A novel Mesa layer which is parallelizable over sequence length and flexibly allocates test- time computation: We adapt the previously proposed Mesa layer (von Oswald et al., 2024) to allow for chunkwise parallel training. We leverage an equivalence of the conjugate gradient (CG) method over multiple time steps with gated linear self-attention, which allows using established hardware-efficient training (Yang al., 2024a). During inference, the layer reallocates test-time compute dynamically as different sequences lead to varying CG iterations to reach stopping criterion, allowing to trade off test-time compute and performance.
The MesaNet a strong language model: We train 140M, 440M and 1B parameters MesaNets, see Figure 1, on the SlimPajama dataset (Soboleva al., 2023). On all of these scales, the MesaNet reaches lower validation perplexity compared to models such as Mamba2 (Gu & Dao, 2024). xLSTM (Beck et al., 2024), DeltaNet (Yang et al., 2024c), Gated DeltaNet (Yang et al., 2024a) and Transformers (Vaswani et al., 2017) with the same base architecture.
find that while reaching the same or better perplexity on language modeling, all RNN models reduce perplexity remarkably differently, namely focus on early tokens in the sequence while transformers excel at later tokens. We further disentangle downstream language benchmarks according to their need for global or only local language modeling, through controlled Sliding- Window Attention ablations. We find that MesaNet outperforms all modern RNNs on global reasoning, in-context learning & in-context recall benchmarks, but unsurprisingly still lack behind Transformers in in-context recall.
In-depth analyses of modern RNNs including MesaNet:
Intriguingly,
2 A PARALLELIZABLE MESA LAYER
We consider autoregressive sequence modeling tasks where the objective is to predict element ex+1 E RR given a sequence of token embeddings e = (e+) At present, autoregressive sequence modeling dominated by architectures based on the causally-masked softmax self-attention layer, whose token updates ét f- ét + Aet follow the rule Aet = >n PA Vn, (KKT,19..) where Hh. Wh.get Rh is referred to as a query, each column ky.e' Wh.ter E Re of matrix KE,t E Reaxt as key, and each column th,t' = Wr, E R of matrix Vn,, E R value; this paper, we follow the convention that vectors are column vectors. The parameters of this layer the matrices [(Ph, Wh.g. WR,,, Wr,w))fe for all H heads; for notational simplicity, we omit positional encodings and absorb bias terms, and assume here for conciseness that all heads equally sized. The function applied vector E E returns attention weight vector: in the standard transformer, (a)) softmax(a);
( - exp(ar))) )1 exp(a;) (Vaswani et al., 2017). Since each head is processed independently and only interacts through the summation in Aet", for simplicity we drop the head index h and the projection matrix P in what follows.
2

---

Published as a conference paper at ICLR 2026
Linear self-attention and test-time training. We focus the case where is the identity function. This yields a linear attention layer (Schmidhuber, 1992), which as we will see next turns out to be a linear RNN (Katharopoulos et al., 2020):
224=-
(1)
Unlike its softmax counterpart, linear attention can be implemented recurrently, by maintaining and updating a matrix-valued state E 188xXA, according to the linear dynamics
0 - + B,uck,"
Above, add forget gates "/ and input gates Bg which have been shown to improve performance (Yang et al., 2024a). Both are usually function of the current input et, like queries, values and keys, but bounded within [0, 1J. Importantly, and in contrast sofumax self-attention, linear attention only requires constant memory and compute to predict the next token. As we review below and more extensively in Appendix A, a series of recent bugh-performancee models (e.g., Gu & Dao, 2024; Peng et al., 2023; Beck et al., 2024; Schlag et al., 2021a; Yang et al., 2024c;a) can be cast into the same basic linear self-attention model (equation 1) using variations of equation 2.
Such modern RNNs can also be seen from the unifying perspective of test-time training (Schlag et al., 2021a; Liu et al., 2025; von Oswald al., 2024; Wang et al., 2025; Behrouz et al., 2025b). Under this view, the key-value linear map Φ, _h + " introduced in equation is learned from the data in context Eist. Let us introduce a time-varying loss, from which we will derive a gradient-base -ased dynamics for Ø
+ =(
TT
Above, 4 measures the instantaneous loss incurred at the current time step, and the second term acts as regularizer with strength controlled by symmetric n, x ng matrix A₁- Now, setting (+) = 1, Hopficld
8, (b) == u, Φk; and A, 1-76 I, and letting evolve through online gradient
descent, 9, 2 Be V VoLL(( (equation 2). In passing, we have also connected modern linear attention to classical associative memory models (Schlag et al., 2021a): 2 Hoppield is the energy function that governs continuous-state Hopfield networks, and Φ is learned through Hebb's associative rule (Hopfield, 1984; Hertz et al., 1991). If we take instead the squared error loss (2) = 1,-0" (4) ;= El0i Lbg 12, we recover DeltaNet (Schlag et al., 2021a; Yang et al., 2024c:a), which learns linear model with the online delta rule (Widrow & Hoff, 1960). Recent work has extended the DeltaNet to perform mini-batch updates, and to perform gradient updates on 1-hidden-layer MLP (Sun et al., 2025), and Titans adds momentum to the batche gradient update (Behrouz et al., 2024). We return to this point in Appendices A and B, where we discuss additional related work from the viewpoint of test-time regression, and derive in more detail the update rules above.
n + Borkk we recover gated linear attention
The Mesa layer: optimal test-time regression. this work, we revisit the recently proposed Mesa layer (von Oswald et al., 2024), also referred to as intention layer in the context of non- autoregressive models (Garnelo & Czarnecki, 2023). This layer again updates tokens according to the linear self-attention rule (equation 1) but now defines the linear map o as the solution of test-time optimization problem, where symmetric positive definite matrix A. R x controls the strength of a quadratic regularizer:
$ mesa arg min (,()), with (
Serl leve - 2kel1 + Tr
2
In all our experiments, we take static, diagonal regularizer, with A, = A V, and A₂ > 0. Above, the cumulative forget factor See = 112 II causally weighs the contribution of past losses until the present (t' = I,... t), taking into account the forget factors >r E [0, 1] so far. The output Aemese of the Mesa layer depends on the (unique) solution dimesa, which can be expressed in closed form:
-1
Ae mesa
G.messs q:
Σ Vu
Σ ke + +A) qt
(5)
= G., + -
(6)
We compute @pmss step by step in Appendix D.
3

---

Published as a conference paper at ICLR 2026
The Mesa layer differs from the test-time training models reviewed above in two key ways. First, instead of considering an instantancous loss measured only at the current input é, as in equation 3, the Mesa layer optimizes the cumulative regularized squared-error loss taking into account all data el so far. While at first this may seem impossible to achieve under a constant memory requirement, the Mesa layer circumvents the need to explicitly keep past tokens in memory (as in softmax self- attention) and exploits the fact that C, is a quadratic function of & (Gauss, 1821). Second, instead of taking a single gradient descent step, the Mesa layer learns & to optimality every time point. We note that the related Longhorn model (Liu et al., 2025) also derives a recurrent layer via the minimization of a quadratic loss, but its loss is evaluated only on the latest input as in equation 3, yielding variant of DeltaNet. We further note that concurrent work (Atlas; Behrouz et al., 2025a) variant of the Mesa layer, while also allowing the model to be optimized at test-time to be nonlinear, as in (Sun et al., 2025). We present the update rules and test-time objective functions of these two related works in Appendix B.
corresponds to a sliding-windon
The Mesa layer is the optimal (in the squared-error sense) linear associative memory (Kohonen & Ruohonen, 1973), and it can store a new association instantaneously (one-shot), whereas DeltaNet requires in general multiple pattern presentations to reduce memorization error (Hertz et al., 1991). This fast learning property of the Mesa layer can be further understood by recasting it as a second- order online learner (cf. Appendix H); DeltaNet only uses first-order derivative information to learn.
Von Oswald et al. (2024) proposed to determine @emesa following classical recursive least-squares. Although computationally attractive at inference, we now stress two shortcomings of this approach. First, forgetting (0 S 7t. < 1) leads numerical instabilities, and requires a regularization term A that decays exponentially with time. Second, this original version of the layer is not parallelizable, and it therefore heavily underutilizes current matrix-matrix multiplication accelerators such as GPUs and TPUs during training. We explain this in detail in Appendix H.
new parallelizable Mesa layer with adaptive forgetting and regularization. To overcome these issues, we propose novel parallelizable version of the Mesa layer which allows for dynamic forgetting. Instead of computing du recurrently, we solve a linear system of equations in parallel, for each query qe:
mesa G. + +A) A)- =G,linsolve(H, A, qe)-
The equation above can be computed by maintaining and updating two state variables, S. = (Gr, Ht), through the following linear recurrence relations:
G. 7 + Be
= H + Bekighi
(8)
where before " E [0, 1] is forget gate and 3, [0, 1] is input gate. We adopt the conjugate gradient method to obtain a solution of = linsolve( H, + A. q.) = (H, + A)- 4. (Lanczos, 1950; Hestenes et al., 1952). This yields a numerically stable Mesa layer as linsolve H, + A, q:) is stable irrespective of forgetting strength, albeit at a higher memory cost compared to single matrix state RNN models, as an additional matrix of size n. X ng needs be propagated forward alongside the standard matrix of size n, X no. Although the RNN state size increases, this expansion amounts to less than 1% of the entire memory footprint of models trained in this paper, which includes both state and parameters.
To enable efficient training. we introduce a chunkwise parallelized (Hua et al., 2022: Yang & Zhang, 2024) algorithm to compute equation 7. Our method builds on top of established efficient implementations of GLA, that we briefly review now. First, note that the output of this layer can be written as of GLA G.g. = = S.u,k 9t- Let us chunk a sequence of length T in T/C chunks of size C, with E {0, (O,C. T matrix multiplication and parallelization across time for GLA is that, given a chunked state variable
i-
C). The crucial insight to enable leveraging matrix-
Ge, we can compute the output at time c < t < c C' as of GLA = (G. + Eizot1 Suurk, )ge = Geg El=0t1 Suo,k qo which can be done in parallel for E {c 1, +C}. In matrix notation we write
OGLA GoQo + V. ① (K_ (T)),
where K₁ = [k....., and OGLA, Va, Q. accordingly, and Z, is a upper triangular matrix of size C. x C containing the appropriate forgetting terms.
4

---

Published as a conference paper at ICLR 2026
Now, we highlight that the Mesa layer can be decomposed into two parts:
messa
Swork qi and qi = (H, + '
(10)
The first part is equivalent to GLA, and can therefore be computed efficiently as just described. It therefore remains to shown how to obtain Qh 1 ......... -hhe-hh within a given chunk of size C' in parallel. As we explain in detail in Appendices C & D, the key observation that the compute- intensive part of a CG iteration boils down to Ei= Sr.kkk, J p. with p its current search direction, computation that is once again in the GLA form. Alongside its fast convergence properties, this is the reason for picking the CG method as our solver, as allowed us to leverage existing efficient chunkwise parallel linear attention implementations. The new Mesa layer proposed in this paper therefore admits a parallel training mode with O(T) complexity. alongside the recurrent inference mode with (1) complexity. In Appendix D, further show how to efficiently compute gradients through the layer in chunkwise parallel form. Finally, we discuss details on precision within our CG solver in Appendix G.5.
TRAIN AND INFERENCE TIME OF THE MESA LAYER
Chunkwise parallel Mesa layer leads to competitive train time. In Figure 2, we report training and inference times TPUv5 and H100 for both transformers (MHA), common RNN alternatives and the MesaNet. Despite having to solve t- linear systems of equations per layer during training as well compute gradients through the found solutions, the MesaNet remains competitive train time with respect to MHA and RNN alternatives.
The Mesa layer, applied with static k, is relatively slow especially early in the sequence. We present in Appendix Table an analysis of the memory and computational costs of inference, comparing the Mesa layer to MHA well recently developed RNNs. This overview highlights tension that the MesaNet faces. On the one hand, if the number of conjugate gradient (CG) steps is to zero obtain of = Qr. and recover gated linear self-attention (GLA) and its compute and memory requirement. Thus, we require k >> 0 for the Mesa layer to differ from GLA, which provides lower bound for the computational cost of the Mesa layer. Note that the Mesa layer is, in terms of flops, roughly & times as costly as linearized transformer models such as GLA, Mamba2 and xLSTM and k - times more costly as (Gated) DeltaNet. Furthermore, because the total cost of executing the CG method grows with kng, there is a maximal value of k for which the Mesa uses fewer flops than MHA for given sequence length.
80
Inferace (ms)
Tolcen Throughout:/
Mesa-CG-S
60
20
Mesa CG=15
Mesa-CG=30
40-
Train
10
MHA
Transformer
Gacied Dehaftet
20
GLA
Mesa CG-30
Dellake:
Mess--tim15
256
2k
4k
8k
256 2k
4K
8k
26 16
4K.x
8
1
x
Sequence Length
Sequence Longth
Sequence ength x Batch size
Figure Train and inference time Mesa layer using different number of CG steps. Left: Train time of a single Mesa later on a TPUv5: output the entire sequence, compute the cross entropy loss, and gradients w.r.t layer parameters. We use batch size of 4, key size of 128 and 8 heads. Center: Inference time of a single Mesa layer TPUv5: compute the next token given certain context length. We use batch size of 128, key size of 128 and heads. Righr: Token throughput (in thousands) when training 1B parameter models on H100 (Dao, 2023) transformer implementation with a triton-basee chunkwise
GPU. We compare parallel implementation of RNN models, including the MesaNet which uses 30 or 15 CG steps across all layers. All models use a key size of 128 and share the same backbone, see Appendix G. We observe competitive token throughput H100s of the MesaNet despite using substantially more flops.
We show this in Figure 2 (center) for typical choice of n. 128, where we plot inference time as a function of sequence length for both MHA and the Mesa layer, when varying k. These numbers reflect the runtime of a single layer and might vary across inference use cases and accelerators.
5

---

Published as a conference paper at ICLR 2026
Layer
Recurrence
Memory read-out
Mamba2
G. Gt-1 + veki
0 G.q.
GLA
G. hG + Bevcki
0: Gr
DeltaNet
Ge = Gt-1(T Behich B
0t G.ßt
Gated DeltaNet
G. Gi-1(r: Brkkkk _ kr k
0. G.q.
mLSTM
G. G + Brucki 2 = 7t + Beks
0: Giqs/max(L4 12 qil)
Mesa
C. 7x Beveki .H: = -. + 3e
0r Gylinsolve + A,
Table 1: Overview of recent lincar recurrent models which we compare to in this work, except for LRU layers, see De ct al. (2024).
The Mesa layer allocates test-time compute dynamically. Being a test-time optimizer, the Mesa layer offers principled way for dynamically allocating test-time compute. The number of CG steps k required to reach a given desired error tolerance is generally head,, sequence- and token- specific due to the context-dependence of the linear systems H, + A to be solved. Via utilization of a stopping criterion, the Mesa layer thus exhibits dynamic inference (and potentially training) costs. This dynamic test-time compute feature of the Mesa layer draws both parallels and differences to softmax self-attention: whereas softmax self-attention increases compute (and memory) as a function of sequence length independently of the sequence being processed, the Mesa layer adjusts compute dynamically, according to the incoming data it needs to process. We provide in Section 5 an experimental analysis of this property of the Mesa layer in trained MesaNets.
MESANET IN A LANGUAGE WORLD
Here we present results obtained 1 BB-parameter models trained 50B tokens from the SlimPa- jama (Soboleva et al., 2023) dataset, and refer to Section L for an extended analysis. comparing models ranging from 140M, 440M up to 1B parameters, each on 15B and 50B tokens. Furthermore, we report strong results on synthetic environments in Section K, which we omit for brevity here.
Architecture & baselines. For the main model backbone, we follow the architecture of common transformers, and employ N stacked residual blocks with 1) a sequence modeling part such as fiii-heed-attettton (MHA) or the Mesa layer and 2) a gated MLP block (see Figure 1). As baselines, we compare to a number of other efficient alternatives to MHA based on linear recurrent layers: Mamba2 (Dao & Gu, 2024), Gated Linear Attention (GLA) (Yang et al., 2024b; Katharopoulos et al., 2020), xLSTM (Beck et al., 2024), (Gated) DeltaNet (Schlag et al., 202 Yang et al., 2024c;a) and Hawk (De et al., 2024), see Table 1. The latter differs from the models reviewed in Section by employing vector-valued state, being closer in spirit to a (now linearized) traditional LSTM (Hochreiter & Schmidhuber, 1997). Furthermore we investigate a recurrent hybrid Hawk-Mess model alternating between a linear recurrent unit (Hawk) and the Mesa layer which we motivate in the next section.
Controls. On top of related work, train transformer models with Hiding-Windoww Attention (SWA) (Beltagy et al., 2020) of varying window sizes. These models have constant per-token memory and compute cost. The motivation study SWA models based on the assumption that transformers as well as SWA models have near perfect recall capabilities, at least within their attention window. Therefore, they provide a simple and interpretable control to study language modeling, reasoning and in-context recall capabilities of RNNs.
Setup. We tokenize the SlimPajama datasets using the byte-level BPE tokenizer introduced in GPT-2 (Radford et al., 2018; Brown et al., 2020a) following Beck et al. (2024) and train all modes on a sequence length of 2048 and fixed ordering of training data. For each model configuration, we scan over range of learning rates, and select the model that minimizes perplexity on the holdout validation dataset of SlimPajama. For exact hyperparameters and training specifications for each model, see Appendix G. For all results, unless otherwise specified, we use MesaNets with fixed amount of 30 CG steps. See Appendix M on varying CG steps during training and Section 5 on using the CG stopping criterion to invoke dynamic test-time compute.
We stress that through sharing the exact same architecture backbone, tokenizer, data and data order across all models, while using the same number of parameters and independently tuned learning

---

Published as a conference paper at ICLR 2026
STIM
LMB.
WIKI
PG10
GOV.
QASP:
AVG
HAWK
111 1
131 1
GLA
DELTANET
HAWY-MESA
MIHA
MambaZ
Haok
11.24
12.23
12.23
10.95
10.33
14,89
14.02
14.43
14.58
MAMBA2
GATED-DRLTANET
SWIA-64
GLA
sLSIM
10,99
11.01
26.93
29.77
11.77
10.95
1092
10:00
9,99
13.35
13.52
14,00
14,00
NL. Diffeennce to MHA.
0,04
-Gated DellaNes
DeltaNis
11.01
10,39
26,79
17.08
11.75
11.58
11,00
10.81
10,00
9.58
13,44
13.28
13.57
14,05
0.42
Mesa
10.53
26.78
11,49
19.71
9,50
13.13
13.79
0.00
-RaokMMea
10.78
26.79
11,55
10.60
9.79
13,20
13.75
-WW..
16,46
20.03
19.42
16,42
17.86
29.15
21.54
-0.02
-SWA-4
12.37
37.76
14.14
12.51
11.56
16.77
15.85
SSWA-1024
11.00
27,22
11.78
10.92
9,79
13.11
13.97
0.04
Transfomer
10,56
27.16
11,42
10.74
9,59
12.86
13.19
04256
512
1924 Token Position
2048
22
Takon Postitio
Table 2: Language Modeling Performance (PPL )) of 1B Models (50B Tokens) evaluated on sequence length of 2048). Mesa and Hawk-Mest show strong performance on all benchmarks, matching or exceeding Transformer baseline w.r.t. avg. per-token PPI.. Lambada (LMB.) scores are higher due to significantly shorter sequences ( 256) with an average of 78 tokens.
Figure 3: NLL Difference relative a Trans- former (IB models, 50B tokens) on Slim Sajma. Most recurrent layers show superior language mod- eling performance in terms of NLL up to the 61'th token. MesaNet and Hawk-Mesa extend the advan- tage beyond 512 tokens. The advantage early in the sequence is even more apparent in log scale (right).
rate for all models, we aim to provide fair 1-1 comparison'. This controlled setup should allow to solely assess differences on the sequence mixing layer while reducing noise. Note, however, that this backbone might be suboptimal choice for RNNs, including the MesaNet. Related work has tuned architectures to their specific sequence layers (Beck et al., 2024; Gu & Dao, 2024). However, these architectural optimizations prevent the integration of Mixture-of-Exxeee -f-peertsss layers, a heavily used building block in current language models. Therefore, we carefully evaluate all sequence layers on the same backbone, based on the widespread decoder-only transformer architecture here, the Llama2 model (Touvron et al., 2023), including rotary position encodings (RoPE: Su et al., 2024) when using softmax attention layers. This backbone does not fuse MLPs with sequence layers, allowing for a direct comparisons between layers. Furthermore, we did not attempt to optimize the architecture e.g.. key size and number of heads for the Mesa layer.
Comparison to the original mesa layer. We considered comparing to the original sequential-in time Mesa layer (von Oswald al., 2024). However, because this model was already an order of magnitude slower when training at the 400M parameter scale. and suffered a large increase in SlimPajama language modeling perplexity of about 3.2 points (~23% performance degradation) due to the inability to train with forget gates, we did not pursue these comparisons further. These results directly motivate the new Mesa layer introduced in this paper.
4.1 LANGUAGE MODELING (WITHIN AND BEYOND TRAIN SEQUENCE LENGTH)
We measure a model's ' general language modeling capabilities first by assessing average por-token perplexity (PPL) (Jelinek al., 1977) on a set of benchmarks. We report PPL on the hold-out validation set of SlimPajama (Soboleva et al., 2023), as well as Lambada (Paperno et al., 2016), Wikitext-2 (Merity et al., 2016), PG19 (Rae al., 2019), GovReport (Huang et al., 2021), and Qasper (Dasigi et al., 2021) the train sequence length and beyond. Because uniformly averaging over all tokens might masquerade important differences between models, we additionally investigate average per-token PPL conditional on sequence position. As we see below, this turns out to be crucial factor when comparing RNNs to transformers.
MesaNet is a strong language model early in sequences. When evaluating the training sequence length of 2048, McsaNet and Hawk-MesaNet outperform all recurrent baselines on all benchmarks on the common metrie of average per-token PPL (see Table 2). MesaNet matches average the performance of the transformer baseline, while Hawk-MesaNet even surpasses it. Notably, a SWA model with window size of 1024 outperforms the majority of recurrent baselines. However, attaining similar PPL scores does not imply equivalent language modeling abilities at different sequence lengths (Lin et al., 2025). Conditioning on the token position, and assessing the NLL difference relative to transformer, reveals, surprisingly, that most recurrent layers exhibit superior language modeling performance early in the sequence but fall behind later in the sequence (see Figure 3). Recurrent models show especially strong performance on short sequences up to 64 tokens.
'Related work such as Yang et al. (2024a), Behrouz et al. (2024) and Behrouz et al. (2025a) use a single learning rate for all models which likely leads to biased and unfair comparisons. Behrouz et al. (2025a) further inherit baseline results from previous work which use a different tokenizer, confounding the comparison further.
7

---

Published as a conference paper at ICLR 2026
Model
Reasoninx Global (Ac: ()
Reasoninx Local
In-Contest Recall
Scrumable 100-dwe
Translation
Model
Reasning Global
Reasoninx Local
In-Contst
Scruuble 100l-der (Acc T1
Trandiatio
Sbolel
Recall
50-ahot
(Ao:
(Aaz
(Azz
(Nimwch
(4ec
(Ao2 13
(Aoc 13
(bleu 5b
Hawk
$7.42
50.04
21.29
4.70
3.51
Hast
41:17
51:57
25.6
6.49
4.75
Mamis22
17.55 5B
68.10
10:31
1.38
2:55
Mamba2
41.62
50.51
17.67
4.19
4.18
GLSS
39.45
43.56
36.50
530
2.57
GLA
44.34
51.44
1964
722
7.58
38.97
13:90
14.89
335
2.7g
775
7.68
DeliaNi
39.72
+3.91
35.39
2,47
DeluNet
63.25
51.66
40:46
7M1
5.37
Gaced DehuNer
40.19
49.10
33.96
6.17
2.58
Gated DeliaNet
44.34
50.76
39.54
8,0
8.53
Mosa
40.38
49.64
19.30
6.2
3.83
Mox
45.05
50.49
41.70
10.10
8.17
Haw--Mows
Hawk-Moa
SWA-4
28.55
48.30
9.79
0.8
0.14
SWA-4
31,30
49.62
11.87
1,6
0.51
SWA 64
38.07
48.76
24.84
3.65
2.59
SWA
42.35
50.52
26.72
59]
4.70
SWA 1024
41,30
48.54
18.21
5.43
5.00
SWA 1024
45.08
51,35
40.84
6.00
14.17
TranSommer
42.15
43.50
49.95
6.01
3.89
TranSormer
45.34
51.62
52.27
6.95
1361
(a) 400M Params, 50B Tokens
(b) IB Models, 50B Tokens
Table Grouped Benchmark Scores (f) on models trained on 50B Tokens from SlimPajama with context length of 2048. We compare the aggregated performance of models with Linearized Recurrent Unit, Gated Linearized Multi-Head Attention, DeltaNet and MESA layers on different subsets of benchmarks. As a reference, we show the performance of Sliding Window-Attente models (SWA) with varying window sizes.
While Hawk exhibits the best performance up to this depth, the model exhibits a sharp performance decline after that. This finding motivated us introduce and investigate the Hawk-Mesa model, which combines the best short-sequence and sengeeeueecce modeling layers (as measured by negative log-likelihood). Confirming this intuition, the Hawk-Mesa outperforms the remaining recurrent models, with the MesaNet being second best: MesaNet and Hawk-MesaNer not only attain the strongest early-in- teayy-nnth--equeneee the-seqqence modeling ability, but also extend the advantage beyond a depth of 512 tokens.
MesaNet is competitive on length extrapolation with recurrent baselines, but SWA-1024 is hard- to-beat baseline. Next, we evaluate the ability to
HAWK
DELTANET
MHA-SWA-64
MAMBAZ
GATED-DELTANET
MHA-SWA-1024
extrapolate to sequences of up to 32k tokens (see Figure 4). While transformer, Mamba2, DeltaNet
GLA
MESA
MHA
XLSTM
MESA
and HawkMesa fail to extrapolate catastrophically to longer sequences on all evaluated benchmarks, MesaNet exhibits cength-xttrapolation capabilities superior to Hawk, GLA, xLSTM and on-par with Gated DeltaNet on all evaluated long-sequence benchmarks with respect to PPL scores (aggregated and conditional on token positions). However, these results should be tempered by the fact that a SWA model with an attention window of 1024 attains com- petitive benchmark scores, even superior finding is in line with recent criticism that PPL may not distinguish a model's 's ability to capture local vs. long-range dependencies between tokens (Hu et al., 2024; Fang et al., 2024). We refer Section L for detailed score breakdown and results on the Needle-in-thh--hasstcc (NIAH) benchmark (Hsieh al., 2024), where MesaNet shows strong performance.
14
8k
16k
32k
Seq. Length
Figure 4: Avg. Mcan-so-Far PPL on Long- Context Benchmarks (WIKI, GOV, QASPER).
sequence length of 32k on some benchmarks. This
4.2 LANGUAGE BENCHMARKS
We next evaluate MesaNet's ss capabilities on a comprehensive set of downstream tasks, ranging across zero-shot reasoning, in-context recall and in-context learning tasks. We evaluate on various benchmarks considered in prior work (Gu & Dao. 2024: Yang al.. 2024a: Beck et al., 2024), and complement them with few-shot learning tasks involving token-manipulation and translation. We present the aggregated results of 400M and 1B models trained on 50B tokens in ??, and report detailed scores in Section L. Across most evaluated benchmarks, the McsaNet matches or exceeds the performance of the evaluated recurrent baselines.
Transformers & MesaNet > other RNNs. Prior work (Gu & Dao, 2024; Yang et al., 2024a: Behrouz et al., 2024; Beck et al., 2024) commonly reports the average performance of a set of common-sens, reasoning benchmarks to compare models. However, evaluations of SWA models with different window sizes reveal that competitive, or even superior, scores on many of these frequently reported benchmarks be attained with attention
Zero-Shot Common-Sense Reasoning Performance:
8

---

Published as a conference paper at ICLR 2026
window size as short 4 (see Table 13). This observation strongly indicates that some of these benchmarks are exploitable by short-range language heuristics, and do not require longer-range language modeling capabilities to reach competitive scores, or are simply too hard such that we end up measuring noise. To reduce the potential benchmark noise and deconfound the results, we hence report the zero-shot reasoning benchmarks in two separate splits:
The Global Reasoning Benchmark Set encompasses all benchmarks where we observe signifi- cant performance increase with a growing attention window size. This includes Lambada (Paperno et al., 2016). HellaSwag (Zellers et al., 2019) and RACE-(M.H) (Lai et al., 2017). Within both reported model sizes (400M and IB), MesaNet outperforms all other recurrent models on average on these benchmarks. However, MesaNet still underperforms the transformer baseline.
The Local Reasoning Benchmark Set includes all benchmarks where we see little to marginal improvement with growing attention window size. This includes PIQA (Bisk et al., 2020), WinoGrande (Sakaguchi et al., 2021), ARC-(E.C) (Clark et al., 2018). SIQA (Sap et al., 2019), BoolQ (Clark et al., 2019), Open BookB (Mihaylov et al., 2018) and StoryCloze (Srinivasan et al., 2018). Unsurprisingly, we observe very similar average scores for all models. Notably, Hawk, the worst performing recurrent model on global reasoning and in-context recall benchmarks, shows excellent performance on this benchmark subset. This observation supports the hypothesis that these subsets of benchmarks are likely measure different capabilities, and highlights the differences between Hawk to e.g. the MesaNet. These analyses motivate the recurrent hybrid Hawk-Mesa model, which tries to capitalize on the complimentary strengths of the two layers.
In-Context Recall Performance: Transformers MesaNet 2 other RNNs. To gauge the ability to recall in-context information, we follow Arora et al. (2024) and Yang et al. (2024a) and evaluate models on SWDE (Lockard al., 2019), SQUAD (Rajpurkar et al., 2016), FDA (Arora al., 2023b), TQA (Kembhavi et al., 2017), NQ (Kwiatkowski et al., 2019) and DROP (Dua et al., 2019). We adopt the dinimalltransformeed versions of the benchmarks from Arora al. (2024) that adjust for the evaluation of non-instruetiion-tum -uned models. In line with the observations on synthetic benchmarks in Section K, MesaNet outperforms all other recurrent models on these tasks. Moreover, MesaNet exceeds the performance of a SWA-1024, the only recurrent model to do so. However, there remains a gap in performance relative to the transformer baseline with an attention window size of 2048.
Few-Shot Learning Performance: Transformers & MesaNet > other RNNs. Finally, we measure the model's ability to learn from few-shot demonstrations. We evaluate on two GPT3 word scrambling tasks (cycle letters in word, anagrams of all but first and last two characters) (Brown et al., 2020b) and three translation tasks (WMT- 14 FR-EN (Bojar et al., 2014), WMT- 16 DE-EN and RO-EN (Bojar et al., 2016) ). MesaNet demonstrates strong performance exceeds the performance of the Transformer on word scrambling tasks, it fails to do so in translations.
all few-shot learning tasks. While it
TEST-TIME COMPUTE ANALYSIS
In the previous section we showed results from models trained and evaluated with 30 CG steps. We study now the effect of using the MesaNet trained on 30 CG steps but evaluate the model when using a dynamic stopping criterion aiming to reducing the CG steps used at inference time. We refer again to Appendix C for description of the CG method used in this work.
Mesa objectives differ widely across heads and layers. When analysing the internals of the Mesa layer on sequences of the SlimPajama validation set, we observe bimodal distribution of condition numbers of H₁,t + AA, across heads almost in every layer, see Figure 14. In particular, we observe that heads either have 1) large and growing condition number with sequence length. or 2) rather low and constant condition number over the sequence. In every layer, there are roughly 1-2 heads for which the condition number of linsolve H,,, + Ay, 4h.z) (and therefore the number of CG steps) grows with f. This motivates dynamic allocation of CG steps in every head.
MesaNets allocate test-time compute dynamically. We test 1) reducing the number of CG steps of all layers and heads uniformly, and 2) varying the solver's stopping criterion to dynamically allocate test-time compute. As shown in Figure 7, when reducing CG steps uniformly, we observe an increase in negative log- likeliiodoood when comparing later in the sequence. This is in line with our findings on the need for higher number of steps as t grows. By contrast, with dynamic stopping criterion e, increasing yields uniform degradation
model evaluated with 30 steps, especially on tokens

---

Published as a conference paper at ICLR 2026
over sequence length. A model with stopping criterion of 10 performs on-par with the base model using a fixed number of 30 CG steps, while reducing the average CG steps used to = 9.
Reasoning (Global, 0-shot)
Reasoning (Local, D-shot)
In-Context Recall
<-3e-2
e-le-3 cmle-4
e-1e5 CG-4
CG-5
CG-7
CG-20
c=le
CG-6
CG-10
CG-30
<=3c
e-1e2
<*1e3
Maannso-far NLL
0.06
10.4
Avg CS Stees
10
0.04
8-
0.02
0.00
-204-
20
20
24
28.44
Accunccy
Accuracy
Length
Langth
Figure 5: Effect of Number of Conjugate Gradient (CG) Steps on SlimPajama Perplexity within and beyond train context length. We show here the effect of reducing the number of CG steps during inference on token perplexity across token position of a IB MesaNet trained on 50B tokens. We either use fixed number CG steps uniformly across the model or apply dynamic stopping criterion >> 0.
6 DISCUSSION
We present a chunkwise parallelized, numerically stable version of the Mesa layer (von Oswald et al., 2024), and scale it up to 1B parameter language models. This layer generates a prediction by solving an optimization problem, which yields a linear model that best fits a given sequence. Our Mesa layer can allocate test-time compute dynamically according to the stopping criterion. Complex sequences are then modeled by many of such layers, while interleaving them with MLPs, into MesaNets.
This approach has ties to multiple long-running lines of research. It relates to alternatives to end- to-end differentiation based on stacks of greedy local learners (c.g., Hinton et al., 2006; Nøkland & Eidnes, 2019; Veness et al., 2021), bringing these to the fast inference timescale, and then delegating to nonlocal backpropagation- based learning the role of determining which optimization problems must be solved at inference time. This in turn relates to mesa-optimization (Hubinger et al., 2019), since test-time optimization objectives (though not the optimizers themselves) are discovered by (base) sequence prediction loss optimization. The idea of specifying the output of a neural layer through an optimization problem is an old one (Amos & Kolter, 2017: 1; Gould et al., 2021), with roots least to energy- -bsed neural models (Hopfield, 1984). Finally. the Mesa layer is perhaps most related to fast weights of Schmidhuber (1992), replacing Hebbian with locally-optimal learning.
The Mesa layer extends state-of-the-arr recurrent language models such as Mamba (Gu & Dao, 2024), RWKV (Peng et al., 2023), xLSTM (Beck et al., 2024), and (Gated) DeltaNet (Schlag et al., 2021a; Yang et al., 2024c;a), which can also be motivated by an in-context regression loss, but update their fast weights with a slower GD process. In new in-depth evaluation, we show that RNNs, in particular MesaNets, outperform transformers significantly early in sequences, while underperforming in next-token prediction and benchmark performance when longer contexts needed. It should be stressed that it is exactly in the long-context regime, however, that RNNs show advantages over transformers in terms of inference time. In our view, these observations merit further investigation, and may serve as the starting point for novel RNN scaling law analyses.
The biggest shortcoming of the MesaNet in its current form is the increase in test-time compute despite its dynamic nature. One possible way around this may lie on the findings of Figure 14, where we see that heads which require more CG steps often do not forget, i.e. 7 irrespective of the input data. This motivates leveraging the similarity of solutions from neighboring time steps, to warm-start optimization of consecutive steps. Moreover, one could envision hybrid approach where the chunkwise parallel CG method introduced in this paper is used during training, while then reverting back to using the efficient Sherman- Morrisson recursion at inference time, which could work given the almost almost-no-forgetting = 1 condition. We point to additional discussion points in Appendix J and leave investigating these directions for future work.
ACKNOWLEDGMENTE
The authors would like to thank wholcheartedly Sarthak Mittal for his contributions to the work after the official submission deadline. Further, we would like thank the anonymous reviewers for their
10

---

Published as a conference paper at ICLR 2026
time and thoughtful reviews that helped improve this work. We would also like to thank Soham De and Stephen Roller for insightful discussions and advice on model pre-training at the beginning of the project. Moreover, would like to thank Angelika Steger, Rajai Nasser, Maciej Wołezyk, Eyvind Niklasson, Ahmad Beirami, Dimitris Papailiopoulos and the members of the Paradigms of Intelligence team for fruitful discussions and their support throughout the project.
REPRODUCIBILITY STATEMENT
We provide pseudocode for the conjugate conjugate-gradient implementation of the Mesa layer in Section C and Section D, and provide detailed descriptions regarding numerical precision in Section G.5. All other important aspects for training (e.g. tokenizer, data, context length) are given in Section 4. We will furthermore, upon publication, provide triton-based open source implementation of the MesaNet and Mesa layer, as well as educational colab notebooks to further ease reproduction and experimentation with our layer and models. Moreover. focused not only on improving the numbers of our proposed method but scanned hyperparameters of the related works extensively (see Section E). Lastly, we focused on an apples-to-apples comparison between methods by using the exact same backbone while only varying the sequence layer.
REFERENCES
Ekin Akyürek, Bailin Wang. Yoon Kim, and Jacob Andreas. In-context language learning: Architec- tures and algorithms, 2024. URL https: / //waxivvvvv////////////aaaa/aavl org/ abs 2400 12973.
Brandon Amos and J. Zico Kolter. OptNet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, 2017.
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models. arXiv preprint arXiv:2312.04927, 2023a.
Simran Arora, Brandon Yang, Sabri Eyuboglu. Avanika Narayan, Andrew Hojel, Immanuel Trummer, and Christopher Ré. Language models enable simple systems for generating structured views of heterogeneous data lakes. arXiv preprint arXiv:2304.0944 09433,, 2023b.
Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughput tradeoff. arXiv preprint arXiv:2402. 1866, 2024.
Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z. Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. In Advances in Neural Information Processing Systems, volume 29, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: 2014.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 2019.
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael K Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. XLSTM: Extended long short-term memory. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL https: /ttp:ps:://ppereee http://ppppeeeeeeeeeeeeeeeeeeeep net/ forum A
Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning memorize at test time, 2024.
URL https::/ /tappp:://rxiite /arxiviiv. org/ abs abs/2501.. 00663.
Ali Behrouz, Zeman Li. Praneeth Kacham, Majid Daliri, Yuan Deng, Peilin Zhong, Meisam Raza- viyayn, and Vahab Mirrokni. Atlas: Learning to optimally memorize the context at test time, arXiv preprint arXiv:2505.23773 2025a.
11

---

Published as a conference paper at ICLR 2026
Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni. It's all connected: A journey through time memorization, attentional bias, retention, and online optimization. arXiv preprint Sttv:22504.11173, 2025b.
Iz Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long- doc-meet transformer. arXiv preprint arXiv:2004.055555 2020.
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, et al. Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405.14782, 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. Proceedings of the AAAI conference on artificial intelligence, 2020.
Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 2016 conference on machine translation. Proceedings of the First Conference on Machine Translation, pp. 131-198, Berlin, Germany, August 2016. Association for Computational Linguistics. URL http:: www. ac web. /rggaanhoo// ogy/ /t//1/////// 6 /W1 6-001
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling. Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amann Radu Soricut, Lucia Specia, and Ale s Tamchyna. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop Statistical Machine Translation, pp. 12-58, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. URL http: /www. aclweb. org/ aat/alothoooyyyy /W/ /WW4444 4 /114-114-000 -302..
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020a,
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, 2020b.
Ryan Burnell, Wout Schellaert, John Burden, Tomer D Ullman, Fernando Martinez-Plumed, Joshua B Tenenbaum, Danaja Rutar, Lucy Cheke, Jascha Sohl-Dickstein, Melanie Mitchell, et al. Rethink reporting of evaluation results in ai. Science, 380(6641)) 136-1138, 2023.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v 1179. URL https: https///acaaaattthhoooooo wtwp:/loooyyyo1111111111 ott.tt/se///aaaaccooooooo/.////p
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference of Learning Representations, 2021.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 190::/905//000 10044, 2019.
12

---

Published as a conference paper at ICLR 2026
Kevin Clark, Kelvin Guu, Ming- Wei Chang, Panupong Pasupat, Geoffrey Hinton, and Mohammad Norouzi. Meta-leaming fast weight language models. In Yoav Goldberg, Zomntsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference ral Language Processing, pp. 9751-9757, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. doi: 10. 10.18653/v1/2022
Empirical Methods in Natu-
http///2eeeemmmmam....666 URL
battps://aee atttss//aaclanthllogg. org/ 2 1aaapppmaiii ma main.. 661/.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.054: 2018.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL abs 2307. 08691.
pettps:/arriiv..o
Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality, 2024. URL httpt:tttt//tttttt orgg abs 2405 1.445.210655 21060.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of informattion-seeking questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011, 2021.
Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian- Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre, Griffin: mixing gated linear recurrences with local attention for efficient language models, February 2024. URL http::/arxiv...ttt///////t/ [cs].
/a///bb//2440 /2402 19427. UrXiv:2402.19427
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Eukasz Kaiser. Universal transformers, 2019. URL https::/ /twpiv.aiii/v..///////..///.aav. org/ abs / 1807..3819.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint (rX1v:1903.00161, 2019.
Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. What is wrong with perplexity for long-context language modeling? arXiv preprint arXiv:2410.23771 2024.
Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: towards language modeling with state space models. In International Conference of Learning Representations, 2023.
Marta Garnelo and Wojciech Marian Czarnecki. Exploring the space of key-value-query models with intention. arXiv preprint arXiv:2305..0200: 2023.
Carl Friedrich Gauss. Theoria combinationis observationum: erroribus minimis obnoxiae. Societas Regia Scientiarum Gottingensis, 1821.
Zhengyang Geng, Xin-Yu Zhang, Shaojie Bai, Yisen Wang, and Zhouchen Lin. On training implicit models. In Proceedings of the 35th International Conference Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 9781713845393.
F.A. Gers, J. Schmidhuber, and F. Cummins. Learning forget: continual prediction with LSTM. In 1999 Ninth International Conference on Artificial Neural Networks ICANN 99. (Conf. Publ. No. 470), volume pp. 850-855 vol.2, 1999. doi: 10.1049/cp: (1944///p:199922111 19991218.
Stephen Gould, Richard Hartley, and Dylan John Campbell. Deep declarative networks. IEEE Transactions Puttern Analysis and Machine Intelligence, 2021.
Alex Graves. Adaptive computation time for recurrent neural networks, 2017. URL https: //axxiv.org/ab.... 603 08983.
13

---

Published as a conference paper at ICLR 2026
Riccardo Grazzi, Julien Siems, Jörg K.H. Franke, Arber Zela, Frank Hutter, and Massimiliano Pontil. Unlocking saae-ttackkng in linear RNNs through negative eigenvalues. In The Thirteenth Internationaa Conference Learning Representations, 2025. URL httpa://openzevier net/ net/ffoumm: forum? dd=0yTo3tVKks
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024. URL htttps:///ppereeeeeeeeee
net/ centt/fornig forum? ? i=
LEYskwIVY2.
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Com- bining Recurrent, Convolutional, and Continuous-time Models with Linear State Space Layers. In Advances in Neural Information Processing Systems, volume 34, pp. 572-585555 Curran As- sociates, Inc., 2021. URL https: / httttps://ppooceedingg nwwrips. wwtpppper/22///aaeeeeeeeeeeeeeee 05546b0es ab9 75 6 Abstract html.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The International Conference on Learming Representations (ICLR), 2022.
Donald O. Hebb. The Organization of Behavior: A Neuropsychological Theory. Wiley, New York, 1949.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2023. URL https: //arxiv. org/ awwaaabb/// /006 08415.
John Hertz, Richard G. Palmer, and Anders S. Krogh. Introduction to the Theory of Neural Compula- fion. Perseus Publishing, 1st edition, 1991.
Magnus Hestenes, Eduard Stiefel, et al. Methods of conjugate gradients for solving linear systems. Journal of Research of the National Bureau of Standards, 49(6):409-436, 1952.
Geoffrey Hinton, Simon Osindero, and Yee Whye Teh. A Fast Learning Algorithm for Deep Belief Nets. Neural Computation, 18:1527-1554, 2006.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):
1735-1780, 1997. URL http: www.:ppir.ttttrreeeeeeeeeeeeeeee http:///....ttaaaaaaaaaaaaaaaa/a wwwuurals//eccooo neco9. html HotcHHohheiiter99.
John J Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences, (1()))30888-33 -3092, 1984.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654 2024.
Yutong Hu, Quzhe Huang, Mingxu Tao, Chen Zhang, and Yansong Feng, Can perplexity reflect large language model's ability in long text understanding? arXiv preprint arXiv:2405.06100 2024.
Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer quality in linear time. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 9099-9117, PMLR, 17-23 Jul 2022. URL https: / http///cceeeedggn///ee/ .ll...r...... press v1 62 hwt.huaaaaaaa html.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 1419-1436, Online, June 2021, Association for Computational Linguistics. doi: 10. 10.18653/v1/2097 .aa..mmmann...mmim... 112. URL https: ac anthology, org 2021 nwa.caacllmmmm maa m....ani... 112.
Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. arXiv preprint 1906.01820, 2019.
14

---

Published as a conference paper at ICLR 2026
Fred Jelinek, Robert L Mereer, Lalit R Bahl, and James K Baker. Perplexity a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1): S63-S63, 1977.
Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https: //kellerjordan github. ic/ pptptp/s////////s//ooooo /mun/.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: fast autoregressive transformers with linear attention. In International Conference Machine Learning, 2020.
Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, pp. 1999,5007, 2017.
Teuvo Kohonen and Matti Ruohonen. Representation of associated data by matrix operators. JEEE Transactions Computers, 100(7):701-702, 1973.
Ben Krause, Emmanuel Kahembwe, lain Murray, and Steve Renals. Dynamic evaluation of neural sequence models. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2766-2775. PMLR, 10-15 Jul 2018. URL https:ttp////c /te:://proceeee i nng...m.. ml www.ssssssve.............e.....e v krausel8a. html.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint 1xXv:1700..440 2017.
Coreelis Lanczos. An iteration method for the solution of the eigenvalue problem of linear differ- ential and integral operators. Journal of Research of the National Bureau of Standards, 45(4): 255-282, 1950.
Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan Brooks, maxime gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. In-context reinforcement learning with algorithm distillation. In The Eleventh International Conference on Learning Representations, 2023. URL https: /ppperevevvveeeeeeeeeeeeeeeeeeee net har///orump/// deerummid=hyyaaaaee d=hy0a
Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transformers to solve inherently serial problems, 2024. URL https: //tptisss////ri...///s/is org/ ala/ab///22/ 22402. 12875.
Zhixuan Lin, Evgenii Nikishin, Xu Owen He, and Aaron Courville. Forgetting transformer: Softmax attention with a forget gate. arXiv preprint arXiv:2503.02130, 2025.
Bo Liu, Rui Wang, Lemeng Wu, Yihao Feng, Peter Stone, and qiang liu. Longhorn: State space models are amortized online learners. In The Thirteenth International Conference on Learning Representations, 2025. URL https: openreview net. forum?id-a joc.cczee..cccee...oo
Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. Openceres: When open information extrac- tion meets the semi-structured web. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Pupers), pp. 3047-3056, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence of Learning Representations, 2019.
15

---

Published as a conference paper at ICLR 2026
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint 1eXii:2104.08786, 2021.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
William Merrill and Ashish Sabharwal. The expressive power of transformers with chain of thought. In The Twelfth International Conference on Learning Representations, 2024. URL https:
///ppnreview..c htt.riiiww.nee//tteeeeeee id=NN JNG1Ph8W..
William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models, 2025. URL htttp:://www...ogg///.//.///
08819.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.0277 027899 2018.
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Černocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Proceedings of Interspeech 2010, pp. 1045-1048, 2010. doi: 00224437/mtterspechh200000 2010-343.
Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals. In International Conference on Machine Learning, 2019.
Denis Paperno, Germán Kruszewski, Angeliki
Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint 1rXiv:1606.0660 2016.
Bo Peng. Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao. Xin Cheng, Michael Chung, Leon Derczynski, Xingjann Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanislaw Wožniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the transformer era. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics; EMNLP 2023, pp. 14077, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 1/2023, sindings-emnll.... findings-emn 936/.
URL https: http:///aaaaattttaae//eo/ org.p/23......0333
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. Random Feature Attention, March 2021. URL /tap::///axiiiie orgg barrg//bb///e 2103. 02143.
arXiv:2103.02143 [es].
Michael Poli, Armin W. Thomas, Eric Nguyen, Pragaash Ponnusamy, Björn Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, Ce Zhang, and Stefano Massaroli. Mechanistic design and scaling of hybrid architectures. In ICML, 2024. URL https://
openreview net/ forum? d=GDDp Gyd9nf.
Matt Post. A call for clarity in reporting BLEU scores. In Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri. Aurélie Névéol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor (eds.), Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10. (0.18653/v1/W18.2 8- 6319. URL https: / /atp//aclaaaa/ httpaccaaanoooyy...o org/W18 6319/.
Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Baohong Lv, Xiao Luo, Yu Qiao, and Yiran Zhong. Transnormerllm: A faster and better large language model with improved transnormer, 2024. URL https: /ataa://aaxxii arxiv, org/ abs/ 2207 14995.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 188), 2018.
16

---

Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. sl slimpa jama-a- 622--62726 2023. URL wtt---------tttoe---eee-- Published as a conference paper at ICLR 2026
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.c 05250, 2016.
Amal Rannen-Triki, Jorg Bornschein, Razvan Pascanu, Marcus Hutter, Andras György, Alexandre Galashov, Yee Whye Teh, and Michalis K. Titsias. Revisiting dynamic evaluation: Online adapta- tion for large language models, 2024. URL https: /atpiv.ivvv./////v////a////aaavv org/ abs 2403. 01518.
Tanya Rodchenko, Natasha Noy, Nino Scherrer, and Jennifer Prendki. Not every ai problem is data problem: We should be intentional about data scaling. arXiv preprint arXiv:2501.13779 2025.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106,
2021.
Maarten Sap. Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.0972 097288 2019.
Yash Sarrof, Yana Veitsman, and Michael Hahn. The expressive capacity of state space models: A formal language perspective, 2024. URL https: attpaaaiivvvi....... abs/ 2405. 17394.
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, 2021a.
Imanol Schlag, Tsendsuren Munkhdalai, and Jürgen Schmidhuber. Learning associative inference using fast weight memory. In International Conference on Learning Representations, 2021b. URL https: / http///pwwweeeeeeeeeeeeeeeeeeeea net/ forum? d== d= 6agbäätt77.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta.... hook. Diploma thesis, Institut für Informatik, Technische Universität München, 1987.
Jürgen Schmidhuber. Learning to control fast-weight memories: an alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139 1992.
Mark Schöne, Babak Rahmani, Heiner Kremer, Fabian Falck, Hitesh Ballani, and Jannes Gladrow. Implicit language models are RNNs: Balancing parallelization and expressivity, 2025. URL
https: //axiiv///vvvv////// otgo ats 2502. 07827.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and
Jeff Dean. Outragcously large neural networks: The sparsely-gated mixture-of-experts layer, 2017. URL wttps::t/aaaaae aawxiv org/ /tt.a////11.. 06538.
Jack Sherman and Winifred Morrison. Adjustment of an inverse matrix corresponding to a change in one element of given matrix. The Annals of Mathematical Statistics, 21(1): 1950.
Daria Soboleva, Faisal Al-Khateeb. Robert Myers, Jacob R Steeves,
Joel
co /ttpa/atteteeeeeeeee///e///eeeee
627B.
Siddarth Srinivasan, Richa Arora, and Mark Riedl. simple and effective approach to the story cloze test. arXiv preprint arXiv:1803.055477 2018.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (learn at test time): RNNs with expressive hidden states, 2025. URL /tRhhttss:///xit.... //tpxvv.orggg////// abs/2407. 04 620.
17

---

Published as a conference paper at ICLR 2026
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models, 2023. URL https://arxix: org/ abs / 2307. 08621.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: / arxiv. org /abg/bb//// 2307 09288.
Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhut- dinov. Transformer dissection: a unified understanding of transformer's attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
Joel Veness, Tor Lattimore, David Budden, Avishkar Bhoopchand, Christopher Mattern, Agnieszka Eren Sezener, Jianan Wang, Peter Toth, Simon Schmitt, et al. Gated linear networks. In Proceedings of the AAAI conference on artificial intelligence, 2021.
Max Vladymyrov, Johannes von Oswald, Nolan Andrew Miller, and Mark Sandler. Efficient linear system solver with transformers. In Al for Math Workshop @ ICML 2024, 2024. URL https: http:///ppeeeeeeeeeeeeeeeeeeeeee .nt/ forum? pezzumiddgccaddlaae
Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning. 2023.
Johannes von Oswald, Maximilian Schlegel, Alexander Meulemans, Seijin Kobayashi, Eyvind Niklas- son, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Blaise Agüera Arcas, Max Vladymyrov, Razvan Pascamu, and João Sacramento. Uncovering optimipptii algorithms in transformers, 2024. URL https: wawpivivvvv../////v//.aa////aavv org/ abs 2209 05858.
Johannes von Oswald, Seijin Kobayashi, Yassir Akram, and Angelika Steger.
Learning random- ized algorithms with transformers. In The Thirteenth International Conference Learning Representations, 2025. URL https: hwtp//pwnneeeweeeee/ee / 10t
Ke Alexander Wang, Jiaxin Shi, and Emily B. Fox. Test-time regression: unifying framework for designing sequence models with associative memory, 2025. URL httttttss///xii org/ abs/2501 12352.
Jos Westhuizen and Joan Lasenby. The unreasonable effectiveness of the forget gate. arXiv preprint arXiv: 04849, 2018
Bernard Widrow and Marcian E. Hoff. Adaptive switching circuits. In IRE WESCON convention record, volume 4, 1960.
Songlin Yang and Yu Zhang. FLA: triton-based library for hardware-efficient implemen- tations of linear attention mechanism, 2024. URL https: /ttpp://gitte /ttph/gtttt/ut///// com/ fla-org/ flash- aeess--ireer-attente
18

---

Published as a conference paper at ICLR 2026
Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXiv:2412.06464, 2024a.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. In Proceedings of ICML, 2024b.
Songlin Yang. Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024c. URL https: / opwpreeeeeeeeeeeeeeeeeee net forum? Rid= y VVRM..
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 2019.
Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems 32, Vancouver, Canada, 2019. URL net / wttt//efeeeenceeeeeeeee pttp:/dffidsidd S1-Aaq..frrr
Yu Zhang, Songlin Yang, Ruijie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, Peng Zhou, and Guohong Fu. Gated slot attention for efficient linear-time sequence modeling. In Proceedings of NeurIPS, 2024.
19

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Linear Attention. As already described above, Tsai et al. (2019) demonstrated that the softmax attention mechanism can be linearized by replacing the softmax kernel k(k; q) = exp(k* q) with a surrogate kemnel k' (or(k). ( The resulting linear attention mechanism iteratively accumulates the outer product of key-value pairs into a recurrent state that is queried at each step, resembling RNNs (Katharopoulos et al., 2020). Since then, numerous works have proposed different designs of the feature map (()) (Katharopoulos et al., 2020; Choromanski et al., 3021; Schlag et al., 2021a; Peng al., 2021; Sun al., 2023; Dao & Gu, 2024) and key-value normalization (Yang al., 2024c; Schlag et al., 2021a; Sun et al., 2023). Notably, more general form of (unnormalized) linear attention was introduced in the early "90s Fast Weight Programmers (Schmidhuber, 1992; Schlag et al., 2021a; Ba et al., 2016), connected to Meta-Learning (Schmidhuber, 1987).
Test-time regression. Contrary to softmax attention, linear attention variants are only capable of storing a finite number of key-value associations. Given key dimension dayy. there exist at most dey orthogonal keys, and therefore, retrieval beyond dRey tokens cannot be error-free. Inspired by the error-correcting delta rule (Widrow & Hoff, 1960), Schlag et al. (2021b;a) proposed to interpolate the value with the previously stored association, yielding the DeltaNet. The DeltaNet update rule is equivalent performing gradient descent step with respect to the recurrent state & on || v/ ? Yang et al. (2024a) demonstrated that the DeltaNet is parallelizable and achieved strong language modeling performance when embedded into modern architecture. Motivated this online regression loss, other works derived the same update rule as the DeltaNet. Instead of a parallel implementation. Liu et al. (2025) approximate the update with a diagonal matrix, while Sun et al. (2025) perform the DeltaNet update on a per-chunk basis, implicitly performing batched gradient descent. Building on this, Titans (Behrouz et al., 2024) adds momentum to the batched gradient descent update. Wang et al. (2025); Behrouz et al. (2025b) unify numerous efficient foundation models from the perspective of test-time regression. Extending Titans, concurrent follow-up work Atlas Behrouz et al. (2025a) is effectively a bliding-window variant of the Mesa layer. It is worth highlighting that this line of research is an instance of Dynamic Evaluation (Mikolov et al., 2010; Krause et al., 2018; Clark et al., 2022: Rannen -einen-Triki& et al., 2024), where model weights are updated at test time via gradient descent steps on prediction loss.
Models with recurrent depth. The MesaNet is related a broader class of models building on fixed point iterations. Universal Transformers (Dehghani et al., 2019) apply transformer blocks iteratively, using Adaptive Computation Time (Graves, 2017) to make the number of recurrent steps token-dependent. Deep Equilibrium Models (DEQs) (Bai et al., 2019) take this idea further by directly solving the corresponding fixed point iteration using quasi Newton methods. More recently, Schöne et al. (2025) introduced an implicit State Space Model that also relies on fixed-point iteration, which trainable in parallel utilizing the Phantom Gradient technique (Geng et al., 2021). contrast to DEQ-style methods, the Mesa layer benefits from the linear structure of fast weight memory, which allows for a more efficient optimization using conjugate gradient steps.
Linear RNNs with forgetting. Forget gates were first introduced by Gers et al. (1999) within the framework of Long Short-Term Memory networks (LSTMs) (Hochreiter & Schmidhuber, 1997), and have since become part of the standard LSTM architecture. Even more, studies on simplified LSTM variants, such as the Gated Recurrent Unit (Cho et al., 2014), have shown the forget gate to be fundamental for the effectiveness of recurrent sequence models (Westhuizen & Lasenby, 2018).
Compared to LSTMs, modern linear attention variants have adopted more coarse grained forgetting mechanisms on the matrix- valueed recurrent state. RetNet (Sun al., 2023) and TransNormerLLMM (Qin et al., 2024) both utilize a trainable decay factor on the recurrent matrix. More recent work found that data data-dependent forgetting improves language modeling performance, although the data dependency is usually limited to the current input, but not the recurrent state, to allow for parallel training. Using an input-dependent decay factor as in this work is the de-facto standard in modern linear attention variants, such as Mamba-2 (Dao & Gu, 2024), XLSTM (Beck et al., 2024), and Gated DeltaNet (Yang et al., 2024a). Gated Linear Attention (Yang et al., 2024b) opts for a data-dependent decay vector, effectively using separate forget gate for each row of the matrix- matrix-valued recurrent state. Similarly, Gated Slot Attention (Zhang et al., 2024) applies separate input-dependent forget gates to each row of both matrices of fixed size Key-Value cache.
20

---

Published as a conference paper at ICLR 2026
State Space Models. State Space Models (SSMs) (Gu et al., 2021; 2022; Fu et al., 2023; Gu & Dao, 2024) build upon first-order differential equations used to describe dynamical systems, which then discretized for sequence modeling. In linear time-invariant (LTT) SSMs, the recurrent state can be obtained through a fixed linear combination of previous recurrent states, which allows for parallel mode using convolutions. Gu et al. (2022) identified the computation of the convolution kernel as the primary bottleneck and proposed Structured State Space Models (S4), a parametrization for LTI SSMs that enables efficient computation. Mamba (Gu & Dao, 2024) introduces selectivity to State Space Models, making the recurrent state transitions dependent on the input. Since the resulting time-varying SSM cannot leverage global convolutions, the authors propose hardware- efficient parallel scan implementation. Mamba-2 further constrains the transition matrix to scalar times identity, and demonstrates that the resulting State Space Model is equivalent to (gated) linear attention (Dao & Gu, 2024).
B DERIVATION OF PREVIOUS TEST-TIME TRAINING RULES
For completeness, we discuss in more detail the update rules for a number of closely related previous sequence modeling layers discussed above and in the main text section 2. Like the Mesa layer, the update rules these models perform some form of test-time learning by optimizing sequence of objective functions (Le )==- We summarize in Table 4 the update rules and corresponding online objective functions that we cover below.
Layer
Objective function
Update rule
GLA
L. 1/ kr + 1-th Tr(
. p 33 VoL
23.
DeltaNet
L, = 1t= kkz 1
&, = &:-1 VoVLlel (1,1)
LongHorn
L. = (u-- diag (:) (16-
-
4, = arg min. (
2 Tr( - $;-1) T (T 1,-1)
Atlas
L, Ei=z-o+1 Se l0y MM
i 0,0,-1 + V (
T % 0 Br NewtoonSchulz (Φ.)
Mesa
L. & Ex- Serllver 1k:rr12
4 arg min (o)
+ + TH(AA,
Table 4: Overview of test-time training recurrent layers, whose update rules can be derived from an online learning objective function.
GLA and DeltaNet update rules. For convenience, we first restate equation 3 below:
( = 4(
2 Tr(
(11)
We show in detail how to obtain the basic GLA and DeltaNet update rules by letting D, follow an online gradient- based learning dynamics,
Φ. Φ,- 3, (o.
(12)
where the input gate 3, plays the role time-dependent step size.
For GLA, we choose I, to be the quadratic continuous-state Hopfield energy,
(1( Hoppiell ()) - Φkr,
and we set the quadratic regularizer depend on the forget gate 7 and input gate 3, follows:
7/
A 3e
Now, plugging 4 and A, into equation 12 yields
t = + 3.
+
1- 7t
Tr(TTT
(13)
23,
= - - (1 Tb, + B.rkk
(14)
= + TT
(15)
21

---

Published as a conference paper at ICLR 2026
which corresponds to gated linear attention as defined in the main text (equation 2).
To obtain DeltaNet, we choose instcad l to the squared error loss,
( 4
0 2kell
and we disable the regularizer (A, = 0), as it was not included in the original DeltaNet model (Schlag et al., 202 Performing again the same computation as above, but now with this squared error online loss, yields the DeltaNet update:
[ 221le0 1kg 12
p = P2-1 VV
(16)
2--1 + 3c - -
(17)
LongHorn update rule. Yet another recent method called LongHorn (Liu et al., 2025) can be derived as online learning on sequence of loss functions (4.). Its update rule can be derived by minimizing an objective function:
2 = arg min L.
LongHorn
(18)
1 arg min 7(r xke) diag(8.)(v; 2k,) 1 Tr( 2,-1) 2.-1),
(19)
+
with 3, now a vector of the same dimension as Ut, instead of scalar, determining an elementwise squared error precision. The solution can be obtained in closed-form, following the derivation provided in Appendix C of (Liu al., 2025):
+ = - + diag(e:) - t k
(20)
Bi with Cr TT, This is a variant of DeltaNet with particular diagonal input-dependent step size that is both a function of k, and 3, (which is chosen to be a vector in this model, as opposed to the sealar gates used in our DeltaNet and in our current MesaNet implementation). For computational efficiency, the actual implementation of LongHorn approximates the update above with a simpler rule that makes use of elementwise multiplications, denoted here by O:
Φ = (1 (k o k.) O Φ.-1 + (ex v.)kk
(21)
where If is matrix of ones. Like the DeltaNet, the LongHorn objective still only takes into account the instantancous squared eтror for the current key-value pair, with an additional memory quadratic potential pulling towards the previous solution to avoid forgetting it entirely through the full arg min. By contrast, the Mesa layer explicitly optimizes the full forget-weighted sum of squared errors from the beginning of the sequence until the present (t' 1 to f).
Omega/A A tlas update rule. Concurrent work by Behrouz et al. (2025a) investigated online learning layers that are intimately related to the Mesa layer. The paper focuses sliding window variant of our objective function:
L
Omega
Σ
llue MM (kx))
(22)
V=--c+1
where is the sliding window length, and Gee determines the cumulative forget at time step for the past loss f', as in the Mesa layer objective. The authors further allow Mg be 1 1-hidden-layer MLP with parameters Q, similarly to (Sun et al., 2025), and unlike the Mesa layer, which derives a specialized update exploiting the fact that M is a linear model. Behrouz et al. (2025a) optimize the sequence of loss functions (l.) online using a second-order Muon method (Jordan et al., 2024):
$. 0, क. + V 10ega ($:-)),
(23)
t = 76 T - 3, NewtonSchulzy
(24)
where NewtonSchulzi denotes the execution of & steps of the NewtonSchulz algorithm, &; is an auxiliary momentum gradient accumulation state variable, and 0, is a dynamic (time-dependent) momentum decay factor, which determines the retention of past accumulated gradients.
22

---

Published as a conference paper at ICLR 2026
C RANK-ONE UPDATE CONJUGATE GRADIENT METHOD
In the next two sections, we describe how we can use the conjugate gradient method to obtain solution for (H, + AA) of for many in parallel. As we will discuss below, the aim is to show how one can do this without materializing H, for all time steps as this would lead to unnecessary memory overhead, see Yang et (2024b) for a detailed discussion of this problem and a "chunkwise parallel" solution. We therefore aim to show here, materializing Hf, = I + kic and only relying on H₁-1 as well as on y, and k2. This will eventually allow us, see the next Appendix section D, to compute and materialize H, only every T/C steps with train length T and chunksize times, leading to a drastic decrease in memory usage. We will do this while approximating Qr = 19 ........... Ic+cl numerically in parallel by only materializing He where E {0, C, C).
starting point, how to compute q without
We opted initialize the conjugate gradient method with x + q: diag(II, + As) this work.
Algorithm 1 Rank-One Update Conjugate Gradient Method
Tt, kt, qt, 6, kimax) 2: Input: Symmetric positive-definite matrix H₃-1 E Raxx, forget strength Yt E (0, 1), key
1: procedure
he I", query 4 E IR", tolerance O, maximum iterations kmax.
3: Output: Approximate solution x.
4:
5:
+ q: diag(H.-11 Ao)-1
Initial guess I
6:
&- (Hr-17t + kichi + Ar)or
D Initial residual r Initial search direction p >> Squared norm of the initial residual Store initial squared norm for relative tolerance
7:
p ttr r
8:
bold t rr
9:
do + 8old
while 1: < kmax do
b Loop until max iterations reached
11:
[H,-177 + ki + Ar)
Matrix-vector product (H,-17h + k,,, + Ax)p
12:
+ Agid
D Step length
pº
13:
+ ap
Update solution
14:
r +- r aqq
> Update residual r
15:
8 f- r
Squared norm of the new residual, 8
16:
if Vnew S Sevōō then
Check relative convergence:
||rxtil| </|ro|] >> Converged
17:
break end if'
t- knew
Improvement factor
bold
20:
p + r + 3p
Update search direction Store new norm as old for next iteration Increment iteration counter
21:
dold +- knew
22:
4- +
23:
end while
24:
return
> Return the approximate solution
25:
end procedure
top of H4-1P, all other parts of the (Ht-17 + KRT At)p computation can be reduced to one vector inner product k well as element-wise products and a final addition of the results. One can therefore approxiamte qi numerically without materializing He, which we will extend in the following to chunks i.e. compute Q: [g'+1..... co in parallel without explicitly materializing H, with e < t c C. This will become obvious after realizing that the computation of (H2-17 + kak? + Ap equivalent computation proposed in Yang et al. (2024b) of GLA.
GLA, therefore allowing for the chunkwise parrallel
23

---

arg min Published as a conference paper at ICLR 2026
Note that the most flops during inference are spend in the matrix-vector product Hip where apply
the CG method simply to (He + A,) 11 qe (and not do not use the "rank-one" update formulation above) resulting in the ((kn)) of Table 5.
We refer to Appendix G.5 for further details about numerical precisions considerations within our CG solver.
D CHUNKWISE PARALLEL FORM OF GATED LINEAR ATTENTION AND THE MESA LAYER
Mesa layer forward pass. The main Mesa recurrence (Equation 7) can be rewritten as follows, considering only one head and assuming without loss of generality that input gates absorbed in keys and values:
H₂ = Hi-17 + kg
G. = Gr-1% + 1ykk
(25)
q. = (H, + A
0r = G.ye
Note that H4 is symmetric, and A is symmetric positive definite, so H4, + A is also symmetric. Let's define
Se. = M
%
ift
0
otherwise
with which the computation of o, (unrolling the definition of G.) has the following form:
Σ Gookk gi.
(26)
To connect to Section 2 where the Mesa layer is defined through a set of optimized linear model fast weights L we note that this is equivalent to minimizing the following objective w.r.t. 4
1 Σ Sullu, 2k,11 + TT((AT")
mesa
(27)
&
and then computing the output through or 4- q: To see why this is the case, let us compute the stationarity condition
2 Gullu; kk; + 1 IT
(28)
A
Ga(u: Dbi
(29)
4A
Σ (GrUrkT - DCukkkk)
(30)
ΦΛ + KK,KK V.KT
(31)
V.R. (K. A)-
(32)
E GakiRT + +
(33)
To simplify the calculation introduced the auxiliary matrix variables Vi and K,, which absorbed square roots of the cumulative forget factors G. We denote the above (unique, for > 0) solution by Tinesa
24

---

Published as a conference paper at ICLR 2026
Now, the recurrence relation for the state variable He can be solved analytically, yielding
H, -- + kickie
Σ
II 75
hish
Σ Cakik
(34)
assuming Ho 0. The same holds for the other state variable, Gt 7GG11 + vekt Ei Seick
Therefore, as claimed, we recover equation 25:
0t $.messo Qt
(35)
Σ SukchT +1) ) qe G. + +A) A)-
Suocky
(36)
(37)
= G
(38)
Chunkwise form. We remark that if 9, is given, this computation is equivalent to a Gated Linear Attention (GLA) layer Yang al. (2024b), and thus be efficiently computed on GPUs and TPUs by splitting the sequence in blocks of opportune sizes C resulting a "chunkwise parallel" form of the layer. In short, given Ge, where E (0, ....., T C} dividing the training sequence length T
in T/C chunks of size C, we can compute the output at time e < S c C as
0: (G
Σ i=ct1
Sauck )4r G.d
Σ i=ct1
Guuk,
(39)
Similar to softmax self-attention, this computation can be done in parallel for t E {o l,... + C} which becomes clearer when using matrix notation
O₂ = + Ve(Ze 2 (K) Q:))
(40)
where Kg Ik., ketc] and Oc, V accordingly. Ze is upper triangular matrix of size C' x C' with Zeli, = Sctiecti. Please see for Triton-based implementation of this chunked parallel formulation of GLA https://ithub..oo/flaar////////p www.//aas--innea-atteeeeeee linear-attention.
We differ from GLA as the Mesa layer replaces which is the standard query 9: = Wye: by q." = (H, + A)- which, as we alluded to above, can as well be computed equivalently to GLA in chunkwise parallel form. Indeed, shown in the previous section, the conjugate gradient method relies purely on simple vector additions and multiplications which can be trivially realized in chunkwise parallel form without extensive memory overhead, with the exception of (H; + A)p. This operation suffers from the same memory problems as a naive GLA layer implementation as storing H, for all time steps is costly which we therefore wish to circumvent. Fortunately, this can easily be done with the exact same chunkwise parallel trick just discussed, which now leverage to compute
(H, A)p H.p + A Σ Gaskih KJ
p + A A.p.
(41)
which is required in the conjugate gradient algorithm.
Note that the first term 1= k, p is in equivalent form of GLA (by replacing with k;) for which we just established that fast chunkwise parallel formulation exist, if we again store only some intermediate states H.. We conclude that the computation of qf (H₂ + A)-²q qq and therefore the whole Mesa layer can be approximated by repeatedly applying in chunkwise parallel computation leveraging matrix-matrix accelerators GPUs TPUs.
Mesa layer backward pass: Let e, be the error coming from future layers at time and be the final loss. Then we have the following:
25

---

Published as a conference paper at ICLR 2026
C
(H,
+ A) G.
dL
do: ei
dq.
dqi
dL
do,
d Ct Ci.i
dA
do
e etttshig (Hr + A) 'q
du;
dL
Σ Steeige1k ki
dv,
dI
Σ Stalii Uo -i his 26 e/ kia)
dk,
dT. - G.-1et ei HH H,11
d%
This is a time-reversed version the formulas to compute the derivatives with respect to vs and kg. Note that dL and dL again be computed in chunkwise parallel manner as they sums dv, dk, of expressions which are all GLA formulation equivalent. et is also chunkwise parallel compatible since, as we just established, running conjugate gradient (chunked) parallelized time is possible.
It remains to see how to quickly compute the derivatives with respect %e To that purpose, let us consider the first term in the equation defining the derivative, as the second can be handled similarly; have that:
Σ Sta - Tr(G G.-1c,]
Σ 12
TT(Gg-iGueett
=
-Tr G.-1 Suerg
This already gives way compute the derivatives that is linear in sequence length (as it is sufficient to accumulate the 1-dependent part as s decreases). However, for maximum efficiency we would like also split the computation into blocks and make use of matrix multiplication units for this computation.
Let F. Stoegg We now explain how to compute the value above simultaneously for block of indices L
G--1 = Gelo-Ic+ + Go-ipihh
LEPEN Stoeegt + GuoFu
Σ Steege = Σ
sSteH
= ((
(
Σ CcoergiT + +Sanful
Th G Σ T
Gebo-Ict Σ So-sptphip
C<pRs
= Tr (GeFFuuGueG--dd +Tr Gebs-1e Σ Stoeigt
sSteu
+ Tr Fucus E So-1pupk
+Ir
Σ
Sesegg" TT Σ
Se-IpUphT
LEpes
26

---

Published as a conference paper at ICLR 2026
=Tr [GePul G1
E
Staha- Tr (Geerd"
esicu
+ Σ SuoSo-1p Tr [But
Σ
Stabs-1p Tr [eige upkp
Lepes
Repes
For computing the last term, make use of the fact that Sab 0 if to rewrite it in the equivalent forms
Σ E So-1p( up) E<p<ss asrcH
ee GelSe= = EE Sc-ppli
( (p et)Ses
REPCL DereH
which can be computed as the product the three matrices Z* QQ with Zig G-- Qis
(a)Tv) (.)(k, ej), Zy = Sifi the requested values appear then as the main diagonal of this matrix.
The second term can be similarly rewritten
G-
GreA))) C5-16 #StCU
Σ Geettas
CETCH
which can be computed by multiplying the vector Pt = J Ge by the matrix defined above, and then by doing point-wise vector multiplication by Ca-1C-
Finally, the first term can computed simply by computing the trace once and then doing a point-wise vector multiplication, and the third term can be computed as the second.
E A FULL DESCRIPTION OF THE MESA LAYER, RELATED WORK AND THE MESANET
For completion, we repeat the Mesa layer computation which described throught the following equations
Ae mesa
Σ
Pr.
i mesa
The Σ R,,,n,iinoovveece + Ar. (h,)).
(42)
The equation above depends on two state variables, Sh, {Gr,e: Hne) which we obtain through the linear recurrence relations:
G = Gh,t-17h.t + Un,ck,,
Hn, = Hi,t-1Th. kn.ck
(43)
where before Th,, [0, 1] is forget gate and 3h,s E [0, 1] is input gate, where we adopt the conjugate gradient method as the solver (Lanczos, 1950; Hestenes et al., 1952). Before the Mesa layer computation, compute the keys, queries, values as well as input and forget strength in the following way.
First, normalize the embeddings with RMS norm e; RMSNorm(e.). After projections k2 = Wier, 4 Wy 1/2 = W. we convolve them in time with a window size of 4 e.g. k2 < Ei b with learnable parameters b,,...,y, b4 Furthermore, after applying Si:.U(x:) = I* (x) non-lincarity we normalize the keys and queries (but not values) to have L2-norm i.c.
f + SiLU(k;)/l (ttxk)//s(sLL(t)))))iaa and g. +- SiLU (
For the forgetting and input gate we simply squeeze the RMS normed et projections through sigmoid i.e. 3 oee,Wo and % aee:W, After computing the output of every head, apply a RMS norm i.e. the actual output of the Mesa layer amounts to
H Pa (aassNoommGG,lllinooovel + Ans (h,))).
Ae mesa
(44)
hm1
The regularization parameters simply send through softplus function to ensure positivity i... A₁ + softplus (A,). We did experiment with input time dependent regularization strength but in this work opted for fixed lambda over time, Section
27

---

Published as a conference paper at ICLR 2026
Comparison to related work: To ensure 1-1 comparison with related work, use the exact same parametrization of the keys, values and queries as well as forget and input strength parametrization for the GLA, Mamba2 and (gated) DeltaNet. Here, only the state update well as output computation differ depending on the rule, see Table for an overview. The mLSTM layers, which we also compare to, have different parametrization of the forgetting well input strength and keys and quries not normalized by their L2 norm, see Beck et al. (2024).
Layer
Output & state update
Memory
Flops output & state update
MHA
O2 Ei= Bra qQ)
(V ke y
2nat
O(n.t) O(1)
GLMHA
0 Pege with
P n1
O(nā) - O(n))
4; = 1:16 + Brock?
DN
Digt with
c: n
O O(m.)
Φ,
4 (ye 1 Bck kt)) + Bev hi
MESA
Equation 7
S: [Gr, H:} - 2nd
O + ((kn)) - (m
Table 5: Flops as well as state size comparison between MHA, gated linearized mutti-nead-attention (GLMHA) such xLSTM Mamba2, (gated) DeltaNet (DN) and the Mesa layer during inference. All softmax attention alternatives require ((rg) flops, with key size na. compute output as as update the state(s). The Mesa layer requires an additional steps of the CG method which costs ((kng) ). For simplicity we assume f2 Ra-
28

---

Published as a conference paper at ICLR 2026
E. MODEL DESIGN
We give an overview over the network architecture for all models compared in this work in bullet points. The only difference is the way how to do the "sequence" mixing of the keys, valyes and queries (and forget and input gates), with an exception of the LRU layer (De et al., 2024), see Table 1.
The model consists of an embedding layer of size ne, which is also shared at the end of the model to compute the logits. We do not apply regularization the parameters of the embedding.
The model is then followed by N number of blocks consisting of sequencing layer e-g. MHA, GLMHA, DN Mesa, described in Section 2, followed by an MLP layer. The input of both the MLP as well as sequencing layer go through a RMSNorm (Zhang & Sennrich, 2019),
Figure/ 1. After computing the logits, apply soft hyperbolic tangent clip with c = 30 with logits =
again following the open source im- plementation of De al. (2024), see https / /ttptttttttttttttt com google google-deepminne deepmind/ recurrent genma / htwpbbllbbb/e///eeeeeeoooc ma in/ htt////eurrrnteennnee.... httprreenneeemma/a/gr//riiiiiiin wwtp//grifiii.../ py.
To compare all different sequencing layers as closely as possible and focus on their ability to incorporate information from the context, MHA, GLA, Mamba2, xLSTM, the (gated) DeltaNet, as well as the Mesa all use the exact same key size and therefore the exact same amount of parameters compute the queries, keys and values. All RNN layers, for direct comparison, additionally only use per head a one dimensional gate for forgetting well writing which we squeeze through a sigmoid function i.e. Bt o (W ges) and ? = oWWeee), except the mLSTM layer. This stands in some contrast to how the models were originally designed e.g. Gated Linear Attention (Yang al., 2024a) or RWKV (Peng et al., 2023) use higher dimensional forget gates. Furthermore, all RNN layers convolve the keys and queries with a window size of 4. This is by now a standard feature of contemporary RNN/SSM architectures, motivated by earlier analyses (Arora et al., 2023a; Fu et al., 2023). Note that for all models, except from mLSTM which uses a special parameterization and normalization, apply SiLU (or swish) non-linearity (Hendrycks & Gimpel, 2023) before we normalize the keys and queries by their L2-norm. The output of each head is independently before the linear projection back to the residual stream send through an additional RMSNorm.
We define Mamba2 as forget-gateds linearized multi-head attention following Yang et al. (2024c), and GLA as its forget- and imput-gated counterpart; both methods with ex-dependent gates.
When using the LRU layer (De et al., 2024), we notice that the layer, in its default hy- perparameter configuration, subsumes more parameters than MHA and the other RNN alternatives, as they use exactly the same number of parameters to each other. We therefore decrease the hidden size multiplier which determines the increase of the RNN state when compared to the embedding size, to match parameter count.
The Hawk-Mesa model simply alternates between blocks that have either a LRU layer or a Mesa layer.
For the MIP layers we follow again De et al. (2024). We create two branches both with dimension of n. 3, apply a SiLU non-linearity to one the branches and merge them by multiplying. We then down project with simple linear layer into ne dimension.
All weights are initialized by sampling them from normal distribution and in "fan in" mode, while scaling the variance the weights which project back to the residual stream by 2.0/N.
F EXPERIMENTAL DETAILS: MESANET IN SYNTHETIC ENVIRONMENTS
F.I MAD BENCHMARK SUITE
We follow the benchmarking procedure detailed in Poli al. (2024) precisely: For each task in the suite, we evaluate the architectures on subtasks of varying difficulty (i.e. varying sequence length, number of training examples, vocabulary sizes and further, task-specific parameters) and
29

---

Published as a conference paper at ICLR 2026
Hyper Parameter
Search
Embedding dimension
128
Number of layers
Number of heads
Key size
16
Epochs
200
Batch size
32
Optimizer
AdamW
Learning rate
[3e-3, le-3, 5e-4, le-4]
Weight decay
[0.01,
.9s
(0.9, 0.98)
Scheduler
Cosine Scheduler with Warmup
Minimum learning rate
le-5
Warm-up start learning
le-7
Warm-up steps
750
Table 6: MAD benchmark suite hyper-parametten taken from Poli et al. (2024),
Hyper Parameter
Search
Embedding dimension
[64, 128, 256, 512, 1024]
Number of layers
[1, 2, 4, 8, 12]
Number of heads
[1. 2. 4]
Epochs
50
Batch size
32
Optimizer
AdamW
Learning rate
[1e-4, 2.5e-4, 1e-3]
Weight decay
[0.01, 0.1]
3s
(0.9, 0.99)
Scheduler
Cosine Scheduler with Warmup
Minimum learning rate
2.5c-5
Warm-up start learning rate
1e-7
Warm-up steps
25000
Table 7: RegBench hyper- parameter search-space taken from Akyüirek et al. (2024). For all models, keep the key size fixed 128 across combinations of embedding dimension and number of heads.
compute the mean accuracy. We further sweep over varying learning rates and weight decay values for each model and report the maximum average task accuracy. For each architecture, we fix a set of hyper-parameters that can be found in Table 6.
F.2 REGBENCH IN- IN-CONTEXT LANGUAGE LEARNING BENCHMARK
Following Akyürek et al. (2024), we report the test-accuracy of the configuration obtained from a grid-search over pre-defined set of shared hyper-parameters for all models, which be found in Table 7.
G EXPERIMENTAL DETAILS: MESANET IN A LANGUAGE WORLD
We follow closely the experimental setup of Beck et al. (2024) as well as De et al. (2024).
G.1
DATA
We train models on SlimPajama Soboleva et al. (2023) and use the GPT-2 tokenizer Radford et al. (2018) which uses vocab size of 50257, in Beck et al. (2024). We pre-tokenize the dataset
30

---

Published as a conference paper at ICLR 2026
and fill up sequences with context length shorter than the train length, which is set to 2048, with other randomly sampled sequences until the context train length full. We separate these separate sequences with BOS token. We follow the same recipe when creating the validation data. Note that this procedure might bias the training as well as evaluation of the model towards shorter sequences.
We train dataset sizes: 15 billion and 50 billion tokens.
G.2 MODEL DESIGN
We give an overview over the network in bullet points.
The model consists of an embedding layer of size Re: which is also shared at the end of the model to compute the logits. We do not apply regularization the parameters of the embedding. We follow again De et al. (2024) and initialize the parameters of the embedding matrix in "fan in" mode but scale back the embedding during inference by ATe leading to variance of 1 in the residual stream.
The model is then followed by N number of blocks consisting of sequencing layer e-g- MHA, GLMHA, DN or Mesa, described in Section 2, followed by MLP layer. The input both the MLP as well as sequencing layer go through a RMSNorm (Zhang & Sennrich, 2019), see Figure 1. After computing the logits, we apply a soft hyperbolic tangent clip with again following De et al. (2024).
c = 30 with logits =
To compare all different sequencing layers as closely as possible and focus on their ability to incorporate information from the context, MHA, GLA, Mamba2, xLSTM, the (gated) DeltaNet, as well as the Mesa all use the exact same key size and therefore the exact same amount of parameters to compute the queries, keys and values. All RNN layers, for direct comparison, additionally only use per head a one dimensional gate for forgetting as well writing which we squeeze through sigmoid function i.e. 3, = (Waee + bg), " o (W,, Cg + by, ). except the mLSTM layer which has a more elaborate parametrization. This stands in some contrast how the models were originally designed e.g. Gated Linear Attention (Yang et al., 2024a) or RWKV (Peng et al., 2023) use higher dimensional forget gates. Furthermore, all RNN layers convolve the keys and queries with a window size of 4. Note that for all models. except from mLSTM which uses special parameterization and normalization, we apply a SiL.U (or swish) non-linearity (Hendrycks & Gimpel, 2023) before we normalize the keys and queries by their 12-nnorm The output of each head is independently before the linear projection back to the residual stream send through an additional RMSNorm.
We define Mamba2 as non-gated linearized multi-head attention following Yang et al. (2024c) and GLA as its gated counterpart with e--dependent forget strength 7
When using the LRU layer (De al., 2024), we notice that the layer, in its default hy- perparameter configuration, subsumes more parameters than MHA and the other RNN alternatives, they use exactly the same number of parameters to each other. We therefore decrease the hidden size multiplier which determines the increase of the RNN state when compared to the embedding size, match parameter count.
The Hawk-Mesa model simply alternates between blocks that have either LRU layer or Mesa layer.
For the MLP layers we follow again De et al. (2024). We create two branches both with dimension of 3ng, apply SiLU non-linearity one of the branches and merge them by multiplying. We then down project with a simple linear layer into n, dimension.
All weights are initialized by sampling them from a normal distribution and in "fan in" mode, while scaling the variance of the weights which project back to the residual stream by 2.0/N.
G.3 TRAINING DETAILS
We train over all the models in this work with batch size of 256, the AdamW optimizer (Loshchilov
& Hutter, 2019) with weight decay strength 0.1, e = 1 x 10 3 = 0.9, 32 = 0.98, and cosine learning rate scheduler with initial learning rate 1 x 10-6, warmup steps of 2000 and peak learning
31

---

Published as a conference paper at ICLR 2026
Model size
Train size
Transformer
Mamba2
GLA
xLSTM
DellaNet
Gated Delanee
Blakk
Masa-Mesa
Mesa
Small
15
0,0025
0.003
0.002
0.0022
0.003
0
0,002
0.0025
0.003
Small
50
0.001
0.001
0.001
0.001
0.001
0.001
0:001
0.001
0.00995
Medium
0,0015
0.0025
0.0025
0:005
8:005
0:0025
0:0025
0,002
0.0025
Medium
0.001
0.001
0.00955
0,0009
0.00885
0,00095
0,0009
0.0009
0.001
Large
15
0,002
0.002
0.02
0.0015
0.0015
0.002
0.002
0,002
0.002
Large
90
0.0008
0.0009
0.000RS
0,0008
0.0008
0,0009
0,0009
0.00885
0.00085
Table 8: Peak learning rate for all models trained for this work determined by learning rate grid scan.
rate of I which is scanned for each experiment, below. We cosine decay the learning rate to 10% of the peak learning rate till the end of the training determined by the train set size. We use as loss the classic cross entropy the next token; we do not compute the loss on the BOS token. We apply gradient norm clipping to norm 1. We apply mixed precision training where the weights are float 32 but activations are bf 1float16 6 following Beck et al. (2024). Interestingly, we find that this actually improves next token perplexity slightly compared to using float 32 everywhere.
G.4 HYPERPARAMETER SCANS
We train model sizes: 140 million, 440 million and 940 million parameters following roughly Beck et al. (2024), As already mentioned, all architectures have by construction almost exactly the same number of parameters for the same architectrual dimensions. All recurrent neural network types have the same parameters as multi-head attention but additionally have two parameter vectors of size na which produce the two gates per head. The Mesa layer has additionally na (fixed in time) parameters for (meta-)learned A regularization. Since the parametrization of the LRU layer is different by construction, we simply adjust the hidden size scaling to 1.25 to match the parameters of the other RNN layers. The different model sizes use key size na 128 and otherwise are setup as follows:
140 million - Small: N = 14 blocks, h = 6 heads, embedding dimension n. = 768.
440 million Medium N 28 blocks, h heads, embedding dimension ne 1024.
940 million Large: N = 288 blocks, / 12 heads, embedding dimension ne = 1536.
The exact number of parameters and peak learning rate can be found in Table 8. For all models, we sean the same range of learning rates: for models trained for 15 billion tokens we scanned {0.003, 0.0025, 0.002, 0.001, 0.0015], and for models trained for 50 billion tokens, observe, similar to Beck et al. (2024), that smaller learning rates were beneficial and thus scan {0.001, 0. 0.00095, 0.0009, 0.0085, 0.0008}. We train all sliding window attention (SWA) models, as they only reference points, with learning rate 0.001.
G.5 NOTES ON PRECISION USED IN THE CG-SOLVER, MESA LAYER DESIGN CONSIDERATIONS OR Why you shouldn' 't scream your Mesa layer
The MesaNet, for the model sizes we consider for the language experiments, solves during training millions of linear systems of equations numerically in one forward pass. Somewhat surprisingly, we did not encounter many training instabilities when setting some crucial hyperparameters and architectural details accordingly. First, follow related work and normalize keys and queries this is a crucial first step to stabilize the Mesa layer. Second, the most important hyperparameter for the Mesa layer, which strongly influences the conditioning number and therefore the number of CG steps needed to solve the linear systems, is the regularization strength A. Due to experimentation when training small models, we initialized A / but restricted its values to be lower-bounded by 0.25. We hypothesize that this lower bound is important to, implicitly, upper bound the condition number. We determined the lower bound by a grid scan when training the medium sized model on 15B tokens. See Figure O for some A values of trained model. We parameterize A through a softplus function ie. = 0.25 + softplus(A) and adjusted the initialization the parameter accordingly.
When training on SlimPajama and using the GPT-2 tokenizer, we noticed that the dataset, especially the sequences which contain code, contains sequences which consist of many repeated tokens such as the empty token "". We call this "screaming at your language model". These kind of inputs the
32

---

Published as a conference paper at ICLR 2026
Mesa layer lead to matrix HA, K,, which contains sums of the same vector outer product which we analysed leads to instabilities when " 1. We therefore upper-bound T/t by by 0.9975 (which might be train length specific) and adjust its value depending the input strength Be: when training SlimPajama, we use " ", with 8y = (1 - (1 by) )y))))) Note that other tokenizers which merge repeated " " should solve this problem partially. This correction improves perplexity in scans small models and so we adopted it throughout experiments.
final comment on the precision of the CG solver: The opted to use FP32 matrix multiplication precision inside the CG solver solely within our Pallas kernel. Note that we used BF16 everywhere else to compare other RNN and transformer models with our MesaNet fairly. This reduces memory loading times we only load data with BF16 precision, compute q in solver with FP32 precision, and cast it down in our solver to FB 16.
Although did investigate in depth FP16 or BF16 precision within the CG solver for which convergence problems are well known, we found the training times when using FP32 acceptable. We leave this important investigation for future work.
We end here with a note of caution when using these lower precisions on GPUs as more work might be needed ensure stable convergence to the approximate solution of the linear solver.
G.6 EXPERIMENTS COMPUTE RESOURCES
We provide here an estimate of the compute resources used for single run of a 1B model. We note that transformers, MesaNets and other RNNs were of somewhat comparable speed on average and so estimate compute by averaging and not differentiating costs across models. We mostly relied on TPUv5 to conduct our experiment. Here we used multi-pod TPUv5s which fit the whole models, without model sharding, and therefore were able to rely solely on batch sharding. For the 1B models, one training run, with sparse intermediate evaluation, when training 50B tokens lasted around 36 hours on average. When training on smaller models, the train time significantly dropped. All Mesa layer investigations were done the 400million scale when training on 15B tokens resulting in train runs which last 3-12 hours depending on the amount of CG steps used and data parallelization applied.
Running our evaluation pipeline for all downstream benchmarks took on average 3 hours on the same hardware, although we note that did not optimize this pipeline for run time.
G.7 TOKEN THROUGHPUT COMPARISONS OF RECURRENT MODELS FOR 1B MODELS
We report in Figure the throughput (in tokens / second) of the IB MesaNet (for different fixed CG steps), Gated DeltaNet, Gated Linear Attention, as well as standard (global softmax-attention Transformers. The MesaNet performs competitively, especially with a fixed number 10 CG steps. We note that 10 CG steps are sufficient to obtain the superior MesaNet results reported in the main text. Gated linear attention, due to the limited flops and matrix multiplications needed to perform a forward pass, reaches significantly higher throughput than all other models. As expected, transformer throughput degrades with increasing sequence length.
H THE ORIGINAL RECURSIVE LEAST-SQUARES MESA LAYER
We now review the original version of the Mesa layer (Von Oswald et al., 2024), where dimesa was determined through the classical recursive least-squares algorithm. The key observation is that demess V.K,T R,' Ei= k. Ei k + +)) and that can calculate the inverse R recursively through the Sherman- Morrison formula (Sherman & Morrison, 1950; Gauss, 1821), R = RT R, A,A,, T.ATRR, R, 1, with Ro = A1 While efficient for sequential inference, I+AT ,,
Sherman-Morrison
this solution is problematic for two reasons. First, when introducing time-dependent forget gates Y E (0, 1] which scale the previous state, i.e., 1R + k,c, ) the matrix inversion for small Y can introduce numerical instabilities as R can grow unbounded. Moreover, note that this Mesa layer version forgets the regularization term A exponentially fast, as it only enters through the initial
33

---

Published as a conference paper at ICLR 2026
Length: 512
Batch Size:
32'
Bon
140
Nn
Nere00
190
Carad Detaiie
Calod Linea 40t-1901
130
Mise 10
Meoe28
160
Datatlt
Gabadl Unoor Azontion
211
2
211
20
Leoghh
Batch
101
Misx-2
Moser 20
207
Misx-33
Mice-3
2
Salch
Lengoh
Figure 6: 1B model throughput (tokens sec) with bfloat16 activation and float32 weight precision on a H100 GPU (top row) using the open source framework https: : //github.bb...////// flash-1 Linear-atte hhar-attention or our custom TPUv5 implementation (bottom row). We show the effect of scaling the batch size (left), while fixing the generation length, or scaling the generation length, while fixing the batch size the token throughput/ sec. For this experiment, averaged 5 iterations reduce noise. On both hardware systems, we see that 1) MesaNet and Gated DeltaNet perform competitive despite MesaNet consuming significantly more flops, 2) Gated Linear Attention outperforming other layers significantly as well as 3) the throughput of the Transformer degrading with larger batchsize and especially sequence length. We chose sequence length for left pancls and batch size for the right pancls small enough, such that the (global softmax) Transformer does Transformer running out memory the largest batchsize.
out of memory for the H100. On the TPUv5 and the left configuration, the
Reasoning (Globai,
Reasoning
In-Context Recall
<-3e.2
E-1o-3
c-1e5
CG+5
CG-7
CG-20 CG=30
c=1e2
c=le4
CG=6
0G=10
<=lez
24
Ie-4
c=lc4-
0.06
sig C5 Stoee
cele5
Cc-0
22.37
0.04
8
CG-5
0.02
6S-19
0.00
-1e-4
30
60
20
40
25
60
Acouncy
Accuraty
Accurcty
Longth
Sco. Longth
Longzh
Figure 7: Effeet of Number of Conjugate Gradient (CG) Steps on SlimPajama Perplexity within and beyond train context length. We show here the effect of reducing the number of CG steps during inference token perplexity across token position a IB MesaNet trained on 50B tokens. either use fixed number CG steps uniformly across the model or apply dynamic stopping criterion >> 0.
34

---

Published as a conference paper at ICLR 2026
state RT=! Second, not aware of way of computing R in parallel-in sareteel-i--imme fashion. This precludes efficient parallel training at scale in current hardware.
The Mesa layer second-order in-context learning method. As reviewed in Sections 2 and A, the closely related DeltaNet model (Schlag et al., 2021a) updates a matrix- vvauue state variable & R x* following online gradient descent squared error loss. Omitting the head index, the dynamics of this layer reads
Φ = f: 3e V
( (+.-1) = . + Be(U: 1
(45)
To make comparison with this layer easier, now express the Mesa layer (equation 4) in similar recurrent form. We assume that we are in the case where the above holds, that we write H as a function of H This requires that forgetting is disabled (Ve7 = 1), or that the regularizer A decays exponentially with time. For simplicity, we assume in what follows that there is no forgetting. Then, using the convention that Ho A, have that
Morrinon recursion explained
Φ: G
(46)
(G--1 vekie )H²²²
(47)
(4.-1H1-1 vrki )H,
(48)
Φ:-1 (He - keki ) H + veki H
(49)
p ks k H + vyk, H
(50)
-:-1 1 (#i-1h: 21 ki H
(51)
f V2 ( V. 1.1.aar (.)
(o:-1),
(52)
recalling that Lg is the cumulative regularized loss (equation and 1, 1-- || from equations 47 48, we used the fact that . G. H From equations 48 49, we used the identity H
& To go
(He - k2 H,
Thus, while the DeltaNet and related layers perform (first-order) online gradient descent on squared error loss, the Mesa layer implements instead an online seccond-ordet Newton descent algorithm.
I A PRELIMINARY INVESTIGATION INTO STATE TRACKING WITH THE MESA LAYER
1.5
1.0
WE(11
Test Accuracy
0.9
Train loss
0.8
WE(-1,1) n E0
0.7
Train length
0.5
Random guessing
0.6
0.5
0.0
192
256
40 64 Sequence length
128
192
256
Train samples (x10*)
Figure 8: Negative 7t and high allow MesaNets to solve parity: When using 7/4 E (-1,1) as well as enforce high A, we enforce the MesaNet into functionality close to GLA as o = q: which allows us to use MesaNet with 7 E (-1, 1) which naive applied does not lcad to a well-defined mesa-optimmizttion problem.
Recent work has investigated the (missing) state-tracking ability of transformers, modern state space models and linearized transformer RNN models, see e.g. (Merrill et al., 2025). It remains an active research direction to study under which circumstances these in-time parallelizable RNN models can better track state than transformers (Merrill & Sabharwal, 2024; Li et al., 2024).
One simple architecture change proposed in Sarrof et al. (2024); Grazzi et al. (2025) which allows layers such as Mamba, GLA or gated DeltaNet to solve certain state tracking tasks is to use forget strength 7t E (-1,1) instead of 'y 1). We highlight that this change naively is not possible
35

---

Published as a conference paper at ICLR 2026
to incorporate into the Mesa layer. Indeed, Nt E (-1,1) could violate the positive definiteness of (He-17t + K + +A) leading to a potentially ill-defined linear system of equations problem. GLA if gf 9. which can be enforced by setting A to very large
The Mesa layer is equivalent values such that (H₃ + A)=1 ~ A-1 and rescaling qe by A. Although undesirably from an online learning perspective, high A should lead to 1 k + A) rendering positive definite even if 2 E (-1,1) leading to state tracking capabilities as observed in Grazzi et al. (2025) for models such as Mamba or DeltaNet with 7t E (-1,1). We show first state tracking results for MesaNets with Yn E (-1,1) or E (0. 1) while initializing A = 50- I and restricting its lower value to 49. These values are chosen by hand, generally a wide range of (large) A actually gave us the same results. When now learning parity, see Figure 8, MesaNets, as hypothesized, start solving parity with perfect accuracy when endowed with "/ E (-1, similar to results presented for Mamba and gated DeltaNet in Grazzi et al. (2025) when using "Y E (-1,1). Although this parametrization showcases the flcxibility of the Mesa layer encompassing the capacity of GLA (and similar layers such Mamba and mLSTM) by enforcing high regularization, we stress that this solution is in our opinion rather a bug than a feature. This is because we actually wish to utilize the extra flops spend to compute g.- We leave investigating how the MesaNet could track state while not falling back to GLA functionally for future work.
Experimental details. We train a MesaNet with 2 layers, embedding dimension of 128, and 4 heads per sequence mixing module (each head with dimension 1 128) amounting to roughly IM parameters. For training we sample bitstrings on the fly and compute the respective ground truth parity scores at each sequence position. We then train the model to predict the parity score at cach position in the sequence. During training bitstrings are restricted a length of 40. In final evaluation, we test the trained model on sequences up to length 256. We train on a batch size of 256 and train in the infinite data regime sampling a total of 10000 batches. We use weight decay of 0.03 and learning rate of 0.001. To obtain the results displayed in Figure 8 we initialize A = 50 and lower bound to 49 and train once with positive eigenvalues only (7t E (0, 1)) and once allowing for negative eigenvalues (yu E (1).
J FURTHER DISCUSSION POINTS
We list here some additional discussion points which we couldn't place in the main text because of space constraints
Backpropagation through the conjugate gradient method: Currently, we are computing the gradient through the Mesa layer assuming that we have approximated of numerically well. We believe this current version is shortcoming of the Mesa layer and speculate that it is actually feasible to train the MesaNet to cope better with fewer steps (and not approximate of as well). For this we would use a stochastic number of CG steps during training, ranging for example from to 30, and backpropagate through the unrolled process, potentially obtaining model which is trained to be behave "optiiaaly" given a certain number of CG steps. This would allow for an even better dynamic test-time compute allocation of the MesaNet during inference as users could flexibly decide to spend more compute for a better model. Interestingly, one could additionally condition (e.g. with a set of BOS token indicating the number of CG steps used during the forward pass) the models forward computation and therefore allow the model to learn to adjust its representation at every layer dependent on the CG steps used in the Mesa layers. We speculate that we therefore would obtain MesaNet which behaves on par with e.g. GLA, Mamba or xLSTM with 0 CG steps and outperforms these RNNs when allocating more CG steps.
Architecture considerations: We decided to benchmark related work while using the common transformer backbone allowing for direct 1-1 comparison between all models. This architecture is extremely widespread and has the advantage to allow for direct usage of Eexture-of-Expertt Shazeer et al. (2017) layers. XLSTM and Mamba, see e.g. Beck et al. (2024), use different backbone which notably merges the MLP layer and the RNN layer in one while matching parameter count. This architecture change leads to overall better perplexity but question if the particular RNN layer or the architecture change, or its combination offers better results. We leave an investigation of a fair comparison of the Mesa layer and other related work when changing the architecture backbone for future
36

---

Published as a conference paper at ICLR 2026
work. Generally, we acknowledge that it is unclear if these architecture changes address the shortcomings of RNNs, which we show in the evaluation section, namely to incorporate sequential long range information. We are excited to study the influence of the backbone when optimizing for incorporating long-range understanding and not perplexity.
Learning fast matrix inversion algorithms from data: To obtain (H++A) we decided use the well known and powerful conjugate gradient method. While this algorithm is widespread, we hypothesis that learning a neural network to solve (H, + A) q directly or adjusting the CG method by learned parameterization, could lead to significant speed ups. We generally find extending well-known algorithms with the help of deep learning or using them building blocks of deep neural networks exciting research direction (von Oswald et al., 2023; 2025; Vladymyrov et al., 2024).
Mesa layer to model sequences outside the language domain: We speculate that the Mesa layer is promising layer for sequence modeling of continuous data, where in-context generalization and not memory the driving factor of improving next token prediction. Therefore the Mesa layer might excel in domains which require some form of in-context (control or reinforcement learning) algorithm distillation (Laskin et al., 2023).
The fundamental limit of RNNs with finite memory: (Moder)) RNNs do have a finite amount of state which they use to information for future access. This has two interconnected, intermediate shortcomings when comparing to softmax: The interpretation and the relevance of certain information in sentence can drastically change token. Since softmax stores all information of the past (all input text and its representations in all layers), it can recall information relevant the current query (for example, a particular question about the text. RNNs need to anticipate when processing information which needs saving such that can be accessed later on.
the last
37

---

Published as a conference paper at ICLR 2026
K MESANET TRAINED IN SYNTHETIC ENVIRONMENTS
We evaluate the token manipulation and in-context learning capabilities by training and evaluating MesaNets on two purely synthetic benchmarks: (i) Mechanistic Architecture Design (MAD) (Poli al., 2024) and (ii) RegBench (Akyürek al., 2024). For MAD, we train 2-layer models and sweep over a range of optimization hyperparameters for each task. For RegBench, we follow Akyürek et al. (2024) and sweep over larger grid of hyperparameters for each task, including number of layers and heads, see Appendix F.
MesaNet excels at the MAD benchmark. MAD comprises a suite of recall, memorization, compres- sion, and copying tasks. As shown in Table 9, the MesaNet achieves the highest average performance, outperforming all linear recurrent architectures and matching the performance of transformers. These strong results demonstrate the MesaNet's s efficacy in managing its fixed-size recurrent state to store and retrieve necessary information across diverse manipulation challenges.
MesaNet and Transformers perform on par on the RegBench. This benchmark requires models to infer the underlying grammar of aveudolanguugee defined by probabilistic finite automata (PFAs), solely from context sequences. At test time, this in-context learning capability is tested on token sequences generated with held-out PFAs. Again, the MesaNet surpasses other RNN models and matches transformers, demonstrating its capability infer rules at test time (Figure 9).
IC & Noioy Recall
Farzy Recall
Memorize Train Data
Selective Copy
Compress
Asg
95-
MarbaZ
100
51.2
42.0
05.4
41.3
66.0
GLA
100
39.0
82.5
96.1 I
42.3
72.0
90
XLSTM
100
47.6
79.8
95.4
43.4
73.2
DetaaNet
100
55.5
40.8
98.8
43.3
67.7
Gated DeliaNet
32.7
95.7
45.0
71,0
65
85
hransforme
CLA
Hawk
93.0
13.6
91.3
77.0
47.7
64.5
Miso
Delittet
MosaNet
Cobed Detaset
Bawk-MesaNee
100
301
85.6
99.6
52.3
73.5
80
Mambaa2
Transfomter
Number of DFAs (in 1000)
10
Table 9: Performance (% Accuracy 1) on the MAD bench- mark (Poli et al., 2024). The MesaNet performs strongly compared to other RNNs and matches the transformer.
Figure 9: Performance on RegBench (Akyürek al., 2024). MesaNet outpertorms other linear architectures and closes the gap to transformers.
L EXTENDED RESULTS IN LANGUAGE ENVIRONMENT
L. LANGUAGE MODELLING / PERPLEXITY ANALYSES
The common approach to measure language modeling performance on set of sequences S (si.....Sy sN) } perplexity (PPL), which is defined as the exponential of the average negative log- likelihood per token (Jelinek et al., 1977; Brown et al., 2020b; Biderman et al., 2024):
[
NLL
log P(sgall3,...... 33,8-1)
S
(53)
j=1=1
PPL = exp [NLL]
where |.5] is the number of sequences, 8j is the j'th sequence in S and 83.5 is the i'th token in the sequence 8y. However, all tokens are weighted equally these metrics, independent their token position. This is especially critical, the magnitudes of the log- ligelikeldoood scores tend to be quite different for early and late tokens a sequence. As consequence, interesting differences between models might be masked in these aggregated metrics, especially when comparing different model families with different inductive biases. Therefore, one needs to condition on the sequence position pinpoint qualitative model differences in quantitative manner.
Mcan-so-far {NLL, PPL). To investigate whether models exhibit different language modelling capabilities at different sequence depths k; we therefore assess mean-so-far NLL and PPL:
[S] min(|sy (.8)
Mean-so-far-Niia
log P(.).1831,..... (8,111)))))))))) (54)
Σ) min m)n((s}l,k) j=1
Mean -or--ssoff-r----- exp Mean-- so- far -
38

---

Published as a conference paper at ICLR 2026
Intuitively, these metrics can be interpreted as how well are sequences modeled up to length k. While these metrics give more granular picture of the loss behavior dependent on sequence length, they still mask important transition points due to the cumulative aggregation up to position k. For instance, the mean-so-far NI.L could still be decreasing for higher (decreasing slope), despite the tokee-ppostionn-deeeddntt NLL may have already plateaued or increased (Lin al., 2025).
Makken-Position-Deeeddde NLL. Consequently, we follow (Lin et al., 2025) and assess the average negative log-likelihood conditional on the token-position & (for which only sum over sequences with |8y| 2 k):
|s|
NLLK =
log P(s.allsj........
(55)
|
1=1
Difference in Deeeen--ossiioo-eeeeee As the field's main interest is improve upon the current cate-of-the-as transformer architecture, we investigate the difference in token- siten-position-deeddeen NLL with respect to transformer (MHA):
NLL Relative to multi-head-attention transformer.
ANLL NLL
NILA
(56)
where a negative ANLL means superior language modelling ability at position & relative to transformer the model's loss is lower. The same difference can be formulated for the mean- so-far metrics. Certainly, such relative metric requires well-tuned transformer baseline.
L.1.1 WITHIN TRAIN CONTEXI-LENGTII
Here, we expand upon the results shown in Section 4.1 and present within-train-contexx-leeee language modelling evaluations on all evaluated pairs of model sizes (i.e., 145M, 400M and 1B parameters) and number of training tokens (15B and 50B tokens).
PPL. We present the PPL scores the five evaluated datasets in Table 10. Across all model sizes and number of training tokens, Hawk-MesaNet exhibits the best PPL performance on the majority of benchmarks among the recurrent models, closely followed by MesaNet. Notably, Hawk-Mesa and Mesa match or exceed the transformer baseline with respect to PPL on the majority of benchmarks on all model sizes. Furthermore, one can clearly observe the impact the attention window size on PPL based on our SWA baselines. PPL is decreasing with an increasing window size in all settings. Notably, 1WA-1024 reaches competitive performance with the majority of recurrent models, i.e. Hawk, Mamba2, GLA, xLSTM and DeltaNet.
Conditioning on the Sequence Position. As indicated in the metrics description, and shown in Section 4.1, uniformly averaging over all tokens in the PPL computation, independent of token's 's depth in a sequence, may masquerade important qualitative difference between models. Therefore, condition the token position and investigate the difference in ooken-position--epeeeeee relative to a multi-head-attention transformer NLL.modd, demonstrate superior language modelling abilities early in sequence relative to the transformer baseline. However, beyond a certain token position, transformers surpass the performance of all recurrent models.
NLL model As shown in Figure 10, most recurrent models
Which model performs strongest early in the sequence? Notably, MesaNet and Hawk-MesaNet exhibit the strong performance early-in-the-See -eetheeseeeence tokens except Hawk. However, while Hawk exhibits the best performance up to a certain depth, the model exhibits sharp performance decline after that and falls behind most models. See Figure 11 for clearer visualization (cquivalent to Figure 10, but loken-position in log-scale).
Which model offers superior performance to a transformer "for the longest"? While Hawk losses its advantage the carliest, Hawk-MesaNet extends the performance advantage to the largest token depths, closely followed by MesaNet.
For completeness, also show the mean-so-far NLL difference AMean-so-far-Wir
rela- tive to a Transformer in Figure 12. However, as indicated, the cummulative aggregation the metric skews the important token depth transition point where transformer surpasses the recurrent models in terms of language modeling.
39

---

Published as a conference paper at ICLR 2026
HAWK
HAMBA2
DELTANET
MESA
MHA-SWA--
MHA--WWA2256
MHA
SVA
Models:
Models::
NL Diferrenca to MNA MLL Diferrince to MHA
150 Token!
0.250
0.000
0.090
0.025
0.035
0.025
0.800
0.000
0.000
0.022
012
1.24
2040
1024
2045
1024
2008
S0B Token:
0.25
0.000
0.00
0.00
-0.013
-0025
-0.02
-0.050
-0
64
512
1004 Token Position
2018
64
512
1024
7048
64
512
1024
2008
Token Position
Tokenn Position
Figure 10: NLL Difference (per token-position) ANLL model relative to a Transformer on Slim SlimPajama Validaton Dataset. Most recurrent models demonstrate superior language modelling abilities early in sequence relative to the transformer baseline, across all settings. However, beyond certain token position, transformers surpass performance all recurrent models.
HAWK
MAMBAZ
DELTANET
MESA
NHA SVA
MHA SWA 256
NHA
Models:
Models:
Models::
0.075
3.075
358 Token::
0.050
0.050
0.002
Diference Difrerencce
0.025
0.025
-0.025
-0.000
-0.250
-0.055
2
21
21
211
0.075
27
SOB Token::
0.000
0.050
0.090
0.025
0.023
0.025
0.002
0.30
0.050
0.098
-0.05
21
25
2
-0,075
27
21
2
-0.075
25
Tokeo Position
Token Position
Tokenn Posttion
Figure 11: NLL Difference (per token- (okeen-psition) ANLL model relative to Transformer Slim SlimPajama Validaton Dataset in log-scale. MesaNet and Slawk-MesaNee exhibit the strong language modeling performance sellyii-thee-squenme sequenee tokens except Hawk. While Hawk exhibits the best performance up to certain depth, the model exhibits a sharp performance decline relatively early in the seq. depth.
GLA
XLSTM
GATED DELTANET
HAWK MESA
MHA SWA 64
MHA SWA 1024
Models:
158 Taken!
Difference MMYA
0.04
0.00
0.02
0.20
0.00
0.00
0.02
0.04
0.04
0.04
1024
2046
1.011
2040
1024
2008
SOB Taken:
Dfferrncce MNA
3.54
0.04
0.61
0.02
0.50
0.00
0.00
-3.04
-0.04
-0.04
04
1224
2048
1024
2040
512
2024
2248
Sequance Langth
Sequance Langth
Sequence Langth
Figure 12: Mean-so-far NLL Difference AMean-so-far-Mian model relative to Transformer on SlimPa- jama Validaton Dataset. The cummulative aggregation in the mean-so-far metric skews the important token depth transition point where transformer surpasses the recurrent models in terms of language modeling.
40

---

Published as a conference paper at ICLR 2026
1SB Tokens
50B Tokens
SLIM
LMB.
WIKI
PG19
GOV.
QASP:
AVG
SLIM
LMB.
WIKI
PG19
GOV.
QASP:
AVG
rpl 1
ppl
pel
101
ppl
rpl
ppl
ppl
ppl
ppl
rpll
145M
Howk
19:73
38.94
23.00
19.87
19.23
29.66
25.08
18.34
37.43
21.25
18.49
18.17
27.83
23.59
Mamba2
18.29
40.34
20.36
19:17
17.03
23.71
23.33
17.05
38.22
19.24
17.87
15,90
22.
21.73
-GLA
17.37
37.96
19.57
18.11
15.86
22.37
21.87
16.30
36.20
18.43
16,90
15.02
20.91
20,62
-XLSTM
17.35
37.97
19.57
18.12
15.88
22.50
21.90
16.20
36.19
18:31
16.97
14.91
20.85
20.57
-DelaNer
19.29
21.68
16.17
36.55
18.08
-Goled-Dellaae
15.55
16,05
Mesa
15.44
21.47
16.05
36.17
17.95
20.57
Mesa
16.81
57.20
18.87
17.34
15.29
21.62
21.15
15.82
35.51
17.70
16.19
14.55
20.38
20.02
Transformer
16.95
38.69
18.65
17.47
15,00
20.80
23.26
15.81
36.54
17.35
16.25
14.04
19.33
19.89
Hawk
12.87
Mamha2
13.27
31.05
14.28
13.28
-GLA
13.69
31.64
15.01
13,89
12.96
17.08
17:24
12.61
29.93
13.75
12.25
11.52
15:27
16.05
-1LSTM
13.71
31.70
14.95
13.68
12.28
17.10
17.37
12.56
29.
13,60
12.72
11.49
15.72
15.98
DellaNet
13.80
31.98
15.07
14.01
12.51
17.20
17.43
12.59
30.00
13.68
12.70
11.49
13.57
16.00
Ginled-DeetNNeer
13.48
31.40
14.71
13.59
12.16
16.64
17.00
12:44
29.57
13.45
12,52
11.31
15.42
15.79
Mesa
13.44
31.38
14.65
13.51
12.00
16.56
16.93
12.34
29.57
13.36
12.40
11:15
15.19
15,67
Hawk Mesa
13.37
31.10
14.55
13.32
12.07
16.68
16.85
12:30
29.38
13.33
12.30
11.28
15.32
15.65
-WA-4
23.36
38.65
29.79
23.51
26.94
48.24
31,66
19.32
33.76
23.45
19.35
21.50
35:41
25.46
SWA 64
15.98
32,97
18.39
16.31
15.20
23.08
20.10
14.04
30.51
16.35
14.19
13.25
19.37
17.95
-SWA-256
14.69
32.64
16.99
15,04
13.42
19.36
18.69
15:23
30.36
14.94
13.38
12.08
17.09
16.85
SSWA-1024
13.95
32.63
15,40
14,09
12.36
17.05
17.58
12.52
30.13
13.71
12.56
11.12
15.26
15,88
TTansfoemer
13.64
32.25
14.71
15.73
12.06
16.51
17.15
12,40
30.10
13.25
12.42
10.96
14.84
15.66
Howk
26.67
Mamba2
12.78
30.30
13.97
12.92
11.58
15.97
16.27
11.39
28.02
12.23
11.42
10:42
14.02
14.58
GLA
12.28
29.13
13.29
12:35
11.08
15.20
15.55
10.99
26.
11.77
10.95
9.99
13.52
14.03
XLSTM
12.38
29.21
13.43
12,40
11.16
15.35
15.05
11.01
26.93
11.81
10.94
10.00
13.55
14.04
TheltaNet
12.23
29.13
13.20
12.28
11.04
15.11
15.50
11.01
27.08
11.73
11.00
10.02
13.44
14,05
GGntod-DehuuNe
15.25
Mosa
12.02
28.57
12.92
11.96
10.76
14.76
15.17
10.83
26.78
11.49
10.71
9.80
13.13
13.79
Honk-Mesa
11.91
12.79
10.72
15.05
10.78
SWA-4
20.27
34.06
24.56
20.33
22.98
40.37
27.20
16.46
29.93
19.42
16.42
17.86
29.15
21.54
SWWA-64
14.08
30.01
16.47
14.33
13.34
19.78
18.00
12.37
27.76
14.14
12.51
11.56
16.77
15.85
SWA 256
12.98
29.63
14.76
13.18
11.82
16.82
16.53
11.60
27.39
12.89
11.71
10.58
14.69
14,81
10.92
27.22
11.78
Transformer
12.16
29.55
12.00
12.10
10.58
14,47
15:31
10.56
27.16
11.42
10.75
9.69
12.86
13.79
Table 10: PPI. at a Maximum Sequence Length of 2048. The score the hest recurrent model with respect to PPL on each dataset is highlighted, and PPL scores from SWA and the transformer baseline are shown as reference. Across all model sizes and number of training tokens, Hawk-Mesa exhibits the best PPL performance most benchmarks, closely followed by Mesa.
41

---

Published as a conference paper at ICLR 2026
L.1.2 BEYOND TRAIN CONTEXT-LENGIA
PPL. We present the PPL scores for context lengths of 4k (see Table 11) and 32k (see Table 12) respectively on all model sizes and number of training tokens.
155 Tokens
5001 Tokens
WIKI
QASE.
QASE
101
101
pol
101
pS1_
pp0
101
101
spl.
145M
Havk
23.50
24.23
19.64
30.09
24.44
21.90
22.63
18.54
25.10
22.79
MMamh22
24.24
27:31
70.07
77.53
34
16:13
27.85
22.55
79:17
25.95
GLA
20.07
22.14
15.68
21.38
19.32
18.83
20.70
14.73
19.95
18.55
XLSTM
20.04
22.13
15.58
21.43
19.70
18.68
20.67
14.63
19.59
18.46
-DelsNaa
19.85
12:05
18.65
Gated-DellaNal
19.64
21:75
15.23
21.03
18.45
20.47
21.00
15.10
18:35
Hask-Mosa
19.33
20.85
15.00
20.69
18.95
18.15
19.72
14.53
19:48
17.91
-Tunddomner
27.68
34.15
33.59
30.77
20.06
52.12
65.55
47.93
$0.57
$6.25
Hook
15.61
17.35
13.00
19.25
14:20
15:45
14:56
Manbh2
15.30
20.59
17.94
-SLSTM
13.00
DehuaNet
15.59
17.07
14.09
15.50
.Gited DehaMet
14.99
11:64
15.75
13.75
15.33
11.04
14.90
Hawk Mesa
14.90
15.15
11.82
15.96
14.68
13.67
14.85
11.05
14.54
13.52
-WA-44
30.09
20.63
28.80
30.69
34.82
34.31
24.55
22.85
37.16
27.25
-WWA-54
19.58
20.23
15.05
25.35
19:71
16.95
17:48
13.55
19.44
16.35
-SWA-156
17.54
18.41
13.59
19.20
12:31
15.47
16-44
12.19
16.85
15:35
SWA 1024
15.90
17.25
12.32
15.58
15.52
14.22
15.41
11.27
14.90
13.95
-Trnndoeneer
33,17
46.81
34.34
41.51
38.96
74.74
130.23
122.52
142.67
117.54
15.11
12.01
17:10
11:35
Manbb2
15.00
18.03
13.33
17.85
17.56
20:00
12.11
9.70
DeliaNai
14.30
12.11
0.84
Gittd Detalier
15.20
10.55
14.01
12.98
9.62
12.54
13.21
10.50
13.93
11.78
12.90
9.57
Howk: Mesa
13:08
14.27
10.29
13.85
12.92
11:81
12.72
9.60
12.53
11.06
-WA--
25.40
25.64
26.56
42.51
15.5%
701
20:71
18.99
50.44
27.55
SWI--4
17.05
17.20
13.74
20.02
17.13
14.65
15.34
11.51
15.54
14.66
-SWA-156
15.25
15.11
11.95
15.71
15.01
13.95
14.34
10.65
14:40
19.18
-SW-
13.80
15.09
10:84
14.45
19.56
12.20
13.27
0.75
12:71
11:08
Transoomer
24.00
31.00
28.06
30.51
37.04
64.04
74.90
Table 11: PPL at Maximum Sequence Length of 4k.
150 Tokens
S0BI Toleass
WIKI.
PG:19
GOV
QASIP:
AVG
WIKI.
P4219
GIW
QASP:
ANG
pel..
poll
spl
psl
502.1
2yp
psll_
psl
145M
Howk
23.93
20,16
30.75
26.05
27.62
19:03
28.74
24:35
Maanin2
$7.56
96.95
44.95
38.47
$4.45
40.51
174.03
106.47
50.52
95.13
-R.A
20:24
27.32
16.31
21.40
21
11.30
18.90
26.30
15.23
201195
21:15
KLSTM
20:30
28.02
15.91
21.61
18.78
25.25
15:13
30
20.04
DeliaNei
979.34
43.10
24.35
258
247.16
Miaa
19.00
26.67
15:36
30.61
25:73
14.65
-ool-Moa
19.72
26.79
15.99
20.97
20.79
15.44
25.09
14.60
19.90
19.90
Transforiee
42.42
72.04
43.19
41.54
40.82
525.05
4435.78
2029.43
324-54
1829.77
34.32
297.00
.LLA
15.43
23.05
12.76
16.35
20.16
11:74
15.05
-LSTM
15.34
20.85
13.00
16.20
16.11
16.00
20.31
11.29
14.90
15.12
DeliaNei
18.50
487.01
25.09
19.28
136.24
19.13
359.50
31.71
17.98
107.18
Guted DettaNet
15.16
21.19
12.27
15.85
16.12
13.82
20:72
11.37
14:07
15.14
Mesa
12.31
15.95
19.55
11.17
14.51
Hool-Moa
15:43
22.30
12.56
16.40
16.85
14.04
31.51
12.25
15.04
16.24
30.07
31.49
38.40
-SWAA-6
19.00
25.07
16.01
21.50
16:98
21.55
19.81
19:81
16.04
SWAL 256
17.63
23.43
13.52
19.62
15.38
15.59
20.07
12.37
17.17
16.30
SWI,, 1024
16.01
21.02
12.40
16.75
16.54
14.48
19.01
11.39
15.36
15.16
Trannfommec
118.6
538.89
158. 16
94.22
335.03
426.15
4312.79
2013.32
413.55
1806.95
Hook
15:07
21:33
23.29
39.52
107.40
GELA
1861
18.72
10.96
14.44
14.41
12.11
16.45
9.95
12.88
12.95
ALSTM
13.74
18.35
10.1
14.5%
14.00
12.20
15:95
10.02
13.03
13.15
DilisNei
14.75
165.22
17.33
15.54
48.21
$4.85
150.00
21.92
14.95
50.00
Gated DeliaNes
83.25
17.75
10.55
13.97
13.83
11.7
15:37
9.80
12:53
12:44
Moa
13.35
18.17
10.50
14.06
14.00
11.92
15.20
9.71
12.35
12.63
Howak- Misa
15.99
194
19:41
14.55
6.
92.91
17.50
12:51
15:13
15
SWALA
25.35
32.75
25.33
43.92
31.85
20.15
25.44
19.55
31.00
26.41
SWI, de
17.10
21.53
14.05
20,40
19.37
14.08
18.35
12.03
17.21
15.09
15:51
19.51
12:17
17:00
17:38
14.04
1024
18.15
1050
16.04
12.57
4841
109.55
36.09
53.95
90
276
1336.99
363.97
356.90
598.0
Table 12: PPL at a Maximum Sequence Length of 32k.
42

---

Published as a conference paper at ICLR 2026
Globul Subeet
Local
LMB.
Hella,
RACE-M
RACE-H
AVG
PIQA 140
Wine
ARC-E
ARC-C
SIQA
BOOLQ
OBQA
SC.
AYG
100
acc
140
auc
4MMM Partmetarrs 15B Tokkes
-WW-44
4,62
34,97
25,97
25,93
22.87
56,81
40.33
43,51
24,23
39.92
57.31
50,00
63.76
46,89
S5WL16
27.31
37,30
25,18
28,04
30,13
67.63
52,64
43,52
21,81
39,71
54,89
27,60
65,82
46,95
39:35
52.17
44,44
58.56
29.20
64.09
47.61
66.00
SWA-1024
41,43
40,00
67,90
$2,80
44,49
60,37
50,20
66.38
45.19
Parameters/ Tokoms
SWA 4
18.28
39,00
29.56
27.66
28.63
87.85
51,93
64,69
24.83
39.71
59.25
32,40
06.14
88,30
SWA-16
35,08
41,52
29,01
28,33
33.47
58,99
52.72
45,58
24.32
39.56
$7.40
33,00
67.54
45,66
-SWA-44
42,14
44.14
$4.51
$1,62
36.17
09,53
55.75
45,24
24.74
40,76
56.45
$3.60
68.40
63,76
SWA-156
43,36
45.31
40.96
90.34
52.51
45.79
23.55
40.25
57,00
57,40
08.96
69. 06
-SWA-102 0134
45,00
66.43
$8,95
54.74
41.90
99.64
52.25
45,71
25,00
40,07
$7.92
53,20
67.92
63.84
Transfomer
64,56
06.30
41,64
35,89
42.15
09.91
52.64
45.96
24.00
40,48
57.31
30,40
09.64
63.00
1B Parumeters/ 15D Tokens
SWA-4
8,46
38,56
67,95
30,40
47,81
SWA 16
33,81
41.52
25.73
27,66
32.93
58.77
52,64
47.26
24.32
40.28
55.26
33,40
67.41
48,67
SWA-64
09.18
SWA-156
47,26
25.26
33,40
40.94
30,50
69,49
Tranodormnr
45,31
66.65
41,16
35,79
42.23
70,78
52.25
45,19
23.55
40,25
$3.91
51,40
67.96
45,42
Parameters// Tokens
44,90
27,88
52.49
45,19
60.5
52,50
68.56
09.62
SWA-16
39,03
45,10
23,73
29,47
36.33
72,09
53.04
45,99
25,43
(1.15
$3,39
32,80
70.76
49.71
$3.35
$0,90
Tranndormer
48,90
51.63
45.54
54.02
09.41
25.24
40.07
60.75
55.00
72.25
50,00
Random
25.00
30,00
30,00
Table 13: Reference Scores of Sliding Window Attention (SWA) Models on Common-Sent Reasoning Benchmarks. On LAMBADA, HellaSwag and RACE-M and RACE-H, we observe significant performance increases with growing attention window. On the remaining benchmarks, we only observe marginal perfor- mance differences between Transformer with sliding window-size of 4 (SWA-4) and full-window attention Transformer (attention window 2048). We highlight the scores of the first short-range SNA model (window sizes (4.16,64)) that matches or exceeds the Transformer performance.
L.2 DOWNSTREAM BENCHMARKS
To evaluate the performance of the investigated models on downstream task, we investigate three classes of benchmarks:
Zero-Shot Common-Sense Reasoning Benchmarks (Section L.2.1)
In-Context Recall Benchmarks (Section L.2.2)
Few-Shot Learning Benchmarks (Section L.2.3)
Within each benchmark section, we report all raw numbers on all model sizes and number training tokens, and complement them with reference scores of BlddnggWinddw Attention models with varying iitentiionwindoww sizes.
L.2.1 ZERO-SHOT COMMON-SENSS REASONING BENCIIARRKS
When tracking the performance of "many models" on "many benchmarks", is common to resort to aggregated benchmark scores. However, aggregated scores tend to masquerade important sub-trends and limit our understanding (Burnell et al., 2023). For instance, prior work (Gu Dao, 2024; Yang et al., 2024a; Beck et al., 2024) averages over set of common-senss reasoning benchmarks. However, evaluations with 400M and IB Sliding-Window Window Attention (SWA) models with different attention- window sizes reveal that competitive, even superior, scores subset of these benchmarks can be attained with an attention windows as short as 4, 16 or 64 (see Table 13). This observation strongly indicates that subset of these benchmarks and do not require longer-range language modeling capabilities to reach competitive scores, or are simply too hard such that end up measuring noise.
either exploitable by short-range language heuristics,
43

---

Published as a conference paper at ICLR 2026
Splitting Reasoing Benchmark into Two Groups. To reduce the potential benchmark noise and deconfound the results, we aim to split the benchmark into two subsets. Therefore, we employ the following benchmark splitting protocol:
1. Reference Scores.
Run every selected benchmark on SWA-(4, 16, 64) models and a trans- former model (attention window of size 2048) on 400M and IB parameters trained on 15B or 50B tokens each.
Splitting Conditions. We then assess the following splitting conditions:
Condition 1: Analyze for every benchmark whether benchmark scores increase with in- creasing attention windows (from SWA-4 to SWA-64).
Condition 2: Verify whether no short-range SWA model (window sizes = 4, 16 and 64) outperforms the transformer baseline with an attention windows of 2048.
3. Benchmark Grouping. Finally, split the benchmark into two subsets:
Local Reasoning Benchmark Set: One of the above conditions violated.
Global Reasoning Benchmark Set: None of the above conditions is violated.
We refer to Table 13 for detailed score breakdown, including two additional SWA reference models (SWA-256 and (WA-1024) 4). Additionaly, we want highlight that these findings, and the bench- mark splitting, are based on experiments 400M and IB models trained on SlimPajama (Soboleva al., 2023). The benchmark splitting likely to change slightly when training with bigger model sizes or different datasets.
Results on all Model Configurations. We report the full set of benchmark scores on all model configuration (model sizes and number of training tokens) in Table 14. Across all settings, we observe similar trends MesaNet and Hawk-MesaNet show strong performance especially on the global reasoning benchmark set. Among the remaining recurrent models, only Gated DeltaNet reaches competitive scores with MesaNet on this benchmark subset. In contrast, we do not observe much score variation on the local reasoning benchmark set. Hawk, the worst performing model on the global set, reaches competitive even close-to-best scores within this set average. This confirms the hypothesis that this set benchmark are likely to measure different aspects of language modeling, or are potentially noisy, not suited for our models they might be still too challenging.
44

---

Published as a conference paper at ICLR 2026
Global
Subset
Local Subvet
Model
LMB.
Hella.
RACE-M
RACE-H
AVG
PIQA
Wino
ARCE
ARC-C
SIQA
BOOLO
OBQA
AVG
15B I.
Tawk
21,87
33,54
29.01
28.52
28,23
64.64
50,83
40.24
21.95
39,
59,11
27.50
12..5
45.76
Mamba2
27,83
33,21
32.04
30,53
30,90
64,44
50,36
39.27
22:27
59,00
51.44
26,40
62.13
44,42
31,05
34,30
33.43
28,71
31.84
63,66
52,00
41.41
21.76
56.85
28,80
63.97
45.03
.LSTM
31,19
34,41
30.94
29,47
33,50
65,13
52,37
40,77
21.76
58,79
56,64
27.40
63.44
45.76
DeltaNee
32,02
33,89
32.04
30,43
32,10
65,45
30,91
40,52
39,15
60,89
28.00
03,97
96,35
Ganed DehuNer
31,65
34,53
33.99
29,09
32,51
64,53
51,07
41,62
21.59
59,05
50,05
26.40
63.21
46.19
34,49
32.87
30,43
32.36
66.43
31.85
40,03
22.27
38,43
56,73
27.40
63.34
45.81
Hask-MMsa
32,14
34,99
32,67
31,96
32,99
65,40
52.96
41,16
23,55
39,05
55,25
28,00
02.89
46.03
Transformer
33,84
33,91
35.91
30,62
33,57
65,
52,49
39,27
22.46
39,10
99.63
26,40
63,76
06.31
Models/ 5o0
Howkk
28.18
66,80
51,62
41.33
22.87
39.
63.97
Mambs2
20.23
34,34
33.15
20.86
$1.62
65,78
51,46
41,00
21.67
50.82
90,30
25,00
61.74
46.11
32,16
35,57
32.04
29,86
65.56
51.07
43.18
23,81
39,82
52.23
63,72
46.10
XLSTM
32,74
25,89
32.87
30,14
32
06.59
31.54
41.57
25.12
39.15
58.65
27.00
64.25
46.49
DeleaNer
32,89
35,39
32.32
31,67
33,07
66,10
51.93
40,53
22.76
58,74
57,65
29.00
64.29
46.36
Guted DeltaNet
32,85
36.15
33.15
31,96
33.53
66,76
31,22
41,92
23.55
38,35
00.43
29.00
64,10
46.92
34.53
30,24
33,33
65.40
51,70
41.63
22.61
Llakk Mesa
34,31
26,40
32.04
31,20
33,69
66,21
31.93
41,54
22.55
38,54
53,57
30,00
64,74
46.38
Transferer
35,40
36,01
35,08
31,10
34,40
64,58
52,00
41,4
22.01
40,12
99,79
30,20
64.23
46,80
400M Models/ 15B T.
Howk
$2,06
35.13
68, 66
44,53
25.00
59.66
Mambal
32,25
51,70
43,31
39,71
59.54
47.88
GLAA
40.09
42,40
34.59
30.54
37,41
68.61
51,78
44,90
24.91
50.61
60. (0.40
28:40
568.30
48.37
37,46
68,50
52.05
45,12
23,46
59,72
DellaaNer
59,28
41,49
36.46
32,34
37.99
69.26
51,70
46,00
23,81
59.76
$2,51
31,20
67.47
47.71
Gated DelanNet
39,98
42,55
32.87
33.68
37,27
69.
52.33
45,20
25.17
40,02
59, 14
29.40
67.60
48.56
33,21
37,65
67.79
50,51
45,12
22.87
47.00
Howk-Mess Mesa
39.84
43,15
34,81
31,67
37,37
69, 84
52,17
45,33
22:27
40,22
$8.04
29.50
67.41
48.11
SWA--4
4,62
34,97
25,97
25.95
22,87
66,81
49.33
43,81
24.25
59.82
57,51
31.00
63.78
46,89
-SWA-64
38.54
39.35
12,87
30,34
35,25
68, 91
52,17
44,40
22.87
59. 06
58. $8.55
79.30
64.99
47.61
-SWA-1004
41,43
40,90
37,57
34,26
38.54
67,90
52,80
44,49
22.61
40.58
60,37
30,70
66.58
48,19
Transformer
41,12
41,27
37,79
34,45
38.53
68.21
51,07
44,28
24.57
40.23
58. $8.10
26.40
66.58
47.68
40UM Modcls 508 T.
Howk
36,70
47.00
33.45
32,54
37,42
71,93
52,25
47.26
24.06
40,89
59,91
34.20
69.85
50,04
Mamba2
37.58
68.72
52.17
45.33
40,74
54.31
36,04
40,07
56.57
29.80
89.96
48,86
-TSTM
41.82
46.23
34,53
33,30
$8.97
68.00
53.35
46,00
23.46
41.61
53.43
31.00
69.33
48.90
DeltaNet
42,25
45,92
37,02
33,68
90,72
70,18
52,72
45,24
24.23
40.48
57,37
32,20
68.87
48.91
Gased DelaaNer
43,99
46.57
35,36
34.83
40.19
70,18
51.85
46,38
25.77
40.5%
$4,89
32,64
70.53
49.10
Mosa
43,39
46,93
38.05
34,26
40.88
70,73
54,46
46,21
24.91
41,10
$7.89
32,40
69.38
49.64
Howk- Mesa
43,94
46,96
3M.12
33,69
40.13
70,46
34.78
46,46
25.51
40,74
$7,80
30,00
70,46
49.53
SWAA4
18.28
39,00
29.56
25.66
25,63
67.85
51.93
44,49
24.85
59.71
58.23
32:40
66.144
48.20
SWA 64
42.34
46,14
34,55
31,07
38,17
69
53.75
45.24
24.74
40,25
56,45
31.60
08.49
48.76
SWA 1024
45,08
46,43
38,95
34,74
41,30
09.64
52.25
45,71
25,00
40.07
$7.92
32.20
67.92
45.84
Transformer
44,96
30
41.44
35.89
40.15
69.01
52,64
45,96
24.06
40.48
57,31
30.40
699 64
48,80
IR Models 15B
Howk
37,98
47.71
35.08
33,25
38,25
71.98
50.43
48.61
25.43
41.90
58.53
31,50
70.59
49.85
Mamba2
39,63
45,06
36.74
34.35
38.95
70.13
52,33
46.97
25.43
59.41
57.34
31,50
70.34
49.22
GLA
43.24
47,20
33.43
33,68
30.39
70,95
52,41
46,97
25.00
41.15
58.59
33.00
70.34
49,80
-1STMM
35.91
30.86
DeltaNee
43,45
47,47
36,46
33,50
40,17
70,78
$2,80
48,48
25,09
39.92
66,44
31,20
69.00
49.72
Ganed DeliaNee
45.37
48,49
35.36
34.07
40.83
71.60
$3,09
48,57
24.83
40.07
51,76
37,40
70.46
49.46
47,70
33,49
40,60
70,89
54.46
47,56
25.26
41,04
32,20
49.71
Howl-Moss
44,05
48,70
39.23
33,40
41,54
71,23
53,20
49.54
24.74
40.89
$1.93
32,00
70.78
40.29
-SWA-4
8,46
38.56
27,62
27.18
25,46
67.95
51,
46.72
23.72
40.17
56.73
30.40
65.50
47.81
-SWA-64
42,60
44,04
31.49
50,72
37.21
69.91
51,
46,72
24.66
41.10
58.5
33.20
67.98
49.18
-SWA-1024
45,06
46,23
39,50
34.74
41,38
70,29
53.99
47,39
24.15
40.94
99.54
30.50
69,00
49.49
Transfumer
45,31
46,65
41,36
35.79
42.23
70,78
52,25
48,19
23.55
40.28
$2.91
31,40
67.98
48.42
1B Models/ S0B
Howk
41,80
54,25
34.25
34,35
41.17
72.91
52,33
51.52
28.75
40.84
56.51
35.00
24.67
$1.57
Mambal
42.15
51,46
37.85
35.02
41.62
71.76
53.95
48.95
26.54
40.5%
55.90
33.60
73.99
50.51
GLAA
47.27
41.44
44.34
72,25
54,14
27.56
ALSTM
46,57
53,08
37.57
34.74
42.99
72,52
34,62
49,45
27.05
41,76
59,75
35,80
72.06
51.50
DehtaNee
47,08
41133
34.83
43.86
72,20
54,30
48,19
27.90
40.84
64.49
34,40
74.28
51.58
Gated DeltaiNet
49,19
54,10
39.78
36,27
44,84
71,93
34,06
51,22
26.38
41,35
53,27
34,20
73,14
50.76
Mesa
45,83
53,58
401.8
36,84
45.00
71,71
53.59
49,37
25.68
40,58
53,30
35.64
74.09
50,49
40.61
44.62
72.52
56.04
50,90
40.17
56,02
35,60
74.03
24.63
44,90
23.05
52,49
48,
39.56
66:15
68.56
-WWA-64
46.11
$1,30
38.40
33.49
42.53
71,87
53.35
62
26.71
40.74
71.74
50552
SWA--100
50.38
53.89
41.44
35,22
45.68
72.47
53.35
49,41
27.13
41.61
62.20
32.60
72.06
$1.35
Trandtencr
45.92
53,63
42.27
37.52
45.54
72,31
54,63
49,41
40.17
66.73
35,20
72.25
51.62
Table 14: Benchmark Scores on Common Reasoning Benchmarks on all model configurations. Best scores among the recurrent models are highlighted for each training setting.
45

---

Published as a conference paper at ICLR 2026
L.2.2 IN-CONTEXT RECALL BENCHMARKS
To evaluate in-context recall, adopted the minimal-transformed version of the benchmarks from Arora et al. (2024) to allow evaluation of non-instruction-tunee models. We truncate inputs to 2000 tokens, and sample greedily until either 48 tokens new-line delimiter is generated. We then parsed whether the target was contained in the generation non-ccae-sensitivee as in Arora et al. (2024).
Sliding-Wiin Window Attention Controls. As expected, we observe consistent score increases with growing attention window size (see Table 15). However, we observe that the SWA-1024 is consistently better on SQUAD than the transformer baseline with an attention window of 2048. Closer inspection the SQUAD benchmarks reveals that the tokens-to-recall are most frequently located in the last Ik tokens of the sequence. Similarly for FDA, most tokens-to-recal are located at the very beginning of the sequence with an average length 2000. Hence, we observe significant performance increase from SWA-102 the transformer baseline with attention window of 2048.
Results on all Model Settings. McsaNet consistently attains best, or in few cases second-best, performance scores average across all evaluated model settings (see Table 16). Moreover, observe that our insights from the PPL analysis in L. directly translate to the observed results in here, e.g., Hawk attaining the worst in-context recall performance.
150 Tolons
SIB Tokens
SWDE
SQUAD
FDA
TQA
NQ
DROP
AYG
SWOR:
SQUAD
TQA
NQ
DROP
AVG
32c
302
axc
20x
300
acc
900MT Models:
7.35
5.00
14.51
10.56
7.77
21.27
9,79
-SWA--15
9,63
10.82
0,27
24,58
4.85
15,33
10,97
13,05
15,30
1.09
33.35
6.59
17.35
14.95
-SW 644
13.14
26.74
10,07
39.34
5,23
19.12
18,94
19,37
38,44
11,43
48.76
7,25
23.96
24.84
SWWA-266
21,89
40,92
12,38
50.95
0,87
25.67
26,06
40,66
47.19
14.20
56.15
10.10
34.70
79.73
10.80
26.45
38.21
Transfrma
37.13
79.13
16.57
26.90
48.33
36,93
75.86
58.95
20.37
49.95
Models:
9.00
6.33
17.06
11.00
13.05
10,06
26.54
11,87
9.51
12.85
18.15
SWWA-64
16.74
30,56
16.61
44.55
7.19
20.45
22.59
22,32
39.65
12.70
$1,90
9,63
23,91
26.72
17.09
30,00
35.23
-SWA--004
60.76
40,65
24,33
$6.90
11.88
27.65
37.03
69,73
47.68
76.68
61.43
15,52
10,04
40.84
Trassfomer
77.04
29.47
70.96
52.27
-aandom
22.
0:0
20.0
70.0
10.00
22.0
7..0
10
70.0
3::0)
1:.0
Table 15: Reference Scores of SWA Models In-Context Recall Benchmarks. The pattern of best scores (highlightreded) is very consistent across the evaluated settings. As expected, increasing performance with increasing sizes attention windows. Except on SQUAD, the transformer commonly attains the best scores.
46

---

Published as a conference paper at ICLR 2026
15B Tokens
SHI Tokens
SQUAD
TQA
FDA
DROP
145M Medels:
HHok
11:45
11:09
0.21
10.35
18.18
11:51
10:04
0.39
35.25
5.38
13:35
24.33
40,17
7.67
25.17
GLA
37,08
38.20
14.07
44.73
5.58
23.35
27.67
29,69
30.46
15.88
48,16
10,50
23.86
28.14
25,00
46.75
19.96
13.43
DeliaNet
33.57
27.83
32.31
30.83
38,07
32,44
48.34
27.77
Mesa
36,50
34,35
14,35
447.22
10.20
25.08
28.21
00,50
24,99
15.79
47.04
13.97
23.77
28.18
43.04
4MM Medsls:
-ook
16.47
23:86
1,00
42.42
5.01
19.65
18.56
23,05
23,85
1.45
48.03
10.83
30,60
31.29
Mumba2
43.11
29,85
20.43
47.04
11.47
22.50
29.12
51.04
29.76
22.25
$2.90
12.58
24.77
32.21
-GLA
$2,30
10.55
70156
50L12
14.16
78.41
14.17
54.10
61.59
26.21
55.04
10.00)
26.17
16.50
XLSTM
$1.67
38.94
23.32
$1.13
14.76
23.45
33.88
50,86
38,87
25.23
$3.67
15,09
24.63
14,89
DedinNec
90.33
$5.02
37.40
50.00
14.38
35.16
15,90
$3.00
$5,59
17.40
$3.90
15.11
35,67
15,19
Gateal DeltaNict
$3.20
35.15
27.04
51.72
15.06
24.82
34,65
56,53
37,25
29.09
53,55
15.01
23.95
35.95
Mesa
53.11
38.54
52,13
99.05
47.05
57.17
29.30
Haok Mesa
52.6
39.95
23015
52,78
13,02
26.20
34.72
53,65
39.95
25.14
55,51
15,02
27.25
26.23
Tomelomeer
77,50
17.15
79.11
55.08
15.50
16.90
48.31
74,06
96.95
75.96
58.95
15.94
70.51
49.05
Mannba2
33,65
37,67
-GLA
90.68
41.29
55.04
25.01
17.8
10.00
19.24
15,59
25.25
19.64
-XLSTM
$7.61
39.11
24.50
$4,50
15,17
26.64
36,26
63.37
38.91
31.58
55,00
15,06
25.59
39.25
58,15
Gueed DeiuNNt
39.59
39.45
37.30
55,86
17,30
25.87
39.25
60.22
39.81
32.12
59,54
13,56
26.98
30.54
Mesa
19.64
41.79
Hzok-Mesa
61.08
41.35
27.77
34.74
15.33
37.08
00:11
65.51
28.68
01.13
17.61
27.20
19.99
Trandcence
70:31
42.76
36.99
63.21
52.37
Table Benchmark Scores for In-Context Recall Benchmarks on Model Settings. MesaNet consistently attains the best or second- becond-best score on average across all evaluated model settings.
47

---

Published as a conference paper at ICLR 2026
L.2.3 FEW-SHOT LEARNING BENCHMARKS
To evaluate the few-shot learning ability, tested two distinct types of few-shot tasks, (i) word scrambling tasks introduced in (Brown et al., 2020b) and (ii) couple of language-fo-language translation tasks.
Word Scrambling Tasks. We report the few-shot performances in Table 17 for 0-,1-,10- and 100- shot settings, As few-shot evaluation tend be sensitive the selection and ordering of few-shot examples (Lu et al., 2021), we report the mean performance over 10 randomly drawn few-shot prefixes. We observe consistent improvements with an increasing number of fewshots for all models except for SWA-4. MesaNet attains the strongest performance scores in most settings, and outperforms the transformer baseline significantly.
While we evaluate on all five word scrambling tasks introduce in Brown et al. (2020b), we observe only observe signal (performance above 1%) for models the ranges 145M to on two tasks: gpt /eccclylllleeeeeeeeeeeeeeeeeleee ing tasks, we observe performance score close to 0%, in line with the results Brown et al. (2020b), and hence omit the scores here.
win_worrd and gpt 3 3/mim/mmmddddddo wttm///ddodd...aaaaaa.aa wa..r-daaaraamms On the three remain-
gpt3/cycle letters
in word 1h0-
3pt3/mid word 2 anagrams
O-shol
1-shat
145M Models
Hlak
0.2 0.410.2
1.3.0.5
1.7.0..5
0.2
0.410.1
0.8.0.22
0.7.10.2
Mamba2
0.0
0.240.2
1.7=04
1.420.3
0.0
0.240.3
0.6:0.22
0.310.1
GLA
0.1
0.2.0.33
2.7
3.0.0.4
0.2
0.110.11
1..10.4
1.510.1
XLSTM
0.1 0.4+0.5
2.8-06
3,8=0.5
0.3
0.1+0.2
0,940.3
1.+0.11
DeltaNet
0.1 0.5+0..4
26-099
3.2+0.6
0.1
0.2+0.1
1.240.3
1,1+0.2
Gated DeliaNet
0.1
0.8.1..6
2.5
3.4.0.66
0.0
0.410.4
14.0.2
1,710.2
Mesa
0.1 02400.3
22-00.5
3.340.5
0.1
0.2402
1..40.3
L.740.1
0.310.2
1.410.2
Transformer
0.1
0.51004
2.6.0.5
3.720.33
0.1
0.2:0.22
1.2:2033
1.710.2
400M Models
Hawk
0.1
1.741.2
5.3-1.2
6.6-044
0.1
0.940.7
2.400.1
28+-02
Manba2
044
2.011.4
0.5
5.1.0.55
044
0.9.0.5
16.10.3
161001
GLA
0.0
1.7+1.1
5.2-100
7.6-0.3
044
1,0007
2440.2
26+02
ALSTMM
0.0
2.31133
5.7
8.210.5
0.2
1.110.5
25.00.3
29103
DeltaNet
0.1
1.5+10
5.7-1.3
7.6-0.66
0.0
1,100
2440.3
2.6+033
Gated DeliaNet
0.1
2.111.7
6.5_1.
9.0.1.88
0.1
0,910.5
26110.3
3.410.2
Mesa
0.4
2.2+1.2
6.6=1.0
9.2=0.6
1.1+05
2.6+0.33
3.240.2
Hawk-Mesa
0.0
1.39099
4.0-14
7.320.5
0.1
0.920.7
2.6:10.3
3.110.1
0.0
0.3++.33
0.9.0.3
-SWA-64
0.1
2.5+1.5
4.-1.1
4,7+099
0.1
1,2+0.5
2.7+0.2
2.7+0.1
2.541.6
1,2++0.8
2.400.4
3.140.3
Transformer
0.4
2.411888
6.7=1.2
8.5=044
0.5
1.4=0.77
3.3:20.4
3,6#0.2
1B Models
Hawk
15+10
6.8-1.5
9.2=0.6
0.940.8
3.5
3.420.2
Mambu2
0.8
3.7.11.7
6.3-08
6.420.77
1.1
18810.3
2,4:1.33
2.010.4
GLA
4.142.11
8.4-122
10.320.5
2.3+0077
3.220.5
4,240.2
ALSTM
0.0
2.2.11.3
73-18
11.0:0.4
0.3
18810.
3.9.10.3
4.610.3
DeltaNet
2.9:119
8.7-13
11.7=0.8
1.620.8
3.720..5
4.14033
Gated DeliaNet
0.3
4.01118
89-14
11.820.7
0.5
2.51099
4.7.10.6
6.110.4
Mesa
0.5
3.342.0
9.7-13
14.0=0.5
LI
2.11111
4.740..6
6.240.4
2.141.5
7.2-1.5
2.0+0.9
SWA-4
0.1
L140,9
1.5007
2.0=0.88
0.2
0.6+0.55
1,4:0,3
1,4+033
-SWA-64
35+18
6.3-1.3
7.8=0.6
1.0
2,4+0.7
38-00.3
4.0+0..3
SSWA-1024
0.1
3.4188
7.5=13
9.0=0.5
0.1
1.920.9
4.3:204
4,3+0.2
Transfoemer
0.0
3.022.2
6.8=1.7
9.2-0.6
0.1
2,420.6
4.240.4
4.7402
Table 17: Few-Shot Performance (Accuracy + Std.) GPT-3 Word Scrambling Tasks (Brown et al.,
2020b) of Models Trained 50B Tokens. Best 50-shot scores are highlighted, and standard deviation reported over 10 random drawn few-shot selections. MesaNet attains the strongest scores most settings, and outperforms the transformer bascline significantly.
48

---

Published as a conference paper at ICLR 2026
Sangguage-to-Langaage Translation. We evaluated a model's capability to translate from three different languages to English: (i) French to English (Bojar et al., 2014). (ii) German to English (Bojar et al., 2016) and (iii) Romanian to English (Bojar et al., 2016). We follow the exact prompt setup of Brown et al. (2020b) evaluate with (0.1,5.10) -and 50-shots, and report the performance in Table 18 with respect to BLEU-sb (Post, 2018) for models trained on 50B tokens.
We observe scores of different performance magnitudes across the three languages, which is most likely caused by the multi-lingual distribution of the training data corpus and French being more prevalent than German and Romanian. MesaNet attains superior scores among the recurrent models. However, MesaNet, and more general all recurrent models, fail to match the transformer performance by a relatively big margin, especially at the scale of IB models. This finding is non -urppissii given the impact of the attention mechanism on the field of machine translation (Bahdanau et al., 2014), indicating that pure model- and data-scaling based on recurrent models will not be enough to match the performance of attention-based architecture (Rodchenko et al., 2025).
WMT14
TR-SH
6 DE-EN
M RO-EX
10
145M Models:
Hook
0,61
0,31
0.25
0,08
0,11
0.49
0.16
0.30
0.19
0.19
0.44
0.16
0.06
0.19
0.29
MMama22
1.68
0,56
0,75
0.73
0.19
0,40
0.25
0.32
0.44
0,50
0.46
.GLA
1.47
0.21
0.52
0,35
0.44
0.24
0.12
0.51
-XLSTM
1.64
0:07
0.73
0,87
0.67
2,09
0.63
0,33
0.81
0,77
1.68
0,22
0.34
0.50
0,35
-DeltaNer
1,57
0.20
0.78
0,90
0,59
1.68
0,49
0,32
0,73
0,81
1,56
0,80
0,55
0,58
0,61
-Guedd DeltaNei
1.31
0,25
0,28
0,42
0,35
0.61
0.19
0.35
Mesa
1,26
0,66
0,35
1,30
1,06
1,53
0,89
0,58
0.46
0,62
1,56
0,32
0.51
0.45
0.52
Howk-Mesa
1.62
0.19
0,80
0,77
0.94
2,03
0,51
0.67
0.47
0.79
1,77
0,24
0.54
0.49
0,90
Transformer
1.55
0.05
0,59
0,70
1,90
0.39
0,86
0.61
0.71
1,75
0,28
0.30
1.41
0.50
400 Models:
-Howk
1.54
2.28
3,95
4.25
4,97
1,36
5,24
3.87
3,67
0:91
1.29
2,00
1.52
1,39
-Mumha2
2.15
4.05
6.07
4.55
3,49
2.17
1,43
3,13
3,19
2.52
1.68
01.66
1,36
1.94
1,64
GLL
1,83
3,20
2,74
4,83
4,23
2.15
2,60
1.58
2.04
2.19
1.72
0.62
1,42
1.96
1,30
xL.TTM
2.14
3,08
3,48
3,66
3.28
2.29
2,06
2.68
2.79
2.77
1.65
1,10
1.37
2.20
2.15
DelitaNet
1.72
3,09
4,43
3,89
3,49
1,84
1,55
3,52
2,83
2.47
1,79
1,67
1,63
1,29
1,45
Gased DeliaNet
1.87
3,92
3,86
4,16
3,27
0,85
3,35
3.18
2.94
1,80
1,05
2.56
2.13
2,22
Mesa
2,23
2.75
4,33
5,05
5.33
2,06
0,80
2.62
3,11
3.70
1,75
0,68
2.09
1.63
2.47
Howk-Mess
1.90
2,83
3.89
4,54
4,27
2,00
2,55
3,66
3,76
5,50
1,74
0.68
1,71
1,71
2,78
-WA-4
0.34
0.13
0.14
0.13
0,12
0.25
0,19
0,76
0,21
0,26
0,29
0.10
0,06
0.07
0,05
SWA 64
1.35
3,82
4,94
4,92
2,17
1,66
1,57
1,18
1,10
1,38
1,29
SWA-
4.09
4,55
7.77
3,66
4,57
5.14
1,96
1,52
2.67
Transformee
2.61
8,27
8,77
8.92
9.63
2,04
3,13
5,73
5,34
5.49
1,94
1,02
1,29
2,23
2.56
1B Models:
Hawk
3.72
5,88
8,55
7.35
4.17
3,33
3,79
3,77
5.20
5,86
2,37
2.89
4.39
4,22
4.17
Mamha2
4,20
11.81
11,90
11,28
5.83
3,07
3,62
6.79
8,18
3,35
2,04
4,27
6,83
4,75
3,38
GLA
3,15
10,60
11,87
10.50
10.51
7,,90
7.46
6.00
4,30
4,95
xl. STM
11.71
10.32
10.56
4.74
3,00
-DelaNett
5.24
8.34
10,79
10,08
6,91
8.72
6.01
5,66
2.29
1,01
4.32
3.39
2.58
Gaated DelmNee
4,71
8,24
10,05
11.25
11.31
4,31
7,59
9,077
8.60
8.76
2,45
4,63
5,67
5,33
5,51
Mosa
3,58
12.44
11,57
11.64
6,98
2,96
5,07
Howk-Mesa
3,68
7,99
10.58
13.16
12,01
2,92
8,05
10,50
8,67
8.43
2,36
4,73
4.91
5.81
5.99
SWA-4
0.54
0.72
0,72
0.72
0.74
0.49
0,75
0.89
0,87
0.72
0,22
0.12
0.14
0,11
0,06
-5WA-64
3,12
3,05
SWA- 1024
8.75
18,09
18,70
19,83
5,99
10,85
14.91
14,30
4.19
10.05
8.38
Transformee
8.30
18.49
17,81
17.70
19.14
6.10
13,06
11.99
13.99
13.85
3.54
5,92
7.11
7.35
7.82
Table 18: Performance Scores (in BLEU-sb) on three Translation Tasks on Models Trained on 50B Tokens. Best 50-shot scores among recurrent models are highlighted. as well as Transformer reference scores. While MesaNet attains the best-score among the recurrent models in most settings, it under-perforin performs transformer by relative big margin.
L.3 NEEDLE IN THE HAYSTACK (NIAH) RESULTS
Setup. We conducted a sweep of experiments on single-neede tasks (NIAH) from the RULER benchmark (Hsieh et al., 2024) suite for 1B models trained on 50B tokens. We ran experiments for both haystack types (noise and essays) for all key/value combinations (both can be in the form of: words, numbers or uuids) context lengths 2048 and 4096.
Results. As scores are quite sensitive to the chosen key and values types, we report mtd percent accuracy over all 9 key/value combinations, with 1000 evaluation samples for each setting. On the "noise" haystack, MesaNet demonstrates strong scores with very low fluctuations across key/value combinations. On the "essay" haystack, we observe relatively high score fluctuations across key/value combinations for all models which makes it hard to form conclusions. However, we would still like to highlight the strong performance of Hawk-Mesa on the essay haystack.
49

---

Published as a conference paper at ICLR 2026
NIAH Noise
NIAH-EssHV
L=2048
L=4096
L=2048
L=4096
Hawk
5.0
1.7 2.9
3.2
2.1 1.6
Mamba2
79.7 17.9
0.7
51.3 22.3
0.0
GLA
9..2
68.5
18.9
73.5
41.4226.9
XLSTM
94.8
804
14,9
69.1 20.5
24.3 9.9
DeltaNet
1.0
6.3
32.3
27.9
Net
9.1
9..
52.1 33.7
11.0
MesaNet
9.5
66.8 + 28.9
1.0
-ask-Mesa
97.6 3.5
65.3
10.5
55.5
28.5
SWA-1024
51.8 0.9
24.3 1.3
47.5 # 11.8
21.6
MHA
99.7
0.0 0.0
98.2 4: 2.5
0.0
Table 19: NIAH Benchmark results for 1B models trained 50B tokens.
M VARYING THE NUMBER OF CONJUGATE GRADIENT STEPS WHEN TRAINING MESANETS
Here present the effect when training the MesaNet less than 30 steps. We opted for training with 30 steps, as we were not optimizing for training flops but first investigate a fully converge Mesa layer, and because of early experiments after 30 steps.
400million model which indicated little improvement
As shown in Figure 13, sequence length when comparing to a model which is trained on 30 steps. Only when dropping the number of CG steps below 10, see a more drastic jump loss increase. As we have show in section C, the backward pass also relies on running the CG method to solve linear systems of equations and we leave investigating for future work varying the number of steps in the forward and backward pass.
small, interestingly, uniform increase of training loss across the
DG-3)
CG-20)
Meca (Thin CG-5, Inference 301
Meea (Trin CG 25) Mesa (Train 0G-50)
Mesa (Trinn CG+10)
Mesa (Train CG-15)
0.02
Meza
1024
2045
Taken Pocition
Figure 13: We compare the validation loss across the sequence of 400 million parameter MesaNets trained on 15B tokens, when varying the number of conjugate gradient steps during training. We observe slight uniform increase of validation loss across the sequence length when comparing to model which trained 30 steps. Only when dropping the steps drastically we see a substantial increase loss.
N EVALUATION METHODOLOGY
For given question we assess for all possible options y the loss (1.((yfa:) of the option conditional on the question, and then normalize by the number tokens of y. contrast to related work (Gu & Dao, 2024; Yang et al., 2024a; Beck al., 2024), we do not heuristically choose between byte-normalized and non-normalized scoring schemes as we have a fixed tokenizer across all models.
Mulitple Choice Tasks:
Greedy Matching Tasks. For a given input x and an expected target sequence (e.g., one or multiple tokens), check whether t would be matched under greedy sampling. This is done by obtaining the logits for the concatenated input of x + y. and checking whether all tokens belonging to are matched by taking the argmax over the logits.
50

---

Published as a conference paper at ICLR 2026
In-Context Recall Tasks. We follow closely the setup of (Arora et al., 2023b). For a given input I, we sample greedily a completion from the model until either 48 tokens or a new-line character is sampled. We then check whether the target y is contained in the output men--as--senstive)
O
AN INTERNAL ANALYSIS OF THE MESANET
Layer
1.0
8
1.0
Cond +
Number OS steps
sorted Avlluss
1.0
00560.007
0.4
0.4
0.5
0.0
1.00
2000
3.0
1000
2000
1000
2000
1000
2000
100
0.0
1000
2000
Huad 0
Head
Head
Head
Head
Head 5
Head 6
Head 7
Layer
8-
Cond +
Number O3 steps
sortec voluuss
00560. 0
3.0
1.0
1.0
50
40-
10
0.0
3.4
0.4
0.0
1200
2000
3.0
1000
2000
10000
2000
1000
2000
100
0.0
1000
2000
Head 0
Head
Head
Head
Head
Head 5
Head 5
Head
Layer 36
3.0
8.
1.0
x
500
CondiMe +-A
Number 03 steps
sortec Avluuus
10
00560.007
300
0.0
3.4
200
0.4
100
2
0.0
12000
2000
2.0
1000
2000
10000
2000
1000
2000
1.00
0.0
1000
2000
Head 0
Head
Head
Head
Head
Head 5
Head 5
Head
Layer 17
3.0
8.
1.0
CondiMu +X
Numbar OG steps
sortec Avaluus
1.0
ccs1o.
150
100
0.4
3.4
0.4
30
0.0
1200
2000
3.0
1000
2000
1000
2000
1000
2000
1000
0.0
1000
2000
Head 0
Head
Head
Head
Head
Head 5
Head 6
Head
Layer 28
3.0
8
1.0
x:
CondiMu +X
Number 0G steos
somtec Avluues
10
c0510. o()
200
150
0.4
3.4
100
0.4
0.0
1200
2000
3.0
1000
2000
1000
2000
1000
2000
1000
0.0
1000
2000
Head O
Head
Head
Head
Head
Head 5
Head 6
Head
Figure 14: input strength 3, forget strength 7, regularization strengths A as well as other internal statistics of a 400M parameter MesaNet trained on 50B tokens averaged over 500 sequences from the Slim Pajama validation set. We observe that high Re values usually lead to the condition number of the be inverted matrix K,K?" mesa. We also observe (outer right plot) that usually these heads lead to higher cosine similarity (cos) between or, the output of the layer if no CG steps are applied which corresponds to gated linear attention, compared to the Mesa output of We compute the number of conjugate gradient steps are computed by measuring the steps of the conjugate gradient method to reach an error of 0.001. We sort the heads for plotting purposes according to their average gamma values.
increase over time. which in turn leads to more CG steps required to obtain output for the
51


Reviews:
[{"review_id": "GlKdNgqVEi", "reviewer": "Reviewer_MDkC", "summary": "This work introduces a novel layer, called Mesa layer, guided by the theory of online learning/ test time training. The layer’s conception is similar to modern Linear RNNs/ SSMs like Gated DeltaNet, which allows for recurrent inference and chunk-wise parallel training. The difference lies in a principled online learning objective and, consequently, the recurrent state update rule, which features not just one, but two states and is computed via several steps of conjugate gradient descent. This yields competitive results in several benchmarks including large-scale LM with other models, such as the most related Gated DeltaNet. The authors conduct thorough theoretical analysis of their method and robust validation.\n\nI’m convinced this work is a valuable contribution to the field of efficient alternatives to Transformer/ online learning models and vote for its acceptance.", "questions": "See section “weaknesses”. Also:\n\n1. I’m curious why the regularization term in equations 2 and 4 is $\\Phi^{\\top} \\Lambda \\Phi$ and not $\\Phi \\Lambda \\Phi^{\\top}$?\n\n2. What are your thoughts on why MesaNet becomes significantly superior to Gated DeltaNet on in-context recall tasks from Table 14 after prolonged training (50B tokens vs 15B)?\n\n3. Does MesaNet indeed have linear computational complexity w.r.t. sequence length T? Lines 449-450 highlight “the need for higher number of steps as t grows” indicating that it may not be the case.\n\n4. Lines 1399-1403: Do new sequences always start with fresh documents or new documents can cross sequence boundaries? I.e. if document A has more tokens than 2048 and therefore cannot be fitted into  sequence 1, will the next sequence 2 begin with the leftover part from document A?\n\n5. Do Transformer and SWA variants used in the experiments have any differences in architecture from LLama 2? Specifically, do the models employ RoPE?\n\n6. Why can't you use Woodbury identity for exact matrix inverse in chunk-wise parallel form instead of costly and approximate conjugate gradient descent steps?\n\n7. Will you release the data preparation/ configs/ pre-training code and, importantly, the weights of different models you've pre-trained for experiments in your paper? That would be greatly beneficial for the advancement of the field and provide very valuable playground and baselines for future research.\n\nEdit: Upon second thought, I see that Woodbury identity (https://en.wikipedia.org/wiki/Woodbury_matrix_identity) in Q6 wouldn't provide any benefits because the problem of finding inverse of $d \\times d$ matrix translates into finding inverse of $N \\times N$ matrix where d is head dimension and N is sequence length. However, I'm still not sure why you don't use exact matrix inverse algorithm with readily available PyTorch or CuBLAS methods (or some JAX/ TPU alternatives). For dense matrices, both conjugate gradient and ordinary exact matrix inverse methods have the same $O(d^3)$ complexity. Have you considered comparing speed of an exact method against your implementation with a commonly used head size d=128?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "First, I stress that I liked this paper a lot and I'm excited about the latest developments in the field of linear-time Transformer alternatives, and in particular the direction where this and related works are steering the field. \n\n* The proposed method (online learning objective, corresponding update rule, chunk-wise and recurrent implementation algorithms) is novel and original.\n\n* The paper is very rich in content, be it new theory, connections to other literature, and especially experiments. \n\n* The scope of the experiments and ablations is very vast and comprehensive, to the extent that it leads to important and valuable observations about the other models and the field of linear alternatives to Transformers themselves (see e.g., summary in lines 81-84). \n\n* The paper is self-contained. For example, Appendix J explains the background from previous work leading to Mesa layer, so it’s not required to be familiar with it beforehand.\n\n* I’d like to highlight specifically that the paper provides a detailed description of the pre-training settings and hyperparameters, including e.g., dataset preparation, which allows for greater transparency and reproducibility.", "weaknesses": "The below comments do not represent major weaknesses and don’t denigrate the quality of the paper. They aim for improvement of presentation or consider the pieces of text which the authors could just omit without detriment of the exposition. \n\n1. Please state in the beginning of the paper that all vectors are column vectors to avoid confusion because many related works such as GLA or Gated DeltaNet use row vectors as a convention.\n\n2. Lines 121-123: How do you derive the expression $\\gamma_t \\Phi_{t-1} + \\beta_t v_t k_t^T$ out of equation 3? Also, the statement on line 127 (“we recover DeltaNet”) is not supported.\nPlease provide full derivations or references to a specific place in the paper(s) where the statements are proved. In the same vein, could you describe how we get update rule (equation 5) out of online loss (equation 4)?\n\n3. Among several recent algorithms specifically designed with test-time training/ online optimization in mind, the paper discusses in depth only (Gated) DeltaNet variants, both theoretically (lines 1577-1603) and empirically. It would be great to explain in more detail MesaNet’s differences and similarities with such algos as Test-Time Training (https://openreview.net/forum?id=wXfuOj9C7L, not mentioned in the paper), LongHorn (https://openreview.net/forum?id=8jOqCcLzeO), Titans (https://openreview.net/forum?id=8GjSf9Rh7Z), Atlas (https://arxiv.org/abs/2505.23735), and Miras (https://arxiv.org/pdf/2504.13173). Lines 145-148 briefly touch on some differences, but further corroboration is required. E.g., could you provide formulas demonstrating why Atlas “corresponds exactly to a sliding-window variant of the Mesa layer”? Additionally, if you have the bandwidth and capacity, would it be possible to pre-train one of these related models and add it to comparisons (although I doubt that it would be significantly different in performance)? \n\n4. Also, It would be informative to include a table with comparison of online-learning objectives for different models, similar to the Table 4 in LongHorn paper (https://openreview.net/forum?id=8jOqCcLzeO).\n\n5. Lines 1452-1453 – The remark about Mamba 2 being non-gated is incorrect. Mamba 2 is gated in the same sense as GLA ($S_t = G_t * S_{t-1}$), but it calls the same mechanism “decay” and uses notation $A$ instead of $G$.\n\n6. There are no code listings for the new layer in the paper, nor accompanying code archive in the submission. The paper would benefit from an additional appendix with a high-level implementation of the core algo in PyTorch or Jax.\n\n7. I may have overlooked that but it seems the $O(T)$ training and $O(1)$ inference complexities aren't explicitly stated in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iQOuWYj2O4", "reviewer": "Reviewer_oZt9", "summary": "The paper proposes MesaNet, a sequence layer that replaces recurrent fast-weight updates with a per-token optimal linear regression solved by conjugate gradients (CG). The layer maintains two matrix states\n$G_t = \\\\sum_i \\\\zeta_{ti}, v_i k_i^\\\\top$ and $H_t = \\\\sum_i \\\\zeta_{ti}, k_i k_i^\\\\top$ (equivalently via gated recurrences $G_t=\\\\gamma_t G_{t-1}+\\\\beta_t v_t k_t^\\\\top, H_t=\\\\gamma_t H_{t-1}+\\\\beta_t k_t k_t^\\\\top$), and outputs $ o_t = G_t (H_t+\\\\Lambda)^{-1} q_t $ computed with a chunkwise-parallel CG solve implemented with GEMMs. Models at 140M/440M/1B on SlimPajama match or surpass strong linear-RNN baselines and are competitive with a transformer in average perplexity, with notably stronger early-sequence performance and solid results on some global-reasoning and few-shot tasks.", "questions": "1. Accuracy–efficiency summary (1B). Using the existing 1B checkpoint and kernels, please provide a compact table for a representative long sequence reporting validation perplexity, mean per-token latency, and peak memory for a few $k$ values (e.g., $0$ and two others) or dynamic stopping with $\\\\epsilon$ (report average CG steps). Also state hardware, precision, batch size, sequence length, and measurement procedure (warmup, repeats). Throughput optional.\n2. Sensitivity/stability ablation. How sensitive are convergence and perplexity to the softplus scale on $\\\\Lambda$ and to the diagonal preconditioner or $x_0$ initializer? Please report CG steps to a fixed tolerance and validation perplexity, and provide recommended defaults.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Clear objective with a closed-form fast-weight map and practical CG solve; principled dynamic test-time compute via stopping criteria.\n* Strong empirical controls (same backbone/tokenizer/data order) enabling clean comparisons.\n* Useful diagnostics: early-sequence NLL gains, length extrapolation, grouped tasks; dynamic stopping achieves near-parity to larger fixed $k$ at substantially fewer steps on average.", "weaknesses": "* Missing a concise accuracy–efficiency summary at inference (for one representative long-sequence setting), including per-token latency and peak memory under fixed batch, hardware, and precision.\n* Stability/conditioning analysis is qualitative; small ablations on the softplus scale for $\\\\Lambda$ and the diagonal preconditioner or $x_0$ initializer are needed to establish sensitivity and recommend defaults.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SgE2TRD9Go", "reviewer": "Reviewer_JXGc", "summary": "The paper introduces a numerically stable, chunkwise-parallel Mesa layer. At each timestep it uses a conjugate-gradient (CG) solver to compute optimal fast weights, yielding a recurrent alternative to attention with dynamically adjustable test-time compute.", "questions": "Please see the concerns detailed in the Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is well-motivated. It presents a numerically stable Mesa layer that solves $q^*_t=(H_t + \\Lambda)^{-1} q_t$ per timestep via a CG solver with gated state recurrences, yielding a well-posed recurrent formulation.\n2. Results are competitive with strong RNN-style baselines and broadly comparable to a Transformer at similar scale, and the accompanying analysis is generally sound.", "weaknesses": "1. The paper shows per-layer timings and training throughput (Fig. 2), but there’s no single, end-to-end table that reports latency (ms/token), tokens/s, and GPU memory alongside quality across CG step counts or the stopping policy, and across multiple context lengths on the same hardware.\n2. It’s unclear when to prefer Mesa over MHA. Although the paper acknowledges that compute grows with CG steps and may exceed MHA past some step/key sizes, it lacks concrete recipes or tables mapping k/tolerance choices to context length, latency targets, and hardware, making deployment decisions difficult.\n3. On mean-so-far long-context perplexity, a strong SWA-1024 baseline is often competitive or better, which tempers any blanket \"long-range advantage\" message. Please expand the long-context suite and report where Mesa wins/loses with matched budgets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OfXhSUUzfa", "reviewer": "Reviewer_mKy1", "summary": "This paper introduces Mesanet, which is a linear transformer based on the Mesa optimal regression update rule. The model is inspired by test-time training and the fact that the state update of linear transformers can be seen as an online learning objective. The model is sound, and the chunkwise parallel training makes the model efficient to train; however, the training is more time-consuming than other linear transformers such as GLA and Deltanet due to the solver update. The empirical results are extensive, and the model is evaluated on many different downstream language modeling tasks and length generalization.", "questions": "My most concerns are mentioned above and I mainly have 3 small questions:\n\n**1)** What is the effect of convoulution on q,k,v in Mesanet as it is a cruical component for Deltanet and Gated Deltanet and Mamba2. Also, since it has conections to the TTT paradigm as stated in: https://kexue.fm/archives/11320\n\n**2)** Why does $\\Lambda_t = \\frac{(1-\\gamma_t)}{\\beta_t}$ structured as so?\n\n**3)** How are forget gates $\\beta_t$ and $\\gamma_t$ defined and extracted? Specifically, what nonlinearities and feature mappings are used for them? Are they similar to those in Mamba-2, or do they use a temperatured sigmoid as in GLA or GSA?\n\nLastly, I am happy to increase my score if above presentation issues and clarifications regarding the paper are resolved.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper have several strengths:\n\n- **Model design**: The model design of Mesanet and the connection to test time training is elegant.\n\n- **Experimental setup:** The experimental setup for pre-training in language modeling tasks are sound and followed by the stablished setups such as [1,2].\n\n- **Fairness of comparison:** The paper tunes the learning rate for all different models which isa significant plus point as studies such as Gated Deltanet [3] specifically use the same learning rate for all models, which can indeed be not fair comparison between baselines.\n\n- **Chunk-wise parallel support for training:** In general the training paradigm and chunk wise form is designed nicely and helps parallel training.\n\n\n\n\n\n\n\n\n\n\n\n------\n\n###  References\n\n[1] Gated Linear Attention Transformers with Hardware-Efficient Training: Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, Yoon Kim\n\n[2] Parallelizing Linear Transformers with the Delta Rule over Sequence Length: Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, Yoon Kim\n\n[3] Gated Delta Networks: Improving Mamba2 with Delta Rule: Songlin Yang, Jan Kautz, Ali Hatamizadeh", "weaknesses": "The main weakness of the paper lies in its presentation, which significantly reduces clarity regarding the approach and design. These issues can be summarized as follows:\n\n**1)** **Mesanet's online learning objective** is significantly under-explained, starting from Equation 4:\n\n$$\n\\mathcal{L}(\\Phi) = \\frac{1}{2} \\sum^t_{\\tau=1} |v_\\tau - \\Phi k_\\tau|^2 + \\frac{\\text{Tr} (\\Phi^\\top \\Lambda \\Phi)}{2}\n$$\n\nBy taking the gradient and setting it to zero, we arrive at:\n\n$$\n\\sum^t_{\\tau=1} (v_\\tau - \\Phi k_\\tau)k^\\top_\\tau + \\Lambda \\Phi = 0, \\quad\n\\sum^t_{\\tau=1} v_\\tau k^\\top_\\tau - \\Phi \\sum^t_{\\tau=1} k_\\tau k^\\top_\\tau + \\Lambda \\Phi = 0 \\quad \\text{(Eq. 1)}\n$$\n\nThe above result corresponds to the optimal $\\Phi$. Since in an autoregressive regime the summations above can be expressed recursively, one can write the following recurrences:\n\n$$\nG_\\tau = G_{\\tau-1} + v_\\tau k^\\top_\\tau, \\quad\nH_\\tau = H_{\\tau-1} + k_\\tau k^\\top_\\tau\n$$\n\nFrom here, one can substitute the states $G_t$ and $H_t$ into Equation (1) to obtain the optimal Mesa update rule:\n$\\Phi = G_t(H_t + \\Lambda_t)^{-1}$.\n\nMoreover, by including forget gates in the recurrences above, the online learning objective will also change, which is inconsistent with the paper’s notation. I suggest reformulating the online learning loss (Equation 4 in the paper) to explicitly include these forget gates, as also described in *Gated DeltaNet’s Table 1* [1]. Moreover, is $\\Lambda_t$ time-dependet or not since in equation 4 of paper is suggested as not but exactly on paragraph bellow it is depending on forget gates.\n\n----\n\n**2) The full recurrence of MesaNet is not shown and not compared with other baselines.**\nI recommend moving the recurrence equation of MesaNet (currently on page 21) into the main body of the paper, along with Table 2 for clarity and comparison with other methods.\n\n---\n\n**3) Motivation behind the Mesa layer.**\nSince the main motivation is the connection to test-time training (TTT), I wonder why one should use MesaNet instead of DeltaNet or Gated DeltaNet. My main question is: given that DeltaNet and Gated DeltaNet leverage TTT to improve recall abilities, and indeed show significantly better recall, does MesaNet outperform them in this aspect? MesaNet introduces additional complexity in both architecture and training. As Table 14 suggests, if the motivation for this design is to enhance recall ability, this should be clearly stated and justified in the paper.\n\n---\n\n**4) Hawk-Mesa underperforms significantly compared to many baselines and even MesaNet itself in recall tasks (Table 14).**\nIf the main motivation behind building a model based on the Mesa layer is to introduce an optimal way to structure memory (as suggested by the TTT framework), why does Hawk-Mesa perform substantially worse than MesaNet in Table 14? Why should one construct such a model if it does not improve recall? I ask this because I believe recall ability is one of the primary motivations behind the design of MesaNet and the TTT framework in general.\n\n---\n\n**5) Tables and plots are not clearly visible.**\nMany numerical results presented as bar plots are difficult to read. I highly recommend replacing them with tables, particularly Figure 5, to improve clarity.\n\n---\n\n**6) Missing baselines: Atlas and Titans.**\nI believe the baselines **Atlas** and **Titans** are missing from the paper’s comparisons. These models address a very similar problem and propose closely related solutions, especially Titans, which should be included for a fair evaluation.\n\n\n---\n\n### References\n\n[1] Gated Delta Networks: Improving Mamba2 with Delta Rule: Songlin Yang, Jan Kautz, Ali Hatamizadeh", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)