Weight-Space Linear Recurrent Neural Networks
Authors: Roussel Desmond Nzoyem
Nawid Keshtmand
Enrique Crespo-Fernandez
Idriss TSAYEM
Raul Santos-Rodriguez
David A.W. Barton
Tom Deakin
Venue: ICLR 2026 Poster (2026)
Keywords: physics-informed machine learning
weight-space learning
meta-learning
deep sequence modeling
linear recurrence
test-time training
Primary area: learning on time series and dynamical systems
TL;DR: We introduce a simple yet powerful model that unifies weight-space learning with linear recurrence to redefine sequence modeling.

Abstract: We introduce WARP (**W**eight-space **A**daptive **R**ecurrent **P**rediction), a simple yet powerful model that unifies weight-space learning with linear recurrence to redefine sequence modeling. Unlike conventional recurrent neural networks (RNNs) which collapse temporal dynamics into fixed-dimensional hidden states, WARP explicitly parametrizes its hidden state as the weights and biases of a distinct auxiliary neural network, and uses input differences to drive its recurrence. This brain-inspired formulation enables efficient gradient-free adaptation of the auxiliary network at test-time, in-context learning abilities, and seamless integration of domain-specific physical priors. Empirical validation shows that WARP matches or surpasses state-of-the-art baselines on diverse classification tasks, featuring in the top three in 4 out of 6 real-world challenging datasets. Furthermore, extensive experiments across sequential image completion, multivariate time series forecasting, and dynamical system reconstruction demonstrate its expressiveness and generalization capabilities. Remarkably, a physics-informed variant of our model outperforms the next best model by more than 10x. Ablation studies confirm the architectural necessity of key components, solidifying weight-space linear RNNs as a transformative paradigm for adaptive machine intelligence.

Published as a conference paper at ICLR 2026
WEIGHT-SPACE LINEAR RECURRENT NEURAL NET- WORKS
Roussel Desmond Nzoyem University of Bristol Bristol, UK (rd. nodnnzoyemmuuuguin,
Nawid Keshtmand University of Bristol
Enrique Crespo Fernández University of Bristol Bristol, UK
Bristol, UK
enri enrique.crespoc cew.eueeeeoooffeeaeeee....
.cc.uk
Idriss Tsayem
Raul Santos-Rodrighs University of Bristol Bristol, UK
David A.W. Barton
University of Maryland Baltimore, USA
University of Bristol Bristol, UK
Tom Deakin
University of Bristol Bristol, UK
ABSTRACT
We introduce WARP (Weight-space Adaptive Recurrent Prediction), a simple yet powerful model that unifies weight-space learning with linear recurrence to redefine sequence modeling. Unlike conventional recurrent neural networks (RNNs) which collapse temporal dynamics into exed-dimensional hidden states, WARP explicitly parametrizes its hidden state as the weights and biases distinct auxiliary neural network, and uses input differences drive its recurrence. This iriin-inspired formulation enables efficient gradient-free adaptation of the auxiliary network at test-time, in-context learning abilities, and seamless integration of domain-specific physical priors. Empirical validation shows that WARP matches or surpasses state- of-the-art baselines on diverse classification tasks, featuring in the top three 4 out of real-world challenging datasets. Furthermore, extensive experiments across sequential image completion, multivariate time series forecasting, and dynamical system reconstruction demonstrate its expressiveness and generalisation capabili- ties. Remarkably, physics-infoo thnsics-informed variant of our model outperforms the next best model by more than 10x. Ablation studies confirm the architectural necessity key components, solidifying weight-space linear RNNs as a transformative paradigm for adaptive machine intelligence.
INTRODUCTION
Deep sequence models, which continuously drive progress in machine learning, are limited in their ability operate outside their training distribution [2; 40; 33]. For instance, subsets of Neural ODE parameters [18] necessitate adaptation via gradient descent to maintain performance on out- of-distribution (OoD) sequences [49; 75]. While effective, their explicit gradient calculation cost has recently catalysed research into gradient-fre test-time adaptation methods [86; 69; 43]. This surge of interest is embodied by in-context learning [60; 86), which has recently been shown to perform test-time adaptation since during inference, it implicitly minimises loss objective using gradient information [94; 102]. Another reason for the poor generalisation of discrete deep sequence models is the inability inject domain-specifie priors in their forward pass. In effort to preserve all desirable traits while unleashing breadth of possibilities, we combine two of the most powerful emerging deep learning paradigms: weight- space learning and linear recurrence.
Weight-space learning the paradigm that treats the weights and biases of a function approximator data points for another learning system [85] offers unprecedented potential for extracting prop-

---

Published as a conference paper at ICLR 2026
State Transition Decoding in Recurrent Neural Networks
Standard RNNS
Linear RNNs
Weight-Space Linear RNNs
h; = fo 1,, Y: gv (h+)
h: = Ah,- + Bx:
02 = A0,-1 1 + BB(x - x1-1)
Y: Ch2-1
Yt MLPO,
Figure 1: Background and conceptual comparison between RNN architectures. Standard RNNs
(e.g. [44; 19]) feature non-linear transition function fo unlike their linear counterparts (e.g. [37; 77]). Our proposed weight-space linear RNNs view their hidden state - denoted as 0g - as the parameters of a family of functions. As observed in the bottom-right corner, represents, in the general case, the flattened weights of an MLP at time step t. Its input is a (concatenation of)
coordinate system(s) to maximally make use of the canonical ordering of the sequence.
erties of a trained model solely from its "weights". Applications span from predicting generalisation
error [91] and recovering training data [28] to classifying and editing implicit neural representations
[24]. With the proliferation of model repositories such HuggingFace and CivitAI, developing methods that effectively learn directly from weights has become increasingly vital [48]. To date, the literature has predominantly focused utilizing these weights inputs and outputs to higher-level models, leaving their potential as intermediate representations (e.g., latent vectors, hidden states) in end-to-end training systems unexplored.
Concurrently, linear Recurrent Neural Networks (RNNs) have seen a notable resurgence, largely due to their hardware efficiency and the resulting of training [22]. Linearity enables advanced sequence parallelisation techniques
[87; 71; 100] and has delivered exceptional performance on long-sequence tasks [37; 77]. However, recent findings raise concerns about the information capacity of their compressed state representations [67]. Moreover, a substantial body of work has shown that linear Transformers [50] and Space Models (SSMs) [37] particular instantiation of linear RNNs are fundamentally less expressive than the standard non-lineat RNNs depicted in Fig. [9; 26; 66). Taken together, these results strongly suggest that con-linearities are crucial for the expressivity of deep sequence models. They invite the reintroduction of non- non-lineasisies into sequence models, while maintaining the hardware- friendly nature linear RNNs.
The preceding analyses motivate our examination of weight-space linear RNNs. To harness the strengths of its constituting paradigms, we formulate several research questions: Can the weights of an auxiliary function approximator serve as high-resolution hidden states for linear RNNs? Can that auxiliary function be effectively adapted during inference without requiring gradient computation? Are the non-linearities in the auxiliary function approximator sufficient significantly enhance the expressive power of such models?
We answer these questions in the affirmative by proposing Weight-space Adaptive Recurrent Prediction (WARP) models as powerful expressions weight-space línear RNNs, which we il- lustrate in Fig. 1. Specifically, our original contributions can be summarised follows:
(1) We formulate general framework for sequence modelling in weight-space, blending linear recurrence with non-linear decoding. Rather than relying on direct inputs, we draw inspiration from the human brain and compute signal differences to drive such recurrences. To the best of our knowledge, our framework is the first of its kind to treat weight-space features intermediate hidden state representations in a recurrence.
(2) To train weight-space linear RNNs, we introduce two parallelisable algorithms: a convolu- tional mode and an efficient recurrent mode (with and without support for auto-regression) well-suited for noisy sequences. These algorithms unlock three practical use cases: (i) gradient- -fre adaptation, i.c., the ability to update critical components responsible for the model's adaptation without requiring gradients; (ii) in-context learning, i.e., the capacity to recognise onput-output patterns in the sequence's 's context and adapt model behavior without
Following the convention from [106]. we refer to the learnable parameters of the processed function approximator as 'weights' (or 'weight space' to indicate the space they helong to) and those of the higher-level learning system (c.g., the neural functional) as simply 'parameters'.
2

---

Published as a conference paper at ICLR 2026
WARP
y
Model
D.
Figure 2: (Left) General sequence modelling setting, In the forecasting scenario, for instance, a context of length L informs the prediction of future states. (Right) WARP's unfolded recurrence. The initial hypernetwork & and transition matrices (A, B) - highlighted in orange are learnable parameters, fitted via conventional gradient descent.
finetuming parameters, and (iit) physics-informed modelling, i.c., the ability to incorporate domain-specific continuous physical priors in the discrete linear recurrence. This final core application is evidenced in our WARP-Phys model, which achieves an order of magnitude lower error over WARP on wide set of synthetic dynamical system reconstruction datasets.
(3) We identify extensive suite of real-world benchmarks to evaluate various capabilities of RNNs regarding classification, reconstruction, adaptation, and memory retention. Empirical results demonstrate how WARP consistently matches outperforms traditional RNNs, SSMs, and Transformer architectures. Remarkably, we push the state-of-the-ard by featuring in the top three in 4 out of 6 multivariate time series classification datasets necessitating the understanding of both short- and extremely long-range dependencies.
2 WEIGHT-SPACE ADAPTIVE RECURRENT PREDICTION (WARP)
This section presents the core ideas underpinning weight-space' linear recurrence, novel framework for deep sequence modelling that operates by directly modulating, in response to sequential input differences, the weights of function approximator [6]. Out of simplicity and consistency with the related literature in Appendix A, we focus in the remainder of this paper on the WARP model, which modulates feed-forward neural network [65]. We begin by establishing the problem setting, followed by WARP's architectural and training details.
2.1 PROBLEM SETTING
Our framework addresses the general sequence modelling problem, wherein computational model must establish mapping from an input x, E IR to a corresponding output yr E TR, with t [O...., T - 1} denoting the time step index. The integer T represents the training sequence length, which remains invariant across all sequences within a training batch. We assume that all input sequences are sampled at the same uniform intervals. Ignoring the batch dimension for simplicity, models establish mapping from T'x to 7 D, (see Fig. 2).
In the regression setting of time series forecasting, have predict future tokens conditioned on a preceding sequence of tokens, designated as the "context" X {xx (e........... where L denotes the context length. Critically, we desire the ability to perform ruto-regressive rollouts during inference. For classification tasks, only the final token yr-1 is treated as a softmax-activvan activated logit to assign a label to the sequence.
Xt+11
objective is to
Whhile high-level model parumeters must frozen, in-context learning may still require gradients finetune some weights.
We that 7 may different for testing sequences. D. is the dimensionality of the subscripted quantity.

---

Published as a conference paper at ICLR 2026
2.2 ARCHITECTURE
While traditional recurrent networks update obscure hidden states h2 VI {1, {1...., 1}. weight- space linear RNNs such WARP update the weights and biases of auxiliary "root" neural network 0r effectively learning a dynamics model in weight-space (see Figs. and 2). Specifically, we define the recurrence relation and the subsequent decoding:
D₂ = =A0,1 + BAxt,
and
YE = MLPp,
where the hidden state 8g R represents the flattened weights of the root neural network at time step 1, and Ax, = x x,- is the input difference. A E R DoDD is the state transition "weights- to-weights" matrix, and B Do the input transition "data-to-weights" matrix. To compute the output yt, the vector 0e is unflattened and combined with non-linear static activation functions to reconstitute the MLP root network. This decoding function is applied to T, a coordinate system (or concatenation thereof) that suitably informs the model of the canonical ordering of the sequences hand. Powerful examples of coordinate systems (see Appendix B.2.1) include normalised pixel locations (for images viewed as sequences), normalised training time T = t/(T - 1), or the general positional encoding to facilitate generalisation beyond T [92].
Compared other RNNs, 0, plays both the roles of the hidden state and the parameters of the decoder, effectively decoding itself. Such self-decoding significantly saves on learnable parameter count.
Importantly, all hidden states can be precomputed efficiently thanks to the linear recurrence in Eq. (1), using for instance, the parallel "scan" operator [87]. Once materialised, the 0, can be reconstituted and self-decoded independently. This allows model to combine the efficiency of linear recurrence with the expressivity enabled by incorporating non- oon-lineasities.
Another key aspect of our formulation is the of input differences Ax, rather than direct inputs Xt, which is a choice Kidger et al. [54] theoretically motivated for continuous-tin timm RNNs. When inputs change slowly remain constant, the weight updates become proportionally smaller, and vice-versa. WARP essentially learns to convert input differences into neural network updates, a critical sele-supeo ability for continual learning and test-time adaptation [10].
Architecture of the root network. The root network 0, is implemented as a fixed-width mul- tilayer perceptron (MLP) [65] with a D,ddimensionall input, and output dimension either D, or 2 x D, depending on whether uncertainty measures are required in the pipeline. When mod- elling uncertainty, the network predicts in addition to mean i4 RDy, quantity o; E RD, on which positivity-enforeig function is applied to obtain an element wise standard deviation ã, = max (softpluuus)(( min)) where min is fixed positive problem-dependent lower bound for numerical stability.
Initialisatinn of learnable parameters. Similar prior work [58], the state transition matrix A is initialised as the identity operator ID, D.. This emulates gradient descent and residual connections in ResNets [42], thereby facilitating gradient flow during backpropagation through time [98]. We find that initializing the input transition matrix B as the zero matrix Op, x D. is useful to ensure that the sequence of weights O, does not diverge early on in the training. This strategic initialisation also imposes a critical constraint wherein the initial hidden state 0g must encode semantically rich information applicable to the entire sequence.
The initial weights On are determined by processing the first observation: 0n ((xo), where the "initial network" Φ a hypernetwork [39] defined as learnable MLP with gradually increasing width (see Fig. 2). On sequence modelling problems with fixed or mildly-varying initial conditions, we sidestep & and directly learn 6g which is initialised with classical techniques such as Glorot [30] or He [41] (and subsequently flattened into 1D vector).
2.3 TRAINING & INFERENCE
Analogous to SSMs [37] and subsequent linear recurrence architectures [77; 71J, WARP supports dual training modes: convolutional and recurrent. The former is accomplished through systematic unrolling of the linear recurrence formulated in Eq. (1), enabling the derivation a convolution kernel K such that for K * Axp.r. Comprehensive notations, algorithms, and rigorous mathematical
4

---

Published as a conference paper at ICLR 2026
derivations are elaborated in Appendix B.2.2. In recurrent mode, we distinguish the uuto-regressive (AR) and the relatively nemory-expenstie non-AR settings. The non-AR setting never sees its own predictions, making it ideal for classification tasks wherein (.)-) only generates logits.
The recurrent AR setting is particularly advantagcous for noisy forecasting tasks that necessitate accurate modelling of the sequential data distribution p(y: (yy)) To mitigate exposure bias [84], implement teacher forcing with scheduled sampling [11], wherein the model incorporates uncertain- ties by sampling yt .N W o using the reparametrisation trick [55]." Selection between ground truth y, and predicted yr follows a Bernoulli distribution with probability Pforsing which we define as training hyperparameter. That said, we consistently use y 1 in the input difference seen in Eq. (1).
During inference on regression problems, the model operates fully auto-regressively. i.c., Ptorcing = 1 within the context window, and Pforcing 0 in the forecast window, regardless of the training mode.
Although other loss functions can be considered, our WARP models are trained by minimizing either the sqean-s error (MSE) for deterministic predictions, or the simplified negative log-likelihood (NLL) for probabilistic predictions:
LMSF Σ
T-1
yy 8x112:
CNL..
lly -vell 2
log o.
As for classification problems, we use the categorical cross-entropy LOce & Ee y(e) log ( where y is the one-hot encoding of the true label class, and C' is the number of classes.
Our learning pseudocodes are detailed in Algorithms and 2 of Appendix B, outlining the strong connection to the fast weights and test-rime training literatures [83; 7; 101]. At each training step, the clow-changing RNN parameters A, B and ó (or 0g) are updated once using gradient descent to minimise one of the loss objectives above. The fast-changing weights 0,, however, updated T times using Eq. (1), i.e., not using gradient descent. This distinction is central to our model's gradient-free adaptation process.
EXPERIMENTS
We evaluate WARP on real-world multivariate time series datasets, 2D images, and physical systems. Our experiments elucidate empirical questions regarding forecasting, classification, and dynamical system reconstruction and generalisation. Additional experiments allow us to demonstrate WARP's in-context learning abilities. Theoretical results are presented in Appendix B.2, and experimental details can be found in Appendix D.
3.1 IMAGE COMPLETION, ENERGY PREDICTION & TRAFFIC FORECASTING
In the first part of our experiments, we focus on forecasting applied first raster-scanned pixel-by- pixel image completion, followed by real-world electricity and traffic flow.
Table 1:
Lowest test MSE (L) and BPD (L) achieved on MNIST (Top) and CelebA (Bottom). The best along the columns is reported in bold, while the second-best underlined.
Image Completion Image completion is cast as a prediction of pixel intensities. We focus on two datasets: MNIST handwritten digits [59], and the
MNIST
L
100
300
L.
G00
MSE
BPD
MS5
BPD
MSE
HPD
GRU
0.074
0.623
0.044
0.573
0.015
0.885
celebrity face attributes CelebA [63] addiiionna image datasets are considered in Appendix E. 2D images are flattened into ID sequences with length
LSTM
0.074
0.652
0.077
0.611
0.027
0.539
ConcNNP
0.330
0.01
0.019
0.520
WARP
0.071
0.615
0.042
0.516
0.014
0,416
T = 784 for MNIST and T 1024 for CelebA.
CelebA
L 100
L
300
L 600
MSE
BPD
MSB
BPD
MSE
BPD
Following [8]], the completion task is condi- tioned on contexts of variable length L. We com- pare WARP against long-establisher baselines
GRU
0.063
24.14
0.488
60.30
IU027
71.51
LSTM
0.051
3869
0.053
7.276
0.012
7,909
0.100
0.132
WARP
0.852
0.040
-0.03
0U027
-0.122
'Athhough equal to AR computational complexity, the recurrent non-AR setting requires more memory
hecause, like the convolutional mode, it materialises all high-dimensional hidden states 6.
We remark that this sampling required during inference smooth sequences like dynamical systems.
5

---

Published as a conference paper at ICLR 2026
like GRU [19] and LSTM [44]; against state-of-the-art (SOTA) SSMs like S4 [37]; and against the ConvCNP convolution- based meta-learning baseline [31] specifically designed for image comple- tion. All models are trained with the NLL loss in recurrent AR mode to ensure fair comparison, and feature nearly the same number of learnable parameters: approximately 1 1.68M for MNIST, and 2M for CelebA. Results in Table 1 demonstrate the generative performance of WARP measured by the MSE and the uncertainly-wware bits-per-dimin per-dimension (BPD) metrics, We focus on the top performing models across three runs, with corresponding qualitative comparisons best captured by the BPD in Appendix F. For instance, Fig. 3(a) shows that at small parameter count, WARP is the only model accurately generate digits without substantial artefacts.
m11
1
h1
h2
6880
3.66
0.90
4.08
7.33
9
3.42
1.27
3.07
7.02
2
9
2.06
0.86
3.56
6.44
(a) MNIST
(b) ETT
Figure 3: (a) Comparison of a GRU [19], LSTM [44], S4 [81], and WARP on the MNIST image completion task with I, 300 initial pixels. All models are roughly at the same size of 1.7M parameters, with architectures described in Appendix C.2. The leftmost column represents target images with context (in white) and ground truths (in green). Predicted forecasts are drawn in red. (b) Heatmap of test MSEs (L) on the ETT task, with best results enclosed and second-bes underlined.
Energy Prediction We evaluate WARP's performance long-range energy forecasting tasks with the Electricity Transformer Temperature (ETT) dataset [108]. Following established methodological protocols [72], utilise the open-source TSLib to obtain preprocessed data splits which we further normalise using train set statistics (additional data processing details can be found in Appendix C).
The models are tasked with predicting 96 time steps based context of length L 96, with performance evaluated using the mean MSE across three runs. The results are shown in Fig. 3(b), where the best along the columns is reported in box while the second-best is underlined. It demon- strates WARP's superiority, achieving the best performance on all subsets except the ETTh subset, where it ranked second. These results are particularly noteworthy given WARP's straightforward design. Indeed, WARP offers an elegant balance between architectural simplicity and predictive power, Additional results on the ETT dataset are presented in Appendix E.
Table 2: Performance on PEMS08 [88]. SOTA baselines leverage spa- tial information, as reported in [62].
Traffic Flow Forecasting We conduct extensive experiments on the PEMS08 real-world traffic network [88]. The network consists of 170 nodes, from which features collected at 5-minute intervals over two months. The standard task is to predict the traffic flow for the next hour (12 steps) given the historical data from the previous hour (12 steps). Given its chunk wise forecasting
MODEL
MAE
RMSE
GMAN [103]
14.57
24.71
which significantly differs from the setting in Fig. 2 - we employ the non-AR mode to train and test WARP. Additionally, we preprocess the input sequence with non-causal convolution, as detailed in Appendix D.
D'STGNN [104]
14.35
24.18
STIDGEN
13.45
23.28
10.10
As demonstrated in Table 2, our model achieves a MAE of 6.59 and RMSE of 10.10. These results represent significant improvement over the current state-of-the of-the--rt on the PEMS08 benchmark [62], reducing MAE by over 50% compared to the puslisubld model. that our model achieves this performance without using the inherent graph structure, outperforming complex Attention and Graph Neural Network (GNN) architectures that are specifically designed to leverage this spatial information.
particularly noteworthy
httpp:://gi thub. cww/thuml / me- Serieerresse Series-l.ibe harres-llibbrry.co

---

Published as a conference paper at ICLR 2026
Table 3: Average test MSE (L) and MAE (4) for dynamical system reconstruction, Best results are reported in bold, while the second-best are underlined. All are reported with x x10-² scaling, except for SINE" with x10- SINE indicates that metrics are computed upon training on its "Small" data split. WARP-Phy- indicates the variant of WARP with physical constraints in the root network.
MSD
MSD-Zero
LV
SINE
MSE
MAE
MAB
MSE
GRU
1.43 = 0.09
5.05
0.55: 0.75
3.27+ 0.13
5,834 0.37
13.1 0.42
4,90s 0.45
179 9.23
LSTM
1.46 0.14
5.43
0.57
3,46:00.8
6,18
13.6 061
9.48 0.12
248 3.45
Transformer
0.34 n.12
2.25 0:2
0.48 0.34
2,90 0.12
11.27 0.0
18.62 0:49
1728 108
2204g 20.0
WARP
0.94 = 0.00
3.04 =: 0.11
0.32= 0.02
2.59 0.07
4.72 a: 0.25
10.9: 0.45
2.77+ 0.09
125 8:46
WARP-Phys
0.03:a044
0.66: 0.12
0.04: 0.01
0.75 6.83
X
0.62: 0.01
6.47: 0.51
3.2 DYNAMICAL SYSTEM RECONSTRUCTION
As our final forecasting benchmark, we evaluate WARP's capabilities on dynamical system recon- struction (DSR) [32]. The experiments presented in this section highlight the challenge of Ool) generalisation to physical parameters, research area that has recently experienced significant surge in interest [76; 15].
We establish four DSR benchmark datasets: (1) Mass Spring Damper (MSD) characterises challeng- ing damped oscillatory dynamics through physical parameters (m, k. c), with trajectories of length T 256, of which L 100 states serve as context; (2) MSD-Zero is version of MSD which varies, in addition to the significant relative scales and wide ranges of (m, k, c). the initial condition Xp: (3) Lotka- Volterra (I.V is parametrised by coefficients (a. 3, 8); (4) SINE comprises sine curves - sin (2r7 + yp) with varying phases yo (we set 7 = 16 and T. = 1, resulting in an initial value problem). Each test set incorporates out-of-distribution parameters, except for SINE, which primarily tests model performance under sample size constraints. Comprehensive data generation protocols for all four datasets are detailed in Appendix C. We benchmark against two established RNNs and the Time Series Transformer (TST) from HuggingFace [46] specialised for forecasting. We evaluate WARP in a black-box setting - which embeds no explicit physical knowledge in the root network followed by the more interpretable grey-box.
Black-Box Setting Our results, presented in Table 3, highlight how weight-space linear RNNs consistently outperform all baseline models across problem domains. Importantly, the standard WARP configuration, which uses a root black-box MLP, ranks within the top two in three out of four problem settings. We observe that TST denoted simply as Transformer - exhibits significant performance degradation on the SINE* dataset (which comprises only 10 sequences), corroborating the documented limitation of Transformer models to overfit in data-scarce regimes due to their inherently high parameter complexity [27].
Injecting Physical Bias (Grey-Box) A principal advantage of WARP is its capacity to incorporate dopain-specific knowledge into the root network, exemplified on the SINE* experiment by embedding the explicit mathematical formulation ~ sin(2x7 + ()) in its forward pass, where & is predicted by a MLP. The resulting architecture, WARP-Phys. demonstrates substantial performance improvements relative to WARP (more than one order of magnitude on MSD). Notably, the incorporation of such a powerful physical prior on SINE* underscores the value of an expressive but data-efficient initial network é whose task it is to capture a representation of yp. Indeed, all models, including WARP and WARP-Phys, perform poorly on the extreme "Tiny" data split (not reported in Table 3). We provide additional details ablations in Appendices and E.7.
LLL.
Repeat-Copy of Physical Systems We evaluate our model's pattern mem- orisation capabilities on the Kotka-Volteraa (LV) dataset, which constitutes a continuous analogue of the established repeat-copy benchmark [89; 77]. To gener- ate the output shown in red in Fig. 4, we triplicate a concise segment of the input, separating the repetitions by 10-token long sequence of -1s. In this challenging problem, WARP demonstrates superior performance relative to all basclines, with the GRU achieving the second-highest performance metrics (see Table 3). These
Figure 4: Sample LV input/output.
7

---

Published as a conference paper at ICLR 2026
Table 4: Test-set accuracies (1) averaged over 5 training runs on the UEA classification datasets. Dataset names are abbreviated: EigenWorms. (Worms), SelfRegulationSCPI (SCP1), SelfRegulation- SCP2 (SCP2), EthanolConcentrraion (Ethanol), Heartbeat, Motorlmagery (Motor). Best results are reported in bold, and the second-best are underlined.
Worms 17,984
SCP1 896
SCP2 1,152
Ethanol 1,751
Heartbeat 405
Motor 3,000
Seq. length
# Classes
5
2
4
2
NRDE
77.2 = 7.1
76.7 = 5.6
48.1 = 11,4
31.4 = 4.5
73.9 = 2.6
54.0 + 7.8
NCDE
62.2 = 3.3
80.0 = 2.0
53.6 = 6.2
22.0 1.0
68.1 5.8
51.6 + 6.7
LRU
85.0 = 6.2
84.5 4.6
47.4440
29.8 = 2.8
78.1 7.6
51.9 8.6
S5
83.9 4.1
87.1 37.12211 2.1
5..1
25.6
3.11
53.0
Mamba
70.9 + 15.8
80.7 + 1.4
48.2
27.9 + 4.5
7
47.7 4.5
S6
85.0 + 1.2
82.8 + 2.7
49.9
26.4 + 6.4
7
51.3
+ 4.2
Log-NCDE
82.8 2.7
82.1 = 1.4
2.6
3.1
7.22
57.2
+ 5.6
LinOSS
95.0 4.4
87.8 2.6
58.2 18.2699
29.9
75.8
60.0 7.5
FACTS
86.7 3.0
73.3 - 2.8
70.3 1 8.8
28.2 3.3
70.3 8.8
49.8 1 3.8
Griffin
79.5 1 5.1
80.0 - 1.5
43.1 1 5.3
24.0 3.5
77.7 + 2.9
43,8 1: 3.3
WARP
70.93 1 2.7
83.53 1: 2.0
57.89 1 1.4
36.49 2.8
80.65 1.9
56.14 1 5.1
findings suggest that the high-resolution weight-space state representation exhibits enhanced pattern retention capabilities compared to conventional methodologies. We note that this particular evaluation protocol is incompatible with the WARP-Phys variant due the deliberate introduction of artificial discontinuities in the temporal sequences. Comprehensive analyses of additional results pertaining to this task, alongside other dynamical system reconstruction benchmarks, are presented in Appendix
3.3 MULTIVARIATE TIME SERIES CLASSIFICATION
We now consider the classification setting. We consider six datasets from the University East Anglia (UEA) multivariate time series classification archive (UEA-MTSCA) [8]. The six datasets are selected and preprocessed following the criteria of known difficulty for deep sequence models and data abundance, with sequence length ranging from 405 to almost 18k [96]. Our model is compared to both discrete and continuous recurrent baselines [70; 54; 77; 87; 96; 36; 80; 72; 23]. All models are trained, validated, and tested with the 70: 10:15:15 split. Additional details the dataset preprocessing, the baselines, and the positional encoding used for T are provided in Appendix C.
Table 4 presents test accuracy metrics across all six benchmark datasets for WARP (trained in recurrent non-AR mode) and competing methodologies as reported in [96]. Our analysis reveals that WARP demonstrates exceptional performance across the majority of tasks, establishing new state-of-the-ae accuracies on the SCP2 Ethanol and Heartbeat datasets, and competitive top three on four datasets. Despite not being designed with long-range dependencies in mind, WARP displays impressive potential on extremely long sequences such as SigenWorme and Motor, outperforming established models such as Mamba [36] and NCDE [54], FACTS [72], and Griffin [23]. This overcoming of the dell-documented vanishing and exploding gradient problems in recurrent architectures [112] is attributed to our careful initialisation scheme in Section 2.2, and the positional encoding scheme using sines and cosines with variable frequencies [92]. These empirical findings substantiate WARP's efficacy robust classification framework for diverse real-world time series applications.
3.4 IN-CONTEXT LEARNING WITH RANDOMLY GENERATED KEYS
key strength of WARP illustrated the classical in-context learning (ICL. setting of [102], where the objective is to find a shared vector w R D_-1 that linearly maps N randomly generated keys X; E D- to their corresponding values y; E IR this setup, WARP learns the weights of the root network that approximate this mapping. We adapt the task by transforming the input sequence into its cumulative sum along the temporal dimension, followed by the prediction of the value corresponding to the query Xy (see Fig. 5(a)). This preserves the underlying function while allowing the model to exploit key-value pairs dependencies. WARP is trained in its recurrent, non non-autoregressive mode with MSE loss over the entire ID output sequence of length T N + = 32. Critically, we do not
8

---

Published as a conference paper at ICLR 2026
employ hypernetwork in this task, Og is fitted directly (see Section 2.2). The results, shown in Figs. 5(b) and 5(c), highlight WARP's ability to perform sub-quadratic in-context learning and generalize effectively.
Sp
Ove. Sire
70
(a) Sequence transformations
(b) Values vs. keys
(c) Query values
Figure 5: Pipeline and results for in-context learning. (a) Cumulative sum transformation and subsequent processing of the imput matrix. (b) Linear mappings learned between scalar keys and values of the same sequences (D2 = 2). (c) Ground truth vs. query point predictions (D, = 8).
key advantage of this approach is that once the model has learned from the context, the final root network 0r-1: E X; + X,, + Σm U4 + Bg. which is equivalent to 0r X2 - dq. can be extracted. This allows it to process subsequent queries without needing re-evaluate the entire sequence from scratch. This method yields significant computational savings compared to other models capable of ICL [60].
DISCUSSION & CONCLUSION
4.1
CORE ADVANTAGES
WARP demonstrates outstanding results across multitude of data modalities, both in-distribution and outro--distribution, as evidenced by the extensive empirical results on time series forecasting and classification we have presented (see Tables to 4 and Figs. 3(a), 3(b) and 5). Additional results showcasing 93% classification accuracy sequential MNIST, along with ablation studies and further results on synthetic datasets are provided in Appendix E. Specifically, Appendix E.3 illustrates the excellent computational efficiency of our approach, as measured by wall-clock training time per epoch, peak GPU usage, and parameter counts.
By letting the data directly interact with the weights in Eq. (1), WARP showcases the appealing in-context learning ability to fine-tune an auxiliary network without gradients at test-time [10; 97]. Additionally, WARP is the latest scientific machine learning [21] technique that seamlessly integrates interpretable physical knowledge into its predictions, a feature standard RNNs have overlooked. This demonstratively allows for sample-efficient training and improved generalisation.
Finally, the WARP architecture, through its input difference, bears resemblance to synaptic plasticity in biological neural networks, specifically Spike Timing-Dependent Plasticity [16], wherein the weight of a synaptic connection is strengthened or weakened depending on the time difference between spikes from pre- and post-synaptic neurons. This neuromorphic quality enables more biologically plausible learning dynamics.
4.2 LIMITATIONS
Some design decisions that strengthen WARP equally face limitations that we outline as promising avenues for future work. First, the size of the matrix A limits scaling to huge root neural networks. Our experiments conducted
RTX 4080 GPU with 16GB memory could only support moderate Dg values, leaving open the question of how expressive WARP models can become if scaled further. Second, more theoretical research is needed supplement the current state of the weight- space learning literature. Our work remains mostly empirical, despite introducing theory-informer algorithms in Appendix B.2 and leveraging the underpinnings of NCDEs universal approximators generalizing RNNs continuous time settings [52]. Lastly. WARP still struggles to achieve SOTA classification performance extremely long sequences with intricate dependencies such images,

---

Published as a conference paper at ICLR 2026
and remains untested on language modalities. Future work would seek first principles to improve long-range performance while reducing the memory footprint of the matrix A, by exploring low- rank complex- complex-valued valued diagonal parametrisations [38], neuron permutation equivariance [105]. or block-diagonal decompositions.
4.3 CONCLUSION
In this work, we introduced Bright-Space linear RNNs, a novel family of sequence models that operates directly within the weight space of neural networks, offering a distinct paradigm from traditional recurrent architectures. We argue that the high- tigh-diimessioal weight space can be used for intermediate representations, resulting in memory. Our comprehensive experiments demonstrate that our models exhibit superior expressivity and generalisation capabilities, enabling a powerful form gradient-free adaptation response to sequential input differences, and showing exceptional abilities when integrating domain-specific knowledge from physical systems. Our framework draws intriguing parallels to neuromorphic learning principles, leading step further towards human-leve artificial intelligence.
RNN hidden states and high-capacity
BROADER IMPACT
While their benefits are evident from Section 4. malicious deployment of our self-adaptable models in scenarios they were not designed for could lead to serious adverse outcomes. Additionally, high- energy costs from high-dimensional weight-space computations could increase disparities our field, To limit the potential for such outcomes and to improve the democratisation of AI, our code is openly available at bttph: /ttt:///ggthtbb com/ ddrous /warp.
ACKNOWLEDGMENTS
This work was supported by UK Research and Innovation grant EP/S022937/1: Interactive Artificial Intelligence. EPSRC LEAP Digital Hcalth Hub grant EP/X03134990 and EPSRC program grant EP/R006768/1: Digital twins for improved dynamic design. We thank Hengshuai Yao and Yasin Abbasi- Yadkori for valuable discussions culminating in ideas that helped improve the appeal and performance of weigh-space linear recurrent neural networks.
REFERENCES
[1] Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint 1rXiv:1803.083 2018.
[2] Shadi Alijani, Jamil Fayyad, and Homayoun Najjaran. Vision transformers domain adaptation and domain generalization: a study robustness. Neural Computing and Applications, 36(29);17979-1888 18007, 2024.
[3] Marcin Andrychowicz. Misha Denil, Sergio Gomez, Matthew W Hoffmann, David Pfau, Tom Schaul, Brendan Shillingford, Nando Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
[4] Martin Arjovsky. Out of distribution generalization in machine learning. PhD thesis, New York University, 2020.
[5] Matthew Ashman, Tommy Rochussen, Adrian Weller. Amortised inference neural networks for small-scale probabilistic meta-learning. arXiv preprint arXiv:2310. 1578666, 2023.
[6] Midhun T Augustine. A survey on universal approximation theorems. arXiv preprint arXi: 12895, 2024.
[7] Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Jocl Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. Advances neural information processing systems, 29, 2016.
[8] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint 181 2018.
10

---

Published as a conference paper at ICLR 2026
[9] Maximilian Beck, Korbinian Pöppel. Markus Spanring, Andreas Auer. Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory, arXiv preprint arXiv:2405.045 2024.
[10] Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning memorize test time. arXiv preprint urXiv:2501.0066 00663, 2024.
[11] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances neural information processing systems, 28, 2015.
[12] Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-leamiing with differentiable closed-form solvers. arXiv preprint 1805 08136, 2018.
[13] Olivier J Bousquet, Amit Daniely, Haim Kaplan, Yishay Mansour, Shay Moran, and Uri Stemmer. Monotone learning. In Conference on Learning Theory, pp. 442-866. PMLR, 2022.
[14] James Bradbury, Roy Frostig, Peter Hawkins. Matthew James Johnson, Chris Leary. Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Menderman-Milngg and Qiao Zhang. JAX: com- posable transformations of Python+NumPy programs, 2018. URL httphttp:/// /ttp:::////t/////// httpg/gggttttooo/ooo///o jax.
com/ coom/goole. e/
[15] Manuel Brenner, Elias Weber, Georgia Koppe, and Daniel Durstewitz. Learning interpretable hierarchical dynamical systems models from time series data. In The Thirteenth International Conference on Learning Representations, 2025. URL https::/// /ttp///ppeerreeeeeeeeeeeeeeeeeee net forum? www.ummdddvoo00aeaemam
[16] Natalia Caporale and Yang Dan. Spike mning-dependent plasticity: a hebbian learning rule. Annu, Rev. Neurosci., 31(1):25-46 46,,,466222 2008
[17] Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A Clifton. A brief review of hypemetworks in deep learning. Artificial Intelligence Review, 57(9)2550, 2024.
[18] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[19] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcchre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mn encoder-decoder for statistical machine translation. arXiv preprint arXiv: /406. 1078, 2014.
[20] Peng Cui and Jinjia Wang. Out-of-distriution (ood) detection based on deep learning: A review. Electronicx, 11221)35000. 2022.
[21] Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine learning through are and what's next. Journal of Scientific Computing, 92(3):88, 2022.
informed neural networks: Where we
[22] Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024.
[23] Soham De, Samuel L Smith, Anushan Fernando. Aleksandar Botev, George Cristian-Muraru Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, ct al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402. 2et.00...14277 19427, 2024,
[24] Luca De Luigi, Adriano Cardace, Riccardo Spezialetti, Pierluigi 7ama Ramirez, Samuele Salti, and Luigi Di Stefano. Deep learning on implicit neural representations of shapes. arXiv preprint arXiv:2302. 2023.
[25] DeepMind, Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Antoine Dedieu, Claudio Fantacci, Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Hennigan. Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, lurii Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzic, Vladimir Mikulik, Tamara Norman, George Papamakarios, John Quan, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Laurent Sartran, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Miloš Stanojević, Wojciech Stokowiec, Luyu Wang, Guangyao Zhou, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL http: hhw..bb.../tgg////// com/ ggoglee le deepmind.
[26] Gregoire Deletang, Anian Ruoss, Jordi Grau Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A Ortega. Neural networks and the chomsky hierarchy. In The Eleventh International Conference on Learning Representations, 2023. URL https: /ttp:///pppeeeee/eeeeeeee ew. net/ forum? wtw.ooonnnnotoitttoteeoeoo.eeeee d=wbxHArke@en.

---

Published as a conference paper at ICLR 2026
[27] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words; Transformers for image recognition at scale. arXiv preprint arXiv:2010. 119299 2020.
[28]
Emilien Dupont, Hyunjik Kim, SM Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. arXiv preprint arXiv:2201.12204 2022.
[29] Mohamed Elsayed, Qingfeng Lan, Clare Lyle, and A. Rupam Mahmood. Weight clipping for deep continual and reinforcement learning. RLJ, 5:2198-22211 2217, 2024.
[30] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
[31] Jonathan Gordon, Wessel P. Bruinsma, Andrew Y. K. Foong, James Requeima, Yann Dubois, and Richard E. Turner, Convolutional conditional neural processes. In International Conference Learning Representations, 2020. URL wwtpt:ttt///...... httttp::///pperrevieeee htt.eeeeeeee.rrrrreeeeeeeeeeeeee
[32] Niclas Alexander Göring, Florian Hess, Manuel Brenner, Zahra Monfared, and Daniel Durstewitz. Out- of-domain generalization in dynamical systems reconstruction. In Forty-first International Conference on Machine Learning, 2024. URL https::tt///.....tm /twp:////eeeeeeeeeeeeeeome.oeee. net forum? i d=xTY ADD2NDD.
[33] Mark S. Graham, Petru-Danicl Tudosiu, Paul Wright, Walter Hugo Lopez Pinaya, Petteri Teikari, Ashay Patel, Jean- U- Im,, Yee H. Mah, James T. Teo, Hans Rolf Jäger, David Werring, Geraint Recs, Parashkev Nachev, Sebastien Ourselin, and M. Jorge Cardoso. Latent transformer models for out-of-distribution detection. Medical Image Analysis, 90:102967, 2023. ISSN 1361-8415, doi: https: //doi.org/ 10. 101 mww....meaaaiaia.m.a.a.aaa.aaaaa 2023. 2023. 102967. URL https: httttpp://wwwwcece. wwwp:/www..........e.e....t. rectt wttp/cccccccceccccceeecccccccccc httpll/eepp///1eeee 84 0415230022222
[34] Riccardo Grazzi, Julien Siems, Arher Zela, Jörg KH Franke, Frank Hutter, and Massimiliano Pontil. Unlocking state-tracking in linear through negative eigenvalues. arXiv preprint (xXiv:241.125533 2024.
[35] Riku Green, Grant Stevens, Zahraa Abdallah, et al. Time-series classification for dynamic strategics in multi-step forecasting. arXiv preprint 240.022402.0833,, 2024.
[36] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, In First Conference Language Modeling, 2024. URL. htt.hhttts://oenneeee eew net/ forum?id tRYakwlVe VY2.
[37] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with struetured state spaces, arXiv preprint arXiv:2111.0033 2021.
[38] Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems, 25:22982-22994, 2022.
[39] David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint arXiv:1609.088 f6t.::/60000006, 2016.
[40] Ryuichiro Hataya, Kota Matsui, and Masaaki Imaizumi. Automatic domain adaptation by transformers in in-context learning. arXiv preprint arXiv:2405. 166119995...9.,,.. 2024.
[41] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 0036-1034, 2015.
[42] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778 2016.
[43] Christoph Jürgen Hemmer and Daniel Durstewitz. True zero-shot inference of dynamical systems preserving long-term statistics. arXiv preprint arXiv:2505.131 13192, 2025.
[44] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1780, 1997.
[45] Sneha Jadhav, Jianxiang Zhao, Yepeng Fan, Jingjing Li, Hao Lin, Chenggang Yan, and Minghan Chen. Hime-varying sequence model. Mathematics, 11(2), 2023. ISSN 2227-7390. doi: m0.3390maathl 1 1020336. URL. https: // /wwww/www/.////////////// www..dpp.....www.mmwww.mmw.m.... 2ooom/227--220 7390/ /2/336.
12

---

Published as a conference paper at ICLR 2026
[46] Shashank Mohan Jain. Hugging face. In Introduction transformers for NLP: With the hugging face library and models to solve problems, pp. 51-67. Springer, 2022.
[47] Samy Jelassi, David Brandfonbrener. Sham M Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying. arXiv preprint XXiv:2402.010322. 2024.
[48] Jonathan Kahana, Eliahu Horwitz, Imri Shuval, and Yedid Hoshen. Deep linear probe generators for weight space learning. arXiv preprint arXiv:2410. 10811, I, 2024.
[49] Armand Kassai Koupai, Jorge Mifsut Benet, Jean-Noël Vittaut, and Patrick Gallinari. Geps: Boosting generalization in parametric pdc neural solvers through adaptive conditioning. 38th Conference on Neural Information Processing Systems (NeurIPS 2024), 2024.
[50] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp. 5156-5166. PMIR, 2020.
[51] Nawid Keshtmand, Raul Santos- Rodriggz,d and Jonathan Lawry. Typicality-based point ood detection with contrastive learning. In Northern Lights Deep Learning Conference, pp. 120- 129, PMLR, 2024.
[52] Patrick Kidger. On neural differential equations. arXiv preprint arXiv:2202.0244: 2022.
[53] Patrick Kidger and Cristian Garcia. Equinox: neural networks in JAX via callable Py PyTrees and filtered transformations. Disserentiable Progranming workshop at Neural Information Processing Systems 2021,
2021.
[54] Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696-6707 2020.
[55] Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013.
[56] Matthicu Kirchmeyer, Yuan Yin, Jérémie Donå, Nicolas Baskiotis, Alain Rakotomamonjy, and Patrick Gallinari. Generalizing to new physical systems via context-informed dynamics model. In International Conference on Machine Learning, pp. 11283-11301. PMLR, 2022.
[57] Bernard O Koopman. Hamiltonian systems and transformation in hilbert space. Proceedings of the National Academy of Sciences, 17(5):315-318, 1931.
[58] Quoc V Le, Navdeep Jaitly, and Geoffrey Hinton. simple way initialize recurrent networks of rectified lincar units. arXiv preprint 504.04150..00911, 2015.
[59] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the JEEE. 86(1 11:2278-2224. 1998.
[60] Ivan Lee, Nan Jiang, and Taylor Berg-Kirkpatrick. Is attention required for icl? exploring the relationship between model architecture and in-context learning ability. arXiv preprint arXiv:2310.08049, 2023.
[61] Ruikun Li, Jiazhen Liu, Huandong Wang, Qingmin Liao, and Yong Li. Weightflow: Learning stochastic dynamics via evolving weight of neural network. arXiv preprint arXiv:2508.0045 00451, 2025.
[62] Aoyu Liu Yaying Zhang. ppatial-tempone dynamic graph convolutional network with interactive learning for traffic forecasting. IEEE Transactions on Intelligent Transportation Systems, 25(7) 7660, 2024.
[63] Ziwci Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[64] Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9()):4900, 2018.
[65] Warren S McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 1:1113,3, 1943.
[66] William Meriil and Ashish Sabharwal. The parallelism tradcoff: Limitations of precision transformers. Transactions of the Association for Computational Linguistics, 1::31-5555, 2023.
[67] William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in space models. In Forty-forst International Conference on Machine Learning, 2024. URL https: / /pw.reeeivveeeeeeeeeeeeeeevee eew net./ forum? (d-g%.g9.17p....
13

---

Published as a conference paper at ICLR 2026
[68] Igor Mezić. Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dynamics, 41(1):309-322 2005.
[69] Rudy Morel, Jiequn Han, and Edouard Oyallon. Disco: learning discover evolution operator for muuli-physscs-agnosse prediction. arXiv preprint arXiv:2504.14499 2025.
[70] James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. International Conference on Machine Learning, pp.
PMLR, 2021.
[71] Sajad Movahedi, Felix Sarnthein, Nicola Muca Cirone, and Antonio Orvicto. Fixed-point mns:: From diagonal to dense a few iterations. arXiv preprint arXiv:2503. 10799, 2025.
[72] Li Nanbo, Firas Laakom, Yucheng XU. Wenyi Wang, and Jürgen Schmidhuber. FACTS: A factored state-space framework for world modelling. In The Thirteenth International Conference Learning Representations, 2025. URI https: openrev ew, nwt/ forum? ?1 d=dmCGjeryhh:
[73] Roussel Desmond Nzoyem and Stephane Labbe, Fracturation de floes de glace par percussion dans un modèle granulaire. Gitlfub, 2021. URL https: httttp:///itttt. /gith/btthh.e/gee com/ /d//ddddouss///////// floes/blob/ paateerrrpoots//ntteeenee http://s/...tteeeaaaaiittt////aa
[74] Roussel Desmond Nzoyem, David AW Barton, and Tom Deakin. Extending contextual self- modulation: Meta-learning across modalities, task dimensionalities, and data regimes. arXiv preprint arXiv:2410.01655, 2024.
[75] Roussel Desmond Nzoyem, David A.W. Barton, and Tom Deakin. Neural context flows for meta-learning of dynamical systems. In The Thirieenth Internatiooaa Conference on Learning Representations, 2025.
URL https::////.. http:///ppeeeeeeeeee/eeeeeee net forummmrummm/ wi=
[76] Roussel Desmond Nzoyem, David AW Barton, and Tom Deakin. Towards foundational models for dynamical system reconstruction: Hierarchical meta-learning via mixture of experts. arXiv preprint arXiv:2502.05335, 2025.
[77] Antonio Orvicto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcchre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. International Conference on Machine Learning, pp. 26670- 26698. PMLR, 2023.
[78] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming
Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation pytorch. 2017.
[79] Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Swish: a self-gated activation function. arXiv: Neural and Evolutionary Computing. 2017. URL https:: /atp:///ppiia
org Corpus
1 1961 58220.
[80] T Konstantin Rusch and Daniela Rus. Oscillatory state-space models. arXiv preprint 4rXiv:2410.03943. 2024.
[81] Alexander Rush and Sidd Karamcheti, The annotated s4, ICLR Blog Track, 2022. URL https://iclr-blog-- tra.ttrackk..o htt.auu.../222/////////tttttaae. blog- htt.kk..wuhubboo2222222a/aaaasse
https:/ficlr-
[82] Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International conference machine learning, pp. 9355-9366. PMLR, 2021.
[83] Jürgen Schmidhuber: Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139, 1992.
[84] Florian Schmidt, Generalization in generation: A closer look at exposure bias. arXiv preprint
191.00191, 2019.
[85] Konstantin Schürholt, Michael W Mahoney, and Damian Borth. Towards scalable and versatile weight space learning. arXiv preprint arXi::2466.0900 09997, 2024.
[86] Louis Serrano, Armand Kassai Koupai, Thomas X Wang, Pierre Erbacher, and Patrick Gallinari. Zebra: In-context and generative pretraining for solving parametric pdes. arXiv preprint arXiv:2410.03437, 2024.
[87] Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In The Fleventh International Conference Learning Representations, 2023. URL https: / //openrevi ew. nerruu A 8HW3Axqks.
14

---

Published as a conference paper at ICLR 2026
[88] Chao Song, Youfang Lin, Shengnan Guo, and Huaiyu Wan. Spatial-temporal synchronous graph convolutional networks: new framework for spatial-temporal network data forecasting. Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 914-921, 2020.
[89] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Yang. Sebastian Ruder. and Donald Metzler. Long range arena A benchmark for efficient transformers. In International Conference on Learning Representations, 2021. URL https: openreviee. /eett
[90]
Emmanuel Trélat. Contrôle optimal: théorie & applications, volume 36. 2005.
[91]
Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and llya Tolstikhin. Predicting neural network accuracy from weights. arXiv preprint arXiv:2002. 14488 2020.
[92] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[93] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapcau, Evgeni Burovski, Pearu Peterson, Warren Weekesser, Jonathan Bright, Stéfan van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, F. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SeiPy 1.0: Fundamental Algorithms for Scientific Computing in Python, Nature Methods, 17:261-211 2020. doi: 10.1038/s41592. 019-0686-2.
[94] Johannes Von Oswald, Eyvind Niklasson, Eutoe Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning,
PMLR, 2023.
[95] Johannes von Oswald, Nino Scherrer, Seijin Kohayashi, Luca Versari, Songlin Yang, Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf. Oliver Sieberling, Alexander Meulemans. Rif A. Saurous, Guillaune Lajoie, Charlotte Frenkel, Razvan Pascanu, Blaise Agüera y Arcas, and João Sacramento. Mesanet: Sequence modeling by locally optimal test-time training, 2025. URL https: http///xxxvv.rr/ee/e /
2506. . 05233.
[96] Benjamin Walker, Andrew Mcl.cod, Tiexin Qin, Yichuan Cheng, Haoliang Li, and Terry Lyons. Log neural controlled differential equations: The lie brackets make a difference. arXiv preprint 240i::2402.181112. 2024.
[97] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
1981 Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE,
78 1550010550 1990.
[99] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion- mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
[100] Songlin Yang, Bailin Wang. Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. In The eiihty- Annual Conference on Neural Information Processing Systems, 2024. URL attps::/ httttp:://ppppreeeee /tpp//opoiiiieeo// net / Eorum?ld=y8Rut VNRFF
[101] Tianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang. Fujun Luan, Songlin Yang. Kalyan Sunkavalli. William T Freeman, and Hao Tan. Test-time training done right. arXiv preprint arXiv:2505.2384, 2025.
[102] Yedi Zhang, Aaditya Singh, Peter Latham, and Andrew Saxe. Training dynamics in-context learning in linear attention. arXiv preprint arXiv:2501.16266 2025.
[103] Chuanpan Zheng, Xiaoliang Fan, Cheng Wang, and Jianzhong Qi. Gman: A graph multi-ateention network for traffic prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 1234-1241, 2020.
[104] Chuanpan Zheng, Xiaoliang Fan, Shirui Pan, Haibing Jin, Zhaopeng Peng, Zonghan Wu, Cheng Wang. and Philip S Yu. Seatioltemporal joint graph convolutional networks for traffic forecasting. IEFE Transactions Knowledge and Data Engineering, 36( ):372-385, 2023.
15

---

Published as a conference paper at ICLR 2026
[105] Allan Zhou, Kaien Yang, Kaylee Burns, Adriano Cardace, Yiding Jiang, Samuel Sokota, Zico Kolter, and Chelsca Finn. Permutation equivariant neural functionals. Advances neural information processing systems, 36:24966-24992, 2023.
[106] Allan Zhou, Kaien Yang, Yiding Jiang, Kaylee Burns, Winnie Xu, Samuel Sokota, Zico Kolter, and Chelsea Finn. Neural functional transformers. Advances neural information processing systems, 36: 77485-77502, 2023.
[107] Allan Zhou, Chelsea Finn, and James Harrison. Universal neural functionals. arXiv preprint arXiv:2402.05222, 2024.
[108] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference artificial intelligence, volume 35, pp. 11106-11115 2021.
[109] Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without normaliza- tion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
[110] Juntang Zhuang. Tommy Tang. Yifan Ding, Sekhar C Tatikonda, Nicha Dvomek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances neural information processing systems, 33:18 13:18795-18806 2020.
[111] Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-leaming. In International Conference on Machine Learning, pp. 7693-7702. PMLR, 2019.
[112] Nicolas Zucchet Antonio Orvieto. Recurrent neural networks: vanishing and exploding gradients are not the end of the story. In The Tiirtt-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: /twp//eeeeeee..ee///////e net forum? id=4 6Jr4sg TWa.
[113] Nicolas Zucchet, Robert Meier, Simon Schug, Asier Mujika, and Joao Sacramento. Online learning of long-range dependencies. In seirtt--eventhh Conference on Neural Information Processing Systems, 2023. URL. www.hhttt:///teee wwt.https///ppeneee ex wwt//fouunm ? d-Wa GG.ajUnn
16

---

Published as a conference paper at ICLR 2026
Weight-Space Linear Recurrent Neural Networks Supplementary Material
A RELATED WORK
18
B METHODOLOGICAL DETAILS
20
B.1 MOTIVATION
20
B.2 TRAINING ALGORITHMS
20
B.2.1 RECURRENT MODE
20
B.2.2 CONVOLUTIONAL MODE
21
B.3 IMPLEMENTATION CAVEATS
22
C DATASETS, BASELINES & METRICS
24
C.I DATASETS.
24
C.2 BASELINES
25
C.3 METRICS
26
D EXPERIMENTAL DETAILS
28
D.I IMAGE COMPLETION.
29
D.2 IMAGE CLASSIFICATION
29
D.3 ENERGY FORECASTING
29
D.4 TRAFFIC FLOW FORECASTING
30
D.5 IN-CONTEXT LEARNING
30
D.6 DYNAMICAL SYSTEM RECONSTRUCTION
30
D.7 TIME SERIES CLASSIFICATION
30
E ADDITIONAL RESULTS
32
E.1 2D IMAGE EXPERIMENTS
32
E.2 SPIRALS & NEURAL CDES
32
E.3 COMPUTATIONAL EFFICIENCY COMPARISON
32
E.4 NORMALISED TIME CORRELATION ON DYNAMICS RECONS TRUCTION
33
E.5 SPECTRAL ANALYSIS
33
E.6 ROBUSTNESS TO NOISE
33
E.7 ABLATION STUDIES
34
F
VISUALISATIONS
38
17

---

Published as a conference paper at ICLR 2026
A RELATED WORK
The problem of sequence modelling. long dominated by Transformers [92], is experiencing renewed focus on recurrent architectures, particularly for their efficiency and unique modelling capabilities [37; 23]. Our model, the Weight-space Adaptive Recurrent Predictor (WARP), intersects with several active research areas.
Weight-Space Learning The concept of leveraging the weight space of neural networks is not new; for instance, optimisers and hypernetworks inherently process weight-space features [3; 107; 39], treating them outputs of learning algorithm. Other works explore weight features as inputs for model analysis [91; 85] or for implicitly representing data [28]. WARP distinguishes itself by directly evolving the root network's weights as its intermediate state, without explicitly specifying test-time loss function to minimise. This test-time regression view is similarly observed with research on linear attention [100; 101; 95] and fast weights programming [7].' 7. In the autoregressive forecasting setting, WARP bears striking similarities to WeightFlow [61] which uses graph Controlled Differential Equation [54] to model the continuous-time evolution of the weights, and to the "delta" rule [82], which equally updates weights based on the difference between the prediction and the target. WARP can thus be viewed as a generalisation to broader problem settings that include classification.
Modern Linear RNNs and SSMs Linear RNNs and SSMs have re-emerged as powerful tools, largely due to their parallelisable and hardware-awarc training [100; 71], with impressive performance on long sequences. Notable architectures like S4 [37] and Linear Attention [50] have massively catalysed recent advancements. While WARP builds on the efficiency of linear recurrence, its core innovation lies in its unique state parametrisation rather than solely the recurrent mechanism
which includes non-lincarities for improved expressivity.
Non-Linear Recurrent Mechanisms The landscape of sequence modelling is rich with innovative designs. Hybrid models like Griffin [23] merge recurrences with attention, while Movahedi et al. [71] seek to compute dense linear RNNs from diagonal ones via fixed-point transformations. Frameworks
like FACTS introduce structured memories [72]. Brain-inspired architectures [113; 7]. including time-varying decoder architectures [45], seek to learn evolving relationships between model inputs
and outputs. WARP contributes to this evolving field by introducing novel mechanism - viewing
the RNN hidden state the weights and biases of time-varying root neural network which
results in non-linear self-decoding.
State and Memory in Recurrent Models A central debate revolves around the true state-tracking and memory capabilities of various recurrent architectures. While some SSMs and even Transformers face theoretical limitations in solving certain tasks [67; 47], improvements like incorporating negative cigenvalues in linear RNNs aim to enhance state-trackin [34]. Other works explicitly include neural memory modules so that surprising events are more memorable [10]. The growing test-time training community [101; 95] proposes to combine recurrence with associative memories for improved sequence modelling- WARP's use of a high- high-dimensional weight space for its states is direct attempt to provide richer "infinie-ddimensiona
memory capacity and more expressive temporal dynamics compared to conventional compressed state representations. This has parallels with the fast weights literature [83; 7].
Gradient- Free Adaptation and Zero-Shot Learning Effective adaptation to -iitr--distriuution
dynamics in continual learning settings is significant challenge [35]. For instance, standard
Neural Ordinary Differential Equations [18] struggle with distribution shifts and need retraining or
fine-tuning for adaptation [56; 49]. With its gradient-free formulation, WARP facilitates test-time
generalisation - a problem explored in meta-lcarning frameworks like Neural Context Flows [75] through differentiable closed-form solvers [12]. or in-context learning [94]. WARP can be viewed as a meta-learning model given its progressive refinement of a shared initialisation &g at test-time, with strong connections to amortised inference [5].
In Fig. 6, we explicitly compare our integration with existing fast weights programming methods.
"The hidden state a function which lives in "infnite-dimensionna" space.
18

---

Published as a conference paper at ICLR 2026
Koopman Operators Our method can also be viewed application of Koopman operator modelling. As is the case with nonlinear dynamical systems [57], the challenge is to identify the correct set of eigenfunctions) that linearise the dynamics. WARP addresses this by effectively using the neural network learn data-driven approximation of the Koopman operator, technique explored in modern dynamics and machine learning [64; 68].
theory to
observable functions (the Koopman
Fasi weight programmerr
WARP
FW speate nile
PW,
F7
A
Hypernetwork
H
6.
Figure 6: Schematic comparison of adaptive weight architectures. (Top Left) The fast weight programmer [83] modifies its processing dynamics by iteratively updating the fast weights FW, using a specialized FW update rule acting on the input 2; and the previous weights FW--1 (Bottom Left) A standard hypernetwork [17] generates the weights 0, for target function fo. by passing context code through a higher level network H. (Right) The proposed WARP architecture, where weights are initialized 0, via hypernetwork are driven by linear recurrence: the previous parameters 0:-1 are processed by block A, and input changes Ar are processed by block B. These components are summed to produce the current weights O, which parameterize the function fa, used to map coordinate inputs to the output ye.
conditioned on o subsequent weight updates
19

---

Published as a conference paper at ICLR 2026
B METHODOLOGICAL DETAILS
B.I MOTIVATION
The main motivation behind WARP (Weight-space Adaptive Recurrent Prediction) is gradient-free adaptation to out-of-distributtin (OoD) settings. Relative to OoD detection which has always been a central problem in machine learning spanning decades of research interest [20; 51], OoD adaptation is a recent but growing field rich in new and stimulating ideas [4; 75J. WARP mimics the dynamics of an idealised "smooth" gradient descent observed through projection of 5998-dimensiona space into 2-dimensional PCA space in Fig. 7(a). This offers promising avenue for OoD adaptation with
minimal cost.
10
WARM
Grudiest Depcent
100
1.50
PC1
Step
(a) PCA of weights trajectories
(b) Norm of weights differences
Figure 7: (a) Principal components of the weight space of WARP vs. weight trajectory fitted via Gradient Descent strategy single trajectory; (b) Norm of the difference between updates go through the time steps for WARP, and the gradient steps for Gradient Descent.
B.2 TRAINING ALGORITHMS
Algorithm 1 Recurrent training algorithm for WARP in its non-AR form.
Input: Training sequences {xi] (E.......N11) T- Output: Trained model parameters (A, B,
Algorithm:
Initialise A = I, 0, and
2. for each training epoch do
3. for each batch of sequences do for each sequence (o....
batch do
Initialise fij = ((xx))
6. for t- 1 to T do
Update 0; = A0 + B(x; - xi-1)
Compute output y; = MLPp: (T)
(var Eqs. (3) to (5))
end for
10.
Compute sequence loss L using Eq. (2) end for
11.
12.
Update parameters A, B, using gradient descent
13.
end for
14.
end for
B.2.1 RECURRENT MODE
The recurrent training pipeline is illustrated in its non-AR setting in Algorithm 1, where N indicates the total number instances in the training set, indexed by The quantity is constructed from the components in Eqs. (3) to (5). Typically, is formed by considering normalised time alone. However, depending on the specific use case, normalised time is concatenated with pixel coordinates (for image data), or with positional encoding using sines and cosines (e.g., for time-series analysis).
20

---

Published as a conference paper at ICLR 2026
Normalised Time. This component consists of the normalised time step, where T is the total sequence length:
(3)
T
Normalised Pixel Coordinates. For image data, spatial information is encoded using normalised pixel coordinates. Given pixel at position (w, h) in an image of total size (W, H), the coordinates are:
h
Positional Encoding with Sines and Cosines. The components this matrix 7 PE IR 7xd are defined as:
sin cos
if k 2j if 2j +1,
(
Casla
where d is the encoding dimension, and C' is hyperparameter that controls the frequency of the sinusoidal functions [92].
For the AR setting trained with teacher forcing, x in line 7 is replaced, with probability 1 Prercing with sample from . M(a, of) which taken wiss using the classic reparametrisation trick as outlined in Section 2.3." The batch sequences (lines 4 to 11) is processed in parallel using vectorisation as per the implementation details below.
B.2.2 CONVOLUTIONAL MODE
Like [37], WARP supports convolutional training mode where the sequence of weights is computed efficiently using Fast-Fourier Transforms (FFTs) on modern hardware [81] using Theorem 1. We use the Pythonic notation Ug.7 & {u: ) TT1 R2' and the * to denote the convolution operation.
The summarised convolutional training algorithm is provided in Algorithm 2.
Theorem 1 (Convolution Mode). Assume B IR Dy D= is full row-rank matrix. There exists Axo D and length-TT kernel K such that 0o.r K AXo...
Proof. It follows straightforwardly that the linear recurrence relation 0, A0,-1 + BAx, can be
unrolled as
0: = A'60 +
B'x
Vt f
(6)
2=0
Since B is of full row-rank, the mapping u + Bu is surjective, and EAx, E RDr such that
8, BAx..
(7)
Substituting this into equation 6, we get
0,
Σ
A'
VE (0....,TT [O....,T-11,,
(8)
E=
from which the large kernel extracted:
the sequence of columns of the Kalman controllability matrix [90]
K= KBBBAAB AB, ....B.......1... AT-1B).
(9)
to form the relation
00. = AXo.T
(10)
In the main text, the superseripts omitted for clarity.
21

---

Published as a conference paper at ICLR 2026
Algorithm 2 Convolutional training algorithm for WARP, where line 6 can be computed with (inverse) FFTs and the convolution theorem. All decoding sequence steps (lines 7-9), as well as the individual sequences (the batch from lines 4-11) processed in parallel.
Input: Training sequences (xt (o......
(--1... Output: Trained model parameters (A, B, Ø)
Algorithm:
1. Initialise A 11 0, and
for each training epoch do
3. for each batch of sequences do for each sequence {x] I-
batch do
Initialise fi = xxx)) and Axo
(Esr Eq. (7) and Theorem 2) (Eer Eq. (10))
6.
Compute G. K Ax
for to
do
8.
Compute output ył MI.P,
end for
10.
Compute sequence loss L using Eq. (2) end for
11.
12. Update parameters A, B, e using gradient descent
13. end for
14.
end for
In practice, however, we find the assumptions Theorem too restrictive to be applicable. Indeed, with the weight space typically larger than the input space, i.e. Dg > Da, the mapping u Bu is not surjective. For such cases, we leverage the initial network á to enforce additional constraints into the learning process. Theorem guarantees the existence of suitable initial input difference Axo to use as input in the convolution equation 10.
Theorem 2 (Existence of an Initial Input Difference). Fix & as a locally linear operator with B Vo(x,), and assume ker o # 0. There exists RP, such that Axo- X and Eq. (10) holds.
Proof. The proof is straightforward by remarking that 0g o(xx). Using Eq. (7), find that
0u BAx1
o(x,) = 0 BAX
Since ker ó + 0, Ev such that d(v) = 0, and since we' 've fixed B = Vo(x)), this leads to ((x,) = ((v) + Vo(x))
Since is locally linear, this relation can be identified with its unique first-order Taylor expansion near Xg, from which identify
Axp: equivalently Axo Xy u.
B.3 IMPLEMENTATION CAVEATS
The difference between a successful WARP training and a failure may lie in small implementation details. We recommend clipping several quantities increase the chances of success.
Prediction Clipping During training, found it important to constrain the outputs of the root network to avoid divergence and blow-up. This can achieved through final activation applied to the mean component of the output, with e.g. min-max symmetric clipping:
X₁ max(min (x,, dum); dim),
22

---

Published as a conference paper at ICLR 2026
with hyperparameter dum 0. Another powerful approach which has demonstrated great success in the realm of Transformers is the dynamic tanh [109] with learnable scalars a, b, a, B:
Xx
X - a tanh
+ 3,
with (a, a) initialised as the largest value encountered in the training datasets, and (b, 9) both as zero. This ensures output scaling that is consistent in shape with the classical tanh activation.
Weight Clipping In some problems like MNIST, we found not enough to constrain the root's output within certain bound, as the predictions kept diverging. In such cases, mechanisms like directly clipping the weights in between time steps provided an additional form of non-linearity helpful for the model. Our weight clipping strategy differs from traditional approaches discussed in continual learning contexts [29] as it does not consider initialisation:
0, = c p(0,, - Wiim: Wlim),
where Whm a hyperparameter, and clip shorthand for min-max clipping as discussed above. This clipping operation serves as an implicit activation function in weight space, preventing un- bounded growth weight values and stabilizing training.
Gradient Clipping As customary with recurrent networks training with the backpropagation through time algorithm [98]. we observed the classical problem of exploding gradients [112]. which was mitigated by clipping the gradient norms within specific bound captured by glim 10-7
23

---

Published as a conference paper at ICLR 2026
C DATASETS, BASELINES & METRICS
C.I DATASETS
Table 5: Problems and their corresponding training datasets with specifications. Details about the UEA datasets are presented Table 4 and not repeated here. The term (varies) indicates that further splits the datasets were made.
PROBLEM
DATASET
SAMPLES N
SEQ. LENGTH T
CONTEXT LENGTH
FEATURES Dz
MNIST
60,000
784
(varies)
Images
Fashion CelchA
(varies)
162,770
1,024
(varies)
ml
34,369
192
m2
34,369
192
ETT
8,449
h2
8,449
192
96
MSD
20,480
256
100
MSD-Zero
20,480
256
100
Dynamical Systems
LV
15,000
256
100
SINE
(varles)
16
Spirals
We describe various datasets used in this paper. Our description delves into the details of pre-existing datasets and the data generation script of synthetic toy datasets. This section complements the summary we provided in Table 5.
Image Datasets Both MNIST [59] and Fashion MNIST [99] datasets were loaded using the well- known PyTorch interface [78]. The values were then normalised that pixel values ranged [-1, 1].
The CelebA dataset [63] was loaded using the API from [74] itself inspired by [1 [1]]]. Training was performed on the train sets (see attributes Table 5), with validation and testing the predefined
test sets.
Electricity Transformer Temperature (ETT) For the electricity data [108], we further normalised the preprocessed data from TSLib to place all values in the range [-1, 1] in order to facilitate learning dynamics. We did not use the predefined "test" set because of its 144-step-long forecast window, which is much longer than the 96 steps all models saw during training. Consequently, used the "validation" set to evaluate our models as well as all the baselines.
University of East Anglia (UEA) On the UEA dataset [8], we follow the procedure from [96] and reuse the same dataset. (We note that this exact experimental protocol was recently observed in [80]). We extracted the necessary scripts for reprodueibility and provide them as part our code under appropriate license.
Dynamical Systems Continuous autonomous dynamical systems can be conceptualised as mul- tivariate time series governed by a deterministic vector field (x, p) x,, with p encompassing physical parameters affecting the dynamics. Given an initial condition Xg. one can systematically simulate and subsample the trajectory Xg:r- To complement provide the vector field used for each dataset in Table We summarise their physical parameter ranges in Table All trajectories are obtained with SciPy's 'RK45' adaptive time-stepping numerical integrator [93]. The five training data splits of the SINE are "Tiny", "Small", "Medium". "Large", "Huge", with respectively 1, 10, 100, 1k, and 10k samples. All datasets are normalised and placed within [-1, 1]. which is calculated using the train set statistics. Comprehensive data generation scripts with physical parameters for all four datasets are provided in our code.
description in Section 3.2,
Spirals The Spirals dataset is additional dynamical system dataset for binary classification tasks.
The training data consists of 10,000 samples, where each sample is a spiral trajectory represented as sequence of 64 2D points (x, y coordinates). Half of the dataset contains clockwise spirals (labelled as 0), while the other half contains counterclockwise spirals (labelled as 1). The spirals are generated using sine and cosine functions with random phase offsets, and the amplitude decreases with time
24

---

Published as a conference paper at ICLR 2026
Table 6: List of considered dynamical systems with their vector fields and/or flow maps.
DATASET
PHYSICAL PARAMETERS
VECTOR FIELD
FLOW MAP
21 D2
Mass Mass-Spring- Spring-Dumppr (MSD)
m. k
used
82
P2
di 021
-22122
Lotka-Volterra (LV)
0,
Non-closed form solution
B2 60122
Sine Curves (SINE)
Ap
Not used
(1) sin (2ar (p)
Table 7: Parameter ranges of several dynamical systems for Train and Test datasets. Test parameter ranges induce OoD trajectories, except for the SINE cases. The relative scale and broad range of parameters values for the MSD problem make this task extremely challenging.
DATASET
TRAIN PARAMETER RANGES
TEST PARAMETER RANGES
m [0.02, 0.04
m E 0.05) k: [2,
Mles-Spring-Dampee (MSD)
(4, 16]
e [0.01, 0.2)
c E [0.01, 0.3
E [20, 50]
E [10, 60]
[80, 120)
E [70, 130]
Lotka-Voltera
[80, 120]
E [70, 130]
[20, 50]
E [10, 60]
Sine Curves (SINE)
E [- a/G, m/6]
E [=#/6,a/6]
create the spiral effect. This dataset serves powerful test case for dynamics classification; it was inspired by [53],"] 10 with few samples visualised in Fig.
Figure 8: Visualisation of a few samples of the Spirals datasets. For each sample, we plot the y and the x sequences of coordinates against each other to observe the (counter- Jclockwise ccounter)Jclokkwsse direction.
C.2 BASELINES
All models are trained based on the same hyperparameter tuning protocol in order to ensure fair comparability.
Standard RNNs We consider two powerful RNN baselines including the Gated Recurrent Unit (GRU) [19] and the Long Short-Term Memory (LSTM) [44]. Both are trained in recurrent AR mode for forecasting problems and recurrent non-AR mode for classification. Both are unidirectional and have a single layer match WARP model. Depending the experiment, vary their hidden size to match the total parameter count of WARP. The remainder of the experiment details such as
An example usage can be found at https: //occs:///ccccc/ cwr.ooccs.icieo wtwperssitddfffffffeeee
neural_cd
25

---

Published as a conference paper at ICLR 2026
training procedure presented in Appendix D. We attach both implementations, using Equinox [53], to our code.
Time Series Transformer (TST) We consider the Time Series Transformer from HuggingFace [46]. This baseline provides a SOTA baseline, leveraging one of the most transformative sequence mixing processes to date: Attention. The specific model used was the TimeSeries TereSSerresrannfomeFFFoorro reeditroenn,on which adds a distribution head on top of the vanilla encoder-decoder Transformer [92]. This means that the model learns a distribution from which take the mean to be used for time-series forecasting. The next token prediction is obtained by randomly sampling a window of context length L plus prediction length 7 from the target time series. This prediction window is subsequently masked for the next token prediction task.
Convolution Conditional Neural Process (ConvCNP) The ConvCNP [31] is secoder-based meta-learning approach that doesn't t require gradients in order to adapt to novel scenarios. The ConvCNP is trained for on-the-grid image completion with 100 random shots (time steps. We adapt the data loading process to allow the ConvCNP to operate on raster-scan-ordered pixels at test time.
Structured SSM (S4) We use the powerful Structured State Space Model S4 with the implemen- tation of [81]. We particularly apply it to the MNIST experiment, where the goal is to forecast contiguous range of future predictions given a range of past contexts. To that end, we simply concatenate the entire context with a sequence of masks set to the length of the forecast window. This input is single sequence of length that is run through the deep S4 model, which maps an output of length T. We then use the last T - L tokens as the forecasted predictions. Unlike other models in this work, the MNIST image completion problem with S4 is trained with a 256-way cross-entropy loss, pixel intensities take integer values in the range [0, 255). This limits the fair applicability of S4 on the CelebA dataset, since images contain all three RGB channels.
Neural Controlled Differential Equation (NCDE) NCDEs or (Neural CDEs) [54] provide a continuous- time framework for processing as continuous path. By using the path as a control for a neural differential equation, NCDEs can naturally handle missing data and irregular sampling. The continuous nature of NCDEs makes them strong baseline exclusively for classification tasks.
seggularl--saplled time series by interpreting the data
Baselines for classification Mamba, S6, Log-CII CDE, NRDE, NCDE, LRU were all reported from [96]. where we direct the reader for further details. We reused the results and the conclusion from that work, as was done by Rusch Rus [80]. The LinOSS baseline [80] reported in Table corresponspods to the more powerful LinOSS-IM IM variant. We used the official implementations of FACTS [72] and Griffin [23]. Griffin's model size was reduced to 5k to fit within our compute budget.
C.3 METRICS
Bits Per Dimension (BPD) The (BPD) is used to evaluate the quality of generative models, particularly for images. It quantifies how many bits are needed on average to encode each dimension (e.g., pixel) of the data, with lower BPD values indicating better model. The BPD is derived from the negative log-likelihood (NLL) of the data under the model's predicted distribution. For a given ground truth pixel value y, and its corresponding predicted mean ý, and standard deviation d,, the overall NLL over the image is calculated as:
(y, 81)²
NLL
O2 3)
0,²
The BPD is obtained by converting the NLL from natural units of information to bits:
BPD NLL logz(e).
Mean Abslute Error (MAE) The MAE measures the average magnitude of the errors in a set of predictions. For sequence of true values y, and predicted values yr, the MAE is given by:
MAE A
y 8rl-
T
1=0
26

---

Published as a conference paper at ICLR 2026
Mean Absolute Percentage Error (MAPE) The Mean Absolute Percentage Error (MAPE) ex-
presses the average absolute percent error. The MAPE is given in percentage points by:
100
Y1 Y1
MAPE
Yt
27

---

Published as a conference paper at ICLR 2026
D EXPERIMENTAL DETAILS
We begin this section by sharing experimental details shared across all problems. Subsequent subsections will delve into the specifics of each completion, forecasting or classification problem.
WARP setup. Although specific details may vary depending the problem, the root network is consistently chosen as an MLP for all problem sets in this paper. Given the quadratic memory cost O(D)), we can vary its layers to balance batch size with capacity. Image completion and forecasting problems use the ReI.U activation [1], while smooth dynamical system reconstruction uses the Swish the root network given in Table 8. The initial hypernetwork -used for all problems except Image Completion, MSD, and LV- is made up of two hidden layers of width Aiy3 + 2.2ha//3 and 2/1/33 /3 + Nes/3 neurons, respectively. The positive integers hin and hoor Da are the number of input and output neurons, respectively.
[79]. Complete details
Table 8: Root MLP configurations for the datasets in each problem.
PROBLEM
DATASET
WIDTH
DEPTH
ACT. FUNCTION
MNNSTT
24
3
ReLU
2D Images
Fashion MNIST
24
3
ReLU
CelebA
24
3
ReL.U
148
m2
148
I
ReL.I
ETT
148
ReLU
148
ReLU
MSD
Swish
Dynamical Systems
MSD Zero
48
3
Swish
3
SINE
48
3
Swish
Spirals
24
1
Swish
Worms
128
1
ReLU
SCP1
SCP2
48
2
ReLU
UEA
Ethanol
32
2
ReLU
Heartbeat
72
2
ReLU
Motor
32
2
Rel.U.
WARP-Phys setup. For the SINE experiment, the root network predicts the phase & to feed into the sinusoid 7 + sinTT + Φ). For the challenging MSD and MSD-Zero problems, we embed knowledge of the general analytical solution and the initial condition with + E(T)xo, where E(-) E R with its four coefficients predicted by the root network, and is known throughout. E(T) is viewed as the exponential AA, where is the constant matrix characterizing the mass- spring-damper dynamics:
(-k/m -c/m) [73]; its poor conditioning - consequence of the large parameter scales and ranges detailed in Table 7 - would destabilise the training if A was directly learned. We note that stronger levels of physics may be embedded into the root network: predicting rescaled time-invariant constants (m, k; c). parameterising the signal as damped sinusoids, deccoposooo,iiionn, etc. The physics-inff informed strategy we present is the one that produced the biggest improvement over WARP in our experiments.
Optimisation & Core baselines. Our WARP framework (along with our custom GRU, LSTM, ConvCNP, and Neural CDE) is implemented with the JAX framework [14] and its ecosystem: Equinox for neural network definitions [53], and Optax for optimisation [25]. We use the AdaBelief optimiser [110], and clip the gradient norm with Otim 10-7 We apply the "reduce
plateau" rule where the learning rate is divided by if the average loss I doesn't 't evolve after 20 epochs. For most problems, we the initial learning rate at 10-5 All GRU and LSTM models have single layer to match WARP We tweak their hidden size rather than the number of layers in order to increase or reduce parameter count, thus keeping in check the complexity of the models under consideration (see Table 9).
The average heing caleulated over 50 iterations.
28

---

Published as a conference paper at ICLR 2026
Table 9: Size of the hidden state in standard RNNs for each dataset.
PROBLEM
DATASET
LSTM HIDDEN UNITS
GRU HIDDEN UNITS
MNIST
750
750
Imuges
Fashion MNIST
ColebA
700
825
920
920
m2
920
920
ETT
h2
920
920
MSD
2450
2850
Dynamical
MSD Zero
2450
2850
Systems
IV
2450
2850
SINE
2280
2280
TST and S4 baselines setup. As for the TST model for forecasting, it is implemented PyTorch [78] and uses the Adam optimiser with constant learning rate of 6 x 10 beta values of 0.9 and 0.95 and weight decay coefficient of 10-1 For the S4 model, we used layers, each with hidden state of size 512, batch size of 50, learning rate of 10-3 and the traditional weight decay with coefficient 0.05. Like [81] I, we used prenormalisation, but we did not use dropout. These hyperparameters were selected based on the validation set when available, and the test set if not (e.g.. all synthetic toy problems).
Hardware. The WARP, GRU, LSTM, ConvCNP and S4 models are run workstation fitted with RTX 4080 GPU with a memory capacity of 16 GB. The TST was trained on RTX 3090 GPU with 24 GB memory.
D.1 IMAGE COMPLETION
On these problems, since trained with the NLL loss from Eq. (2), we use a dynamic tanh activation function the mean prediction, with (a, b, a, 3) initialised (1, 0, 1, 1). Only experiments run MNIST and Fashion MNIST use weight clipping, while CelebA does not." For CelebA, we use Tlim 10-1 (see Section 2.2) while find the unusually large Ttm 0.5 suitable for MNIST and Fashion MNIST. We compare WARP to the MNIST baselines roughly at the same parameter counts: GRU (1.694 M), LSTM 1.69 M), S4 (1.717 M). and WARP (1.687 M). We train for 200 and 250 epochs in batches of 640 and 1256 for (Fashion) MNIST and CelebA, respectively. We apply the recurrent AR mode with Plorcing 0.15, while directly feeding the mean prediction back into the recurrence (i.e., the reparametrisation trick is disabled both during training and inference).
As inputs to the root network, while the normalised pixel coordinates are better suited for this task. we report our mate-of-the-an results using the normalised time 1/(T 1). In fact, all results presented in the paper only use the normalised time coordinate system, except for the time series classification the UEA dataset presented below.
D.2 IMAGE CLASSIFICATION
For this task, we use the same hyperparameters as the MNIST task described above, with the only difference that the training now performed in recurrent non-AR mode.
D.3 ENERGY FORECASTING
For these experiments, all models share identical architectures across the four datasets (ETT-h1, ETT-h2, ETT-ml, ETT-m2). The learning rate differs between hourly and minute-level datasets: 10-5 for h1/h2 and 10 for m1/m2. For hourly datasets, we train for 500 epochs, while minute-level datasets require only 250 epochs (corresponding to roughly 1.5 hours of training). All models are trained with batch size 3600 in autoregressive mode with stochastic sampling (non-AR), and
Apprtt from (Fashion) MNIST, other problem in this work used weight clipping.
29

---

Published as a conference paper at ICLR 2026
Prorcing 0.25. No final activation is applied to the root network's mean output, while the typical positivity-enforring from Section 2.2 is applied to the standard deviation with Olim 10-1
D.4 TRAFFIC FLOW FORECASTING
Our model architecture disregards the explicit spatial connectivity provided in the PEMS08 dataset [88]. Instead, consider the features from all nodes independently, creating flattened feature vector for each time step. This results in an input of shape (12, 510), where 12 is the number of historical time steps and 510 represents the 170 nodes, each with features. Before the input sequence is used in the linear recurrence, its features are transformed by ID-convolution with 510 input channels, 4080 output channels, and kernel length of 36. The model is trained predict a single feature per node for the future 12 time steps.
D.5 IN-CONTEXT LEARNING
The setting is the elegant in-context learning setting developed by [102], where the goal is learn the linear mapping between several key-value pairs. The keys {x; iem....,N vectors of dimension D2 - 1. and the values {ii)a Li.,,, are scalar, both concatenated to form a state of dimension Dz- final query key is given, and the model must predict its corresponding value. This missing value is substituted by 0 in the input sequence, as depicted in Fig. 5(a).
Importantly, retain consistency across the literature, we preserve the notations from [102], even though they conflict with those established in our problem setting in Section 2.1. To revert back to our original setting, one can replace the existing inputs with X A concat( X(+1 ( for 1 =
T - 2; and -1-1 concat(X,. 0). As for the outputs, yr+1, for = 0.... T 2: and O, O....,T- YT 3gg.
D.6 DYNAMICAL SYSTEM RECONSTRUCTION
For the dynamical system reconstruction tasks, since uncertainties are not required, models are trained without NLL loss (i.e., with the MSE loss defined in Eq. (2)). Consistent across all dynamical system experiments, no weight clipping is employed, and the predictions are enforced in the range [-1, 1] with mnitiinitialise dynamic tanh. All losses are computed on the normalised test set.
Mass-Spring- -amperr (MSD and MSD-Zero) For both the MSD and its MSD-Zero variant, the experimental setup is largely identical. A learning rate of 10-5 used. Training proceeds for 1000 epochs using batch size of 1024. WARP, GRU, and LSTM models are trained in an auto-regressive mode with a teacher forcing probability Proncing = 0.25.
(LV) The LV experiment is performed for 1500 epochs with a batch size of 1024. Training is conducted with teacher forcing probability Pioncing 1.0, meaning the model is always fed the ground truth inputs during training. This is because this a memorisation task, and the goal is for the model to predict the next token knowing the previous one. LSTM and GRU use the same hyperparameters, except with hidden states of sizes 2450 and 2850 respectively (see Table 9).
Sine Curves (SINE) Across the various SINE datasets (Tiny, Small, Medium, Large, Huge), consistent configuration is maintained. The learning rate is set to 10-5, Models are trained for 1000 epochs in single batch (as large 10000 on Huge). Similar MSD, training is autoregressive with Pforcing = 0.25. No final activation is applied to the root network's 's mean output. The inference process for SINE datasets begins with a very short context, just I time step.
D.7 TIME SERIES CLASSIFICATION
For time series classification tasks, encompassing both the UEA datasets and the Spirals dataset, models are consistently trained in the non-AR mode, with the categorical cross-entropy loss. Across all these classification experiments, root weight are evolved without weight clipping, and no dynamic tanh activation is applied to their final outputs. Key training hyperparameters exhibit some variation across these diverse datasets: the learning rate is 10-5 for the Spirals dataset and most UEA datasets
30

---

Published as a conference paper at ICLR 2026
(e.g., Ethanol, Heartbeat, Motor, SCP1, SCP2), with the Worms dataset being exception 10-6, 6. The number of training epochs varies widely, ranging from 800 for the Worms dataset, 4000 for Spirals, up to 6500 for the Ethanol UEA dataset, with other UEA datasets generally trained for several thousand epochs. Given our limitation of 16GB available VRAM memory, batch sizes also differ significantly; for instance, the Worms dataset uses batch size of 40, other UEA datasets batch sizes typically in the hundreds (from approximately 280 to 560), and the Spirals dataset employs large batch size of 10000, Regarding data preprocessing, input data normalisation is applied for several UEA datasets (specifically Ethanol, Heartbeat, SCP1, and SCP2), but it is not used for others like Eigen Worms and Motorlmagery, nor it required for the Spirals dataset.
This task uses positional encoding in addition to normalised time. The dimension and the denomi- nator constant C' of the positional encoding defined in Eq. (5) [92] and used in concatenation with the normalised time on the UEA dataset, are presented in Table 10.
Table 10: Hyperparameters for positional encoding the UEA datasets.
Worms
SCP1
SCP2
Ethanol
Heartbeat
Motor
Dimension d
20
10
10
10
10
10
Denominator constant C
20
10
10
5
10
31

---

Published as a conference paper at ICLR 2026
E ADDITIONAL RESULTS
E. 2D IMAGE EXPERIMENTS
Similar to MNIST image completion, train WARP, LSTM and GRU to generate items of clothing (Fashion MNIST). The results, presented Table 11 confirm the potency our framework, as previously evoked in Section 3.1. We perform additional classification the sequential MNIST dataset, where we observe a 99.93% accuracy on the subsampled grayscale images in R 14x 14x1,
Table 11: Best test-set MSEs and BPDs on Fashion MNIST across 3 runs with different seeds.
METHOD
MSE
BPD
GRU
0.078
0.66
LSTM
0.082
0.73
WARP
0.064
0.59
Table 12: Best accuracies and walltime comparison for Spirals classification across 3 runs.
METHOD
ACCURACY (%)
WALL TIME / EPOCH (SECS)
Neural CDE WARP
100.0
0.12
99.96
0.41
E.2 SPIRALS & NEURAL CDES
Table 12 reveals several limitations of WARP on the toy Spirals dataset originally introduced to test Neural CDEs [52]. We find that at the same parameter count, WARP not only struggles to achieve 100% accuracy, but is also roughly 4x slower, despite being implemented in the same conditions as the Neural CDE.
E.3 COMPUTATIONAL EFFICIENCY COMPARISON
To provide a comprehensive analysis of computational efficiency, we evaluate several key performance metrics for WARP and our baseline models. The experiments were conducted NVIDIA RTX 4080 GPU, ensuring a consistent hardware environment for all comparisons.
For the MNIST image completion task, report the average wall-clock training time per epoch, peak GPU memory usage, and total parameter counts. To ensure a fair comparison, all models were trained with fixed batch size of 128. The results, presented in Table 13, demonstrate WARP's notable efficiency. Despite having a comparable number of parameters to the Transformer model, WARP requires significantly less GPU memory-on par with the much simpler GRU and LSTM architectures and achieves the fastest training time.
Table 13:
Training efficiency comparison on the MNIST image completion task. We report the average wall-clock time per epoch, peak GPU usage, and the number of learnable parameters. WARP the most efficient terms of both time and memory.
Avg. training time per epoch (seconds)
Peak GPU usage (GB)
Parameters (M)
Model
GRU LSTM S4 Transformer
57.04
4.49
1.69
59.22
4.95
1.70
61.53
12.60
1.71
18.62
10.03
1.69
WARP
45.22
2.89
1.69
A similar analysis conducted for the UEA benchmark datasets, with results detailed in Table 14. For these experiments, the batch size was fixed to 32 across all models. The table provides a detailed breakdown of the training time, memory usage, and model complexity for WARP each dataset.
32

---

Published as a conference paper at ICLR 2026
Table 14: Detailed training metrics for WARP on the UEA benchmark datasets.
Training time per epoch (s)
Peak GPU usage (MiB)
Num. of epochs
Training batches
Parameters
Dataset
per epoch
(M)
Worms
10.29
14598
1000
6
5.697
SCPI
0.92
654
5000
13
0.476
SCP2
3,85
2866
5000
17.34
Ethanol
2.22
1536
6500
12
4.681
Heartbeat
6,50
4354
1500
75.02
Motor
2.46
2558
2000
4.503
It is important to note that implementation of WARP, along with the GRU, LSTM, and S4 baselines, utilises JAX. In contrast, the Transformer and all other baselines is implemented in PyTorch. This difference in framework influence performance measurements. Following standard practice, we exclude any one-time JIT-compilation costs from the reported wall-clock times.
E.4 NORMALISED TIME CORRELATION ON DYNAMICS RECONSTRUCTION
Let's analyse when the root network takes as input exclusively the normalised time. In that case, WARP uses diagonal readout matrix (z(r)
in Fig. 9()) to self-decode the hidden states. This implics that post training, the weights 0g and the time T = t/(T" - 1) should be correlated. We confirm this hypothesis by plotting the correlation coefficient between the vector forr and all time points across all samples in the test set. We observe a strong either positive or negative correlation between the two quantities (see Fig. 9())).
13
486
481
48
200
Time
050
075
110
Correxatio Confiitett
03
(a) MSD readout matrix
(b) Correlation
Figure 9: (a) Example "readout" matrix on the MSD problem for all time steps & at all times highlighting WARP's diagonal decoding direction (7)r: (b) Correlation between the network's weights 0g and the time on the MSD problem; indicating strong linear dependence between the two.
E.5 SPECTRAL ANALYSIS
To understand the dynamic properties and memory mechanisms learned by our model on the MSD task, we conduct a spectral analysis of its state transition matrix A. This analysis crucial for visualizing how the network learns to retain information over long time horizons. We looking for cigenvalues clustered near the unit circle (AAl = 1), as this indicates capacity for long-term memory without vanishing exploding gradients. The analysis in Fig. 10 reveals the model successfully learns to preserve long-term dependencies by maintaining the vast majority its eigenvalues directly on the unit circle. The minor spill-over (|>l > is effectively managed by gradient clipping during training (see Appendix B.3).
E.6 ROBUSTNESS TO NOISE
We evaluate robustness on the MSD dataset by corrupting the input trajectories with increasing levels of Gaussian noise 7), rescaled such that y corresponds to standard deviation of 39 (the maximum absolute value encountered in the dataset). We employ four metrics to compare the generated trajectories to their noisy ground truths: MSE, RMSE, MAE, and MAPE, as defined in Appendix C.3. The results in Fig. reveal that WARP exhibits robust performance from minuscule (n 10-1) up to moderate noise levels (10-1 1 10*), after which all error metries increase
33

---

Published as a conference paper at ICLR 2026
( Elgesvalce Spectrum
Transition
10
1.00
Unit Ciosle
Weight Dimentzion
1000
Imaegnary Part [m.))
2000
-10
0.00
3000
00.25
0.50
4000
-0.75
1000
2000
-10
-1.00-
Weight Dimenston
11.0
-0.5
0.0
0.5
1.0
Real Part Re(a)
Histogram Bigenvalue Magnitudes
(d) Digenmode Analesi (Prequincy Deary)
1400
Unit Circk
1.0
1200
Megnitudo Decesy liste
0.8
1000
800
0.6
0.4
400
0.2
Unit Circle ()2)
0.0
Magnitude
Angle Frequency 4A (radians)
Figure 10: Spectral analysis of the learned state transition matrix A on the MSD task. In both cases, the model successfully learns to place eigenvalues on the unit circle to achieve long-term memory, as observed in subplots (b), (c) and (d). For the visualization in (a), the colormap is intentionally saturated at a low absolute value of 10-3, This is necessary because the diagonal elements (== 1.0) several orders of magnitude larger than the learned off off-diagonal couplings.
10
MSE RMSE
104
Error
102
306
10
10 2
10
10
10
10
10
10
10a
10a
104
Noise Level n (std dev of Gaussian noise)
Figure 11: Evaluating performance in noisy scenarios. These results illustrate WARP's robustness to
increasing levels of Gaussian noise on the MSD dataset, as measured by several metrics.
sharply by several orders of magnitude, indicating critical threshold beyond which the model fails catastrophically.
E.7
ABLATION STUDIES
We briefly discuss several experiments carried out to gain insights into model. For all ablation studies, experimental protocols like training hyperparameters are presented in Appendix D. Figures and Tables in this section captioned with the corresponding paragraph title.
34

---

Published as a conference paper at ICLR 2026
1.00
Neural ODE
0.10
MAPE(%)
Waight dimeessio
MSE
1000
0.01
0.00
10
100 Samples
1k
10k
# Samples
10
100 1k
10k
500
Weight dimenston
1005
1602
2003
2001
(a) SINE
(b) ETTml
Figure 13: (a) Sample efficiency on SINE. (b) dense weights-to-weist matrix on ETTml.
Eliminating the root network. The root network 0g is integral to the efficacy of WARP Although not absolute prerequisite for the WARP- variant, nonetheless persists as pivotal constituent of our framework. Illustrativel,, the omission of 0, in favour of directly fitting yp for the SINE modelling problem results in catastrophic degradation in model expressivity.
105
10
10-4
100
10k
Samples
Figure 12: Eliminating the root network - Test-set MSEs on the SINE problem. The omission of 0, in favor of directly fitting &o (which call WARP-Phys') results in catastrophic degradation in model expressivity. Performance almost as bad as the Neural ODE analysed in Fig. 13(a).
Initial network configuration. Since WARP's weight trajectory is driven by the changes in the signal and not the signal itself (see Eq. (1)), is important to have an expressive initial hypernetwork Xy Og. which embeds the initial tokens into suitable weight spaces [52]. Our empirical investigations reveal that sidestepping this component substantially curtails the model performance on complex synthetic benchmarks, such as MSD-Zero, and on real-world datasets, including ETT.
Table 15: Initial network configuration - Empirical investigations reveal that sidestepping ø in favor of directly learning 6 curtails the model performance on complex synthetic benchmarks, such as MSD-Zero, and on real-world datasets, including ETTml.
PROBLEM
WITH ø
WITH 0g
MSD-Zero
0.32
1.02
ETTml
0.02
1.25
Data efficiency. With L - 1, the SINE benchmark is challenging initial value problem. At equal (root) neural network parameter counts, we vary the number of training samples, and we plot MSE and MAPE test metrics for WARP and the Neural ODE [18]. The results, depicted in Fig. 13(a), not only show improved performance across data regimes, but they also indicate that more data is not necessarily better for WARP's performance, suggesting potential for monotone learning [13].
35

---

Published as a conference paper at ICLR 2026
Dense state transitions & Channel mixing. The total parameter count of our model is quadratic in the root network's 's dimensionality Dg. Specifically, attempts to replace E R Pe Da with diagonal or low-rank approximations have resulted in remarkably less expressive models, thus solidifying its dense nature, as illustrated in Fig. 13(b), as a key component of our framework.
Untrained
Trained
0-
0.0030
0020
-0 0.0025
4-
0.0020
8-
000015
-0.0015
00,0100
10
-0.0010
12-
00,000
12-
-0.0005
14
co0n
1000
Dense state transitions & Channel mixing Attempts to replace E Dex Do with either diagonal or a low-rank approximation Á result in less expressive models. We observe here a low-rank À E 166 16 on the ETTml problem, such that = PÂQ, with all quantities in the right-hand side learnable.
Figure 14:
GT
Recons
Uncerteinty
GT
Recons
Uncertainty
3
3
6
6
6
6
7
2
7
7
(a) Fixed 7 = 0.5
(b) Variable t/(T - 1)
Figure 15: Root network evaluation - When using normalised time as the coordinate system, if we fix the evaluation point 7, we observe mild degradation in the qualitative results. While these figures are shown for MNIST with L = 300, the behaviour is observed across problems, including dynamical systems like MSD (see Fig. 9()). GT stands for the Ground Truth, Recons for the Reconstuutton//oomlltiinn and Uncertainty is the model-outputted standard deviation.
Table 16: Positional Encodings (PE) ablation We report the classification accuracy (%) WARP with and without PE on the UEA datasets. The results show consistent performance drop when PE removed, underscoring its importance for long-range dependencies such as Worms and Motor.
Worms
SCP1
SCP2
Ethanol
Heartbeat
Motor
with PE
70.93 + 2.7
83.53 + 2.0
57.89 1.4
32.91 + 4.2
88.65 + 1.9
56. + 5.1
w/o PE
60.98 = 3.1
80.00 + 2.0
57.89 + 1.5
31.65 + 0.8
77.42 + 2.2
50.88 + 2.3
36

---

Published as a conference paper at ICLR 2026
1.0
W mma
0.5
0.0
Ground
-0.5
Infecence Stort
32
64
96
128
160
192
Time Step
Figure 16: Ablation of the reparametrisatioon trick On the electricity problems, if stochastic sampling during training is not used, the model only predicts the mean of the distribution, thereby ignoring high-frequency components or noise in the signal. Here, this is illustrated with prediction on the ETTml test split.
37

---

Published as a conference paper at ICLR 2026
F VISUALISATIONS
Reconstruction using 100 initial steps
CF
Chaarteay
01
Recom
Uscertics
Becoes
Uncertatoo
OF
Reous
Uacamaizty
2
2
7
6
7
/
9
8
1
8
5
8
6
6
8
6
6
6
Reconstruction using 300 initial steps
Reooss
Recons
Uncortent-
of
Recoas
Recors
7
7
0
/
9
9
1
1
5
6
6
6
2
6
6
Reconstruction using 600 initial steps
02
Recess
Uharteay
07
Recots
Uncertastt
ot
Becoms
CT
Recots
2
2
f
7
2
3
7
7
0
O
/
/
9
9
1
1
7
5
5
6
6
6
6
Figure 17: Completed images from the MNIST test using WARP The same of images is
shown across three settings: (Top) L = 100, (Middle) L = 300, (Bottom) L 600. Along the columns, we show groups of results, each with Ground Truth (GT). Reconstruction (Recons), and Uncertainty. resulting in 12 total columns. As our model sees more steps, its forecasting improves and uncertainty decreases.
Reconstruction using 300 initial steps
Recnre
Becane
Unsestrists
CTT
Torens
CI
Recure
Figure 18: Completed images from the Fashion MNIST test using WARP.
38

---

Published as a conference paper at ICLR 2026
Reconstruction using 100 initial steps
Seteas
Dnortaan
Recoms
Uncertaicty
or
Burces
Recuts
Dasertunt
Reconstruction using 300 initial steps
Reaone
Raccns
Reconstruction using 600 initial steps
Figure Completed images from the CelebA test set using WARP at various context lengths.
Reconstruction using 2150 initial steps
Figure 20: Completed images on the CelebA test set at high-resolution (T = 64 x 64 4096), using
positional encoding [92]. This illustrates WARP's suitability for long-range dependencies.
39

---

Published as a conference paper at ICLR 2026
Reconstruction using 100 initial steps Becons
Recons
Of
Rocons
Roceba
Figure 21: Completed sequences from the MSD test using WARP
Reconstruction using 256 initial steps
Figure 22: Completed sequences from the LV test set using WARP.
Reconstruction using 96 initial steps
Reaura
Disersber
Figure 23: Completed time series from the ETTml using WARP
40


Reviews:
[{"review_id": "lmMAYKkBmE", "reviewer": "Reviewer_T7rz", "summary": "The paper proposes **WARP**, a framework that performs *linear recurrence in weight space* rather than in hidden-state space. At each step, the parameters of a small decoder network evolve linearly:\n\n$$\\omega_t = A \\omega_{t-1} + B(x_t - x_{t-1}), \\qquad y_t = \\text{MLP}_{\\omega_t}(\\varepsilon)$$\n\nwhere \\(A, B\\) are learned transition matrices and $\\varepsilon$ encodes position or context.  \n\nConceptually, this shifts recurrence from feature dynamics to parameter dynamics. It blends (i) the efficiency of linear RNNs/SSMs (e.g., S4, Mamba), (ii) the expressivity of nonlinear decoders, and (iii) in-context or gradient-free adaptation through weight evolution. The model is tested across image completion, time-series forecasting, dynamical-system reconstruction, and classification.", "questions": "• Include direct comparisons to LinOSS, FACTS, and Griffin.\n\n• Explore structured or low-rank A, B for scale.\n\n• Add runtime and memory tables.\n\n• Include a brief stability/spectral analysis.\n\n• Clarify what fundamentally distinguishes this from hypernetworks and fast-weight RNNs.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. Original framing. The move to perform recurrence directly in parameter space is novel\nand quite elegant. It reads as a middle ground between hypernetworks and fast-weight\nRNNs, but with the analytical simplicity of a linear transition.\n\nS2. Range of results. The experiments span diverse domains - MNIST/CelebA completion,\nETT and PEMS forecasting, DSR, and UEA time-series classification. The UEA section\nis particularly strong: comparisons include modern SSMs like S5, Mamba, S6, NRDE,\nand NCDE, with WARP performing competitively across most tasks.\n\nS3. Interpretability and analogy. The weight updates via input differences evoke synaptic-\nplasticity rules, which gives the method a neat biological parallel and some explanatory\nappeal.\n\nS4. Presentation. The paper is clear, visually well-organized, and balances theory with\nintuition. Figures showing progressive reconstruction genuinely help convey how the recurrence behaves.", "weaknesses": "W1. Benchmark depth. While broad, the benchmark is missing some of the newer SSMs\nthat define the current frontier. In particular, LinOSS (Rusch & Rus, 2024)—an oscilla-\ntory, long-sequence SSM—is cited but not compared. Given that LinOSS, FACTS, and\nGriffin all outperform S4 and Mamba on long forecasting tasks, excluding them makes\nthe SoTA claim weaker.\n\nW2. Scalability. The transition matrix $A \\in \\mathbb{R}^{D_\\omega \\times D_\\omega}$ scales quadratically with the size of the decoder, which will quickly become impractical. No structured or low-rank variants are\nexplored.\n\nW3. Theory gap. The paper is mostly empirical. There’s no discussion of spectral properties,\nstability, or representational capacity of the linear map in weight space.\n\nW4. Domain imbalance. Some domains (especially physics and image experiments) use\nsmall or older baselines (ConvCNP, GRU, Transformer). More recent adaptive or physics-\ninformed baselines like Neural Context Flows (ICLR 2025) or ZEBRA (2024) would\nstrengthen those sections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NuwNC5fYaL", "reviewer": "Reviewer_jyKY", "summary": "This paper analyzes recurrent neural networks (RNNs) through a weight-space linear recurrence formulation that unifies several modern architectures — including continuous-time linear RNNs, state-space models (SSMs), and residual recurrent networks — under a single linear operator perspective.\n\nThe authors derive closed-form expressions for training dynamics and generalization in the overparameterized limit, showing that convergence properties and implicit regularization can be understood via the spectral structure of the recurrent Jacobian. They provide:\n\nA linearized weight-space recurrence model that approximates nonlinear dynamics by a low-rank operator with analytically tractable behavior.\n\nA demonstration that generalization error scales with spectral conditioning, extending kernel-based intuition from linear networks to recurrent architectures.\n\nEmpirical validation on synthetic sequence modeling and dynamical-system reconstruction tasks (mass–spring–damper, Lotka–Volterra, PEMS08 traffic), showing alignment between predicted and observed convergence trends.\n\nOverall, the paper contributes to a principled understanding of how weight-space geometry and recurrence interact to shape training efficiency and generalization.", "questions": "1. How sensitive are your analytical predictions to non-normal dynamics (upper-triangular Hₕ) versus the diagonal “normal” case you ultimately focus on?\n\n2. Could your framework accommodate nonlinear activation perturbations (e.g., ReLU linearization) to estimate when linear approximation fails?\n\n3. Have you compared your spectral regularization predictions to empirical spectral shrinkage observed during training (e.g., spectrum compression in Wh)?\n\n4. Could you clarify how your results relate to implicit bias analyses in linear transformers or SSMs (e.g., Merrill & Sabharwal; Orvieto et al.)?\n\n5. Are there regimes where the recurrence’s spectral radius predicts too-rapid forgetting or instability, contradicting observed dynamics?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper builds on well-established analyses of linear networks and extends them naturally to recurrent settings using a spectral-decomposition framework (Schur- and SVD-based). Derivations are internally consistent and clearly documented.\n\nThe proposed linear recurrence view elegantly bridges RNNs, residual-RNNs, and diagonal SSMs, helping clarify connections between recent model families.\n\nExperiments across several dynamical-system tasks (MSD, LV, traffic flow forecasting) confirm the predicted dependence of training speed and generalization on spectral conditioning and effective recurrence length.\n\nThe inclusion of both synthetic physics systems and real-world time-series (PEMS08) demonstrates breadth and internal consistency.\n\nMathematical exposition is detailed; hyperparameters and architectures are listed (Appendix D.4–D.6). Code release and ablation details are promised.", "weaknesses": "1.  The analytic results rest on linear, Gaussian assumptions; nonlinear recurrence effects and gating dynamics are only discussed qualitatively. As such, predictive power for modern gated RNNs or structured SSMs is limited.\n\n2. The analysis centers on the infinite-width, overparameterized limit; it does not quantify where the asymptotic predictions break down for finite models.\n\n3. The authors reference Saxe et al. (2014) but omit more recent theoretical works on curriculum and transfer in RNNs (e.g., Rajan, Kepple & Engleken) and on gradient-flow analyses in recurrent kernels — literature directly related to their spectral-mode interpretation.\n\n4. Although the experiments match qualitative trends, they serve mainly as demonstrations rather than quantitative tests (e.g., no variance or uncertainty estimates, small sample sizes).\n\n5. Generalization is assessed by mean-squared error only; tasks with stochastic noise or long-term dependency tests (copy, addition, character-level modeling) would strengthen claims about recurrence depth and spectral bias.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GyOp2eDthB", "reviewer": "Reviewer_rGWP", "summary": "This paper offers a new weight-space learning technique that iterates weights of a feedforward neural network in a linear state space, enabling in-context learning. The authors evaluate WARP across diverse tasks including image completion, time series forecasting and classification, and dynamical system reconstruction, demonstrating competitive or superior performance compared to standard RNNs, state-space models (S4, Mamba), and Transformers. Notably, a physics-informed variant achieves order-of-magnitude improvements on physical system reconstruction benchmarks.", "questions": "“Rather than relying on direct inputs, we draw inspiration from the human brain and compute signal differences to drive such recurrences.” - from where is the inspiration drawn? Did you try ablating this, i.e. just using x_t as the input to B? Similarily, did you try ablating the random noise applied to observations in AR mode? i.e. p_forcing = 0 (or 1, whatever means noise is never added)\n\nWhat is the operational difference between gradient-free adaptation and in-context learning? The definitions provided seem to be nested on page 2 (i.e. in-context learning implies gradient-free adaptation).\n\n\"This strategic initialisation also imposes a critical constraint wherein the initial hidden state θ_0 must encode semantically rich information applicable to the entire sequence.\" Could you give some clarity here - how can the weights encode information about a sequence prior to observing it?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The core idea of parametrising RNN hidden states as weights of an auxiliary neural network is conceptually interesting and, to the best of my knowledge, novel. The authors test their method on a diverse set of domains and perform a large range of ablations to show the necessity of design choices. The writing is generally accessible, with good intuitive explanations, and goes far to place itself in the larger test-time adaptation literature.", "weaknesses": "Claims are sometimes overstated and/or imprecise. For example, phrases like \"transformative paradigm for adaptive machine intelligence\" (Abstract, Conclusion) and \"redefine sequence modeling\" (Abstract) are not well-supported. The empirical results show WARP is competitive but not uniformly superior. \"Brain-inspired formulation\" (Abstract, page 2) refers only to using input differences, with citation to synaptic plasticity [16], but the connection is somewhat superficial - there is a rich literature concerned with modelling realistic synaptic plasticity rules which this approach to weight-space trajectory modelling does not engage with. \"Infinite-dimensional RNN hidden states\" (page 9, footnote 6) is misleading—the hidden state is finite-dimensional, though it parametrizes a function.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K7gWau6xml", "reviewer": "Reviewer_3Pnu", "summary": "This paper proposes WARP, a novel class of recurrent neural networks (RNNs) that perform sequence modeling directly in weight-space, blending linear recurrence with non-linear decoding. Unlike standard RNNs, which maintain a hidden state that is a result of propagating the sequence through the network, the proposed model’s hidden state is instead equal to the parameter vector of a so called auxiliary (“root”) MLP and is updated over time via a linear recurrence. Each update is driven by consecutive input differences, and the auxiliary MLP provides nonlinear decoding using an input that encodes the canonical ordering of the sequence.  This formulation enables gradient-free adaptation and in-context learning during inference, as well as the injection of physics priors through explicit parameter constraints.\nExtensive experiments are conducted across time-series analysis, dynamical system reconstruction, and multivariate time-series classification. WARP shows consistent or superior performance to state-of-the-art baselines on most tasks. A physics-informed variant, WARP-Phys, achieves significant improvements on physical dynamics reconstruction tasks.", "questions": "1. Have you evaluated WARP’s performance on noisy measurement scenarios for dynamical system reconstruction?\n2. Could you discuss related sequence modeling approaches in weight space and explain why these were not included as baselines in any of the tasks?\n3. For the Energy Prediction experiment: what are the current state-of-the-art results? In general, for all experiments, how were baselines tuned, and how much effort was spent on hyperparameter optimization? Also, performance of S4 is reported only on MNIST, and ConvCNP only on CelebA. Full performance tables in the appendix would improve transparency.\n4. Have you considered benchmarking WARP on long-range dependency benchmarks, such as Long Range Arena (LRA)? This could contextualize WARP’s capabilities relative to other established long-sequence models that were evaluated on these tasks.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Novel conceptual framing:  The idea of treating the recurrent hidden state of a linear state-space model as the weights of another neural network is both elegant and novel. It bridges ideas from fast weights, meta-learning, hypernetworks and structured state-space models while maintaining linear recurrence efficiency. Additionally, it offers a built-in support for gradient-free adaptation, in-context learning, and physics-informed modeling in a single architecture.\n2. Computational Efficiency: Once the model has learned from the context, the final root network can be extracted and reused to process subsequent queries without reevaluating the entire sequence, yielding significant computational savings compared to other in-context learning models. Furthermore, the proposed architecture leverages a dual training mode that combines linear recurrence with a parallel scan operator – a well-established technique in the State Space Model (SSM) literature – to accelerate state propagation. Together, these design choices lead to notable computational efficiency improvements.\n3. Strong empirical results: Competitive or superior performance on time-series analysis, especially Traffic Flow Forecasting (despite ignoring graph priors) and Image completion, as well as dynamical system reconstruction. The inclusion of a physics-informed variant further demonstrates the framework’s adaptability and potential for interpretability.\n4. Clarity and completeness:  The paper is well-written, includes high-quality figures, ablations, detailed appendices, and clear pseudocode.", "weaknesses": "1. Scalability constraints:  The main bottleneck is the large transition matrix which scales quadratically with the number of root-network parameters. Experiments are thus limited to moderate model sizes, raising questions about feasibility for large-scale models.\n2. Limited theoretical grounding:  While the empirical evidence is compelling, the theoretical analysis of representational capacity and stability (e.g., under linear recurrence updates) still remains to be established. \n3. Computational cost reporting:  Although the recurrence is linear, updating and decoding weight vectors remains costly. Memory and compute scaling with model size are not fully quantified and are only provided in the appendix, but entirely missing from the main body of the paper.\n4. Limited setting for dynamical system reconstruction:  While the possibility of making the network physics-informed is compelling, the shown examples illustrate this for relatively simple systems with a small number of parameters. While the proposed method clearly allows for in-context learning, and hence does not need to retrain a network for each new dynamical system (from the same category), the setups are done for what appears to be noiseless input-output data, and a low number of parameters. One could, instead of learning the entire mapping of the system, learn only its phase or exponential mapping for any other sequence model that allows for ICL, in the same way as demonstrated in this work. This alternative formulation would serve as a fairer baseline for comparison.\n5. Novelty relative to prior work:  There is conceptual overlap with other concepts briefly outlined in the related work (e.g., fast weight RNNs), though the authors’ framing is distinctive. A more explicit comparison to those baselines would help to better position WARP’s contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MIiGSYD6YS", "reviewer": "Reviewer_rzHm", "summary": "The authors seek to incorporate domain priors and in-context reasoning into linear RNNs.\nThey show that (i) linear RNNs can act as effective hypernetworks, (ii) these RNNs can engage in in-context learning or physics-informed modeling, and (iii) these hypernetworks exhibit significant improvements over baseline models.\nModels are evaluated on forecasting / reconstruction, classification, and in-context linear regression tasks.\nBaseline models include gated RNNs, linear SSMs (S4), and transformers.", "questions": "I suggest the following:\n- Traditional long-range dependency experiments like Long-Range Arena benchmarks.\n- Text prediction and classification experiments to support claims of expressivity.\n- Connections to biology, especially in the main text, should be clarified.\n- As stated by the authors, future work should explore ways to reduce the density of hidden-state transition matrices used in WARP. \nComplex diagonal recurrences may be fruitful in this regard.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Fantastically written, very clear.\n- Related work and appendix are thorough.\n- Experiments show clear improvements of the proposed WARP RNN over baseline models, both in accuracy and compute.\n- Authors are clear about limitations, especially the important limitation that the hidden state transition matrix is dense.", "weaknesses": "- The introduction does not (although the related work does) mention selective SSMs (e.g., Mamba), which have re-introduced nonlinearities into linear RNNs.\n- CelebA reconstruction seems a bit contrived as a task: the annotated S4 article referenced does indeed reconstruct MNIST and similar image data, but then proceeds to spoken digit data, which the authors did not try reconstructing.\nMoreover, CelebA face reconstructions are substantially corrupted.\nWhile I do not deny WARP's improvement in image reconstruction over baseline, I am unsure about the salience of image reconstruction as a metric for evaluating RNNs.\nI think there are more relevant tasks to measure long-range dependencies, such as Long-Range Arena benchmarks.\n- There are no experiments on text prediction or classification, which are some of the most relevant tasks for evaluating new RNN architectures - does WARP have limitations or inductive biases that prevent it from expressively processing text?\nIf so, the authors should state it, as this is an important limitation, especially to claims of in-context learning.\nIf not, then the authors should evaluate their model with text to support the expressivity of WARP.\n- WARP's connections to biology are not sufficiently explained in the main text, although the authors do discuss it further in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)