Transformers as Implicit State Estimators: In-Context Learning in Dynamical Systems
Authors: Usman Akram
Haris Vikalo
Venue: Accepted by TMLR ()

Abstract: Predicting the behavior of a dynamical system from noisy observations of its past outputs is a classical problem encountered across engineering and science. For linear systems with Gaussian inputs, the Kalman filter -- the best linear minimum mean-square error estimator of the state trajectory -- is optimal in the Bayesian sense. For nonlinear systems, Bayesian filtering is typically approached using suboptimal heuristics such as the Extended Kalman Filter (EKF), or numerical methods such as particle filtering (PF). In this work, we show that transformers, employed in an in-context learning (ICL) setting, can implicitly infer hidden states in order to predict the outputs of a wide family of dynamical systems, without test-time gradient updates or explicit knowledge of the system model. Specifically, when provided with a short context of past input–output pairs and, optionally, system parameters, a frozen transformer accurately predicts the current output. In linear-Gaussian regimes, its predictions closely match those of the Kalman filter; in nonlinear regimes, its performance approaches that of EKF and PF. Moreover, prediction accuracy degrades gracefully when key parameters, such as the state-transition matrix, are withheld from the context, demonstrating robustness and implicit parameter inference. These findings suggest that transformer in-context learning provides a flexible, non-parametric alternative for output prediction in dynamical systems, grounded in implicit latent-state estimation.

Published in Transactions on Machine Learning Research (02/2026)
Transformers as Implicit State Estimators: In-Context Learning in Dynamical Systems
Usman Akram
wwm.n.dkrom&ouuciinnueee.e..
Deportment of Electrical and Computer Engineering The University of Teaas at Austin
Haris Vikalo
Cenkaao@@ce.utezzaue edu
Department of Electrical and Computer Engincering The University of Texas at Austin
Reviewed on OpenReview: https: // openreveew net/ pet/ffrum? id= MIMKMMKGUG
Abstract
Prodicting the behavior of dynamical system from noisy observations of its past outputs classical problem encountered across engineering and science. For linear systems with Gaussian inputs, the Kalman filter the best linear minimum mean-square error estimator of the state trajectory optimal in the Bayesian sense. For nonlinear systems, Bayesian filtering is typically approached using suboptimal heuristics such the Extended Kalman Filter (EKF), numerical methods such as particle filtering (PF). In this work, we show that transformers, employed in an in-context Icarning (ICL) setting, can implicitly infer hidden states in order to predict the oulputs of a wide family of dynamical systems, without test-time gradient updates or explicit knowledge of the system model. Specifically, when provided with a short context of past input-output pairs and, optionally, system parameters, frozen transformer accurately predicts the current output. In linear-Gaussian regimes, its predictions closely match those of the Kalman filter; in nonlinear regimes, its performance approaches that of EKF and PF. Moreover, prediction accuracy degrades gracefully when key parameters, such as the state-transition matrix, are withheld from the context, demonstrating robustness and implicit parameter inference. These findings suggest that transformer in- context learning provides a flexible, non-parametric alternative for output prediction in dynamical systems, grounded in implicit latent-state estimation.
1 Introduction
In-context learning (ICL), particularly in the form of few-shot prompting (Ygatama et al. 2019 has emerged as prominent research direction in natural language processing (NTP). In this framework, large language model (LLM) performs new tasks by conditioning on a small number of input output examples provided in the prompt. One of the carliest works to demonstrate this capability was by Brown et al. (2020). who evaluated GPT-3 across a range NTP datasets under "zero-shot", "one-shot' and "few-shot" settings. Zhao et al. (221)) identified majority-label bias, recency bias, and common-token bias as sources of instability GPT-3's accuracy under few-shot prompting, and proposed contextual calibration a remedy. A theoretical
explanation for ICL as form of implicit Bayesian inference is offered in Xie et al. (2021). Min al. (2022 explore why ICL works in practice, showing that its success is not dependent on having access to ground-truth labels; it further emphasized the importance of label space, input distribution, and prompt format. In parallel, [chllag et al. (2021) provide a theoretical analysis showing that transformers can act as fast weight programmers.
Early work on using standard transformer decoders for in-context learning of autoregressive models includes (Garg et al. 2022), where the ability of transformers to learn function classes from examples is empirically

---

Published in Transactions on Machine Learning Research (02/2026)
investigated. A model is said to learn function class F over a domain X' if, for any E F and any inputs 21, 22, ,N, qqueryy drawn i.i.d. from X, the model can predict (Xquery) given the sequence T1, f(r1), f(z2)..... IN Tquer-- The function classes studied by Garg et al. (2022) range from simple linear functions to sparse linear models, two-layer neural networks, and decision trees.
Tquery-
Two parallel works by Von Oswald et al. (2023) and Akyürek et al. (0023) examine which algorithms transformers implicitly implement when learning function classes in-context. Von Oswald et al. (2023) build on Schlag et al. (2021) and show that linear self-attention layers can be interpreted as performing step of gradient descent. Specifically, they prove that for a single-headed linear attention layer, there exist key, query, and value matricos such that forward pass corresponds to one step of gradient descent with an &2 loss applied to each token. In contrast, Akyürek et al. (2023) introduce a raw operator framework that supports operations such as matrix multiplication, scalar division, and memory-like read/write. They demonstrate that a single transformer head equipped with suitable key, query, and value matrices can approximate this operator, implying that transformers are, in principle, capable of performing lincar regression via cither stochastic gradient descent or in closed-form.
Our work investigates whether transformers, using in-context learning (ICL), can learn to perform filtering in dynamical systems described by noisy state-space models. Specifically, we study whether a transformer, conditioned on a short context of past the current output of the system. The model is pre-trained on synthetic trajectories generated by randomly sampled system parameters and evaluated without gradient- -based updates at test time. Our goal is to understand which classical filtering algorithms the transformer's behavior most closely resembles under this setup. To that end, begin by examining the structure of the Kalman filter (Kalman 1960), the optimal linear estimator for systems with Gaussian noise, and ask whether its operations can be replicated by a transformer architecture. We show that Kalman filtering can indeed be expressed using operations that are readily implemented by a transformer, and provide both a proof-by ber--byyyooottttiooo and empirical evidence that, when sufficiently scaled and provided with enough context, the transformer learns to emulate Kalman-like behavior. Notably, its performance remains strong even when some parameters, such as the stale-transition matrix, are omitted from the context, suggesting robustness and an ability to infer missing information implicitly. We then turn our attention to nonlinear systems and empirically demonstrate that transformers can in-context learn to achieve output prediction accuracy comparable to that of classical nonlinear filters such as the Extended Kalman Filter and particle filters. While our analysis and experiments primarily focus on output prediction, we also include preliminary evidence that transformers can, in some cases, explicitly recover latent states, thus highlighting their potential for more general inference tasks.
pairs (and, optionally, system parameters), can predict
Scope and framing. This paper focuses on the empirical investigation of whether transformers trained via in-context learning can perform inference in dynamical systems. In particular, our goal is to examine whether standard transformers, when trained on synthetic input-output trajectories from randomly generated systems, can learn to perform filtering tasks without explicit model supervision or architectural modifications. The study is guided by constructive arguments about the representational capacity of transformers. While we do not offer convergence guarantees, our results show that with sufficient size and context, transformers can approximate Kalman-like behavior in lincar settings and extend nonlinear regimes, sometimes outperforming classical filters. Performance may degrade under limited model capacity or severely reduced context, discussed in Section 4.
1.1 Related work
Prior research at the interface of deep learning and Kalman filtering include Deep Kalman Filters (Krishnan et al. 2015) and KalmanNet (Revach et al.) 2021), which learns the Kalman gain in a data-driven manner using a recurrent neural network. A follow-up study (Revach et al. 2022) extends this approach using gated recurrent units lo estimate both the Kalman gain and noise statistics while assuming fixed system parameters. In contrast, our model is trained on data generated from randomly sampled system parameters, encouraging the transformer to learn the filtering procedure itself rather than memorize input-output mappings for a specific system. Other works have explored connections between attention mechanisms and structured state-space models (SSMs). Dao & Gul 2024) reformulate SSM computations as matrix multiplications on
2

---

Published in Transactions on Machine Learning Research (02/2026)
structured matrices, drawing parallels with efficient attention variants. Sieber al. 2024 propose unified dynamical systems framework encompassing attention, SSMs, RNNss, and T.T However, neither of these works address state estimation, filtering, or in-context learning. (ooel & Bartlett 2024) show that softmax self-attention can approximate the Nadaraya- Watson kernel smoother, which turn resembles the Kalman filter. In contrast, our work directly largels in-context learning for dynamical systems and builds on the raw operator framework of Akyürek al. (2023) to show that transformers can implement the exact operations needed for Kalman filtering, as supported by both theoretical constructions and empirical validation.
1.2 Summary of contributions
We present, to our knowledge, the first study demonstrating that transformer, pre-trained on trajectories generated from randomly sampled system parameters, can in-context learn to perform filtering in dynamical systems. Specifically, show that frozen transformer, when conditioned on short context of input outttttuuuutt pairs (and, optionally, system parameters). can predict the current output without test-time updates or direct model supervision. Our contributions are as follows:
We provide a proof-by-constuue by-construction showing that the Kalman filter can be reformulated using operations readily implementable by transformer. Using the mean squared prodiction difference (MSPD), empirically demonstrate that transformer can in-context learn to emulate the Kalman filter tailored to individual systems.
Beyond linear systems, we show that transformers can in-context learn to perform accurate output prediction in certain nonlinear dynamical systems. This includes a challenging maneuvering target tracking task with unknown turning rate, where the performance is comparable to that of the Extended Kalman Filter and particle filtering.
We evaluate robustness by withholding portions of the system model from the prompt. Notably, in the absence of the state-transition matrix, the transformer approximates the operations and predictive accuracy of the Dual Kalman filter, demonstrating implicit parameter inference and context-level adaplability.
We observe that transformer behavior depends on scale: Small models and short contexts tend to emulate classical regression methods (e-g., SGD, Ridge, OLS), which do not involve latent state inference, while larger models and longer contexts cxhibit filtering behavior that suggests implicit recovery hidden states, approximating Kalman, Extended Kalman, and particle filters.
The remainder of the paper is organized as follows. Scction 2 learning and filtering. Section 3 introduces the system model and presents constructive arguments showing that transformers in-context learn to implement Kalman filtering under white observation noise. Section 4 presents simulation results, including applications to non-lincar systems and robustness to missing model parameters. Section 5 concludes the paper. The code used to generate experimental results is available here
reviews relevant background on in-context
2 Background
2.1 Transformers
Transformers, introduced by Vaswani al. 2017), are a class of neural network architectures that rely on an attention mechanism to capture relationships among elements in an input sequence. This mechanism enables transformers to model long-range dependencies and has been central their success sequence- to-sequeeee tasks. The experiments in this paper utilize the GPT-2 architecture [Radford et all 2019), decoder-only variant of the transformer.
A brief overview of the attention mechanism sets the stage for the discussion that follows. Let G(I-1) denote the imput to the 1h layer. single attention head, indexed by 7, consists of key, query, and value matrices denoted by WK WQ, and W.Y. respectively. The output of head y is computed as
b4 Softmax (W (W ((-((()))) ((()))
(1)
3

---

Published in Transactions on Machine Learning Research (02/2026)
The softmax matrix in equation assigns attention weights that indicate how strongly each token attends to others in the sequence. The outputs of all B heads are concatenated and projected via WP, yielding
A' = W WTbp 16 by be,...., l/l.
(2)
This result is added to the layer imput and passed through feodforward block to produce the output of the thh layer
G = W1o (W22 (A' -(--)) A' + G(--1)
where À denotes layer normalization and a is the activation function. In our experiments, we use the Gaussian Error Linear Unit (GcLU) activation (Hendrycks & Gimpel 2016
2.2 State-space models for linear dynamical systems
Linear dynamical systems can be described by a finite-dimensional state-space model involving hidden states 2t E I and observations y/ E Rm" related by the equations
T++1 Fixz + q:
(4)
y1 H + Tt.
(5)
where E Rn and E RR denote zero-mean white process and measurement noise, respectively (for simplicity, we assume no external control input u.). The state equation (4), parameterized by the state transition matrix F. E R and the noise covariance matrix Q, models the lemporal evolution of the latent state. The measurement equation (5), parameterized by the measurement matrix Hy E R and the noise covariance R, defines how observations are generated from the underlying state via linear transformation.
State-space models are widely used across machine learning (Gu et al. 2021 computational neuroscience Barbieri et al. 2004), control theory (Kailath 1980). signal processing Kailath et al., 2000), and economics Zeng & Wu 2013). A central problem in many of these domains is the estimation of the latent state sequence 2y and its functions (e.g., the system's output) given noisy observations z/r and the parameters the model.
2.3 In-context learning in the absence of dynamics
When F- Inxx Q = 0, H = h4 E 1x (scalar measurements), and zo x, the state space model simplifies to
Ft 3
(6)
= h4 + Tt:
(7)
i.e. the state becomes time-invariant and the system reduces to linear measurement model. At the crux of estimation problems in this setting is the inference of the unknown random vector ø from past measurements Vn, .2, .., UN and the corresponding measurements vectors b hy hy,..,............... ......., hy. This problem can be addressed using several classical methods:
Stochastic Gradient Descent. Initialize with &o = 0nx1, and update recursively as
f = - - 2a(hī_ AT-
(8)
where denotes the learning rate. Once a pre-specified convergence criterion is met, the final estimate is set to $ fN
Ordinary Least Squares (OLS). Form the matrix N Nxx with rows hy, h2, ... JN, and let Y= [31, 12 "N The OLS estimate is
Sols= = (
(9)
4

---

Published in Transactions on Machine Learning Research (02/2026)
Ridge Regression. To reduce overfitting, OLS can be regularized
i Ridge = (1I H + A
(10)
where denotes the regularization parameter.
Note that if x= = f. where o² is the variance of the measurement noise and 7 is the variance of the prior on the latent vector Zo = z, then ridge regression yields the lowest mean square error among all linear estimators of v, iee, those that linearly combine measurements y1,... VN form L. Moreover, xo ~ NN( (0,,) and Tr o(,,02)) then the ridge regression solution coincides with the minimum mean square error (MMSE) estimate, ie., f -[x
A pioneering study that explored the ability of language models to learn linear functions and implement simple algorithms was presented by Garg ct al. 2022 Building on that work, Akyürek work.ckkyürck et al. (0023) examined whether GPT-2 based transformers can in-context learn the setting where 2; = x as in ( Specifically, they investigated which algorithms the transformer implicitly learns to implement when tasked with predicting yN, given an input formatted as the matrix
0 1f
11
9/2 O
UN-1
N
In Akyürek et al. (0023) the transformer was trained on batches of examples constructed from randomly sampled latent states and measurement parameters. The authors observed that the model's behavior depends on both the architecture size and the context length. For small models trained with short contexts, the transformer approximates stochastic gradient descent (SGD). For moderate context lengths (up the dimensionality of the latent state) and moderately sized models, its behavior resembles ridge regression. Finally, with large architectures and context lengths exceeding the latent dimension, the transformer's performance approaches that of ordinary least squares (OLS).
A major contribution of Akyürek et al. 2023 was to theoretically demonstrate that transformers can approximate the operations required to implement SGD and closed-form regression. This was accomplished with the RAW (Read Arithmetic Write) operator (see Appendix GD, parameterized by matrices Wo, Wa. and W, and with an dement-wise operator (). The RAW operator maps the input to layer I, denotod G()) the output ((+1) using index sets s, r, w, time map K, and token positions 1,..., 2N ..N as
((4+)) L,w
W,
[ kE
W. Σ
[G
(11)
G ((+)
(1) Gijdd"
(12)
i.j
A single transformer head can approximate this operator for arbitrary W., Wa, W, and o. With specific parameterizations, it can implement primitives such as affine transformations, matrix multiplication, scalar division, dot products, and memory-based read/ write operations.
3 In-Context Learning for Dynamical Systems
We begin by outlining an in-context learning procedure for the generic linear state-space model defined in M4 5), assuming a time-invariant. state equation (i.e., F, = F + J, Q * 0). For simplicity of presentation, we first consider the case of scalar measurements. In this setting, the optimal causal linear estimator of the state sequence 21, in terms of mcan-square error, the well-known Kalman filter [Kalman] 1960). The procodure begins by specifying the initial state estimate and its corresponding error covariance matrix, denoted & and Pt respectively. (In our experiments, we initialize these as at and P I Inxn-) Subscquent estimates and covariances are computed recursively using the Kalman filter's prediction and update steps, as detailed by the expressions below.
5

---

Published in Transactions on Machine Learning Research (02/2026)
Prediction Step:
& F
(13)
P.- FP P" +Q
(14)
Update Step:
K, P, (H, R) at = 2. + K
(15)
(16)
P.. = -I-KKH
(17)
In the case of scalar measurements, where H, = hy (row vector) and R= x2 (scalar), the matrix inversion in 15 simplifies to scalar division, an operation that can be readily approximated by a single transformer head. The update equations in this scalar setting reduce to
PT hg
a5 = 35 +
(yr h,55)
(18)
h.P, h? + 0²
Pr 1uf h4
pr= (1 h.P, hT + o PP
(19)
These simplified forms highlight the modular arithmetic structure the Kalman update, which we later exploit in constructing transformeer-execulabie analogs.
To examine whether transformer can in-context learn approximate the behavior of the Kalman filter, design training procedure based synthetic data generated from lincar dynamical systems with scalar observations. For each training instance, we randomly sample system parameters including the state transition matrix F, the mcasurement vectors bi....,Ny: the process noise covariance Q, and the measurement noise covariance o² along with the corresponding outputs ,/ These quantities are arranged into a structured input matrix dimensions (n + 1) x (2n + 2N + 1), formatted as
0 F
0 Q
2
0 4T
91
122 O
UN-1
(20)
h
The transformer, denoted by O, trained to predict the output at every alternate column starting from position (2n + 1)st where the first column is indexed as 0. In particular, the objective is to minimize the mean squared error over the prediction horizon using the loss function
N
1
y To(h1, 31 h V1-1, hr F,, F,Q,
(21)
N
1=1
This formulation explicitly tests whether the transformer can, based only on a short trajectory of input output pairs and system parameters, learn predict the current output in a way consistent with Kalman filtering.
As shown by Akyürek al. (2023). there cxists parametrization of transformer's 's head that approximate the Read Arithmetic Write (RAW) operator defined in equations [1] 12 Building on this, we identify set of basic matrix operations, each implementable using the RAW operator, that are sufficient to reformulate the Kalman filter's prediction and update steps. These operations are defined over specific index subsets of the transformer's 's input matrix. For example, consider the system matrix F; the indices corresponding to F in the input matrix of expression (20) are denoted by
I impat
(1, (L.,), 1), (n, 0), (n,, 1), (n, n 1)}.
Further details constructing these index sets are provided in the appendix.
To facilitate aforementioned reformulation of the Kalman filtering recursions, we define the following primitive operations:

---

Published in Transactions on Machine Learning Research (02/2026)
M,)(I, Multiplies submatrices at index sets I and J, and writes the result to index set K. Div(I,j, K)). Divides each element at indices in 7 by the scalar at index j, and stores the result to index set K.
Aff(1, J, K, W1, W2). Performs the affine transformation W1 mat(!) + W2 mat(J) and writes the result to K, where mat(T) and mat(J) denote submatrices at index sets and J, respectively.
Transpose(I, J). Computes the transpose the matrix at and writes it to J.
With the primitive operations defined above, the Kalman filter recursions can be reformulated using brausformer--eecutaabl instructions. To enable this, we introduce auxiliary notation for managing in- termediate computations. We assume that the imput to the transformer can be prepended with a matrix of identity and zero submatrices, denoted by Aappend- The full input then Acat [Aappend; Amput). Within Acat, we define the following index for intermodiate variables:
Jm: an n x n identity submatrix;
IB2, la9: two n n submatrices of zeros;
BB3 1 x n row vector of zeros;
I84, JBst two column vectors of zeros;
IBS, IBS, I87: scalar zeros.
We also define the index sets Ir, IQ, and Io to refer to the locations of F, Q, and 0² in Acat, respectively.
These buffers provide writeable memory slots for storing the evolving Kalman variables (such as the state, covariance matrix, and intermediate expressions) through the course of the algorithm. The complete recursive implementation is detailed in Algorithm [ A line-by-line explanation of how cach transformer operation symbolic abstraction
maps Kalman step is available in Appendix A Note that Algorithm ! serves describing how generic transformer architecture can, in principle, emulate a Kalman filter using the Mul, Div, Aff operators that are special instances of the unified transformer primitive RAW operator. These derivations are not convergence or learning guarantees, but rather proofs of representability illustrating what the transformer architecture can compute in principle. The steps outlined in Appendix A illuminate this derivation in detail and walk through the mathematics behind cach symbolic update. The empirical investigation presented later in this paper uses standard transformers, without any custom architectural modifications.
The above framework naturally extends to systems with vector-valued observations and uncorrelated noise. Suppose E Hm and rt ~ (0, R), where R is a diagonal matrix with positive entries of oz,,.,, o Let H. denote the measurement matrix, si denote the jth component of yt, and H, denote the jth row of He- Then the Kalman update equations become Kailath et al.) 2000
H (1+ = 27. + H PT H +o?
(1)
(22)
p(1)+ = -(1-
P. H H H
P.-
(23)
H!! P.- ((U +
p(s 1)) ()T
a00++ 211 (1-1)+ +
1/ -H
()) 20 211-1)+)
== 2,..... .m..
(24)
H ()) g 1( 1)+ H11 + +o²
اان b(i-1)+ 1)+ H H (T)T 11,5))
p000
Bld
1= 2, 2,....,m
(25)
5 10-1) 1)+ 11 1,,00 o²
at 11 (m)+
(26)
P p(m)+
(27)

---

Published in Transactions on Machine Learning Research (02/2026)
While equations 13
17 present the standard Kalman filter in its classical recursive form, equations 22
27
re-express the same Kalman filter updates in an unrolled format that makes them amenable to transformer implementation. This recursive structure enables sequential updates for each measurement dimension and is readily encoded in a format similar to that in the scalar case. In particular, the in-context learning can be performed by providing the transformer the input formatted as
o
02
V1
UN
0
mm
V1
U
(28)
Q
(L)T
0
N
0
H
Im)
0
(m)
This layout enables a direct extension of the scalar Kalman filtering implementation to the multivariate sctting, using the same transformer--cxooutbbbl primitives (e.g., Mul, Div, Aff, and Transpose).
Algorithm Formulating the KF recursions using elementary operations implementable by transformers. Input: Acat, Ip, lQ: La, 1 B1, B2, LB3, BA, 1s, Lbo, LB., 1s, Ip9 Initialize I [xour +- (1 n, 2m)-
for i= to N do
Ix X + (1: n, 2n + 21)=
T x- [1: n, 2n 2i 1):
(0, 2n 21):
Transpose(Tr: IF, IB2)
Mul(IF. Ixcmn 1xmx))
Mul(IF, IB1, IB1):
Mul(IB1, IB2, IB1):
10
(TI Iq. IBI, W1 = Inxn, W2 = Inxn)
11
Transpose(In, J,, Ina)
12
Mul(I In. IBi)
13
Mul(T TBA TB))
14
(IB Io. IBS, W1 1, W2 =))
Div(In4, TH. Tna):
16
Mul(I,, 1mmn' LBT)
17
(I, IB7, I87, W1 1, W2 -1) Mul(IB7. IBA, 1B8):
18
(I IBS, I Ixmm'
Wi 1, W₂ 1):
20
Mul(T TB3: [p9)
Mul(IB9, IBI, IB9)
21
22
Aff(Im, Ing, I8I, W1 Jaxn; W2 -Iax)
& Xxaxs
4
Simulation Results
4.1
Experimental setup
For transparency and reproducibility, we build upon the code and model released by Garg al. (002).. We adopt curriculum learning, starting with a context length of N 10 and incrementing it by 2 every 2000 training steps until reaching N = 40. In our setting, context length refers to the number of past measurements
8

---

2. Strategy 2 (Guaranteed Stable Dynamics): The state transition matrix is constructed as Published in Transactions on Machine Learning Research (02/2026)
and associated parameters available to algorithm at inference time. Unless stated otherwise, the presented results are obtained for the hidden state dimension set to = 8.
Training is performed using the Adam optimizer Kingmaa 2014 with a learning rate of 0.0001 and a batch size of 64. For each training example, Ty and the measurement matrices H1, H..... Hy are sampled from isotropic Gaussian distributions. The process noise qt is sampled from (N, Q), where Q = U Here, Uq is a randomly sampled 8 x orthonormal matrix and Ee is diagonal with entries drawn from the uniform distribution [4(0, oz]. A training curriculum gradually increases oz over 100,000 steps, after which it is held constant at 0.025. Similarly, the measurement noise I; is sampled from (0, R), where R is diagonal with entries of...., o drawn from 410, o2), with of also incrcasing over the first 100,000 training steps to fixed value of 0.025. Both Q and f are re-sampled independently for each training example.
We consider two strategies for generating the state transition matrix F.
1. Strategy 1 (Unitary-lnterpolated Dynamics): We generate the state transition matrix as F = - a) + +UFF,
where ([, I] and Up is a randomly sampled orthonormal matrix. As a result, the eigenvalues of F are generally complex and can be expressed as ped For fixed a, the phase range @ E [- Φal 0 and 1, all eigenvalues lie
expands from 0 to on the unit circle; for intermediate values of a, they may also lie inside it.
increases from 0 to 1. At the endpoints
Since F may have cigenvalues on near the unit circle, systems generated under this strategy are not necessarily stable. In fact, we empirically observe that when a = 1, the transformer's loss fails to decrease, indicating poor convergence. To address this, we implement a training schedule in which a gradually increased from 0 to 1 over 50,000 steps and then held constant.
F UF.
where Up is a random orthonormal matrix and Er is diagonal with entries sampled from 1]-1,1]. This ensures that all eigenvalues of F lie strictly within the unit circle, yielding a stable system.
To evaluate the transformer's ability to approximate bascline estimators, we report both mean-squared error (MSE) and mean-squared prediction difference (MSPD). The MSE quantifies the average squared error between the transformer's s predicted output and the true output. MSPD, on the other hand, compares the predictions of two models directly, regardless of ground truth. For two algorithms Ai and Ag, and for a given context D [H....., .....,,N-ii HN-1] ~ p(D), the MSPD is defined as:
(s), = EDAAN~D(M)
(29)
4.2 Results on linear systems
We start by evaluating the transformer in an in-context learning setting on linear dynamical systems. The transformer model used in these experiments has 32 layers, 4 attention heads and hidden size of 512. Unless otherwise noted, evaluation is performed on batches of 5000 randomly sampled examples. The noise covariances are fixed at og = o² = 0.025, and the Kalman filter baseline is initialized with a zero state estimate and identity error covariance.
Before analyzing in-context output prediction performance, we first demonstrate that transformers can explicitly estimate the latent state sequence from scalar observations provided in the format of expression (20). To this end, we train a transformer under the unitary-iterppllteed dynamics regime (Strategy 1) to directly predict the hidden state using the loss function
N
1
[
(o V1. h- U1- h, F.Q,
(30)
N

---

Published in Transactions on Machine Learning Research (02/2026)
-CL
KSl Kalmmnn Filar
Kamman Filer
KCL and SIGD
0.8
0.
E2l and Fodge
MSE
02
0.2
0-
5
10
15
30
35
40
5
10
15
20
25
30
56
40
Content Length
Contect Lengh
(a) Mean squared error (MSE)
(b) Mean-squared prediction difference (MSPD)
Figure 1: Performance of the transformer in explicit state estimation from scalar measurements (Strategy 1). The transformer's predlcctoons are compared to Kalman fillering, SGD, ridge regression, and OLS.
Figure a shows the mcan-squared error (MSE) between the transformer's prodiction of IN and the ground Iruth, compared against several baselines including Kalman filtering, ordinary least squares, ridge regression (A 0.05), and stochastic gradient descent (with learning rate of 0.01). The rationale behind these parameter choices is detailed in Appendix E.3. The corresponding MSPD values between transformer predictions and each baseline are reported in Figure [b). b). The Kalman filter achieves the lowest MSE, as expected. As context length increases, the transformer's performance converges to that of the Kalman filter, distancing itself away from the regression based estimators which do not account for temporal dynamics. A notable trend observed for both ridge regression and SGD basclines is a sharp risc MSE al simall-to-innermmeiiale context lengths, followed by recovery at longer horizons. This behavior, recurring across many experiments, is consistent with the double descent phenomenon (Schaeffer et al. 2023) wherein models initially transition from an over-parameterized to under- pedde--prameetrrized regime. When the number of imput features (i.e., context length) grows beyond the model's capacity to disambignate noise from signal, overfitting leads to increased test error. As context continues to grow, the model eventually gains sufficient. statistical leverage to suppress noise and recover performance. This effect is more pronounced for OLS, while the degradation in Ridge regression is milder due to the stabilizing effect of regularization.
We next assess the transformer's ability to perform in-context one-step output prediction. Given context. of past inputs and outputs, the model is tasked with predicting the system output Y/ each step. The resulting MSE and MSPD for Stralegy 1 are shown in Figures 2(a) and 2(b), respectively; their counterparts for Strategy 2 are shown in Figures ((c) and 2(d) Under Strategy 1, performance of the transformer is closest to that of the Kalman filter. Under Strategy 2, at short context lengths, the transformer's predictions most closely resemble those of stochastic gradient descent (SGD) with learning rate 0.01. As the context length increases, its behavior progressively aligns with that of the Kalman filter. The differences between methods are more pronounced under Strategy where the system may be unstable, and narrower under Strategy 2, where the dynamics are constrained to be stable.
To assess the robustness of the transformer to incomplete context, we repeat the previous experiment while withholding the noise covariance matrices Rt and Q. As shown in Figure 3 the transformer's performance remains stable in both strategies, with minimal degradation in MSE or MSPD. These findings suggest that the transformer may be implicitly inferring the missing noise statistics as part of its in-context learning process. Note that the Kalman filter bascline still has access the full model paramcters, including noisc covariances. If this information were withheld from the Kalman filter well, estimation would require a more complex approach such as expectation maximization (EM) to recover the unknown covariances.
We next evaluate the transformer's ability to perform in-context lcarning in systems with multi-dimensional (non-scalar) measurements. Specifically, we consider observalions of dimension two and white Gaussian noise. The transformer's input formatted according to expression 28), and includes the full parameterization of the state-space model. Figure 4 shows both the mcan-squared error (left) and the mean-squared prediction difference (right) between the transformer's predictions and those of several baselines. The results confirm that the transformer emulates the Kalman filter in this more general setting.
10

---

Published in Transactions on Machine Learning Research (02/2026)
KG, Ke Filler
Karmnn Filer
KG not RID
and Fidge
0.8
0.8
MSS
1/6 MISPO
0.2
0.2
0-
10
15
20
30
35
10
15
20
25
30
56
40
Context Length
Contect Lengh
(a) Mean-squaared (Strategy 1)
(b) Mean-squared prediction difference (Strategy 1)
Kalman Filor 90D
12 8GD
0.5
Ragg
0.8
1Cl and Podge
MSS
88
MSPD
0.8
02
10
15
20
30
35
40
5
10
16
20
25
30
36
40
Contect Langth
Conteet Lenghn
(c) Mean-squared error (Strategy 2)
(d) Mean-squared prediction difference (Strategy 2)
Figure 2: Comperison between transformer and classical estimators in one-step output prediction for scalar measurements.
Our final set of experiments involving linear dynamical systems investigates the transformer's ability to perform in-context learning when no model parameters (i.e., neither the state transition matrix nor the noise covariances) are provided in the context. To keep the setup minimal, we revert to scalar measurements. Figure 5 presents both the MSE and MSPD of the transformer relative to various baselines. As seen in the figure, when faced with the challenging task of both capturing the state dynamics and implicitly estimating unknown state transition matrix, the transformer struggles and achieves performance that most closely resembles performance of SGD. However, if this task is rendered somewhat easier by reducing dimension of the state vector from n = 8 to = 2, thus consequently reducing the number of parameters needed to specify unknown transition matrix, the same transformer model succcods in closely emulating performance of the Kalman filter that provided the information withheld from the transformer. Remarkably, despite the absence of model-speciiic information, the transformer's performance progressively approaches that of the Kalman filter as the context length increases. This behavior is reminiscent of the Dual Kalman Filter (DKF) Wan & Nelson 1996), which alternates between estimating the hidden state and the unknown state transition matrix. Specifically, the transition matrix is treated as latent. variable and estimated via a secondary Kalman filter that uses regressor constructed from prior state estimates. Appendices B and C present theoretical arguments and an implementation blueprint showing how a transformer can emulate the DKF in this setting.
Appendix presents results evaluating the transformer's robustness distribution shifts between training and inference. For instance, model trained under Strategy with measurement matrices H; drawn from (N0 is tested on systems where H, ~ [[ 3]. Despite this substantial change in the observation model, the transformer's MSPD relative to the Kalman filter remains low, indicating strong generalization. Appendix E further evaluates robustness to variations in state dimensionality. Finally, Appendix D demonstrates that the transformer's in-context learning capabilities extend naturally to systems with control inputs, and provides implementation details and input formatting for this setting.
Remark 1: While in all of the reported figures the MSE and MSPD curves appear to flatten beyond context length 30, both metrics continue to decrease, albeit at slower rate. This diminishing improvement reflects
11

---

Published in Transactions on Machine Learning Research (02/2026)
KSl Kaltman Filar
Kammnn Filer
KCl and SIGD
0.8
0.
and Fodge
MSE
1/
0.2
0.2
0-
10
15
20
30
35
10
15
25
30
56
40
Context Length
Contect Lengh
(a) Mean-squared (Strategy 1)
(b) Mean-squared prediction difference (Strategy 1)
Kalman Filor 90D
K2 80D
0.5
Ragg
0.8
1cl and Podge
88
10 MSPD
0.8
02
10
15
20
30
35 40
10
16
20
25
30
36
40
Contect Langth
Contect Langth
(c) Mean-squared error (Strategy 2)
(d) Mean-squared prediction difference (Strategy 2)
Figure Performance of in-contert learning with transformer that not provided noise covoriances Q and R. (a) Mean-square error (MSE) under Strategy 1. (b) Mean-squared prediction difference (MSPD)
relative to the baselines under Strategy 1. (c) MSE under Strategy 2. (d) MSPD relative to the baselines under Strategy 2.
the interaction between context length and model expressivity. As context increases, the transformer has more signal to infer the latent structure, but unless the model architecture is sufficiently expressive, its predictions may remain biased relative to the Kalman filter. This behavior is further studied in nonlincar systems settings and corroborated by the results reported in Tables and of Section 4.3, which show that beyond a certain point, error reduction limited not by context length but by the transformer's architectural capacity to emulate optimal inference.
Remark 2:
We emphasize that multi-step prediction is feasible in our setup by recursively leeding the predicted latent state back into the input stream. While this introduces some accumulation of prediction error, results in Appendix (.rruddxxA4]eeee demonstratt that the transformer maintains strong performance and closely enulates the Kalman filter at horizons of up to 5 steps.
4.3 Results on nonlinear systems
We now whether transformers can in-context learn to emulate filtering behavior in nonlinear dynamical systems. As discussed in Appendix F key operations of the Extended Kalman Filter (EKF) be implemented using transformer primitives, suggesting that transformers are capable of learning nonlincar filtering strategies via in-context learning (ICL). To further contextualize performance, we also compare it against particle filtering (PF), which widely used for nonlinear systems with non-Gaussian posteriors. Note that PF relies on stochastic sampling and resampling steps, which do not lend themselves as naturally to transformer architectures.
We compare the performance of transformer with baseline algorithms two representative systems:
12

---

Published in Transactions on Machine Learning Research (02/2026)
KS Kalmmnn Filar
Kammnn Filer
KCL and SIGD
08-
0.8
F2L and Fodge
MS
0.0
1/
02
0.2
0-
0-
5
10
15
20
30
95
5
10
15
20
25
30
56
40
Context Length
Contect Lengh
(a) Mean-squarre error (Strategy 1)
(b) Mean-squared prediction difference (Strategy 1)
Kalman Filor
K2 8GD
0.5
90D
Rage
0.8
1CL and Podge
1in MSE
08
0.8
02
02
5
10
15
20
30
35
40
5
10
16
20
25
30
36
40
Contixt Langth
Contect Length
(c) Mean-squared error (Strategy 2)
(d) Mean-squared prediction difference (Strategy 2)
Figure 4: In- In-contezt learning (ICL) with a transformer for systems with 2D measurements. The transformer receives the full system specification and performs one-step output prediction.
System (Nonlinear State Evolution): A system with nonlinear state dynamics and linear measurements:
F(+1 F tanh(2n;) g
(31)
yt H43, Ttb
(32)
Here, the state and measurement dimensions are fixed to 2. The matrix F is given by F 0.81+ where UF has entries drawn from a standard Gaussian distribution. The process noise covariance is set to Q 0. + 0.12g), with Zg also drawn from standard normal distribution. The observation noise covariance is R = 0.017, and the measurement matrices H, are sampled from an isotropic Gaussian distribution. Variations of this system with different state transition functions and dimensionalitics are explored in Appendix F.
System 2 (Maneuvering Target with Unknown Turn Rate): nonlinear tracking model with time-varying dynamics and measurement equations. Full model details are provided Appendix F.
The transformer model used to obtain results in Figures G 7 has 16 layers, 4 attention heads and hidden size of 512. Figure 0 reports the mean-squared error (MSE) and mean-squared prediction difference (MSPD) for System 1, where the variance of the process and measurement noise is o² = o² = 0.0125. The results demonstrate that the transformer closely tracks the performance of both the EKF and PF, indicating that it Iearns to emulate nonlinear filtering behavior in this regime.
Figure presents results for System 2, a more complex maneuvering target model with nonlinear observations. Interestingly, in this setting the transformer achieves the best performance (i.e., the lowest MSE) among all the considcred methods. This indicates that in-context learning generalizes effectively even when key latent parameters, such turn rate, are unobserved Linear regression and SGD are omitted from this comparison due to the nonlinearity of the measurement model.
1 Note that EKF and particle filter incorporate the turn into the vector and thus estimate recursively details, please see Appendix F.2.
13

---

Published in Transactions on Machine Learning Research (02/2026)
0.5
0.6
(UL Kallnan Fra
Kalman Filer gm12)
1C2 and 560 (enQ)
50D (an))
04
ledge (ym2)
(m+3)
++I0L GLS
MSSE
KSl gwellt
MSPP
0.3
ICL and tolmnn Filse (0x8)
1/
0.2
10
15
30
95
10
15
25
30
56
40
Context Length
Contect Lengh
(a) Mean-squarrd error (Strategy 1)
(b) Mean-squared prediction difference (Strategy 1)
0.0
0.0
Kainan Faxz)a=2)
ICL and 800 (n-2)
077
800 (y-2
0.7
ICL and Plage (1-41
Podge (red)
08
08
SGD (ye5)
111 Nalman Fibe (0+5)
0.
IC. (mm())
05
8.4
033
031
02
02
0.1
0.1
10
15
20
30
35
60
5
10
20
25
30
36
40
Contoot angth
Conteet Langth
(c) Mean-squared error (Strategy 2)
(d) Mean-squared prediction difference (Strategy 2)
Figure 5: Performance of in-context learning (ICL) with a transformer under fully missing model parameters (scalar measurements). No information about the state transition matrix noise covariances is included in the contert.
12.
IC. and EKF
SGC Fldgee
and Partcie Filar
MSKO
0.3
0.22
10/1
0.1
10
20
25
30
301
40
10
15
20
25
30
25
40
Connent Length
Conghh
(a) Mean-squared error (MSE)
(b) Mean-squared prediction difference (MSPD)
Figure 6: Performance of in-contert learning (ICL. System with nondinear state evolution and linear mensurements. Transformer achieves performance par with Extended Kalman Filter (EKP) and Particle Filter (PF).
Finally, we fix the context length 40 and investigate how the transformer's behavior in System 1 depends model capacity, varying both the number of layers and the embedding dimension. The results, summarized in Tables [ and 2 reveal that smaller transformers exhibit behavior more similar to that of SGD and Ridge Regression, while larger models yield outputs that incrcasingly align with those of EKF and the particle filter, i.e., methods that incorporate knowledge of the system's dynamical structure. This transition underscores the role of model capacity in enabling the transformer to approximate sophisticated recursive inference algorithms. Notably, even modest increases in either depth or embedding dimension significantly improve the transformer's alignment with classical filters, measured by MSPD. These findings suggest that architectural scale is a key factor in unlocking in-context learning of dynamics-aware estimation procedures. In short,
14

---

Published in Transactions on Machine Learning Research (02/2020)
10
una BXF
Darget Teoting dau (Frge) (Tage Tacogg
and Partile Filor
MSPD
10
10
25
40
Combest Lenghh
Content
(a) Mean-squared error (MSE)
(b) seean-squared prediction difference (MSPD)
Figure 7: Performance of in-contezt learning (ICL) on System a nonlinear trucking model with unknown turn rate. The transformer consistently outperforms particle filtering and other methods, especially at longer horizons, demonstrating its ability to handle uncertainty and perform nonlinear inference in an in-contert fashion.
while the operations required for filtering are representable in principle, sufficient model capacity is necessary to learn them reliably from data.
Number of Layers
ICL and EKF
and Particle Filter
ICL and
ICL and SGD 0.05
ICL Ridge and Ridge
L.0281
0.9504
0,2662
1.0010
0.7484
0.7463
2
0.2447
0.2274
0,2433
0.1806
0.1891
0.1689
4
0.1251
0.1078
0.3598
0,1800
0,2448
0.2146
8
0.0526
0.0337
0.3970
0.1921
0.2892
0,2591
0.0531
0.0336
0.4273
0.2199
0.3191
0.3189
Table 1: Effects of transformer depth MSPD in System 1. The table reports mean-squared prediction difference (MSPD) between the transformer and various baselines as the number of transformer layers increases (embedding dimension fixed to 512).
8 ICL and EKF
Embeddiig Dim
[CL and Particle Filter
[CL and SGD 0.01
ICL aud SGD 0.05
ICL and Ridige 0.01
ICL and Ridge 0.05
L.068
1.011
0.283
1.036
0.795
0.793
64
0.104
0.067
0.366
0.170
0.264
0.263
0.411
0.310
512
0.053
0.034
0.341
0.192
(1.280
0.288
Table 2: Effect of embedding dimension MSPD in System 1. The table reports MSPD between the transformer and various basclines as the embedding dimension increases (number of transformer layers fixed at 8).
5 Conclusion
This work investigated whether transformers, trained via in-context learning on syuthetic trajectories from randomly sampled dynamical systems, can implicitly learn to perform filtering without test-time gradient. updates or explicit access model equations. We provided constructive arguments showing that the Kalman filter can be expressed using operations natively supported by transformer architectures, and empirically demonstrated that, in linear systems, transformer predictions closely track those of the Kalman filter when given sufficient context and model capacity. The nature of the learned algorithm was found to depend on both model scale and context length smaller models or shorter contexts emulate linear regression or stochastic gradient descent, while larger models with longer contexts converge toward optimal filtering behavior.
Beyond linear systems, we demonstrated that transformers can generalize nonlinear dynamical systems well. In particular, we showed that they can match the performance of the Extended Kalman Filter and the particle filter, and in some settings even outperform them. These results suggest that transformers do
15

---

Published in Transactions on Machine Learning Research (02/2026)
not merely replicate fixed algorithm but instead learn flexible, data-driven inference strategy through in-context learning.
The robustness of transformer-basee filtering to missing context, such as unobserved noise covariances or system parameters, further highlights its potential as a general-purpose alternative to manually designed filters. However, performance degrades under limited model capacity and/or missing inputs, emphasizing the importance of expressiveness and informative prompts. Future work will explore extensions to temporally correlated noise and examine the emergence of internal representations that support in-context filtering.
References
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. Tn The Fleventh International Conference Learning Representations, 2023.
Riccardo Barbieri, Loren M Frank, David P Nguyen, Michael Quirk, Victor Solo, Matthew A Wilson, and Emery N Brown. Dynamic analyses of information encoding in neural ensembles. Neural computation, 16 (2):277-300) 2004.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurol information processing systems, 33:1877 1901,191,,1, 2020.
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In Forty-first International Conference on Machine Learning, 2024.
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 15:30583-30599, 2022.
Gautam Goel and Peter Bartlett. Can transformer represent kalman filter? In 6th Annual Learning for Dynamics & Control Conference, pp. 1502 1512. PMLR, 2024.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint 0rX00:211..00366, 2021.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.084115 2016.
Thomas Kailath, Linear systems, volume 156. Hrentice-Hall Englewood Cliffs, NJ, 1980.
Thomas Kailath, Ali H Sayed, and Babak Hassibi. Linear estimation. Prentice Hall, 2000.
Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint orXiv:14126698. 2014.
Rahul G Krishnan, Uri Shalit, and David Sontag, Deep kalman filters (2015). arXiv preprint ar XXX.:1511..5121, 2015.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.1 12837, 2022.
Robert Piché, Simo Särkkä, and Jouni Hartikainen. Recursive outlier-robust filtering and smoothing for nonlinear systems using the multivariate student-t distribution. In 2012 IEEE International Workshop Machine Learning for Signal Processing, pp. 1-6. IEEE, 2012.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Guy Revach, Nir Shlezinger, Ruud JG Van Sloun, and Yonina C Eldar. Kalmannet: Data-driven kalman filtering. In ICASSP 2021-2021 TEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3905 3909. IEEE, 2021.
Guy Revach, Nir Shlezinger, Xiaoyong Ni, Adria Lopez Escoriza, Ruud JG Van Sloun, and Yonina C Fldar. Kalmannet: Neural network aided kalman filtering for partially known dynamics. IEEE Transactions Signal Processing, 70:1532 1547, 2022.
Rylan Schaeffer, Mikail Khona, Zachary Robertson, Akhilan Boopathy, Kateryna Pistunova, Jason W Rocks, Tla Rani Fiete, and Oluwasanmi Koyejo. Double descent. demystified: Identifying, interpreting & ablating the sources of deep learning puzzle. arXiv preprint arXi::203..2 14151, 2023.
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9355 9366. PMLR, 18 24 Jul 2021. URL https://proceedings mlr. wttpppeeeseeee/pieaaao httsss//1113///aaaa2laaaaa..
Jerome Sieber, Carmen Amo Alonso, Alexandre Didier, Melanie N Zeilinger, and Antonio Orvieto. Un- derstanding the differences foundation models: Attention, state space models, and recurrent neural networks. Xiviv preprint orXit:2405.1177 15731, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Leurning, volume 202 of Proceedings of Machine Learning Research, pp. 35151 35174. PMLR, 23 29 Jul 2023. URL https: 7/proceddings. mlr wttppprs//v22//oooooooaaaaaaaaaa
html
Eric Wan and Alex Nelson. Dual kalman filtering methods for nonlinear prediction, smoothing and estimation. M.C. Mozer, M. Jordan, and T. Petsche (eds.), Advances in Newral Information Processing Systems, volume MIT Press, 1996. URL https://procceddng. www.:.s.nnrisss.ccppppeeee httppp/r.iillaeaap///9/////// 14770 4477220074444555555111166333333
pdf
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint srXfc:2111.02080 2021.
Dani Yogatama, Cyprien de Masson d' A'Attmme, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dycr, et al. Learning and evaluating general linguistic intelligence. arXiv preprint ar Xiv: 11373, 2019.
Yong Zeng and Shu Wu. State-space models: Applications in economics and finance, volume 1. Springer, 2013.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pp. 12697- 12697112706. PMLR, 2021.
17

---

Published in Transactions on Machine Learning Research (02/2026)
A Details regarding the implementation of Algorithm 1
We claborate on the operations in Algorithm 1, focusing for simplicity on the case of scalar mcasurements.
Let the state dimension be n.. Define the auxiliary matrix
Bi
B2
Bg
B
B4
Bg
0nx1
Onxt
Onx1
Aappend =
(33)
01xn
01xn
01xn
0
0
B5
B6
B7
For illustration, consider 2, where
0
0
0
0
0
0
0
0
0
Aappend
0
1
0
0
0
0
0
(34)
0
0
Form the concatenated matrix Azat [Amppedd Ainputl; and define index sets such as IBI
{(0, 0), (1, (0, (1,,1)) 1b2 = {(0, 2), (1, 2), (0, 3), (1,3)}, and so on. These index sets specify mem-- ory regions corresponding the working blocks in the transformer's computation, where operations like matrix multiplications, transpositions, and affine combinations
carried out.
To build intuition, walk through the first iteration of the FOR loop in Algorithm 1. We initialize a =0 and carry out a sequence of transformer-reddabe operations:
Initialization: Set Xxcur +- (1 : n, 2n).
2. Pointer sctup: For i = 1, set X + (1 2n + 2).
3. Measurement access: Define
Jn + (1 : 2n + 1) (for Mif))
l, - (0, 2n + 2) (for y1)
4. Transposc: Write FT to B2 using Transposc(Ir, IB2).
State prediction: Compute कं Fat via Mul(IF, Xxeur Xmma).
Covariance prediction:
Compute FP+ B1
Multiply by FT to get FP FT in Bi
Add Q via Aff lq, (BI) to form PT
Kalman gain computation:
Transpose hT to B3
Compute PT hT to Ba
Compute scalar hy P1 of to B3
Add measurement noise o²: AIs Ia. Ins)
Divide compute K₁: Div(JBA, Tps. I
8. Measurement update:
Compute predicted observation hy to By
Subtract from y1 to get residual in B7
Multiply K1 with residual to get Bg
Update &t 27 Ki(m1 1T to X
Covariance update:
Compute outer product by K1 to Bg
18

---

Published in Transactions on Machine Learning Research (02/2026)
Multiply with PT to form h1 PT
Subtract from PF- to form Ptt in Bi
10. Pointer update: Set Xxaur Lx Xxexi"
Each of these steps is directly implementable via composition of the primitive operations defined in the main text: Mul, Div, Aff, and Transpose.
B A brief summary of the Dual Kalman Filter
Consider the state-space model
T++1 F1 + qt
(35)
V/ H + Tt:
(36)
where both the latent state 2, and the transition matrix F; must be estimated. The Dual Kalman Filter (DKF), introduced by Wan & Nelson (1996). addressos this by alternating between two Kalman filtering recursions one for given an estimate of F,, and one for Fę given an estimate of 2t.
Let fa E TR=" be the vectorized form of F,, and define the matrix X₃ E TR""" as:
0
0
X₂
(37)
This formulation enables the re-expression of the system as a linear model in fs:
ft = ft-1,
(38)
3t 1
(39)
where H. H, and ry.= Hiq. re, with rya N(O, and R, HQQH? +
The DKF then proceeds with standard Kalman filter prediction and update steps applied to ft.
Prediction Step:
ft = ft
(40)
PT. P
(41)
Update Step:
f. PT, PT, T Rg)
(42)
ftt Jr. + -yylys-llfhh -lgaff)
(43)
Pt = (I - -K)
(44)
In the scalar measurement case, Hfa H HT. H.QHE, and R reduce to scalars. Thus, the Kalman gain simplifies to
f.t
K
Hy p- HT. + R2 P- Fr.fff..
(45)
19

---

Published in Transactions on Machine Learning Research (02/2026)
C Transformer Can In-Context Learn to Perform Dual Kalman Filtering for a System with Scalar Measurements
We now consider the setting where the state transition matrix is not provided as part of the context. Let the transformer's input be
0 2
0
91
1/2
UN-1
(46)
Q
0
11
0
11 0
0
1
One can argue analogously to the bronstuyyco used for the Kalman filter that transformer can learn to perform Dual Kalman Filtering in context, even in the absence of explicit dynamics. To support this, we extend the previously introduced set of elementary operations with a new one, MAP J), which takes the vector at indices I and transforms into a matrix of the form shown in expression 37 result at indices J.
storing the
In addition to the index sets defined in the main text (i.e., Acat; IF, Iq: Io, Iøı through Iø9), we introduce the following index sets: TBio indices of an n² block initialized to zeros; TBI1: indices of a x n² vector block; IB12 and J814: indices of n² x n² matrices; Ibı3 and fhe..' indices of n² x 1 vectors.
The complete Dual Kalman Filter recursion can now be expressed using operations implementable by transformers; the pseudo-code below shows one iteration of the resulting algorithm.
1. Initialization: Set Xxour + (1 (1:n, 2m).
Pointer setup for iteration i 1:
Xxaxst (1: n, +
I + n1 1)
I < (0,2n 22)
3. State prediction:
Transposc(Tr, IB2): Write FT to B2
Mul(Tr, Tx [x ): Compute #7 Fag
Covariance prediction:
Mul(Ip, IBI, IB1): FP*
Mul(TBI, T22 TBI): FP, FT
Aff(/BI, le, IBI, W1 W₂ 1): PT
Kalman gain computation:
Transpose(In. Is)): h1 Mul(IBI, ls LBA): PT hT
Mul(TBa, TAA TBS): by NT
Afs lo, IBS, W1 =1=1, W2 = 1): s = h,P + 2
Div(Ins, Ins, In4): Kalman gain Ki
Measurement update:
Mul(T,, Xnaa TB7): predicted 5/
Aff(Ty, Inz, I87, Wi = 1, W2 = -1): residual y - 91
Mul(Ip7, 1BA, Is): Ki(yi 51)
Aff(I Xmma LBs, Ixmea W1 = 1, W₂ = 1): at
7. Covariance update:
20

---

Published in Transactions on Machine Learning Research (02/2026)
Mul(I IsS, Is9): h1K1
Ma,([ LBL [p)) h KPP
(TI I89, JBI, W1 I, W2 --I:: Pt
8. DKF step: latent transition matrix estimation
MA X IBIO): Build X1
Mul(B LIo, 4B1): H K1
Transpose(IBu: IB13)
Mul(I IB12, IBL1): H/1P
Mul(Inn, /13, IBS): H H
Aff Lo, Lb.
1, W2 1): add Ry
Div LBS, 1B13:: Kalman gain for f
Mul(I M13, TBS)
Aff famas I8S, fmext
W1 = 1, W2 1): ft
Mul(I mmo IB1))
Mul(TB13, IB11, IB14)
Mul(IB14, B12, TB14)
III Im4, I12, Wi I, W2 -I): Pt.
MAP(I; Juess Ip): overwrite F for next round
9. Pointer update: Set &ecur - &
D Experiments with Control Input
The filtering setup studied in the main text can be extended to systems with control inputs, provided that the measurement noise remains white. In this case, the state-space model becomes
1++1 Far: B q
(47)
1/ Hęr +
(48)
and the Kalman filter prediction step is modified accordingly:
f = F + Bui-
(49)
The derivation showing that transformers can implement Kalman filtering operalions under this model follows the same constructive arguments presented for the zero-input case and is omitted for brevity. Instead, we report empirical results for scalar measurements and nonzero control inputs.
In these experiments, the control matrix 8x8 is generated as = UB where UB is a random orthonormal matrix and ΣB diagonal matrix with entries sampled from 11-1, Control vectors ut R are sampled from a standard Gaussian distribution and then normalized to unit norm.
The input to the transformer in this setting is formatted as
0
0
0
0
0
0
(50)
FF
Q
1
0
0
uN
Here, F is generated using the Unitary-Interpolee Interpolated Dynamics strategy (Strategy 1), and the remaining settings match those of the baseline experiments without. control.
Figure shows the results. The transformer maintains low MSE relative to the ground truth and exhibits MSPD behavior similar to that of the Kalman filter, indicating successful in-context learning the presence of control inputs.
21

---

Published in Transactions on Machine Learning Research (02/2026)
GL KARKER FiRo
SGC
KL and Rkdee c.01
500
1adge 0:11
2.00
XL OLS
0.4
20
25
35
40
85
40
Conneot Langth
Content Langh
(a) squared enror (MSE)
(b) Mean-squered prediction difference (MSPD)
Figure 8: Transformer performance on linear systems with control inputs. The model retains effective filtering
behavior even with added input terms.
E
Additional Experiments and Further Details
E.1 An illustration of the performance out-of-sample parameters
To evaluate the performance of the transformer systems with parameters sampled from distribution different from that used during training, train the model with F generated using Strategy and H; ~ 1), 1). We then evaluate the trained transformer under two representative distributional shifts:
1. The measurement matrix H; is sampled from a uniform distribution 21(0, 3];
2. The state transition matrix F is generated using Strategy 2 instead of Strategy 1.
As shown in Fig. the transformer maintains low MSPD in both settings, indicating robustness of in-context learning to shifts in both measurement geometry and underlying dynamics.
10
RCD
ane 81
and 5C0 606 Hagget
KL 56 66 KKL OLS
1/ 1
(a) MSPD under uniform sampling H. (0, 3).
(b) MSPD under shift from Strategy Strategy for generating F.
Figure 9: Evaluation of a transformer trained on Goussian-distributee systems under representative out-of- distribution shifts.
E.2
Transformer performance across varying state dimensions
We next assess how transformer performance varies with the dimensionality of the latent state. The transformer is trained using default settings and again with F sampled via Strategy 1. During evaluation, we fix the context length to 40 and vary the state dimension from 2 to 8.
Figure 10 shows the MSPD (normalized by state dimension) between the transformer and various baseline methods. We find that:
The gap between the transformer and Kalman filter remains nearly constant across dimensions.
22

---

Published in Transactions on Machine Learning Research (02/2026)
The discrepancy between the transformer and SGD Ridge regression grows with increasing state dimension, reflecting the degraded performance of those baselines.
IC_ and Kalman Filler
IC_ and Ridge 0.01
0.2
aCd Sed.and.S000 S60
Ridge
IC and OLS
1/n MSPD
0.15
0.1
0.05
0
State Dimension
Figure 10: Normalized MSPD between transformer and various baselines as function of state dimension.
Content length is fired to 40.
E.3 Justification of the choice of the learning rates for SGD
identify the optimal learning rate a for stochastic gradient descent (SGD) and the regularization parameter À for ridge regression, we performed a sweep over a logarithmic range from 10-5 10 Based on mean- squared prediction difference (MSPD) between each baseline and the transformer, we found that 0.01 and À 0.01 consistently yielded the best performance across most context lengths, with only occasional exceptions. For consistency and clarity, we fixed both parameters to 0.01 in all experiments reported in the main paper.
E.4 Multi-step Prediction
In this section, we evaluate whether a transformer trained solely for one-step prediction can generalize to longer horizons via recursive in-context inference. Specifically, we consider the task predicting the future output UN-1+7 h given context of N input-outpin pairs and system parameters. The initial context is structured as
0 F
0 Q
91
12
UN-1
(51)
17
0
0
h
N
To perform 7-step prediction, we recursively apply the transformer T times. At each step i, the predicted output û; appended to the input stream. To isolate the prediction challenge, freeze the final measurement matrix hx-1 and reuse it for all future steps t > - For instance, to generate IN-1475 feed the transformer the extended input
0
0
0
2
0
91
122
WN-1
UN
UN ON-1+T-1
(52)
FQQ
F
0
AT
h
N-1
h hw
We adopt similar setup for baseline methods: stochastic gradient descent (SGD) and ridge regression are trained with input-output pairs where the output label corresponds to mcasurement 7 steps ahcad. Experiments are conducted under Strategy for system generation, using three values of process noise variance: o E 0.025, 0.1, 0.5. These control the entries of the process noise covariance matrix Q. Figure summarizes the results. As expected, prediction error increases with both horizon length and noise level og Nevertheless, the transformer maintains performance closely aligned with the Kalman filter and significantly outperforms begre methods, even in challenging multi-step prediction settings.
23

---

Published in Transactions on Machine Learning Research (02/2026)
Ddge (olp
4C1 and Ki
ECL Peolee (e1))
and DC2
ECA (int
ICL and Lones
03-
02-
500
02
Contect Lengh
Contedt Lengts
(a) Mean-squared error ( 0.025)
(b) Mean-squared prediction difference of 0.025)
Golva (rkpe
Calmga
Colerr Tolge (-
Rog (0)
Cormoo Langh
Contol Largts
(c) Mean-squared error (o = 0.1)
(d) Mean-squared prediction difference (o² 0.1)
10
20
25
12
13
20
25
Corbent Langlh
Conkod Langts
(e) Mean-squared error (o² = 0.5)
(f) Mean-squared prediction difference (a² = 0.5)
Figure 11: Multi-step prediction performance on linear dynamicaa systems (Strategy 1). For each value of process noise variance o² we evaluate MSE (left) and MSPD (right) between transformer predictions and baselines over horizons T= 1 to 5. The transformer (ICL) tracks the Kalman filter closely and outperforms Rädge regression and SGD, demonstrating its ability to propagate uncertainty over multiple steps.
E.5 Is the transformer utilizing the parameters of the state-space model?
In the main text, we investigated the transformer's performance on lincar systems when major components of the stale-space model (specifically, F, Q, and R) were withheld from the context. Here, we take a step further and evaluate the case where all parameters of the state-space model, including the measurement matrices Hr, are excluded. That is, the transformer must predict ÅN given only the past measurements 91. .2.....99.. . as context. As shown in Fig. 12, in this setting the performance deteriorates sharply, as indicated by the significant. increase in the resulting MSF.
24

---

Published in Transactions on Machine Learning Research (02/2026)
Kohran Filtt
Kaman Filler
KCL.
SCD (PAAI (NBI HDL Of 0997
5CD
Mh
02
0.2
0-
10
15
30
95
40
10
16
50
56
40
Content Length
Contect Lenghh
Strategy 1
(b) Strategy 2
Figure 12: Performance of in-contert learning for system where the state dimension is under two system generalion stralegies. As key puramelers of the stale-space model are omalled from the context, the transformer's 's performance degrades steadily. The gop between 1CL and the Kalman filter is more pronounced under Strategy 1, where inference is more challenging.
F Extended Results and Analysis for Nonlinear Systems
This section provides additional details for the nonlinear systems analyzed in Section 4.3. Specifically, we consider systems governed by nonlinear state-space dynamics of the form:
T++1 = fn((2t) + q
(53)
= Here + Ft:
(54)
where f-(-) nonlinear transition function parameterized by 17 Im....., nw], and qe. ri denote the process and observation noise, respectively.
A classical approach to estimating the latent states in such systems is by means of the Extended Kalman Filter (EKF), which linearizes the dynamics around the current estimate at each time step. The EKF proceeds via the following prediction and update recursions:
Prediction Step:
.7 (n( Pr = pt FT
(55)
FT +Q
(56)
Update Step:
K, PT (H.P. H Rt-
(57)
ot B.. + Kily, H₂AT)
(58)
p = (T-K,HH,))
(59)
Here, Fr denotes the Jacobian of fn evaluated at at
F.1 System 1: Nonlinear State Transitions with tanh Dynamics
We consider specific nonlinear state transition function f, (x) Im tanh(12x1),... tanh where
the parameters and f22 are independently sampled from the uniform distribution 24[-1, 1, 1). For this function, the Jacobian with respect to the state vector is diagonal and given by
F. 7 diag (1 tanh (1221), tanh
25

---

Published in Transactions on Machine Learning Research (02/2026)
This structure enables efficient computation of the EKF update steps using element-wist operations. We argue that transformer can emulate these operations in-context, provided that the input sequence is formatted as
2
7
1/2
91
UN-1
(60)
0
0
0
Prior work (e.g., Akyürek et al. (2023) has observed that the GeLU nonlinearity approximate scalar multiplication and, with sufficient additive bias, act nearly as an identity function. Building this observation, we construct an expression that closely approximates tanh(æ) using GeLU activations as
2Z 2
Lanh(x cr²) GeLU (v²T) 2z
GeLU V No + No,
where N₂ > is large bias term and 3 0.044715. This identity leverages the approximation of tanh(.) difference of GeLU activations with shifted inputs.
Since transformers can effectively bypass softmax via large biases (making attention weights nearly one-hot), this approximation can be implemented by a single attention head. In the regime we study, x + cr = x, so the net operation approximates tanh(x) directly. When combined with the transformer's ability realize primitive arithmetic operations such as Mul(), Div(), and Aff(), this suffices to emulate the nonlincar update steps of the Extended Kalman Filter for the system under consideration.
F.2 System 2: Target Tracking with Unknown Turn Rate
We evaluate the transformer's in-context learning capabilities on a practical nonlinear system: maneuvering target tracking with an unknown turning rate Piché et al. (2012). The state vector is 2t (at, âe, be, be, we]; where (a,, by) is the position, (ăc, br) is the velocity, and W. is the angular turn rate.
The state transition model is
F1+1 = Fi + qt
(61)
where Fg is nonlinear function of Wh.
sim A.)
postwr -
cos (Wي A.)
sin (wi A
0
Fe
0
coss(1y (wi) A,)
sin(wr,
0
We
We
sin (w, A.)
cos (w;
0
0
0
1
and the process noise covariance is
M
0
A /3
2
Q
0
②1 M
0
M
A /2
A.
0
0
Q2
The measurement vector consists of noisy versions of the polar coordinates of the target, i.c.,
V +
by la + rz.
y/ =
arctan
We set A, 0.1 and draw cach simulation's 's parameters as: 91 ~ 0.10, ~ 400 0.000250,, of ~ 24 0.025], of ~ 10, 0.000016), and the initial state from zo ~ N((O, 10, 0, - -0.053 Q). The transformer is trained to predict yn from an input formatted as
0 2
91
1/2
9N-1
(62)
Q
0
0
26

---

Published in Transactions on Machine Learning Research (02/2026)
To implement the Extended Kalman Filter, we compute the Jacobians
sin(w,
col(w.
àefιt + fef
0
sin A,)
sin (w, A, 0 (,, A.bb
Fe
sin(w,
- + btt
b sin sin(w; Ae)
0
A,cos(ur) )a sinn(w
0
0
and
0
h
H₂
+b
of
b
+0?
b
where
fık wrArroo(uAAi) - sin(wr and f2k - wh Ansisin(rA))) cos(w)Ar)
w/
w/
F.3 RAW Operator Approximation of Smooth Nonlinearities
The set of operations implementable by the RAW operator, including multiplication, affine combinations, and repeated composition, is sufficient to construct any polynomial. Indeed, monomials of the form cx can be computed via one scalar multiplication followed by k - applications of the multiplication operator. Once computed, polynomial terms can linearly combined using the affine operator. As a result, functions expressible via Taylor expansions can be approximated by transformers to arbitrary accuracy. For example:
x
x
x²
x
sin(r) R x
(63)
3!
5!
7!
9!
11!
x.2
x²
x.
8
r
10
x.1
cos(I)) R 1
+
+
(64)
2!
4!
6!
8!
10!
12!
x.3
2x
17x7
62.
1382r
tanh(x) 2 x
(65)
3
15
313
2835
155925
Similar series exist for arclan(x) and Vz, which also appear in the nonlinear systems considered in this work. Since transformers can represent these polynomial approximations with sufficient depth and width, this supports the claim that they are capable of learning to approximate the behavior of extended Kalman filtering in nonlinear dynamical systems.
F.4 Additional Non-linear State Transitions
Figure 13 presents results for several variations of nonlinear dynamical systems, each differing in the form of the state transition function:
System I: fz) F s2n( + q, with n 2 and m 2. System parameters F, Q, and are identical to those used in System 1 of the main paper.
System II f,(x) (2x) + d6, with 2 and m 2, where a(-) denotes the sigmoid function. All other system parameters match those of System
System III fx(x) tanh(2x) + ge, with n and m 1. System parameters remain consistent with System 1.
4. System IV: fz) = tanh(2x) 3 exp(-x²) + qr, with n 2 and 2, and system parameters again matching those of System 1.
27

---

Published in Transactions on Machine Learning Research (02/2026)
In all cases, the entries of the measurement matrix H₃ are drawn from isotropic Gaussian distribution. These experiments confirm that the transformer's performance generalizes across range of nonlinearities. In all four cases, the MSE closely matches or outperforms the Extended Kalman Filter and tracks the performance of the particle filter, particularly as context length increases. This consistency highlights the transformer's capacity to adapt its inference strttegy to diverse dynamical structures.
To skonne
Tunsfore
5GD o.
560 C.1
1idge
18dge
Rigoo
thn
1011
10
Contest
25
15
Contact Lenghh
(a) MSE of various methods for System I
(b) MSE of various methods for System
16
To monoee
-L
550 0.
SGB C01
Ridge
Riige
Higg 18
tie MSE
H90g 003
101
(c) MSE of various methods for System IIT
(d) MSE various methods System IV
Figure 13: Transformer performance on nonlinear systems with varying state transition functions. Evaluation is conducted for each of the systems described above, illustrating that in-contert learning remains effective across a range of nonlinearilies.
G Derivation of the RAW Operator
To make this work self-contained, we briefly summarize how the RAW operator can be constructed and used emulate key computational primitives within transformer architectures. This derivation follows the treatment in Appendix C of Akyürek et al. (002), to which refer interested readers for additional technical details and formal proofs.
G.1 Embedding Layer Design and Positional Control
We assume the existence of a linear embedding layer immediately preceding the transformer layers. This embedding layer can be parameterized to include extra scratch space necessary for intermediate computations. Specifically, use the embedding matrix
I(d+1)x(a+))
0
W.
(66)
which embeds the input into the first d + dimensions and reserves the remaining entries for auxiliary operations.
Following Akyürek et al. 2023
attention is directed using position embeddings that encode keys and queries.
These are structured as
(L) p: Oct1 kQ of ... (L) di O O 2pT
28

---

Published in Transactions on Machine Learning Research (02/2026)
where each kt, of Iº controls the attention for head I at time step This setup allows queries and keys to be extracted via
W = 0 0 pxp 0 ... W = [o
0 Txxp ...].
where 1xxp selects the appropriate query or key block. Using this formulation, two common attention patterns can be encoded:
Attend to the previous token. To enforce attention from time step i to i 1, set
h;j = C2
(67)
qi Ne;-1;
(68)
where e; is the 7th standard basis vector and N >>>>> ensures the softmax selects the intended token.
2. Attend to fixed token. To force every future token to attend to token t, define
{t) i>t
K(1)
(69)
00
otherwise
and set the positional encodings to
-N, i
h;
N. N i i= L1
(70)
G.2 Utilizing Nonlinearities
The nonlinear components of (ransformer layers can be exploited to implement element-wise arithmetic operations with high accuracy. Below, we restate three key results that underpin the construction such operations within transformer blocks.
Element-wise multiplication via GeLU, The GeLU activation can approximate multiplication,
π 2
(GeLU(r + u) - GeLU(x) GeLU(y))
xy xy + 0(2 v²²),
(71)
enabling scalar products to be implemented through combinations of GeLU activations and additive operations.
Bypassing GeLU. The GeLU nonlinearity can be made approximately linear by shifting its input by large constant according to
GeLU(N + + x) N
N S
(72)
This allows a transformer head to effectively implement identity transformations when needed.
3. Bypassing layer normalization. With appropriate padding and large additive constants, layer normalization can be suppressed, i.c.,
V A(xx, N, -N -N 0)) R |x, 2N, -2N
Σxx x, 0],
(73)
where X(-) denotes layer normalization and E R This result shows that layer norm can be effectively neutralized by padding the input with tailored constants.
29

---

Published in Transactions on Machine Learning Research (02/2026)
G.3 Parameterizing the RAW Operator
Recall the transformer layer formulation:
b0 Softmax (W (W. G(
(74)
A WL .e...,, GO Wio (Wad (4 + )( A + G(t-1),
(75)
(76)
We aim to show that transformer layer with parameters 0 (W Wi, W2, (WQ, WK WWm can approxi- mate the RAW operator. Following the construction in Akküreek et al. (023) this can be achieved using just two attention heads. Key and query matrices are set to implement the token access pattern K(i), enabling the appraximation of
W [((1]] kER())
Σ
G
(77)
The first head is configured as follows:
W W encode the attention pattern K(i).
WY is sparse, with nouzero entries (WY Damm) rin] = (Wa),,n m m= mo0=,,... 1,..., Irl.
This ensures that only entries in are modified, while others remain unchanged, i.e.,
(A + G.
Wa Σ
Gf [r], (A.) + (rerrrrr 1. $ t).
(78)
[(()] kEK()
To cancel the residual in i, the second head set up as
(W2, wxK) K(I) = (W )am), r(n) 1.
(79)
The feedforward projection combines the two heads as
(W )xfm), [ml] = 1, (W )ejm), dmm]-H -1
(80)
This yields the attention output matching expression The complete RAW operator is defined
isolating changes to t.
((++)) W.
[Wa
Wa
Σ
1r wo
(81)
[K(i] kE(()
(++) i.jēw
(1) "ijjéw
(82)
Let m; denote the output of the second MLP layer. To emulate the RAW operator, we require
(mi)rew = W.
Wa
Σ
1 o [C
d. A.[w].
(83)
[i]]] keK()
(mi)eet = - (0) [1] Ar(1],
(84)
(mi)rggow =
(85)
The exact structure of the MLP depends whether represents addition element-wise multiplication.
30

---

Published in Transactions on Machine Learning Research (02/2026)
G.3.1 Additive Operator (o +)
Let Wa be the output of the first MLP layer. Since the GeLU non-linearity can be bypassed by adding large bias term N, we aim to configure u, such that
(ui)e = WG [s] + G + A. + N,
(86)
(u:): = - !G + A.(( +
(87)
(us) - (G [w] + + [w]) + N,
(88)
(e(uiiu))) -N
(89)
This is achieved by sctting the first MLP layer's weights W1 as
(Wi) s(n) W m,
(90)
(Wa [W))) t(x) 1,
(91)
(W1 Da(m), 4/m] -11 1,
(92)
(W1 [m[m), w[m) 11,
(93)
and its bias by as
(b1) (b1)e, (b1) = N, (b (b1)ge(uiuu) = -N.
(94)
(95)
The second MLP layer is configured to combine and cancel terms as needed,
(W2) (W2)m(m)) {[p] (Wo)m,
(96)
(W2) (mm) 4[m]] = 1, (W2) w[m] = 1,
(97)
with bias
(b2) = -N Σ (Wo) (Wo)... - N,
(98)
(b2)afm² =NN,
(99)
(b2) O.
(100)
G.3.2
Multiplicative Operator (o = *)
To approximate multiplication, we introduce three auxiliary hidden slots ta tv te. The first MLP layer's output configured
(u:)... WG (s) c A,4
(101)
G(-1t]+ +A([
(u:)es
(102)
(u)).. (
(1) [s] + A+(s))
(103)
N
(u:): (a + A/tt)) N.
(104)
(u)). - (a! [w] + Av(w)) + N,
(105)
(w)),g(uu, N.
(106)
31

---

Published in Transactions on Machine Learning Research (02/2026)
This behavior is achieved by setting the first MLP layer's parameters as
W
m.
(Wi),, [m]] s(u]
s[m] = N (Wi)c.(m),
(107)
N
1
(Wi)esmm) dfml N
(Wi)e.(m), afm] N
(108)
(Wi) (e))) tfm] = -1,
(Wi) w(m) = -1,
(109)
and bias vector
(bi)nun, = O, (bi)eu = N, (br)sg(urruuunneem = -N
(110)
The second MLP layer computes the product via
wff) 4.[n] = (Wo) N2
(111)
(W2) w[m]. 6//o] = -(W)) mn N2
π
(112)
2
(W2)m(m), ta[m] (Wo),,,N N2
(113)
V 2
(W2) (W2)cfm) wfmll w/m] 1, (W2) {/m] 1,
(114)
with bias
(b2)aur N, (b2)je(sum)) 0.
(115)
Division via LayerNorm. approximation
compute y/c from a structured input vector [c, y, 0J one can use the
2 MZMN A (ING, y/M, -Ne- Σx/M, oi) ~ [MN, s/c, -MN - y/c, 0),
(116)
where M,N > 1. The desired quotient y/c can then be isolated through appropriate weight selection in the feedforward layer.
G.4 Expressing Core Operations via the RAW Operator
Three fundamental operations used throughout this paper can be expressed as special cases of the RAW operator. Specifically:
(11))
= -RAW(G: *, W =, Wa =, W, KF r1
w "i","" +1), [= (1)) VI))
(117)
Aff(G; (i,1), (f, (1","")
W2. (" Wi, RAW(G; +. W = WW:,, W. Wa=Wa W. =1,, bn b b,
s s= (i,)), r= ='''j'), w= (i","" + +1), K =
[tt (t)) Vt))
(118)
mov(G; (f. RAW(G: +. W 0 W. =1 W. =,
8= O. r= (i'))) w= (i.j), K= ((,
(119)
These equivalences follow directly by substitution. Moreover, the dot operator naturally extends to parallel execution, enabling efficient matrix-level implementation of Mul.
32


Reviews:
[{"review_id": "EFvRjup6Y7", "reviewer": "Reviewer_WLHd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores whether transformers trained via in-context learning can exhibit behavior similar to classical filtering methods for dynamical systems. It combines constructive arguments about the expressivity of transformer primitives with extensive experiments on synthetic linear and nonlinear systems, comparing transformer predictions to Kalman, extended Kalman, and particle filters.", "claims_and_evidence_yn": "No", "claims_explanation": "The submission presents substantial empirical results and detailed theoretical constructions. However, as a non-expert in filtering and control, I found it difficult to clearly assess whether the evidence fully supports the claims. In particular, the theoretical components appear to focus on representational capacity rather than guarantees about learning or generalization, and it is not always clear how these relate to the empirical findings. The presentation is technically dense, which makes it challenging to confidently judge the strength of the evidence beyond the reported experiments.", "audience_yn": "Yes", "audience_explanation": "The topic is relevant to researchers working on in-context learning, algorithmic behavior of transformers, and connections between neural networks and classical estimation methods.", "requested_changes": "- More clearly distinguish between empirical observations and theoretical claims.\n- Improve accessibility by summarizing the main contributions and limitations at a higher level.\n- Clarify the intended scope of the conclusions, particularly what is demonstrated empirically versus what remains conjectural.", "broader_impact_concerns": "NA", "additional_comments": "My comments are based on limited domain expertise and focus mainly on clarity and scope rather than technical correctness. Clarifying which claims are theoretical versus empirical would help non-specialist readers assess the contribution.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nnc3BKnlxE", "reviewer": "Reviewer_qRSw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explicitly studies whether transformers can be trained to emulate time series filter algorithms by feeding the state-space system to the transformer as inputs. The paper shows that transformers can fit non-linear dynamics well, but struggles a bit with linear dynamics.", "claims_and_evidence_yn": "No", "claims_explanation": "I'm not totally sure what the claims precisely are. We see that transformers can be trained to predict state space systems, but what does this tell us? Does the network actually use the state-space specification input in a way we think it does? The network isn't able to match exact kalman filter in performance: doesn't this mean that the a transformer can't reproduce kalman filters? \n\nSee below for more comments.", "audience_yn": "Yes", "audience_explanation": "This work surely has interest in the community.", "requested_changes": "See below.", "broader_impact_concerns": "No issues", "additional_comments": "The notation in the paper could be made more consistent. In sec 2 the layer inputs are first described as G^l, then x_t, and finally by q^l [which in appendix is again G^l]. Since the paper is proposing a unified treatment of transformers and state-space systems, it would make sense to use a unified notation as well.\n\nHow is the kalman filter of eqs 13-17 different from eqs 22-27? Why do we need both?\n\nAlgorithm 1 is incomprehensible to me. This should be annotated to make sense of this. If alg 1 is supposed to be used as just a code listing, it would be better given as eg. python code.\n\nMoreover, I'm not totally sure how a transformer is actually used in this paper. I don't understand how the mul/div/aff/trans operations now implement a transformer that executes eqs 22-27 or eq 28. I don't see how eq 28 goes into alg1. Do we still use standard transformers in this paper, or is the transformer some special architecture? What kind?\n\nThe network used seems quite large, and the batch size as well. Was it necessary to have such larger transformers and batches to make this work? If transformers in some sense naturally align with kalman filters, shouldn't we expect much simpler networks to work as well?\n\nWhy do Ridge and OLS fare so poorly in fig1? I would assume that these are gold standard estimators, and should give good performance. Can you explain?\n\nIt seems that the transformer can't predict as well as kalman filter in Fig1. Why not? I thought that we expected transformer to be theoretically capable of solving this problem, and since we explicitly train a transformer to accomplish this task, it should do quite well in practise as well. \n\nOverall I'm a bit confused of this paper's setting. The paper assumes a Kalman algorithm, and feeds all the components of it to a transformer as input, and tries to fit the state-space system. This is an unusual way of using a transformer, so what does this precisely tell us? I was under the impression that the purpose of this paper is to show that transformers use similar mechanisms as kalman filters, and are thus theoretically equally capable. I also thought the paper was about showing how a standard GPT2 emulates kalman filters (without knowing that it's supposed to be doing it). \n\nFinally, even if the transformer is able to fit the state space using the kalman algorithm as input, how do we know the transformer is even using the input? What if the transformer is solving the system in it's own \"style\" and disregards, corrupts or abuses some of the in-context input?\n\nThe paper claims that ridge regression suffers from double descent. I'm not sure if I agree. Doesn't the regulariser mitigate the noise fitting problem of double descent?\n\nWhy the context length is limited to 40? Why not go much further, eg 1000? \n\nIntuitively it would make sense to compare to a vanilla transformer that tries to predict state spaces without knowing about the state space equations. Do you think this would make sense?", "extra_scores": {}, "extra_text": {}}, {"review_id": "nVtiJpUhWK", "reviewer": "Reviewer_jnXo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study the ability of transformers to learn dynamical systems using in-context learning (ICL). Given a set of past input-output pairs generated by a linear or possibly non-linear dynamical system, the objective is to predict the current output given the current input. For a linear dynamical system (LDS) with Gaussian inputs, the Kalman filter (KF) is the optimal estimator, while approximate methods such as the extended Kalman filter (EKF) or particle filter are typically used in practice when dealing with non-linear dynamics.\n\nFrom my perspective, I see two main contributions in this paper:\n1. The authors reformulate the Kalman filter using operations implementable by a transformer, demonstrating that a transformer could potentially learn to emulate a KF. They then provide experimental results demonstrating that, indeed, transformers can indeed do this.\n2. The authors demonstrate that transformers can in-context learn to predict outputs in non-linear dynamical systems with accuracy competitive with commonly-used approximate methods such as the EKF and particle filtering.\n\n## Strengths\n- The authors provide a combination of theoretical and empirical contributions demonstrating that a transformer can learn to emulate a KF using ICL.\n- The paper is well written and easy to understand, even for a reader who may not be familiar with the latest research on transformers and ICL. The important elements of the Read-Arithmetic-Write (RAW) framework (Akyürek et al., 2023) used to establish the theoretical results are summarized in an appendix.\n\n## Weaknesses\n- The plots are trying to show too many things. This is especially true for Figure 5, where there are additional curves for $n=8$.\n- The meaning of the context length for the KF and variants is unclear.\n- The authors consider only one-step forward prediction.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, with both theoretical and empirical evidence.", "audience_yn": "Yes", "audience_explanation": "Yes, a large portion of the machine learning community is currently interested in understanding what transformers can and can't learn, and this paper contributes to that understanding.", "requested_changes": "- Is there a need to have both SGD and Ridge with two learning rates along with OLS in the plots? The OLS behavior doesn't seem too different from Ridge, so I suggest including only SGD and Ridge with a single learning rate, and if desired, including all of the curves in additional plots in the appendix.\n- The x axis on all of the figures shows the context length, for which the meaning is clear for ICL, but not for the Kalman filter and variants. Please explain the meaning of context length for the Kalman filter, EKF, and particle filter.\n- The MSE for ICL and MSPD between ICL and the Kalman filter seems to hit a floor beyond context length of around 30 and does not continue to decrease. Do you have some sense for what is happening and whether the error is due to bias or variance in the ICL's predictions?\n- Figure 7 caption: Why do you state that the transformer performance matches the particle filter? It looks to be doing much better than the particle filter.\n- Please discuss whether it is possible to perform prediction beyond one time step. For example, if $H_t$ remains fixed over time, this seems like it should be possible.", "broader_impact_concerns": "None", "additional_comments": "Suggested minor revisions:\n- Line under equation (20): Please add that you are indexing your tokens beginning with 0, not 1. If using indexing beginning with 1, then the $(2n+1)$th token is $\\sigma^2$, not $y_1$.\n- Figure 6(b): ICL and Kalman filter should probably be ICL and EKF.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)