Learning to Adapt: In-Context Learning Beyond Stationarity
Authors: Zhen Qin
Jiachen Jiang
Zhihui Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: in-context learning
gated linear attention
non-stationary regression
Primary area: optimization

Abstract: Transformer models have become foundational across a wide range of scientific and engineering domains due to their strong empirical performance. A key capability underlying their success is in-context learning (ICL): when presented with a short prompt from an unseen task, transformers can perform per-token and next-token predictions without any parameter updates. Recent theoretical efforts have begun to uncover the mechanisms behind this phenomenon, particularly in supervised regression settings. However, these analyses predominantly assume stationary task distributions, which overlook a broad class of real-world scenarios where the target function varies over time. In this work, we bridge this gap by providing a theoretical analysis of ICL under non-stationary regression problems. We study how the gated linear attention (GLA) mechanism adapts to evolving input-output relationships and rigorously characterize its advantages over standard linear attention in this dynamic setting. To model non-stationarity, we adopt a first-order autoregressive process and show that GLA achieves lower training and testing errors by adaptively modulating the influence of past inputs--effectively implementing a learnable recency bias. Our theoretical findings are further supported by empirical results, which validate the benefits of gating mechanisms in non-stationary ICL tasks.

Published as a conference paper at ICLR 2026
LEARNING TO ADAPT: IN-CONTEXT LEARNING BEYOND STATIONARITY
Zhen Qint.l.,,Jiachen
Jiachen Jiang', Zhihui Zhu
Michigan Institute for Computational Discovery and Engineering, Department of Electrical Engineering and Computer Science,
Department of Statistics, University of Michigan 'Department of Computer Science and Engineering, The Ohio State University
ABSTRACT
Transformer models have become foundational across a wide range of scientific and engineering domains due to their strong empirical performance. key capa- bility underlying their success is in-context learning (ICL): when presented with a short prompt from an unseen task, transformers can perform per-token and next- token predictions without any parameter updates. Recent theoretical efforts have begun to uncover the mechanisms behind this phenomenon, particularly in super- vised regression settings. However, these analyses predominantly assume station- ary task distributions, which overlook a broad class of real-world scenarios where the target function varies time. In this work, we bridge this gap by providing a theoretical analysis of ICL under non-stationary regression problems. We study how the gated linear attention (GLA) mechanism adapts evolving input-output relationships and rigorously characterize its advantages over standard linear atten- tion in this dynamic setting. To model statiionnon,. we adopt first-order au- toregressive process and show that GLA achieves lower training and testing errors by adaptively modulating the influence of past a learnable recency bias. Our theoretical findings are further supported by em- pirical results, which validate the benefits of gating mechanisms in non-stationary ICL tasks.
implementing
INTRODUCTION
Transformer- baased architectures (Vaswani et al., 2017) have emerged as powerful and versatile modeling framework, achieving state-of-the-art results across a wide spectrum of scientific and engineering domains. Their remarkable effectiveness has been demonstrated in natural language processing (Radford et al., 2019; Brown et al., 2020). recommendation systems (Zhou et al., 2018; Chen et al., 2019), reinforcement learning (Chen et al., 2021; Janner et al., 2021). computer vision (Dosovitskiy et al., 2020), and multi multi-modal signal processing (Tsai et al., 2019), as well as in more specialized areas such quantum information (Ma et al., 2025) and wireless communication sys- tems (Kim et al., 2023). A particularly notable instance is their pivotal role in the development of large language models like GPT-4 (Achiam et al., 2023), where the Transformer backbone enables highly advanced generative capabilities.
A distinctive and increasingly studied feature of Transformer models is in-context learning (ICL) (Min et al., 2021), which allows the model to perform previously unseen tasks at inference time by conditioning on sequences of input-output examples, without requiring any explicit parameter updates. This emergent capability has spurred a growing body of research aiming to understand the underlying mechanisms that enable such behavior (Brown et al., 2020; Min et al., 2021; Dong et al., 2022: Wies et al., 2023; Zhang et al., 2023; Bai et al., 2023; Li et al., 2024a; Bertsch et al., 2024; Akyürek et al., 2024: Jiang et al., 2025a; Song et al., 2024; Wu et al., 2023; Qin al., 2025). In particular, recent theoretical works have investigated the realization of ICL in supervised regression settings, showing that certain architectural components-sun such as linear attention mechanisms effectively emulate simple learning algorithms. c.g., single step of gradient descent, when the input data distribution is stationary (Garg et al., 2022; Akyürek et al., 2022; Von Oswald et al.,
"Corresponding Author: zhengin@umich.edd

---

Published as a conference paper at ICLR 2026
2023: Zhang al., 2024; Huang al., 2023; Chen et al., 2024; Yang et al., 2024; Zhang al., 2025; Mahankali et al., 2023; Ahn et al., 2023; Li et al., 2024b; 2025; Fu et al., 2024; Ding et al.). These findings offer valuable insights into the algorithmic behaviors implicitly encoded by architectural design, shedding light on the interplay between representation, memory, and adaptation in modern Transformer models.
However, much of the existing theoretical understanding is limited to stationary data settings, where the input-output relationships remain consistent across in-context examples and the query point. In contrast, many practical scenarios-including time-series forecasting, streaming data, and natural language-exhin exhibit non-stationarity, where the underlying target function evolves over time. In such settings, recency bias, or the increased predictive relevance of more recent examples, plays a crucial role accurate prediction. Empirically, linear attention mechanisms are often insufficient for these non-stationary tasks, motivating the introduction of architectural variants that incorporate inductive biases better suited for adaptation, such as gated linear attention (GLA) (Yang et al., 2023; Jiang et al., 2025b), RetNet (Sun et al., 2023), Gateloop (Katsch, 2023), RWKV-6 (Peng et al., 2024), as well as state-space models like Mamba-2 (Gu & Dao, 2023). These methods have achieved strong performance in non-stationary sequence modeling, yet there remains a lack of formal theoretical understanding of their behavior in ICL settings.
Contribution: In this paper, aim to bridge this gap by presenting theoretical analysis of ICL in non-stationary or time-varying regression problems. We investigate how the GLA mechanism adapts evolving input-output relationships and provide rigorous characterization of its advantages over standard linear attention in this setting. To model non-stationarity, we adopt a first-order autoregressive process, which allows us to analytically capture temporal variations in the regression targets. Within this framework, we show that standard linear attention incurs higher training and testing errors due to its limited capacity to adapt to distributional shifts over time. In contrast, GLA exhibits inherent adaptability by dynamically modulating the contributions of past inputs, effectively inducing learnable recency bias. This gating mechanism enables the model to better accommodate time-varying input-output mappings, therehy achieving more robust in-context generalization. Our analysis underscores the importance of architectural components-particulaat) particularly gating-in equipping transformer models with the ability to implement adaptive learning algorithms in non-stationary environments. Experimental results further corroborate our theoretical findings. Collectively, our work contributes theoretical perspective that clarifies the design choices behind transformer variants and offers a conceptual framework for understanding and developing architec- tures suited for adaptive ICL.
Notation We use bold capital letters (e.g., Y) to denote matrices, bold lowercase letters (e.g.- y) to denote vectors, and italic letters (e.g., y) to denote scalar quantities. Elements of matrices denoted in parentheses, in Matlab notation. For example, Y (($1,2) $2) denotes the element in position ($1, of the matrix Y. The inner product of A E IR xd2 and B E Id xd2 can be denoted as (A,
Σde Ede A(81, B ($1,82). | || represents the Frobenius norm of X. Og and Oaxa denote the zero vector in I2 and the zero matrix in [Ædxd, respectively. For positive integer K, [K] denotes the set [I....,K}.
1.1 RELATED WORKS
A growing body of work has investigated the emergent phenomenon of ICL, with a focus on un- derstanding its behavior in stationary regression tasks. For example, (Garg et al., 2022) empirically demonstrated the ICL capabilities of transformers by analyzing prompts where each input is labeled by task-specific function drawn from a predefined function class, such as linear models. Along similar lines. (Akyürek et al., 2022) investigated linear regression and introduced transformer construction capable of performing single gradient descent (GD) step using in-context examples. Building upon this, (Von Oswald al., 2023) designed weight matrices for linear attention-only transformers that replicate GD updates in linear regression tasks, and notably, they observed that the learned weights resemble those obtained through end-to-end training on ICL prompts.
Further progress has been made by studying the convergence behavior of transformer architectures. In particular, (Zhang et al., 2024) showed that, for a single-layer linear self-attention model, gradient flow with carefully chosen random initialization converges global minimum, yielding low pre- diction error on anisotropic Gaussian data. Complementary work by (Huang et al., 2023) initiated the theoretical study of softmax attention, analyzing the training dynamics of one-layer, single-head
2

---

Published as a conference paper at ICLR 2026
transformers and providing convergence guarantees for linear regression. This line of research was subsequently extended by (Chen et al., 2024; Yang et al., 2024; Zhang et al., 2025), who provided sufficient conditions for the convergence of multi-head softmax transformers trained with GD in ICI. scenarios. Alternative theoretical perspectives have also been explored: for instance, (Ma- hankali et al., 2023) demonstrated that transformer performing single GD step on least-squares objective can serve as a global minimizer of the pre-training loss, offering a different interpreta- ICL. Similarly, (Ahn al., 2023) showed that single-layer model,
tion of training objectives when trained on random linear regression tasks, implicitly learns to perform a preconditioned GD step at test time, further reinforcing the connection between ICL and optimization based learning rules. Meanwhile, (Li et al., 2024b; 2025) offered theoretical interpretation of GLA through the lens of weighted preconditioned GD, although their analysis remains limited to stationary regression settings. Beyond first-order methods, more advanced optimization techniques have also been con- sidered; for example, (Fu et al., 2024) analyzed the convergence behavior of second-order methods in ICL, highlighting their potential for accelerated adaptation relative to first-order approaches.
IN-CONTEXT LEARNING TIME-VARE VARYING FUNCTIONS
This work builds upon the well-established in-context learning (ICL) framework introduced in (Garg et al., 2022), which aims to train models capable of performing ICL within specified function class. As discussed in prior work, significant efforts have been devoted elucidating the mechanisms un- derlying ICL. In particular, a number of studies (Garg et al.. 2022; Akyürek et al., 2022: Mahankali et al.; Ahn et al., 2023; Huang et al., 2024; Zhang et al., 2024; Li et al., 2024b; 2025; Zhang et al., 2025) have investigated the dynamics of ICL. in transformer architectures through the lens of lin- car regression tasks, where the target function is typically assumed to take the form f(x) = (w, x). However, these studies commonly rely on the simplifying assumption that the regression weight vec- tor w remains fixed throughout the task. This stationarity assumption creates a theoretical- -pracccccl gap, as it does not faithfully reflect wallwwrldd scenarios in which data distributions are often non- stationary and the underlying regression weights may vary across different input samples.
In-context Learning Time-varying Functions To bridge this gap and advance the theoretical un- derstanding of ICI. in non-stationary settings, we introduce a more realistic framework in which the labels in the training prompt are generated by time-varying functions. Formally, let Dx denote distribution over inputs and Dr; a time-varying distribution over functions in F. A prompt P is defined as a sequence (æ (zx),,.... ... Dn, (x(,,,,, Fquery), where the inputs æ 21.... Bn E Rd and query Equery = In+1 Ed are drawn from Dx and each f. is drawn from Dj. One may consider two canonical types of time-varying functions inspired by the literature:
Deterministic time-varying functions: Here, f+ = f(,i/(n + 1)), where f is assumed vary smoothly over rescaled time. This setting captures gradual and predictable evolution in the un- derlying mapping, as extensively studied in time varying nonlinear regression models (Zhang & Wu, 2012; 2015).
Stochastic fime-varying functions: In this case, the evolution of fa is modeled a stochastic process, allowing for random fluctuations in the function mapping. A representative model is f((z) 7ft 1 (x) + cxr), where 0 < & < is a forgetting factor modeling gradual drift in task mappings and (x) is zero-mean stochastic perturbation.
Definition 1. We say that a model M can in-context learn the time-varying function class Fs up to accuracy with respect to (F), if it can predict fa+1l (I.qurr))) based on the prompt P with average error
Ep [e(M(P), fn+1 ( Sc.,
(1)
where E(:y)) denotes an appropriate loss function, such as squared error.
Within this framework, we then pose the following central question: Question: Can we train a model to in-context learn a given time-varying function class?
In this work, facilitate theoretical analysis while preserving non-stationarity, we consider simple yet expressive instantiation of the function class:
y/ (z;)) = (w, 2.)) E R, [n + +1]. where each weight vector w, evolves according to first-order autoregressive process given by
(2)
w: = 700;1 es [n + 1].
(3)
3

---

Here, 0 < I 1 is the autoregressive coefficient that controls the temporal correlation of the weight vectors, the sequence w, follows a random walk model, which is a widely adopted generative model in signal processing and adaptive filtering literature (Sayed, 2011). To facilitate tractable analysis, Lid. we further assume that the initial weight vector is drawn i.i.d as Wo ~ NNO,, oN0,,.,,)) the noisy terms Lid Published as a conference paper at ICLR 2026
are i.i.d. Gaussian with' et ~ (N, oT), and the input vectors are i.i.d. samples from a zero- iid mean Gaussian distribution with covariance matrix æ; ~ (N), Moreover, we assume that the random variables Wi-1, ei, and are mutually independent. Following a long line of theoretical work on in-context learning (Mahankali et al.; Ahn et al., 2023; Zhang al., 2024; Chen et al., 2024; Yang et al., 2024; Li et al., 2024b; 2025; Zhang al., 2025). we adopt Gaussian assumptions in our analysis. This modeling choice enables sharp and explicit characterizations test errors-rather than only providing loose upper bounds-and is therefore essential for isolating how key quantities such as govern the behavior of the learned in-context learner.
both the training and
Gated Linear Attention In the non-stationary regression setting introduced above, where the un- derlying task weights evolve gradually over time, it is crucial for the model to effectively capture pairwise correlations while adapting to the dynamics of changing tasks. Although standard lin- ear attention mechanisms offer computational efficiency and scalability, they lack the flexibility to modulate the influence of prior context based on its relevance to the current input-an ability that is particularly important in nonstationary environments.
To address this limitation, we employ Gated Linear Attention (GLA) (Yang et al., 2023; Li et al., 2024b; 2025), which enhances linear attention by introducing gating mechanism that controls the flow past information. This structure enables the model to selectively integrate relevant histori- cal patterns while suppressing outdated ones, thereby offering better inductive bias for capturing evolving structures in non-stationary tasks.
Formally, we consider the following implementation of GLA. Let We E R(d+1)x(4+1), Wx E [(d+1)x(d+1) and Wy E [dd+1)x(d++] denote the query, key, and value weight matrices, respec- tively. To streamline the subsequent analysis, we follow prior works (Ahn et al., 2023; Huang al., 2024; Zhang et al., 2024; Li et al., 2024b; 2025; Zhang et al., 2025) and construct the prompt by evaluating each function f. on the sampled inputs and pairing each input with its corresponding output:
x1 y1
Fn+1 Rdd+1)x(n+1),
Z = [21
Zn
2n+1]
=
(4)
For each input 2,, we define the corresponding query, key, and value vectors as q: = Woz;, k; Wx and v; = Wvz, The output of GLA at position i is given by:
o S,q, and S.
(5)
where À E 10 is a forgetting factor that determines how quickly the attention mechanism dis- counts earlier information. For ease of theoretical analysis, we adopt simplified formulation where a single global forgetting factor À is used, rather than assigning separate, data-dependent gating coefficient to each token as done in the original GLA model. By unrolling the recursive update in (5), we obtain:
a+1
n+1
S AS, + Untik
Σ i=1
V,k = Wv
Σ 1=1
x n++1 An+1-iz(2]] W
(6)
which leads to the following expression for the output vector:
n+1
On+1 = Sa+11q+1 = Wv
Σ
WR Woz
(7)
is worth noting that when 1, the weighted sum degenerates into an unweighted accumulation, ie., En+1 2,z, = Zz under which the GLA formulation reduces to the standard linear attention
'One may assume of (1 20 to ensure F|][ - In this work, relax this constraint to allow more general settings.
4

---

Published as a conference paper at ICLR 2026
model. This highlights that GLA generalizes linear attention by introducing a learnable memory decay.
Since the final prediction is taken the last entry of the token vector output by the GLA layer, only a subset of the entries in the weight matrices Wy and Wq. Wx influence the output. To simplify the notation and subsequent analysis, we merge the query and key matrices into single matrix and define
WH 2
w wx1
w 11
KQ] 12
W
E [dd+1)x(d+1) and WKQ=
KQ
T
R R(d+1))(4+3),
(8)
22
w
where WH. 11 decomposition, we express the predicted output as
Rdxd wyz why w12 w2n
KQ KQ
e Rdx1 and w 1. w-
KQ
E R.
Using this
w KQ W221
Tn+1 = 1 (d = w [[
=
Σ Antll'zu 2,z7
In+1-
(9)
Note that only the last row of Wr and the first d columns of WKQ contribute to the final prediction. Therefore, without loss generality, we may set the remaining entries Wy and WKQ to zero in the subsequent analysis.
THEORETICAL ANALYSIS OF GLA FOR TIME-VARYING REGRESSION
In this work, we investigate the convergence behavior, training error, and testing error of ICL linear predictors based the GLA model for time-varying functions. Suppose we are given B independent in-context learning training tasks, where each task prompt corresponds to an embedding matrix Z,, for r I..... constructed according the transformation defined (4):
Ir,1 27,0 2r,4 (,,,,, 2r,1)
T Dr,a (Wr,n; Draa)
Ir.n-1
Z, = Zr,1
(10)
0
where the weight vectors w evolves according to (3).
We denote the prediction produced by the GLA model on the query input of task as Vr. I, whose exact form is given in (9). The empirical risk over B independent task prompts is then defined as:
((0) = 2B Σ (Jrm+1 (WDr,++. ( xr,n+)))
(11)
where the model parameters denoted by 0 (WKQ, Wv). To analyze the learning dynamics, we consider the population risk induced in the limit as the number of training prompts tends to infinity, i.c., B oc:
L(O)= lim ((0) Esc., 1. 12 [0 (w.++1 2n 1n) B-+00
(12)
where we omit the task index for notational simplicity.
We study the evolution of the model parameters under gradient flow, which characterizes the continuous-time limit of gradient descent with infinitesimal step sizes. The parameter dynamics are governed by the ordinary differential equation de - (()). To facilitate the analysis, fol- lowing the approach of (Zhang al., 2024), we introduce an initialization scheme that satisfies the following assumption.
Assumption 1. (Initialization) Let a > be parameter and Q E Rdxd be any matrix satisfying |00 |F = and AQ # Oaxd Rdxd, assume
Wv Oaxx Og E (d++1)((d+1) and Wk) = of
Tee Og E (dd+1)x(d+1) (13)
Under this setup. the following result establishes that the gradient flow dynamics with respect to the population loss converge specific global optimum.
5

---

Published as a conference paper at ICLR 2026
Theorem 1. (Convergence of gradient flow) Consider gradient flow over the population loss in itd.
(12), Assume that < 1, the initial task weight wo ~ NNO I noises e, ~ ((0,o22) and
itd.
imputs æi Ltd. ~ Ni A). Suppose the initialization satisfies Assumption I with initialization scale
satisfying v
2D, where
AdllAll
2 +
1²11-12m)
(1-27)²
1-x= n1
2
de:
A A=
D1 =
2 -2y 22212
>-(1->">")
A 2g2n12
A-y
0
[1-
(x-7)(1-7ン)
02,
X X+7
and À D2(2A trace + D&A with
22
14 )0?
A A=
D2
7 2022 w
22, 2n+2
1 x
(
(2 2
x1-X11²
X 22_ 2n+2
J02:
A x *?? ?
X2
(2#-
(X²
(1-32](1-23)
and
A 22+ 1)0 +
- 22
2(22n+4 nd n2 (1 112--1122) 23m+2 2(m- Joe: A=
11-2272 2272
(1-22)3
Ds
27 222+1 33220+3-22200 22
2 Ar 2 2n
22
)ou + ( 2g 2 -22
-22=22
A(A
11 7/11 225
2(2 2nt2, 2
20 2 22
20112
27 12 2, )o::
A A 7.
(1-721(2-713) 71(1 (111-17)) Xx)
X(X (x+-))(+))
(A- 73(1
Then gradient flow converges to global minimum of the population loss (12). Moreover WkQ(O) and (v) respectively converge to
Wv(() DDDIAAA1 1+00
Oaxd 0
021
and lim WxO(t) = = "IF 1-300
0.
(14)
1
The proof is deferred to Appendix B. Despite the non-stationary nature of the regression model considered in this work, we establish that gradient flow converges global minimum even under random initialization. The closed-form solution in (14) reveals that the location of the global op- timum is explicitly determined by À and 7, highlighting their structural influence the solution. While the main theorem focuses on the regime < A < 1 and 0 < < 1. more general result accommodating arbitrary
and y >> is established in Theorem 4 of Appendix B. Moreover, in the limiting case where À 7 1 and of 0. the result reduces precisely to that in (Zhang et al., 2024, Theorem 4), thereby recovering the stationary setting as a special case of our more general formulation.
Training error We now analyze the training error of the learned network. At the global optimum- i.c., when the parameters converge to lim (t) and lim400o WKQ (t) in (14), straightfor- ward calculation yields the prediction In+1 as follows:
0a+1- | 1]
Σ
n+1 1
2,2,
A 0
Dat1 = D1 Σ X 1=1
w, x, TT,a,, ) - xn11- (15)
This expression confirms that, for sufficiently long prompts, the trained model successfully in- context learns the family of linear predictors. We emphasize that both À and y jointly influence the degree of time variation in the underlying model. We next quantify the training error at the global optimum.
Theorem 2. (Training error) Assuming the conditions in Theorem 1 hold, the recovery error be- tween (15) and (2) is
[[O++1 2 = Dî trace (D2(A trace( AĀ 'A) + 2AĀ 'Ä -'A) AA
+D2AĀ- AĀ AA + DA trace(A) 2D trace (AĀ (16)
where D4 = - 22
20+2 1-7²
o².
The proof is provided in Appendix C. Equation (16) illustrates that the training error depends jointly the parameters À and 7. Consequently, for fixed 7, there exists optimal value of X

---

Published as a conference paper at ICLR 2026
that minimizes the enror. Although the expressions of D; suggest symmetric structure in À and y, it does not necessarily imply that choosing = 2y minimizes the recovery error. In fact, the error involves a subtle balance between the - and o² -dependent terms as well as the trace terms with A-1 When = y, the simplification of D; may amplify certain crise-dependent factors and deteriorate the overall error. This observation highlights that the optimal choice of À depends not only on the apparent algebraic symmetry but also on the interplay between noise statistics, system dimension, and the spectral structure of A.
We now consider a special case with A I, in which the training error in (16) reduces to AD; B[( ya+1) ) dD4 D2+d)D2++;" The first term dD4 only depends the autoregres- sive process and is independent the choice of x. To study how the second term varies according to A, assume 7 1 is fixed and d is sufficiently large, such that (2+d)Da+ Di/D2 {(A).
aDf
To further illustrate how E(A) might vary with X, we consider two particular cases. Case
de < Ow- In this regime, D1 and D2 respectively dominated by X ant1,n+2-x0 2222 o and y2x PPx2+22_X2 22,2422 x² Te. Consequently, E(A) can be well approximated by 2 22014 (AAA0"""()A)))) o
A-7
(A"+ (1A))))())))))) (A->)
which increases monotonically with À on (0, 7) and then decreases monotonically on (ni 1). Case II: Ay n is sufficiently large such that Dj and D2 are dominated by (1-1")(1>>) and x² 22n-2
(1->")(1 1-1
-²11-13) respectively. In this case, E(A) & (1->"[[--App(0--"""-p------ which likewise increases monotonically with À on (0, 7) and then decreases monotonically on (7, 1). Although these results are derived under simplifying approximations, they suggest that the training loss exhibits an inverse U-shaped The subsequent dependence on X, attaining its minimum some À < 1 rather than X experiments provide direct validation of these theoretical predictions.
Testing error In this part, we characterize the prediction performance of the trained transformer when evaluated on test prompt drawn from potentially different task distribution. Notably, the model parameters are fixed at their global optimum obtained from training, and the test prompt may differ in its length, data distribution, and underlying dynamics. We consider test prompts of the form
Tm
Im++ 0
Z
Zm
2m+1]
F1
7m
36
3m (Wm, mmm)
Em
E
(17)
(wi, mi)
where the latent task weights w. )== evolve according to the first-order autoregressive model W. = 7 wi-1 + ε,,i = 1,..., + 1. To distinguish between training and testing distribu-
iid N/O, I)., and the driving noise
tions, we assume that the initial weight vector satisfies Wy e, Lid. T.. The inputs are drawn independently as 2 (O, A), and we assume mutual independence among random variables w.- e, and T.,
iid.
Given forgetting factor X, the prediction T produced by the model at test time (evaluated at the training global optimum)
m
Um++ D1
Σ x W. x. [,,,))A Tm+11
(18)
We now characterize the mean squared prediction on the test prompt: Theorem 3. (Testing error) Under the assumptions in Theorem 4, the expected prediction error of the model the test prompt is given by
Attrcee À - + 2λĀ AA- E [ 12 = D2 trace (A tracef
+ DDAA 'AĀ-1A) AAA + DA trace(A) 2D D1 trace AÅ -A), (19)
where D, for i 1,...,4 4 are defined analogously to the D, constants from training, with the substitution X + X, - 7, 02
o2 o² 0² and -> m.
The proof has been provided in Appendix D. This result quantifies the generalization behavior of the trained model when applied to unseen prompts sampled from a potentially different distribution.
7

---

Published as a conference paper at ICLR 2026
Notably, the prediction error depends jointly on the training and testing task statistics through the in- teraction between A and A. Moreover, the expected error [] - +m++1 1 )o] inherently nonzero due to the stochastic nature of the task troduces irreducible uncertainty in the test labels V. This highlights the importance of employing GLA, which adaptively modulates the influence of past observations and better accommodates tem- poral variations in the underlying regression weights. In the subsequent experimental section, we empirically demonstrate the effectiveness of the GLA mechanism in handling non-stationary tasks.
celluttoon-spccffccall
the noise in the dynamics of w, in-
Comparison with Adaptive Signal Processing The non-stationary regression setting considered in this paper is closely related to classical problems in adaptive signal processing, where the under- lying model parameters evolve gradually over time (Sayed, 2011; Das et al., 2015; Abdolee et al., 2016; Qin et al., 2020; Claser & Nascimento, 2021; Yu al., 2021; Wang al., 2022). To track such non-stationary dynamics, wide range of online algorithms have been developed, including the least mean squares (LMS) algorithm, the affine projection algorithm (APA), and the recursive least squares (RLS) algorithm. These methods are designed to update model parameters iteratively in response to streaming data, with the goal of minimizing instantaneous or long-term prediction error. Under non-stationary models such as the first-order autoregressive process described in (3), the corresponding theoretical error analyses for these methods also indicate that, for a fixed y, there exists an optimal choice of step size (in LMS/APA) forgetting factor (in RLS) that minimizes the tracking error.
While classical adaptive signal processing methods explicitly update model parameters over time based on streaming observations, the paradigm studied in this paper-in-context learning with the GLA model-adopts a fundamentally different approach. Instead of relying on explicit parameter updates, as in LMS, APA, or RLS, the GLA implicitly adapts to task dynamics via internal repre- sentations conditioned on the prompt. In particular, the gating mechanism in GLA enables the model to selectively integrate past information in soft and differentiable manner, thereby tracking non- stationary structures without modifying perspective on learning in non-stationary environments, where adaptation arises not from external optimization procedures, but from the model's forward computation itself.
parameters. This architectural distinction offers new
4 EXPERIMENTAL RESULTS
In this section, we present experiments to validate the theoretical analysis and demonstrate the advantages of GLA in non-stationary models. The experiments lowing settings. The training and testing losses are defined as B Σ 8 (1m+1 - 3m+1)² 22² respectively. Unless otherwise specified, we set d 10, n 100, of 1, of = 0.01. and
conducted under the fol-
(0r.+1 1r,++1)² and
10". The AdamW optimizer adopted with learning rate 10-2, weight decay 0.05, and momentum parameter 0.9. Each model is trained for 2000 epochs with a batch size of 5000 samples. The loss associated with the optimal is highlighted by a star. Although the theoretical analysis imposes constraint on the initialization matrix, our experiments use random Gaussian initialization and still observe the predicted behavior, indicating that the constraint is not necessary in practice.
18
22
2,00
La
0.5
y-0.22 210
31p 1-020
26
04
04
0.2
0.4
0.5
0.2
1.0
0.
022
0.4
0.6
0.5
1.0
00
0.2
0.5
0.5
1.5
0.
02
0.4
3.6
0.5
10
(a) Training loss
(b) Testing loss 10) (c) Testing loss (m = 50)
(d) Testing loss (m 100)
Figure 1: Training and testing performance of the one-layer GLA model with different À and
The first experiment compares the training and testing performance of the one-layer GLA model under varying choices of y and X. As shown in Figure la, when the autoregressive coefficient decreases and the impact of noise becomes more pronounced, an appropriate choice of À is required to attain the lowest training loss. During testing, we evaluate the GLA model trained with X = 0.9 under different sequence lengths m E 500,50, 1003. The results in Figures 1b to 1d show that, across different values of 7, selecting an appropriate À remains crucial for minimizing the test loss.
8

---

Published as a conference paper at ICLR 2026
These results highlight the role of GLA in stabilizing learning under non-stationary conditions. By introducing a gating mechanism into linear attention, GLA effectively regulates the influence of past inputs, thereby mitigating error accumulation and enhancing the model's adaptability to distribu- tional shifts. Consequently, GLAA achieves longer effective memory and improved generalization, underscoring its advantage in handling time-varying data. As mentioned previously, a one-layer GLA model applied to first-order autoregressive process functions analogously to an adaptive fil- ter. To illustrate this, we compare its performance with LMS and RLS algorithms. We set the LMS step size to 0.01 and the RLS forgetting factor to 0.98, train on sequences of length 1000, and perform 10,000 Monte Carlo trials, averaging the results. The training errors for LMS and RLS respectively [0.2639 0.3168 0.6058 1.0072 1.4758] and [0.2555 0.3746 0.6658 0.8881 1. 1.2916] for y [0.8 0.85 0.925 0.95 0.975). Compared to LMS and RLS, which require fixed or slowly adapting parameters, one-layer GLA model achieves lower training errors (see Figure la) because it possesses higher representational flexibility. Furthermore, LMS and RLS adapt only to a single sequence time, requiring retraining for each new input, and therefore cannot leverage cross- sequence information. In contrast, GLA's learnable weights are shared across sequences, allowing the model to generalize and adapt efficiently to new inputs without retraining.
2.85
0.50
0.55
0.25, D.
Layer
25.75
20.0
$1.00
LLyees
Layer
3
P9.0
5.775
Layer
Liyers
0.5.2 2-.2--00 6. 0.225 1-000 8.55,
byyer
Layee
Laysrs
2.5
3.00
Layer
0.00
Loyers
0.3
500
1005
2500
2020
500
1000
1520
2000
Edctts
(a) Training loss
(b) Testing loss (m = 100)
(c) One Layer
(d) Multiple Layers
Figure 2: (a-b) Training and testing performance of the multi-layer GLA model with y = 0.95 and
different A; (c) convergence performance of the one-layer GLA model; (d) convergence performance for GLA models with different layers.
In the second experiment, we investigate the impact of network depth on the performance of the GLA model. As illustrated in Figures 2a and 2b, increasing the number of layers consistently en- hances both training and testing performance, suggesting that deeper architectures can more effec- tively capture mangrange dependencies in non-stationary sequences. Conceptually, each GLA layer implements linear adaptive filter whose effective behavior is determined by its gating weights. When multiple layers are stacked, these adaptive filters operate at different timescales, enabling the network to simultaneously capture short-term fluctuations and longer-term trends in the evolving regression weights. This mulli-timescale structure explains why deeper GLA models achieve better performance under non-stationary regression: single layer can track only one effective timescale of drift, while multiple layers collectively approximate a richer family of dynamic predictors. While formal theoretical analysis for multi-layer GLA models is not yet established, the empirical results underscore the critical role of the adaptive gating mechanism in regulating information flow across layers, thereby mitigating error accumulation and improving generalization.
Under the same experimental settings as the first two experiments, we examine the training conver- gence of the layer GLA with the optimal À corresponding to the minimum loss, and of the multi- layer GLA with X = 0.85. With random Gaussian initialization and a sufficiently large number of training samples, Figure 2c shows that the one-layer GLA achieves linear convergence, in agreement with our previous analysis. Figure 2d further demonstrates that the multi-layer GLA maintains linear convergence, indicating that the adaptive gating mechanism effectively stabilizes gradient propaga- tion across layers. A rigorous theoretical characterization of convergence for multi-layer GLA is left. for future work.
In the third experiment, we assess the ICL capability of GLA and Linear Attention (LA) models on a real-world language task. We focus on sentiment classification using the SST-2 dataset (Socher et al., 2013), which contains 67,349 training samples and 872 validation samples with binary labels (positive/negativee.. To initialize the models, we employ GPT-2 (small) (Radford et al., 2019), which consists of 12 layers, hidden size of 768, 12 attention heads, and approximately 117M parameters. We then replace the original softmax attention with (i) linear attention, resulting in LinearGPT2, and (ii) gated linear attention, resulting in Gatedl iPT2 Both models optimized using AdamW

---

Published as a conference paper at ICLR 2026
with learning rate of 5 10-3, weight decay of 0.05, and momentum parameter of 0.9 for 1,000 iterations. For ICL fine-tuning, we provide 20 in-context demonstrations per instance, computing the loss only on label tokens. During evaluation on the SST-2 validation set, we vary the number of demonstrations K (1,5, 10, 15, 20). Performance is assessed using two metrics: (1) Accuracy, defined as the standard prediction accuracy; and (2) Confidence, calculated for each correctly clas- sified example by converting the model's logits over positive, negative to probabilities (Ppos Poeg) and taking max((pps: Pnog), with the reported value being the average over all correctly classified examples. As shown in Figures 3a and 3b, when À 0.9, GLA achieves the highest accuracy and confidence, outperforming LA by a clear margin. This empirical advantage can be attributed to its gating mechanism: unlike LA, which implicitly assumes stationary linear regression struc- ture, GLA is able to adapt to the non-stationarity of real-world data by selectively integrating or discarding historical information-an ability that proves critical for reliable prediction.
(a) SST-2 Accuracy
(b) SST-2 Confidence
(c) MNLI Accuracy
(d) MNLI Confidence
Figure 3: Accuracy and confidence of Gatedl .nterdiinerrPP2 vs. LinearGPT2 on SST-2 sentiment clas- sification (left two) and MNLI natural language inference (right two) across different numbers of demonstrations.
In the final experiment, we evaluate the ICL capabilities of GLA and Linear Attention (LA) models on more challenging natural language inference task, which requires determining the logical re- lationship between premise-hypoteee hypothesis pair (entailment, contradiction, or neutral) across broad range of text genres. We use the Munti-Genre Natural Language Inference (MNLI) dataset (Williams et al., 2018), which spans multiple genres and contains approximately 393k training examples with three class labels. Following the same setup as the third experiment, we provide 10 in-context demonstrations per instance for ICL loss only on the label tokens. For evaluation on the MNLI validation set, we vary the number of demonstrations K 3,,3,5, 7, 10). As shown in Figures 3c and 3d, GLA consistently achieves higher accuracy and confidence than LA, highlighting the benefit of the gating mechanism.
by context length-and compute the
CONCLUSION
This work presents theoretical investigation of in-context learning in non-stationary regression problems, addressing an important gap in the current understanding of transformer models. Un- der a first-order autoregressive model of non-stationarityy we show that GLA outperforms standard linear attention by dynamically reweighting past inputs, enabling more accurate prediction in time- varying settings. Our analysis provides rigorous justification for the advantage of gating in captur- ing distributional shifts and highlights its role as architectural inductive bias in adaptive learning- These findings not only deepen the theoretical foundations of ICL in dynamic environments but also suggest broader implications for the design of transformer variants in real-world applications characterized by non-stationarity.
A natural direction for future work is generalize the first-order autoregressive assumption broader class of dynamic-weight models. In particular, allowing more flexible temporal evolutions- such as higher-order dynamics, stochastic drift, or slowly varying adversarial changes-would further illuminate how in-context learning behaves in general non-stationary settings. A second direction for future work to develop rigorous theoretical characterization of how gating mechanisms interact across multiple GLA layers. While our experiments show that stacking layers consistently improves performance, a principled analysis of how multi-layer structures capture multiple timescales of drift remains an important open problem. The third future direction is to analyze the global optimization landscape of the GLA model studied in this paper. Our numerical experiments suggest that random Gaussian initialization consistently converges the theoretical initialization conditions are violated. This indicates the existence of a benign global optimum and motivates deeper theoretical study of the model's optimization landscape.
global minimum under gradient flow, when
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The work has been supported in part by NSF grants IIS-2312840 and LIS-24029522 ZQ gratefully acknowledges support from the MICDE Research Scholars Program the University of Michigan.
REFERENCES
Reza Abdolee, Vida Vakilian, and Benoit Champagne. Tracking performance and optimal adaptation step-sizes of diffusion-Ims networks. IEEE Transactions on Control of Network Systems, 5(1): 67-78, 2016.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint ar arXiv:2303.087744 2023.
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to imple- ment preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 06645614-45657 2023.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learn ing algorithm is in-context learning? investigations with linear models. arXiv preprint vrXiv:2211.15661 2022.
Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-context language learningg Archi- tectures and algorithms. arXiv preprint TrXiv:2401.12977 2024.
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems, 36:57125-57211 2023.
Amanda Bertsch, Maor Ivgi, Emily Xiao, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.00 002000, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084- 2021.
Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, and Wenwu Ou. Behavior sequence transformer for e-commerce recommendation in alibaba. In Proceedings of the 1st international workshop deep learning practice for high-dimensional sparse data, pp. 1-4, 2019.
Siyu Chen, Heejune Sheen, Tianhao Wang, and Zhuoran Yang. Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality. arXiv preprint wwX14:2402.19442, 2024.
Raffaello Claser and Vitor H Nascimento. On the tracking performance of adaptive filters and their combinations. IEEE Transactions on Signal Processing, 11:3104-316,, 2021.
Bijit Kumar Das, Luis Azpicueta Ruiz, Mrityunjoy Chakraborty, and Jerónimo Arenas- GarasaG On steady state tracking performance of adaptive networks. In 2015 IEEE International Conference on Digital Signal Processing (DSP), pp. 843-847. IEEE, 2015.
Nan Ding, Tomer Levinboim, Jialin Wu, Sebastian Goodman, and Radu Soricut. Causallm is not optimal for in-context learning. In The Twelfth International Conference on Learning Represen- fations.
11

---

Published as a conference paper at ICLR 2026
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning, arXiv preprint arXiv:2301.00234 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, G Heigold, S Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learn- ing Representations, 2020.
Deqing Fu, Tian-qi Chen, Robin Jia, and Vatsal Sharan. Transformers learn to achieve second-order convergence rates for in-context linear regression. Advances Neural Information Prucessing Systems, 37:98675-9871 16. 2024.
Shivam Garg, Dimitris Tsipras. Percy S Liang, and Gregory Valiant. What transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00755: 2023.
Yu Huang, Yuan Cheng, and Yingbin Liang. In-context convergence of transformers. arXiv preprint 1r.05:2310.05249, 2023.
Yu Huang, Yuan Cheng, and Yingbin Liang. In-context convergence of transformers, Proceedings of the 4lst International Conference on Machine Learning, pp. 19660-19722, 2024.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as big sequence modeling problem. Advances in neural information processing systems, 34:1273-1286, 2021.
Jiachen Jiang, Yuxin Dong, Jinxin Zhou, and Zhihui Zhu. From compression to expansion: A layerwise analysis of in-context learning, arXiv preprint arXiv:2505.17322 2025a.
Jiachen Jiang, Zhen Qin, and Zhihui Zhu. In-context learning for non-stationary mimo equalization. arXiv preprint 2rXiv:2510.087// 2025b.
Tobias Katsch. Gateloop: Fully data-controlled linear recurrence for sequence modeling. arXiv preprint arXiv:2311.01997, 2023.
Seungnyun Kim, Anho Lee, Hyungyu Ju, Khoa Anh Ngo, Jihoon Moon, and Byonghyo Shim. Transformer-based channel parameter acquisition for terahertz ultra-massive mimo systems. IEEE Transactions on Vehicular Technology, 72(1 72(11):15127-15133 2023.
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context Ilms struggle with long in-context learning. arXiv preprint arXiv:2404.00 02060, 2024a.
Yingcong Li, Ankit Rawat, and Samet Oymak. Fine Fine-grained analysis of in-context linear estima- tion: Data, architecture, and beyond. Advances in Neural Information Processing Systems, 37: 138324-138364, 2024b.
Yingcong Li, Davoud Ataee Tarzanagh, Ankit Singh Rawat, Maryam Fazel, and Samet Oymak. Gating is weighting: Understanding gated linear attention through in-context learning. arXiv preprint arXiv:2504.00430 2025.
Hailan Ma, Zhenhong Sun, Daoyi Dong, Chunlin Chen, and Herschel Rabitz. Tomography of quantum states from structured measurements via quantuu-awaeeee aware transformer. IEEE Transactions on Cybernetics, 2025.
Arvind Mahankali, Tatsunori B Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer linear self-attention. arXiv preprint arXiv:2307.03377 03.6000.03.6..... 2023.
Arvind V Mahankali. Tatsunori Hashimoto, and Tengyu Ma. One step of gradient descent is prov- ably the optimal in-context learner with one layer of linear self-attention. In The Twelfth Interna- fional Conference on Learning Representations.
12

---

Published as a conference paper at ICLR 2026
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannanch Hajishirzi. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15944 2021.
Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, et al. Eagle and finch: Rwkv with matrix- valued states and dynamic recurrence. arXiv preprint arXiv:2404 arXiv2240405589,, 2024.
Zhen Qin, Jun Tao, and Yili Xia. A proportionate recursive least squares algorithm and its perfor- mance analysis. IEEE Transactions on Circuits and Systems II: Express Briefs, 68(1):006-510, 2020.
Zhen Qin, Jinxin Zhou, and Zhihui Zhu. On the convergence of gradient descent on learning trans- formers with residual connections. arXiv preprint arXiv:2506.05249 2025.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Ali H Sayed. Adaptive filters. John Wiley & Sons, 2011.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Bingqing Song, Boran Han, Shuai Zhang, Jie Ding, and Mingyi Hong. Unraveling the gradient de- scent dynamics transformers. Advances in Neural Information Processing Systems, 37:92317- 92351, 2024.
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor transformer for large language models. arXiv preprint arXiv:2307.08621, 2023.
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for computational linguistics. Meeting, volume 2019, pp. 6558, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordy- intsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference Machine Learning, pp. 35151-35174, PMLR, 2023.
Yu Wang. Zhen Qin, Jun Tao, and Le Yang. Performance analysis of pris-based time-varying sparse system identification. In 2022 IEEE 12th Sensor Array and Multichannel Signal Processing Work- shop (SAM), pp. 251-255. IEEE, 2022.
Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. Advances in Neural Information Processing Systems, 36:36637-36651 2023.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 conference of the North Amer- ican chapter of the association for computational linguistics: human language technologies, vol- ume I (long papers), pp. 11p.11.. 1112-1122, 2018.
Yongtao Wu, Fanghui Liu, Grigorios Chrysos, and Volkan Cevher. On the convergence of encoder- only shallow transformers. Advances in Neural Information Processing Systems, 36:52197- 52237, 2023.
Songlin Yang, Bailin Wang. Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with harfwate-efficient training. arXiv preprint arXiv:2312.06635, 2023.
13

---

Published as a conference paper at ICLR 2026
Tong Yang, Yu Huang, Yingbin Liang, and Yuejie Chi. In-context learning with representations: Contextual generalization of trained transformers. arXiv preprint arXiv:2408.10147, 2024.
Yi Yu, Rodrigo C de Lamare, Tao Yang, and Qiangming Cai. Tracking analyses of m-estimate based Ims and nlms algorithms. In 2021 IEEE Statistical Signal Processing Workshop (SSP), pp. 1-5. IEEE, 2021.
Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49):1 2024.
Ting Zhang and Wei Biao Wu. Inference of time-varying regression models. 2012.
Ting Zhang and Wei Biao Wu. Time-varying nonlinear regression models: nonparametric estimation and model selection. 2015.
Yedi Zhang, Aaditya K Singh, Peter E Latham, and Andrew Saxe. Training dynamics of in-context learning in linear attention. arXiv preprint arXiv:2501.16265, 2025.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? Advances in Neural Information Processing Systems, 36:17773-17794, 2023.
Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp- 1059-1068, 2018
14

---

idd N(O, oT and they are Published as a conference paper at ICLR 2026
Appendices
ETHICS STATEMENT
This study presents a theoretical analysis of in-context learning in non-stationary regression prob- lems. No human animal data were collected, and all experiments rely exclusively on publicly available datasets that have been widely used in prior research. We recognize that pretrained LLMs may inherit biases from their training corpora. Since our method does not involve additional fine- tuning of LLMs, it does not directly address such biases. We encourage future investigations to place greater emphasis on fairness, accountability, and transparency deploying these models in practical scenarios.
REPRODUCIBILITY STATEMENT
We have taken extensive measures to facilitate reproducibility. Detailed descriptions model ar- chitectures, hyperparameters, and experimental setups included in both the main text and the appendix. In addition, we will release the full codebase, configuration files, and comprehensive documentation upon publication, thereby enabling independent verification and extension of our results.
THE USE OF LARGE LANGUAGE MODELS
Large language models were employed solely for improving clarity, grammar, and overall readabil- ity of the manuscript. They were not used for conceptual development, experimental design, data analysis, or the generation of research content. All theoretical contributions, implementations, and experimental results reported in this paper the authors' own work.
A TECHNICAL TOOLS USED THE PROOFS
iid Lemma 1. Suppose w, 70;-11 + e₁ where we N(O, II e, mutually independent. We have
( (o + minfaa b)o2)I,
7 = 1,
E W. wi
(g + 2 att -la
I
7 +
(20)
Proof. For any and b, we have
W. = -
Cii
(21)
i=1
wh wo +
(22)
Then we have
min(a.b} Σ
w 11
7 a+ 23 B(ese,]
min(a,b)
=gatb 2
y a+b a0+b-20 25 02I
[ + minfa, min(a,b)o?]1
= 1,
(gatbog + 20 2010 -
2010-
(23)
7 4 # 1. I
--
15

---

Published as a conference paper at ICLR 2026
According to (Zhang al., 2024, Lemma 5.1), we have
Lemma 2. Let 2,, 1..... n + be prompts defined in (4). Then the prediction Dn+1 can be written as a quadratic function as follows:
Fn-1 = u Hu,
(24)
Oaxd En+1 1n+1
where H = 2
@
A
Ant1-iz
1-i
2,
with X =
E ((d+1)x(d++) and u =
voc(U) E xx U =
Un
112
E [R((++)x(d+1)) with Un = 11 421 = W22
W.Q
KQ
Un
U-1
U12 =w/ and u-1 = _ defined in (8).
Un
u12
Lemma 3. Let u = vec(U) = vec
as Lemma 2. Consider gradient flow over u-1
Uu21
1
L(u) = Bi(u] Hu 11 )
(25)
2
with respect u starting from initial value satisfying Assumption 1. Then the dynamics of U satisfies
(t) u2 ,AAAAA DIV-IA,
(26)
dt
du (t) -ttrao(u_-AAAnAAA + D2 traco (A2)))
(27)
dt
and u12(t)= = Od, u2 (t) = for all t. 2 0, where
n + m(n+1)
x 1,
2
A-1"+1
2 +
(1+1)A And
*
I-1
2)²
y
2042
+
7-9
(1-7)
2n+2
1=1,,1,
D1 =
2 no?
12
[1-x²)²
1-x-1 1-x="
00
> 1,)= =
A-7 0w
2n11
+
[ー
(x-7)(1-ス1)
2-22
02:
X 1, = 1/7,
0
\1(1-100")
A 2012 (A-x)(1-73)
02
1/7.
x-7
(1->-)(1->n)
and A D2(2A (A) + AzA with
no 2
+
1m+
2 22
1
w
x x2n
(m+1)A 2 2
o +
1- X2
02
D2
2 na
1-7
auta
2
+
nx
2
nya
72 2n+2
1 1 +4
[1 1--27
)02
102
,
1=1,7#1,
1-x²
x
2n 2n
0
[ -111 []
_2 2m
[1
(1--)(1-3 x
\²_ 2
)0?
,
2m
22,
2n+2
(2 2222m 2m
1 2u
102
1, 1/7,
xa x=
[A
and
n(n - 1)02
+ (m-1)a(n+1))
1=1,7=1,
2(x=+1 w
21 23)
e
2(nd x2n44 112n+=((n1111 1)
-
+1 )02
X x
(1- (1- x1²
1) [+))
2
(1- - x292
[I
(I-A
(2 72 2n+1
2 +2
2 +
2
2n+1
2n
7) [+7]
(1-7)²
(1+7)
(1-))²²
23d
(n - 1
A >
6-- 12-1)(1-9 (1-7)
D3-
2 n 1)0² + 2n(A
22441)
2( 2000 -1
+ 1)29)
(00t2(nn11) 1-x
102
1 #1
11-
(1-X3)>
21 20 27 20-1
2-27
)o
+
21 2_2
2(n-1)
A(A (()-7)21 2m 27 2m
(2+77)
x-7
(1-32)(A-7))
(1-77)
2n
X x(A ( 2)
2m 22 2n
2
002 + ( 122
-1- (22 []][- 23
2( 1 1"
2 2m43 21 2,",222 2001
2 2 n+ 22 2n+ 123
+
)02 [A- 911
A 7,
(1 (1 X7)
16

---

Published as a conference paper at ICLR 2026
Proof. We first calculate the derivatives of u following
du(t)
di - Vv)(u) - 2 uu T HHuu + 2 1 Huu.
(28)
Following the same analysis of (Zhang et al., 2024, Lemma 5.2), we can first obtain
2 H un! T Hul
1 E
vec(U)) vec Σ Antl 22,2 1
)UX)
UX) vec ((E (Eattlizz22) )
2
d+1d+1
1
E
((. vec(T))
(29)
2
where T=
E Xm=1-izi 2,27
UX and H, X are defined in Lemma 2.
In addition, we can derive
2 21 _ +n++1 H|u
E
ann (i) s0 wa+1(f)
Σ
n+1
n 1 2, ))
u
1=1
a+
x A \ 212, )))
Σ
E
Fnn (j)
E wn+1(i)
Σ
IE
Oaxd
Bn+ 1
1n++1 (2) (1) n
0
1=1
1+1 A W w x
Di=1 An+1-b w (i) ax.r we 11)..
iiz A w wn+1 xz) wi
E
X Wwn+ (j)e0) 00:00
D1 Σ
d
Oaxd T
A(:55)
O Oaxd
A(:.:)]] A )u, 0
(30)
(:,))
A (t..))
where the second equation follows the fact that {wa} are independent of {a:). and the last line follows wo lid. NNO e; iid. NN o2)), æ; ltd. A), and they are mutually independent. In addition, to compute D1, we apply Lemma to evaluate the cross-covariance E(w,+1w,] =
(o2 + II,
( 03 + 2" -- gig + I
and then get
--11
1n+1= 10 + io
7 = 1 0e),
D1
EE 1n+1 (07--177m (y 2
11l1 In
7 1
no? 10 + 1(
A= = 1,
A-aa1 o² +
(n+1)x2 And et
A 1,=11,
1-X
(1=
2n12 2m+22
2-g -y 2 2222222422 2042
1-7 o
== 11,
(1+7)
X 2 +
1211-
(1~2-1-
22
T-x 12
A 1,1 76
(31)
ow + x-q
2041
I-72 n2
1
(A->)(1 (1
2n+1
> # 1,1= 1/7,
gnth,nt2
2m
2
0 +
\a(1-\"9")
x 1 n-2 2n12
A 1, 1/7
(1-7)(1
Combing (29) and (30), (28) can be rewritten as
du(t)
ΣΣ (((i))U((.))) veo(T) +D, Σ
d d41
(LA 0
Oaxd
A(;
0 Oxx
A(:
) u. (32)
A (:.j)
0
=
17

---

Published as a conference paper at ICLR 2026
Dynamics of U Under the Assumption 1, following the same analysis of (Zhang et al., 2024, Lemma 5.2), we can guarantee (12(t) = 0₂ E R and U21 = Og E I for all time 2 0.
Now, we will respectively analyze dynamics of U11 and u-1 when (12 O₂ R& and u22 (t) O2 E R
Dynamics of Un expand (29) as
First, for k, E [d], we use du(t) ((1 1) (a-
k))
(k: 1) and then
dt
d+1de d+1
E
T(.UU i= j=1
E
T(.))U( + E 1),,)(
(33)
2
For the term T( (1,)) 1), we have (t.
U(i E
x m. (1) w.Fw+111) 67)
Σ
b Bn (1)w mnsza(()))
U(I.))L B.
nu+1 3a (1) wa
x wl 23 (1))
-a=1 2 a- 6 (o + min b)o ja (1) (k))
Y 1,
-(U.JJuLAAG.D
Ea
2 (aatb + 2a 7--1 ot )x (1)x. mnars (A)],
7 # 1, (34)
where the last line follows Lemma 1. For the summation, we have
((.
2
E b 2 + min(a (ks))) u...JUnA(..))
7= 1, (k)a)) a. An,((
J
Ea
En (yatt oz + rots = la-h
x-1
7
(35)
For the term T(d 1, d 1Ku, (). get
1
B[T(d+
+ 1,d ((kk)((
2
1
E
Σ Σ Amt1-a w x Unian+1)
n
b (0)1(1) a)
2
1
2n+2-a-b 2 xs (k) waws u. ( 11
2
Σ
D 2 22-22aab (o + min(a, bfo (n) x x Uni A(;
y=1,
E
En
En (g + poth-gle o² ) (k) u. Un ((, 7
7-11
(36)
Combing (35) and (36), we have
d+1
E
[22 E De
Ex 2 + minfa, bboo)) x... Un)((;1)
1=
22, Di En 2
1eth 10th ----le--0 oz)ms (k))a)))) aa Un1((;1), ---1
7
(37)
18

---

Published as a conference paper at ICLR 2026
In addition, using (30) and following the analysis of (Zhang al., 2024, Lemma 5.2), the ((1 1) + 1) + k)-th element of the second term in (32) can be computed as D (:, Furthermore, we can obtain
(1)
E
Eis
- (o) min(f, m oa UniA |
9=1, 1 7# 1.
=
D
B
2om1
2 ayat 0 2 2- Il-
x--1 loso xat UnIA,
[A² Notice that based on (Sayed, 2011, Lemma A.2), E D aa
+ 2A + traco(A)A,
b"
We can further derive
E
20=11
b + min(a, 22 x.w.). b=
Y= 1 a. X 22 0 + 1019///e 61 [
7 *
72-1
-a=1 2 20
uo (21 2 (r))e(A)A) + +2 2E En
2 +b0 7=1
Em=1 22
- - (2A2 + trace(A)A)
+2 +2 E 20m 2 0 + 2a4s - o²)
7#
72-1
(no + (1+1)m oZ) (2A + trace (A)A) (n(n 1)02 + (n- 1)((+11) o2)
A= 1,7=1,
22-2aut2
1-2² ow + (m+1)2 22222222222022 2n
(2A² + traac(A)) (A)A)
+(( 2( 22) 2
2o (1-1)((12)
2(mx 2204 1123)
(1 (1-A)((AAAA-- 22 2030)
2( -14=-=-
x 1,7 =
[1 2)2
(1-2](1-23)
0 + 1112 (1-92)2
2n+2
)o2
(2A -tracc(A)))
+ (2 [1 (1+7)
20+1
- 27 -2 )o +
(1-2)
9 ( (1-77)
(1-7)²
1
A
1=
17
2n+2 no
nd2n+2
1
10 (2A 2 + trace(A)A ()AA)
1-xx
(1-
22+2 (n
1)0 +
(2( 2 12)
2(2 2n - 1924)
2an
#1,
(38)
11-2")²
(1-13)3
1-1²
2
2n-2
)o (2A* (r))e(A)A)
23 2n
2
2,,2
21 -2x²
2(n-1)x22
(T-
2r
2-2 2n-2
A
1 1,A= 1/7:
MIA
213m
2,, 2n
2,, 22
(1--210 [1 x2 )o (2.2 (race(A)A)
x 2n42
22
2 222m
2011
22
2,
)o + ( 27 -x 22+22
(1- -1-
((44))
21-
201 A0
2m2 1 22 2m
12 2,
)o?) A?
x
X(A
(1-
Now, we can obtain the dynamics of U11 as following
IU) - 1
(39)
dt
where Ā = D2(2A + trace(A) A)T) + D&A with
n + in+lin re:
>=
2a 22 2 (x+1)22 1-=1))==p000 3 22042____
*11,==, 1, A=
12
2n42 I-
22
( 1-
)02
D2
2 not
1-
n 20
1
3
)00
X 11>=
1-x-
x=
2 20 20_ 2n
2 -
x2
111-11111-
20 _
(1-
1- 2m
x2y
)0
X 1, * 1,1=1/11111
2 22 2,,
xa
2
[A=
(1
2
2x
2
11
2n+2
00
X 1,7 1, 1/7.
19

---

Published as a conference paper at ICLR 2026
and
nnn 1 + (m-1)((a-1) de
A= 1-11-=1 1,
203 201 2(A26= 123
(1-2)² 20+1
[1- 10
2(n - -1p-t4-(n-100
(1-A)(1-X)
(n- 11-1))2)
22p
+
20* - )o?,
+
X * 1,7 1,7=1, =1,
(2 (1-> (1+)) - 2
11
0 +
[--
2
(1-
20
(17)
(1-7)²
(18-1)(1 ())-) (n
7-1
1)
A= * * 1,
D₂
A 2 - 1)0 + 2m 12
(1 12
2(220+4
-nd
(1-)230
380+2n(a-1)
1-1²
jo
21 -2, 2, 2n-1
2-27
2n-2
0 +
21 222
2(n 1)yd
1(3 72 (x+
Amy
(1-3
(1-32)(1ー3)
2n 2n-
2n-
ge.
A 1, - 1/7
A(A (2 (1+ x2)
1 2 21 73 2n41
24 x
2n
jo + 2- -2 - 220-2
20 -))-()))))))
(1- 1=)
2(1 2n42g"
22 2 X(A
2y
22 27 13 )o::
(1-7 77
(A-ク)(1-33)
Dynamics of u-1
Next, similar with (33), we expand (29) at (d + 1) element:
d+1 d+1
E
T U
2
E
(.
2
2
u-1
(i.
nutl (,)x woxn+1(ii
Σ 1 "w tw xbσB U U1i )
2
4-1
E
xat1-aw
w
Ba T U1 U11En+1
Σ
b T
U 2a wi)
b=1
u-1
ΣΣ
x 22+2 a- x (1) (1)x0 wa Wi xbzi T Unl
trace
((.)))
2
1=1 j=1
+
u-1
E
trace
n+ w C. T
UNAUTT Σ
xutl 2622 wo
2
aml
0=1
U-1
trace
AUT E
2n+
D W. W/ 2b20s
U11
2
a=1
n
n
+
U-1
trace
ΣΣ
X
2n
22+2-a-20
2 w.
x U AUJ]
2
a=10 6m
u trace AUTT)
(40)
where the last line follows (36) and (39).
In addition, according to (Zhang et al., 2024, Lemma 5.2), the last element of the second term in (32) can be represented as Dj trace A UT). Hence, we have
du dt
trace(u t AU,]) + D1 trace AT
(41)
Based on the same analysis of (Zhang et al., 2024, Lemma A.1), we have
Lemma 4. Consider the gradient flow of L in (12) with respect to u, initialized according to As-
sumption 1. This is equivalent to performing gradient flow with respect to Un and u on the same loss function
22, trace (race( (AAUn D P trace( A AUH))
L( (Un,u)))
(42)
where A and Dj are defined in Lemma 3.
20

---

Published as a conference paper at ICLR 2026
Note that [(U u-1) differs from L(u) by constant and can therefore take negative values. We can further derive
Corollary 1. The loss function L(U11 u-1) in Lemma 4 satisfies
D2
min Uneddd
I(U11, u-1) 2
trace( (Λ²
(43)
M-ICR
and
L(U11,
min Usj eade
I(U11. u-1) A -
-
(44)
Furthermore, L in (12) satisfies that L(U11, u-1) min eedddd
L(U1I, A - I|.
(45)
-12
and the global minimum (U11,011) of I. in (12), when initialized according to Assumption I, satis- fies
u-1U11 DIĀ
(46)
Proof. First, we testify 1 (race(A(u_AA: DςAĀ- "AAAA)(--AA PLAĀ-1)T)
D
Irace(A A -1= ) equivalent to L(U11, u1) Specifically, we have
2
1 trace A(u-IAtUnAAt -DAAA=)ju_AAtuiA -DIĀĀ-)))
2
DJ trace(A2A
1
trace (A(22,ABUAAUTA DIU-AÄ-'AAU,AA
2
-DI -
D trace A (2
22 trace trace (UU) - DIu-1 trace(A UT)
2 [[UU,, u-1).
(47)
where the second equation uses the fact that À and A commute.
Notice that À and (u- IccIAAU AAU DA - inite matrices, we have trace( A (u- A AA () - DバAĀ-1) 2
- are positive semidef-
0. Hence, we have eaddka 1ER
LUU u-1) = trace A -1). addition, (44) follows
D?
trace(AA TAA) A)
|A] for any matrix A. Since L(U11, u-1) i(Un. +-I for some constant
C, condition (45) is satisfied.
To attain the global minimum characterized by (43), it is necessary that A DĄA = 0. Using the identity AĀ-I A 1 A this condition reduces to u- = D
Following the analysis of (Zhang al., 2024, Lemma A.3, Lemma A.4 and Lemma A.5), can directly obtain
Lemma S. Consider the gradient flow of L in (12) with respect u, initialized according to As-
sumption 1. For any t 2 O, it holds that
22 (1) (race(Unn)()) (U)(())
(48)
21

---

rtd. Published as a conference paper at ICLR 2026
Lemma 6. Consider the gradient flow of L in (12) with respect initialized according to As- sumption 1. If the initial scale satisfies
2D:
0<0<
(49)
VdlAAl
then, for any t 2 O, it holds that
o²
w-1(1)
A
(2
(do21IA()) 0.
(50)
2D 2
B PROOF OF THEOREM
The following theorem is established under a more general setting with X, Y > 0.
Theorem 4. Consider gradient flow over the population loss in (12). Assume that the initial task (id.
weight wo lid. NO0 o.I), noises e, ~ o(0,,² and inputs I; ~ N),, Suppose the where
initialization satisfies Assumption with initialization scale a > 0 satisfying a < A
no + n(n+1) 2 A-a+t1
0e:
A -=
I 0 +
2=#
(1 x12
And +12
Oet
A A
n1 g 22
ow + --
(1 (1+)
2n12
de
A= #1,
D1
2
1- )2n)
[1-2232
22
1-x2
de:
\
7-1 2
A-y Ow +
2n+1 1-
(1-2)(1-22)
de:
, 1/76
1ntl, dr 20+2
dr 11
n 2,2
02
(1-7")(->p)
(1-))(1->²)
and Ā D2A2A + (A) + A&A with
n +
(m+1)
2 der
1-1,1=1
2
[a+1)12-}8) 20 22
X²
1-x²
Oe:
D2
2
1-3²
3n+2
0 +
( (m) 22
1-
I-7
n
2
11-23)2
22
2
jo:
0o::
1=
_mm_
x x 0g 122 20
2
T- 21
x2y
Jo2
>11,7
212042 2n+2
x²
F
(x²->* [[
22
2m
-
26+2
10
A ,, 1/7
and
n(n 1)02 (a-1)[(n1) o².
A >=
2(2" 2(2"11 1-1 2n $1 13
2 +
2(1) 12: (m-1)))) (1-2)(1-121)
+ 20²"nuAn-
(1 ( )02
* 1
[1 x>2
(1-1)
(2 (1 2) (1t))
20 27 2t 2n+1
)
(,2²1
7-
-+ 20+1
(1 (2
(1-7) (
(1-
(7 13(1 (n 1
2
0)
\= 1
7-1
Ds
A 20 2 1)o + 2n 2242
(1 22
2n n1
(12230
220+2
1-
)0?
* 1, =
22
2n
x-7 j0
20 212
7) (14-)
11-3"
22 22
+ 20
1 # 1, == 1/7,
M x2))
22 22
2
24
22
jou + 2: 2 1100220-1
A(A (1(4))))
2(1 Aut2, 2 11
2y344 2041
2, n 22mt22n222 2
)o?
x 1, 1 + 7, 7,>#111 1/7.
[1-7XXA- 7XI -x))
M( XA--P-
(X-5)
Then gradient flow converges to global minimum of the population loss (12). Moreover, WkQ(O) and (v-(0) respectively converge to
lim Wv( / D
Oaxd 0g]
and lim WkQ(t) = v D D
'IF'
A-1
0g
(51)
t
0 0
22

---

Published as a conference paper at ICLR 2026
Proof. We begin by deriving lower bound on the squared gradient norm of the loss function
(L(Un((), 20-))
OL(U),(1), (-_
(U,
uL(Un)(),u_10)) aUn(()
u-
2
@Vn(()
Ou-1(1)
IIL2,ÄAUnA D2 A
-² I (u A Aλ A A
d²
Ae II; (2D1 do | |J (ULIABUIAA AÅ "A+ (52)
2D AI
where the third equation uses that A and A commute and the last line follows Lemma 6. In addition, based on Corollary 1, have
L(Un(t).u-1(tt) min Unertex4 #-1ER
L(U11, u-1)
A (u AU DAAR '
2
< AA (1- A DIAN 1)A 12 3 1 - 11?
2
2 AA (U - DAA ))
trace(A A trace(A 1),
(53)
where the first inequality uses that A and A commute.
Combing (52) and (53), we can get Polyak-Lojasiewicz (PL) inequality as follows:
LV u (1)
A (2D AdI)
2
ut) - min
((Un..m1))
A| 2 trace (A A-1A (race(A-))
caded M_
min L(U11,U-1).
(54)
From the dynamics of gradient flow and the condition, we have
d (L(Un)(t), u-)(t)
min egddd
L(U11,
dt
M11ER
dnn(t) (L(Un()),u_1(t))
du-i(t) (t), ut)
dt @t)((t)
dt
du)
(L((U(()) (1),((()
@Un((1)
Ou-1(t)
min Ungggddd M-TER
((U11)
(55)
When + 00, we have
0 S (Unn(t))=1(t))
min Uiredexd 21
L(U11, u-1)
Se ol (0)
min L(U11, - 0.
(56)
Uh endx
which implics lim; L(U (t), 1=)) min min enedxa (U11,
23

---

Published as a conference paper at ICLR 2026
According Corollary 1, Lemma 5 and Lemma 6, we have =-1(t)Ui1(t) = DAÂ-1 a-1(t) [t)lle)] and further obtain
lim Uni(1) - -n(()= -iIl---ilA-- lim u-1(()= = V D1lIĀ-11|R. 1-00
(57)
1-00
This completes the proof.
C PROOF OF THEOREM 2
Similar with Theorem 4, the following theorem is also established under more general setting with X, O.
Theorem 5. (Training error) Assuming the conditions in Theorem hold, the recovery error be- Iween (15) and (2) is
[I( - 9n+1)9 D9 trace (D2(A trace (A- AÄ-
MA'-Ay 'A) 2AA ΛÄ -A-) AA 1AA--A AA + Da trace(A) 2D3 trace (AA-1A),, (58)
+DAAÅ AA
o2 + (n + 1)02
y 1
where D4 = 222 20 1-120 2n
7#1 1
8
Proof. To establish the result, we expand [T[ - Bn+1) ) as
E[
3n++1
27
D1
Σ n w; 0,00
A 1- "xn+1 - 2 Øn+1
DJ E
x w w:a0
A n 22, 20,000 wj
1=1
a
E u T4 12+12
Wn+1| -2p E
Σ
x w. 2;,07
Ā xn+1 2 Wn+1 (59)
For the first term of the last equation (59), we have
E
An+1-i
x
Ā In+
Σ
1ht1- 2x, Ant1-1ze,0
E
trace
2n
Λ aa,,0r]] u0
D2 trace(A trace(A Á -AÄ -AA-1A) 2A -A MAÄ-A 'A) + D3 trace AÄ 1A--1A "AA--A) (60)
AAAA
x b a in (Sayed,
where the last line uses (38) and E [ A
2AAA trrac((A))))))A
2011 1, Lemma A.2).
For the second term of the last equation (59), we can derive
E E[w T un
Wn+1l (eltrace(AAx+11) T+1)]
D2 (A)
(61)
o + (n + 1)
7
where we define D4
2 02 1- 1-1
2x42
which follows the result in Lemma 1.
? *
a
24

---

Published as a conference paper at ICLR 2026
For the third term of the last equation in (59), get
}
E
A A w x A In+ 1 n Wa+1
E trace
Ant1-iAX AA -1 Aw
Di trace(AA" A),
(62)
where the last line uses (31). This completes the proof.
D PROOF OF THEOREM 3
Proof. We first expand E[(Tm+ 9m+1)9 } as
(mm+1 1 1m++) )²]
2 m+ Bm++1 )
E
D1
w. Bi
A
Bm++1
D2 E
W. T
E
Em+
1
m-
A
F w
3
E w8
m+1 Fm
10 1m+1 - -DD, E
W0 T.a.y
A -
E m
20
(63)
For the first term of the last equation (63), we have
m
m
m+
1
W.
T.
Lm
T
Tm+1
J, w
m
Irace
2m
F. A
'AA"
i=1
1=1
=D trace(A trace A AA + 2A
AÄ 'A + D3 trace AÅ 'A -A), (64)
AAA.
+ b
where the last line uses (38) and [ a
a == b in (Sayed,
2AAA trace AA)A,
2011, Lemma A.2). By replacing X, o2 of and in D3 and D3 with X, J, J2 an J2 and respectively, we obtain the modified quantities D2 and D3.
For the second term of the last equation in (63), can derive
E E(w En+ 1 #4 1 E[[race( Awn+ 10 w T+1)
D4 trace A)
(65)
82 + (m 1)
b =
where we define D4
which follows the result in Lemma 1.
22
1-72
de
For the third term of the last equation (63), we get
m
m m+1-i AA
]
Σ
m
W
N.C.
Um+ 1
T Wm+1
E trace
Aw W
D trace AA A).
(66)
where the last line uses (31). By replacing X, 7, 2 o² and n in D1 with T, 7. T2 T2 and m, we obtain the modified quantity D,. This completes the proof.
25


Reviews:
[{"review_id": "mIVKn7zC15", "reviewer": "Reviewer_WfeF", "summary": "In this paper, the authors study the capability of ICL to learn time-varying function class, thus provides a valuable addition to literature. In particular, the authors provide theoretical results on the convergence of the gradient flows, as well as training and testing errors. Numerical experiments are also provided for validation.", "questions": "1. Can you provide comments on the behavior of the gradient flow when the initialization assumptions in both Assumption 1 and the Theorem 1 do not hold?\nOn Lines 146-147, e_i and \\eta_i are not consistent.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors provide rigorous theoretical research on the congruence of the gradient flows for ICL training, and quantify the training and testing errors. The analysis is correct and detailed.", "weaknesses": "1. The function class considered is limited. The analysis is restricted to the case in equation (3), it is desired to demonstrate the implications of the results for more broad classes of functions.\n2. The numerical experiments are limited. Given strong assumptions made on initialization for the theoretically results, numerical experiments offer a good opportunity to explore what happens when those assumptions are violated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JTDD2RuKx0", "reviewer": "Reviewer_xcya", "summary": "This paper provides a theoretical analysis of in-context learning (ICL) under non-stationary regression problems, where task distributions evolve over time. It demonstrates that Gated Linear Attention (GLA) outperforms standard linear attention by adaptively modulating past inputs through a learnable recency bias, enabling better handling of time-varying functions. Theoretical results and empirical validation confirm GLA's advantages in dynamic settings.", "questions": "See the Weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a novel theoretical framework for in-context learning (ICL) under non-stationary regression problems.\n2.  The work exhibits certain technical rigor, with precise theoretical derivations.\n3. The paper is well-structured, clearly defining the non-stationary ICL problem early and motivating it with real-world examples (e.g., time-series forecasting).", "weaknesses": "1. The theoretical analysis focuses exclusively on linear regression tasks with Gaussian inputs, which is a very narrow setting. Real-world in-context learning scenarios often involve non-linear functions, complex data distributions, and high-dimensional inputs.\n2. The paper models non-stationarity using a first-order autoregressive process, which is a highly constrained form of time-varying dynamics. This fails to capture more complex real-world non-stationarity patterns such as abrupt changes, periodic variations, or long-term trends.\n3. The paper briefly shows that deeper GLA models perform better but provides no theoretical justification or detailed analysis of how gating mechanisms interact across multiple layers in non-stationary settings.\n4. While the paper includes a sentiment classification experiment on SST-2, this is only a single NLP task and doesn't demonstrate the broader applicability of GLA to diverse non-stationary scenarios.\n5. While the paper briefly mentions adaptive signal processing methods in Section 3, it doesn't actually compare GLA's performance with established methods like RLS or LMS in the same non-stationary regression setting", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X8wx7piw4g", "reviewer": "Reviewer_V7fJ", "summary": "This paper presents a theoretical analysis of in-context learning (ICL) for non-stationary regression tasks. By leveraging the linearity of both the gated in-context learner and the underlying task, the authors derive closed-form results for the convergence of weight parameters and provide quantitative expressions for training and testing errors. These errors are explicitly characterized in terms of the forgetting factor $\\lambda$, the task dynamics parameter $\\gamma$, and the token length $n$. The framework also unifies linear attention and gated linear attention through a single parameterization by $\\lambda$, enabling a coherent theoretical comparison.", "questions": "Do you expect the analysis can be extended to nonlinear settings, such as softmax attention or nonlinear task mappings ? If so, what kind of interpretation would be obtained. Would it be similar interpretation  to the linear case ?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper leverages the linear structure of both the gated ICL model and underlying tasks to obtain the convergence results and explicit error bounds. The dependence of these bounds on $\\lambda$, $\\gamma$, $n$, provides the meaningful insights for practical model design and parameter selection. Overall, I think this paper makes a valuable contribution, providing the theoretical understanding of how gated linear transformers for non-stationary linear setting.", "weaknesses": "While this paper is theoretically strong and sufficiently novel, it is currently limited to linear models and tasks.\nHowever, I think that these limitations are acceptable for the scope of this paper. Out of interest, I would like to ask the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)