Continuum Transformers Perform In-Context Learning by Operator Gradient Descent
Authors: Yash Patel
Abhiti Mishra
Ambuj Tewari
Venue: ICLR 2026 Poster (2026)
Keywords: neural operators
in-context learning
continuum transformers
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Transformer operators have demonstrated an ability to perform in-context learning for solving PDEs; we demonstrate they do so by performing operator gradient descent in an operator-valued RKHS.

Abstract: Transformers robustly exhibit the ability to perform in-context learning, whereby their predictive accuracy on a task can increase not by parameter updates but merely with the placement of training samples in their context windows. Recent works have shown that transformers achieve this by implementing gradient descent in their forward passes. Such results, however, are restricted to standard transformer architectures, which handle finite-dimensional inputs. In the space of PDE surrogate modeling, a generalization of transformers to handle infinite-dimensional function inputs, known as "continuum transformers", has been proposed and similarly observed to exhibit in-context learning. Despite impressive empirical performance, such in-context learning has yet to be theoretically characterized. We herein demonstrate that continuum transformers perform in-context operator learning by performing gradient descent in an operator RKHS. We demonstrate this using novel proof strategies that leverage a generalized representer theorem for Hilbert spaces and gradient flows over the space of functionals on a Hilbert space. We further show the operator learned in context is the Bayes Optimal Predictor in the infinite depth limit of the transformer. We then provide empirical validations of this result and demonstrate that the parameters under which such gradient descent is performed are recovered through pre-training.

Published as a conference paper at ICLR 2026
CONTINUUM TRANSFORMERS PERFORM IN-CONTEXT LEARNING BY OPERATOR GRADIENT DESCENT
Abhiti Mishra'
Yash Patel
Ambuj Tewari
Department of Statistics University of Michigan Ann Arbor, MI 48104
Department of Statistics
Department of Statistics University of Michigan Ann Arbor, MI 48104 t tewariaS umi ch edu
University of Michigan Ann Arbor, MI 48104
abhit i @ umi ch. edu
yppatel8 @eneelmmmmmicm ch.ichh-. edu
ABSTRACT
Transformers robustly exhibit the ability to perform in-context learning, whereby their predictive accuracy on task can increase not by parameter updates but merely with the placement of training samples in their context windows. Re- cent works have shown that transformers achieve this by implementing gradient descent in their forward passes. Such results, however, are restricted to standard transformer architectures, which handle finiteddimensional inputs. In the space of PDE surrogate modeling, a generalization of transformers handle infinite- dimensional function inputs, known as "continuum transformers," has been pro- posed and similarly observed to exhibit in-context learning. Despite impres- sive empirical performance, such in-context learning has yet to be theoretically characterized. We herein demonstrate that continuum transformers perform in- context operator learning by performing gradient descent in an operator RKHS. We demonstrate this using novel proof strategies that leverage a generalized repre- senter theorem for Hilbert spaces and gradient flows over the space of functionals on a Hilbert space. We further show the operator learned context is the Bayes Optimal Predictor in the infinite depth limit of the transformer. We then pro- vide empirical validations of this result and demonstrate that the parameters under which such gradient descent is performed are recovered through pre- pre-training.
INTRODUCTION
LLMs, and transformers more broadly, have demonstrated a remarkable ability to perform in-context learning, in which predictive performance improves without the need for parameter updates, merely by providing training samples in the LLM context window Minaee et al. (202)); Dong et al. (2022). With workflows increasingly relying on such tne-tunnig in place of traditional weight updates due to its computational efficiency, much interest has gone into providing theoretical explanation of this phenomenon Akküürk et al. (2022): ; Garg et al. (2022): Dai et al. (2022). Such works have demonstrated that, with particular choices of transformer parameters, inference pass through such models is equivalent to taking steps of gradient descent for the in-context learning task.
While LLMs were the first setting in which in-context learning abilities of transformers were ploited, interest is increasing in its broader application to other sequence prediction tasks. In par- ticular, seemingly orthogonal subfield of machine learning is that of accelerating the solution of partial differential equations (PDEs) Li et al. (2020a); y Du et al. (2023); Liu et al. ; ( Dafarzadeh al.) (2024): Oommen et al. (2024): :: You al. (2022). In this setting, the sequence no longer of finite-dimensional vectors, but instead of
functions. Nonetheless, transformers have been adapted to this setting, with architecture known as "continuum transformers" Calvello et al. (2024). More surprising still is that such transformers continue to exhibit in-context learning, whereby related PDEs can be efffciintly solved with the similar placement of solution pairs in the
continuum transformer context window Cao et al. (2024); Yang et al. (0023) Meng et al. (2055)
Unlike in the mnite-dimensional setting, this generalized, functional in-context learning yet remains to be theoretically characterized. We. therefore, herein extend this line of theoretical inquiry to
"Equal contributions

---

Published as a conference paper at ICLR 2026
characterize the continuum transformers that have been leveraged for in-context operator learning Calvello et al. (2024). Our contributions, therefore, are twofold: the insights afforded by such theoretical analysis and the development of mathematical framework for doing such analysis, as we highlight in greater detail in the main text. In particular, our contributions are as follows:
Proving continuum transformers perform in-context operator learning by performing gra- dient descent in a Reproducing Kernel Hilbert Space of operators and that the resulting in-context predictor recovers the Bayes Optimal Predictor under well-specified parameter choices of the transformer. Such proofs required the modeling of a generalized continuum attention mechanism and subsequent novel usages of generalized representer theorem for Hilbert spaces and Gaussian measures over Hilbert spaces.
Proving that such parameters under which continuum transformers implement operator gra- dient descent minimizers of the training process of such transformers, leveraging novel gradient flow analysis over the space of functionals on a Hilbert space.
Empirically validating that continuum transformers perform in-context operator gradient descent upon inference with the exhibited parameters and that such parameters are recov- ered with transformer training across a diverse selection of operator RKHSs.
BACKGROUND
2.1 NEURAL OPERATORS
Neural operator methods, while more broadly applicable, most often seek to amortize the solution of spatiotemporal PDE. Such PDEs formally described by spatial fields that evolve over time, namely some Ω x oo) + R. where x E Ω C F are spatial coordinates and t E [0, oo) is time coordinate. While the more abstract operator learning framework can be formulated as seeking to learn a map G A + U between two function spaces A and U, we are most often interested in learning time-rollout maps, in which the input and output function spaces are identical. In such cases, it is assumed dataset of the form D = {(u, (0) u. (T)) is available, where (0)) - TR is
the initial condition, for which there exists some true operator G such that, for all i, u (2) ((u
While many different learning-basee approaches have been proposed to solve this operator learning problem, they all can be abstractly framed as seeking
min || GllEaquun = =,, 11Ō(uo) G(uo) duo-
(1)
Within this broad family of operator learning, the most widely employed classes are the Deep Op-
erator Networks (DeepONets) Lu et al. (202T) 2019); Wang et al. (202T): Kopaničáková & Kar-
niadakis (2025) and Fourier Neural Operators (FNOs) Li et al. (2020b с at: Bonev al. (2023). FNOs are parameterized as sequence of layers of linear operators, given as kernel integral trans- forms, with standard intermediate ReL.U nonlinearities. Formally, a single layer is then given by aFF @ F(u))), where F denotes the Fourier transform and Rd the learnable kernel parame- ter. This layer encodes a translationally invariant kernel integral transform.
2.2 IN-CONTEXT LEARNING AND CONTINUUM TRANSFORMERS
The standard attention mechanism is parameterized by 0 == { WG, W. where WA IRdxxd W, E Rd, and W, E Rd, for sequential data X E Rdxn Vaswani (017)). Formally,
Attn(X) (W.X)MH (W, (WaX)),
(2)
where M E R a masking matrix and R44, xn x Rd x - Raxn is a nonlinear transform of key-query similarity measures. This resulting matrix K) is referred to as the "attention weights" matrix. Most often, de = dg and H == softmax, i.e. [H(Q, K)]Ks = Q K2) E. attention blocks with residual connections and feedforward and normalization layers.
cxp(( e Cheng et al. (2023). Transformers are repeated compositions such
We now highlight the "continuum attention" mechanism proposed in Calvello et al. (0244) that generalizes transformers to operator learning; "continuum" here highlights that such architecture
2

---

Published as a conference paper at ICLR 2026
models data in its desecrtizztion-agnosiits function form as with other neural operator methods. In particular, in place of F; E IR, 24 E X' for some Hilbert space X. The natural generalization of the attention mechanism for function inputs then replaces the WE, Wg. and Wa matrices with linear operators. For a sequence I E X", the key, query, and value operators respectively map We 2 K.", W x Q", and We X V where K., 2, and V are Hilbert spaces. Notably, the original framing of continuum transformers Calvello et al. (2024) generalizes attention by considering an infinite index set, rather than considering unfiite-dimensione tokens as we do here; we discuss how these two seemingly distinct notions are equivalent in Appendix A In practice, such operators are implemented as kernel integral transforms, as Wg2; (Rg O FF.),,
where R2 is the Fourier parameterization of the query kernel, with Rd and R. similarly defined for the key and value kernels. The continuum attention mechanism assumes = Q and defines
At == (W,X)M softmax X), (WeX)
(3)
where the interpretation of M remains the same as that in Equation ( particular, while the key, query, and value operators are generalized, the resulting attention weights matrix still lies E g=** Transformer architectures, most notably in LLMs, have been observed to exhibit an unexpected behavior known as "in-context learning" (ICL), in which they perform few-shot learning without any explicit parameter updates but merely by having training examples in their context windows. We follow the conventions of offCheng et al. (2023) in formalizing the ICL phenomenon. We suppose the dataset D = {(x,, y:) Hi on which the transformer was trained has samples of the form
y:)
x (1)
(2)
X,
x
2
y1 = y/4
(n+1)
Vi
Vi
O
We then suppose 2/" f1( where f. # fy for i + j. This is the critical difference in the in-context learning setting: in the standard setting, f; = is fixed across samples and matches the f' at test time, meaning the goal for the learner is learn such an f. In ICL, however, the learning algorithm must be capable of "learning" an unseen f' at inference time without parameter updates.
2.3 OPERATOR META-LEARNING
Significant interest has emerged in meta-operator learning for spatiotemporal PDEs, in which single network maps from initial conditions to final states across system specifications Wang et al. 2002); Zhang (2024): [Sun al. (2024): Liu al. (2024): Cao et al. (2024); Chakraborty et al. 2022). Formally, unlike the traditional setting discussed in Section 2.1 here the dataset consists () (3) ) for which the true operator can vary of sub-datasets D U,D., where D. {(u()) u
()) Vj E [n.] but 9, * 9r for + i.. The goal is to across sub-detssets, i.e. 9, satisfies u:z = (u then, given only a limited number of training samples D == ((m6)) un " with < Σ, n, for fixed, potentially unseen operator G', learn approximation 9 R G. Most often, this is done by pre-training a meta-learner GML on D and then fine-tuning IXL on D to arrive g.
Building off of the observed ICL of transformers, the fine-tuning of these meta-leamnrss too can be performed via explicit weight modification or via in-context learning. An entire offshoot of meta- learning known as in-context operator networks (ICONs) has spawned from the latter Cao et al. (2024); y Yang et al. ): Meng al. (2025): Yang & Osher (2024). Recent works in this vein have demonstrated notable empirical performance leveraging bespoke transformer architectures built atop the continuum attention from Equation (3) Cao et al. (2024); Alkin et al. (2024); Cao et al. (2025);
Cao (021) Such behavior has been observed but has yet be theoretically characterized.
3
THEORETICAL CHARACTERIZATION
3.1 IN-CONTEXT LEARNING MODEL
For quick reference, we provide a compilation of the notation used throughout this and the remaining sections in Appendix L We now consider a generalization of the continuum attention, paralleling that of Equation (2) allowing for more general key-query similarity measures. In particular, instead of restricting Q K. as required for Equation (3 to be well-defined, we allow H Q x k
3

---

Published as a conference paper at ICLR 2026
(C()))(+)>x(+)) where L(V) denotes the set of bounded linear operators from V to V; note that
the "n + 1 convention is adopted as data is of the form given by Equation ( This generalization subsumes the softmax form assumed in Equation 3 where for R, c E L(V) is understood
to be defined as cfu for fo E V. from which we can similarly view softmax : Q x K - If X, an m-layer continuum transformer T: x + x consisting of generalized continuum attention layers with residual connections is well-defined as T := Tm To, where
(C(V))(*+))
Xe+1 Te(X)) = X (HVV No.,..))(,..
(5)
where we now view M : yn+1 + yn+1 as mask operator acting on the value functions. Note that we present this using a "double-transpose" notation as compared to Equation (2) to follow the mathematical convention of writing an operator to the left of the function is acting upon. Notably, in the setting of in-context learning for PDEs, the pairs (f(i), x00) are generally time rollout pairs (((), v))) (n((t-1))r) (HA() for some time increment At, elaborated upon more in Appendix B We, thus, assume ((() (00) X, that is. that they lie in single Hilbert space. Following the formalization established in Cole et (2024), we then seek to learn in-context map X' + N. for which we construct a context window consisting of training pairs (()) Z = X @ X:
(f())
f(2)
f(n) mm))
flattl)
(2 x
Zo = (1)
E
(6)
where {(f(), )))) predict y (x+1) - -[T(=)],,n+1 As in the typical ICL formalizaiionn, the key, query, and nonlinearity
input-output pairs, 20 = (f(1), ))),, and T -n+1 + 2n+1, We
operators only act upon the input functions. That is, denoting Xo- [ f(0), ... f(m), f(at1)
xn++1 as the vector of input functions and Xe E g similarly as the first row of the matrix Zi.
Zet1 Ze+ + (FT We && X.),M,,,),W,.. T
with the layer l query and key block operators Wg.t pa+l + Qu+ and WA,e X + K only acting on the restricted input space and Wa,e gn+1 - gn+1 on the full space. The mask
Lnxo 0
0 with Laxn denoting the n x n block identity operator.
block operator matrix M set as
We denote the negated in-context prediction for an input f(n+1) := after layers by
Te(f: (W., Wy. W2)l
(n)) = [Zele,nt1
(8)
The in-context loss given as follows, where we critically note that the plus sign arises from the convention that the final ICL prediction carries negative:
L(W., Wg. Wv) = Ezo,u(+t)) (Il + a 1
(9)
Notably, the very modeling of the generalized continuum layer in Equation (5) by introducing H Q*+1, x k**+ - (n+)))) was a non-obvious generalization of Equation 33 in introduc- ing this operator-valued nonlincarity. The more natural thing would have been to simply consider a scalar-valued function "similarity" measure, i.e. defining H Q=+1 x KC++1 + ( (n+1)))) However, critically, doing so does not lend itself to characterization in an RKHS, upon which our analysis over the remaining section is based. Thus, the mathematical framing of this problem as above is a worthwhile contribution that the broader community can use for further analysis.
3.2 IN-CONTEXT LEARNING OCCURS VIA GRADIENT DESCENT
We now show that continuum transformers can perform in-context learning by implementing op- erator gradient descent. For in-context samples {(f(), z ( the in-context task is to find
0+ minoee L(O)- := minoeo Σ (1) O Naturally, such operator learning problem can be solved by iteratively updating an estimate Oc+1 = O₂ - n) (LO))). Our result, therefore, roughly states that the prediction of (-1 by the l-th layer of continuum transformer, with particular choices of key, query, and value parameters, is exactly Ozf for any test function
4

---

Published as a conference paper at ICLR 2026
f. We defer the full proof of the theorem to Appendix E but here highlight novelties of the proof strategy in demonstrating this result. The strategy involves deriving an explicit form of the oper- ator gradient descent steps and then showing, under specific choice of (W,, Wg. W4 ), inference through a layer of the continuum transformer recovers this explicitly computed operator gradient expression (see Appendix D). While the general strategy parallels that of [Cheng al.) (2023), the explicit construction of the gradient descent expression over operator spaces is not as directly evi- dent as it is over dintesiiinniinnll vector spaces. In particular, we had to invoke a generalized form of the Representer Theorem to obtain this result, as compared to its classical form leveraged therein.
Theorem 3.1. Let X + L(X) be arbitrary operator- kernel and O be the operator RKHS induced by K. Let {(((), u(9))² and I(0)= == Ei=1 || O 1 Lei Oo 0 and let Og denote the operator obtained from the f-th operator-valae valued gradient descent sequence of L with respect to || llo as defined in Equation 23 Then there exist scalar step sizes ....... such that if. for an m-layer continuum transformer T := Tm o ... a Tu. [H(U, W)is k(u( ,l()),
12x
Wo,e (0
0 Wa,t = and Ws,e I for each 0g..., then for any f E X,
-(()
Te(:: Wg, ,..,,,,) -Osf-
3.3 IN-CONTEXT LEARNING RECOVERS THE BEST LINEAR UNBIASED PREDICTOR
In the finite-dimensiae dimensiooa setting, it was shown in Cheng et al. 2023 that if (y(") are outputs from the marginal of a Gaussian process with kernel R(:.-) for inputs {x(1)), transformer of depth m - with (U, W))s,3 k(u()), w()) recovers the Bayes optimal predictor in-context for window of {(x(1) y. Roughly speaking, this result demonstrates the optimality of recovering the true f in-context if the distribution on the function space from which f was sampled is Gaussian. We, therefore, seek establish a similar result for the setting of operator-valued kernels, which requires defining a Gaussian measure on the operator space from which the true O is sampled.
Notably, paralleling how the finite marginals of finiteddimensional Gaussian processes induce dis- tributions over functions, characterizing the finite marginals of a Hilbert space-valued Gaussian process induces a distribution over operators. Doing so, however, requires the formalization of a Gaussian measure on Hilbert spaces. A Hilbert space-valued random variable F is said to be Gaussian if (sp, F) is Gaussian for any yp E X MMenfooglio Petris (2016). We now present the generalization of Gaussian processes to Hilbert spaces from Jorgensen & Tian (2024).
Definition 3.2.
Gaussian Random Variable in Hilbert Space (Definition 4.1 of Jorgensen & Tian 2024)) ) Given an operator-valued kernel K : X x X - L(X), we say UIF ~ NO ((F)),
where U = (u() u u0+ D1, = [f()... "i.....,,(()) p(nt1] and [K(F))&s == x(f()) ((ر if, for all
v11), (2) E X and (f(0, u,))), (f6), w()) E x
( (())) x (0(2), (acl),,())))) u())xx = (()() )(0)))))) (o()) (1))x x ~ N (o, (o61), x(F()  ,())e(
(10)
(11)
Such Hilbert-space valued GPs have been leveraged in Hilbert-space kriging, notably studied in Menafoglio & Petris (2016): (2016)[Luschgy Luschgy (199).. They proved that the Best Linear Unbiased Predictor (BLUP) precisely coincides with the Bayes optimal predictor in the MSE sense, meaning the f :
X"' + X' amongst all measurable functions that minimizes [=[][X,+1 - f(X) for zero-mean jointly Gaussian random variables X (X1.... X,,) E X and X,+1 E X, is the BLUP. Their formal statement is provided for reference in Appendix F.I
We now show that the in-context learned operator is the Best Linear Unbiased Predictor as m + oo and thus, by Theorem F.I Bayes optimal if the observed data are x Gaussian random variables and if Il matches k. We again highlight here the approach and main innovations in establishing this result and defer the full presentation to Appendix F2 Roughly, the approach relied on establishing that the infinite composition Tox == ...o Tm o... To converges to well-defined operator and that this matches the BLUP known from Hilbert space kriging literature. The careful handling of this compositional convergence and connection to Hilbert space kriging are novel technical contributions.
5

---

Published as a conference paper at ICLR 2026
[f()...., ffn+1111111111 = = D Let X x X L(x) be an operator-valued kernel. Assume that U is a k. Gaussian random variable per Definition 3.2
Proposition 3.3. Let F
Let the activation function H of the attention layer be defined as (U, W) = k(u()), w wl)) Consider the operator gradient descent in Theorem 3.1 Then as the number of layers m + oc, the continuum transformer's prediction at layer m approaches the Best Linear Unbiased Predictor that minimizes the in-context loss in Equation
3.4 PRE- TRAINIGGNING CAN CONVERGE TO GRADIENT DESCENT PARAMETERS
We now demonstrate that the aforementioned parameters that result in operator gradient descent from Theorem 3.1 are in fact minimizers of the continuum transformer training objective. This, in tur,, suggests that under such training, the continuum transformer parameters will converge to those exhibited in the previous section. Similar results have been proven for the standard Transformer architecture, such as in Cheng et al.] (2023). Demonstrating our results, however, involved highly non-trivial changes their proof strategy. In particular, defining gradient flows over the space of functionals of Hilbert space requires the notion of -ifhhtt-ciffrremiaabiltt. which is more gen- eral than taking derivatives with respect to matrix. Additionally, rewriting the training objective with an equivalent expectation expression (see Equation (39) required careful manipulation of the covariance operators of the data distributions as described in the symmetry discussion that follows.
The formal statement of this result is given in Theorem 3.6 This result characterizes a station- ary point of the optimization problem in Equation [2) when the value operators W., have the form
To
O rel and the Wa, and Wx,e operators have form relating the symmetry of the data distribu-
tion, as fully described below. If the symmetry is characterized by self-adjoint invertible operator Σ, we establish that there exists a fixed point of the form Wg, = E.12 and Wx, cy E for some constants {be} and {ce). This fixed point relates to Section 3.2 and Section 3.3 since if I, we recover the parameter configuration under which functional gradient descent is performed.
This proof relies on some technical assumptions on the F' and UIF distributions and transformer nonlinearity; such assumptions are typical of such optimization analyses, as seen in related works such as Cheng al. (2023) et al. (2023); (Dutta (2024). As mentioned, the proof proceeds by studying gradient flow dynamics of the Wa.e and Wq.e operators over the PFF, U) distribution. Direct analysis of such gradient flows, however, becomes analytically unwieldy under unstructured distributions P(F, U). We, therefore, perform the analysis in a frame of reference rotated by Σ 1/2 By Assumption G.3 this rotated frame preserves the P(F, U) distribution and by Assumption G.4 also the attention weights computed by the continuum transformer, allowing us make conclusions on the original setting after performing the analysis in this rotated coordinate frame. We demonstrate in Section 5.3 that common kernels satisfy Assumption G.4 We defer the full proof to Appendix G
Assumption 3.4. (Rotational symmetry) Let Pp be the distribution of F [f ..... f(x+1)) and ((F) = (U & UJ. We assume that there exists a self-adjoint, invertible operator Σ ::4 X X
such that for any unitary operator M, 2 1/2 F d F and K(F)- 12 F).
Assumption 3.5. For any Fi, F/2 E x and any operator S X + X with inverse S-1, the function H satisfies H (Fi, F2) H( S-1 F22)
Theorem 3.6. Suppose Assumption 3.4 and Assumption
3.5 hold. Let [(r, Wa. Ws) =
Wo.t = Om+1 x 0 denote the set of (W,, Wx) operators with the property that (Wy: Wo) S if
= {1o
rel rel)] e=o....m
0 E Wa,e, Wa,e) where as defined in Equation 99. Let S C
and only if for all é {O....,m) m)) there exist scalars be, Cg R such that Wy.e 1/2 1/2 and
Wa. = 1/2 Then
m E(o.,)) + VV, Ilis + WV [ias]] 0.
inf
(12)
E=
Here V Ww, and Wie denote derivatives with respect the Hilbeer-Schhiid norm ||| Ilns-

---

Published as a conference paper at ICLR 2026
4 RELATED WORKS
We were interested herein in providing a theoretical characterization of the in-context learning exhib- ited by continuum attention- based ICONs, paralleling that done for mnite-dimensionll transformers Akyürek et al. (2022) Garg et al. ) Dai et al. (0022). While previous works have yet to for- mally characterize ICL for continuum transformers, a recent work Cole et al. (2024) began a line of
inquiry in this direction. Their work, however, studied fundamentally distinct aspect of functional
ICL, characterizing the sample complexity and resulting generalization for linear elliptic PDEs.
Most relevant in the line of works characterizing finite-dimensional ICL is Cheng et al. (2023)
Loosely speaking, they demonstrated that, if kernel k(---) is defined with IN denoting its associated
RKHS and a transformer is then defined with a specific WR, Wg. W, and [H( K)), = x(Q;, K2)
from Equation ( an inference pass through m layers of such a transformer predicts y," -("11) from
Equation 4 cquivalently to f(x(at))) where f the model resulting from m steps of functional gradient descent in I.. They then demonstrated that the predictor learned by such ICL gradient descent is Bayes optimal under particular circumstances. These results are provided in Appendix
Notably, these results required non- non-trivial changes to be generalized to operator RKHSs herein; as discussed, such mathematical tooling is more general than its application to the setting considered herein. In particular, function space directly without requiring discretization required formalizing several notions to rig- orously justify being able lift the proofs from finite to infinite dimensions. We view this significant contribution to the community. Now that we have gone through this formalism, the broader community can directly use these analysis strategies to further study in-context learning or unrelated inquiries. This is highly valuable, since it suggests that theoreticians can often follow their finite-diments dimensionl intuitions and defer to our infinite-din dimeti--dimeiiinall results to rigorously justify their results. In other words, our proof strategies suggest to other researchers working on similar prob- lems that they need not be bogged down in the details of the error convergence minutiae that often arise in approaches relying on finite projections and can instead work with these clean abstractions.
did
approach to performing the analysis in the dimente-ad
Similarly, this work opens up the space for who can contribute to further the theoretical study of op- erator ICL. Much of the classical optimization community, for instance, may not be intimately famil- iar with the mathematical formalisms required around operator spaces. However, with our formal- ism, they can provide insights with little change from how they would approach Emite-dimension analyses. We, therefore, believe this framework, consisting of the generalized transformer and char- acterization of its optimization, is worthwhile contribution to the theory community.
5 EXPERIMENTS
We now wish to empirically verify the theoretically demonstrated claims. We provide setup details in Section 5.1 and then study the claims of Section 3.3 in Section 5.2 and those of Section [34 in Section 5.3 Code for this verification is available at https:/gtthub..co//sssspattfe//a
5.1 EXPERIMENTAL. SETUP
In the experiments that follow, we wish to draw Gaussian Random Variables, as per Definition 3.2 To begin, we first define an operator-valued covariance kernel k. particular, we consider one com- monly encountered in the Bayesian functional data analysis literature Kadri et al. (2016a), namely the Filbeet-Schmidt Integral Operator. Suppose ky X x X + IR and ky 1 Ω - l are both positive definite kernels, where ky is Hermitian. Then, the following is a well-defined kernel:
( ((2 (u)) = (10) / kn k uyulss
(13)
Notably, similar to how functions in a sealar RKHS can be sampled as f = Ee (((()() for (()) ~ N( o2) over a randomly sampled collection { J==1
we can sample operators by sampling a collection {(p(0)) ) and defining 0
Ii,a(0)) (0)) (kz()))), hy w)((e)) (s')d))) with (+) o2) We focus on X C2
7

---

Published as a conference paper at ICLR 2026
Linear
Laplacian
1.75
1.50
1:35
1.00
103
150
200
250
50
100
150
200
250
Gradient RBF
Energy
IraM
DLIP
DLUP
16 00-0
Layers
Figure 1: In-context learning loss curves over the number of layers in the continuum transformer. The kernels of the data-generating processes are given in the titles of the sub-figures. Curves show the mean 11/2 standard deviation from 50 i...d. draws of the operator.
for which the functions sp()), (b(+)) can be sampled from a Gaussian with mcan function 0 and co- variance operator a + -3, where a, 8, 7 E It are parameters that control the smoothness of the sampled functions and A is the Laplacian operator. Such a distribution is typical of the neural operators literature, as seen in Subedi & Tewari (2025): Kovachki et al. (2021): Bhattacharya et al. (202T, and is sampled as
p(a)
Σ 1N/2
1| + 3) -/) jun where 2 ~ NNO0 (14)
where N/2 is the Nyquist frequency assuming discretized spatial resolution of N x N. Such sampling is similarly repeated for vb(*), We consider standard scalar kernels k and ky to define Hilbert-Schmidt kernels. For kz, we consider the Linear, Laplacian, Gradient RBF, and Energy kernels and for ky. the Laplace and Gaussian kernels. The definitions of such kernels is deferred to Appendix H To finally construct the in-context windows, similarly sample functions f6) per Equation (4) and evaluate (أب)) Ofti using the sampled operator. To summarize, a single ICL context window of the form Equation (6) is constructed by defining an operator O with a random (s) (s)
sample {(a} (/ ) sampling imput functions {,("), }, and evaluating u. =
5.2 BEST LINEAR UNBIASED PREDICTOR EXPERIMENTS
We now empirically demonstrate the claim of Proposition 3.3 specifically using the Bilbert-Schmidt operator-valued kernels described in the previous section. In particular, we demonstrate the opti- mality of the continuum transformer in-context update steps if the nonlinearity is made to match the kernel of the data-generating process. We consider four pairs of the aforementioned (kag kyy) to define the operator-vale valued kernels and then fix the model parameters to be those given by The- orem 3.1 Notably, as WA.E and Wy.e are implemented as FNO kernel integral layers, we do so by fixing R..e R 1 (N/2)x((/2) VÊ. The results are shown in Figure As expected, the in-context loss curves to decrease over increasing layers when the kernel matches the nonlinear- ity, each additional layer then corresponds to additional step of operator gradient descent per
find
8

---

Published as a conference paper at ICLR 2026
Poisson Equation
400
linear
laplacion
350
gradient_rdb rbf
energy
300
250
200
150
100
50
Layers
Figure 2: In-context learning loss curves over the number of layers in the continuum transformer for the Poisson equation samples. Curves show the mean +1/2 standard deviation from 50 i.i.d. draws of the operator.
Theorem 3.1 For each setup, we also construct the BLUP to demonstrate the desired convergence, whose explicit prediction is given by
OLUP Σ n(S, (O) Jkyu())))))
(15)
We see the optimality of matching the kernel and nonlinearity across the different choices of kernels in the limit of l 00, namely in converging the same error the BLUP. We see that this result is robust over samplings of the operator, as the results in Figure haee reported over 50 independent tri- als. We visualize the in-context learned predictions for each setup combination Appendix[] which reveals that, when ky matches the lata-generating kernel, the resulting field predictions structurally match the true Of.
5.2.1 POISSON EQUATION
In the previous experiment, demonstrated the optimality of the estimator when its kernel matches that the generatinntnnggggg RKHS. practical settings, however, selecting x in this fashion may not be possible, estimating the kernel of the RKHS from which samples are drawn is often difficult. We, therefore, now study the robustness of this exhibited behavior in a realistic PDE learning setting.
In particular, study the 2D Poisson equation. The 2D Poisson equation is given by Au(x) f(x), where T2 + R is the scalar field of interest and f T + IR is the source field. In u. The source fields were again
this setting, wish to learn the solution operator drawn from GRF, as described in the Section 5.1 and corresponding solutions u computed using an analytic FFT-based Poisson solver. Representative samples are visualized in Appendix J
The results are shown in Figure 2 We see that the exhibited parameters of the continuum transformer continue to display the desired optimization characteristics over layers even in this case where the explicit kernel is unknown. In particular, we find the linear kernel to exhibit optimal performance here; future work exploring the systematic selection of optimal practical settings would be of great interest. Note that the BLUP achieves near-perfect estimation accuracy here, since the solution operator of the Poisson Equation over periodic boundary conditions linear in f.
5.3 OPTIMIZATION EXPERIMENTS
We seek to empirically validate Theorem 3.6 namely that Wa. b.E-1/2 and Wg. 1-12 fixed points of the optimization landscape. Direct verification of this statement, how-

---

Published as a conference paper at ICLR 2026
Operator Convergence Over Training Steps
0.956
0.954
0.952
105-0111
Rx Uneer Laplecian
Energy
140
Training Step
Figure 3: Pairwise convergence of the key-key, key-query, and query-query operators of the contin- uum transformer in Hilbert-Schmine cosine similarity across different kernels kg over training steps. Curves show the mean +1 standard deviation from i.i.d. trials of training procedure.
ever, is not feasible, since do not have an explicit form the 1-1/2 operator. Nonetheless, we verify that, for any &1, lz and i.j {q. k} (indicating whether query-query, or key-query operator pair), (W.,,, W W - 1, where W W/[[W]||s denotes the normalized operator. Since are working over the space X' = L (T and considering Fourier kernel parameterization for the operators, this Hilbert- -cchmit inner product can be computed over the kernels, i.e. (Ri Rj where (+, )) is the Frobenius inner product. The final metric is then
comparing key-key,
cos (t) = 4m
1
Σ
Σ
(Ra, R3
(16)
|
(JE((,g) (1,6,6(
representing the average pairwise cosine similarity at step i of the optimization between the learned operators. Since again consider 250-layer continuum transformer, naively computing Equa- tion (16) is computationally expensive; we, therefore, report this metric over a random sampling of m' To layers of the network. We repeat this optimization procedure over independent trials, where randomization occurs in the sampling of the datasets across trials and network initializations.
As required by Theorem 3.6 we constrain the optimization W., to over operators of the form
T This procedure is repeated across each of the k, kernels considered in the previous section 0 [e
with k,, fixed to be the Gaussian kernel; we demonstrate that the Linear h kernel satisfies Assump- tion [..4 in Appendix K.I Notably, the other choices of k2 do not satisfy this assumption, yet we find that the convergence result holds robustly to this violation. Specific hyperparameter choices of the training are presented Appendix K The results are presented in Figure As mentioned, we see that the operators all converge pairwise on average, validating the characterization of the fixed points given by Theorem 3.6 As the previous experiment, we find results to consistently replicate across training runs, pointing to this finding being robust to initializations.
DISCUSSION
In this paper, we provided a theoretical characterization of the ICL phenomenon exhibited by con- tinuum transformers and further validated such claims empirically. Unlike the language learning context, this insight suggests practical direction for improving ICL for meta-learning in PDEs, namely by estimating x for specific PDE meta-learning tasks and using this directly to parameterize H in the continuum transformer architecture. Such RKHSs are often not explicitly defined but rather induced by distributions of Section (.3 suggest that a stronger version of Theorem (anrrem3..6hshee shoul hold, in which the convergence result independent of proving such generalization would be of great interest.
parameters of the PDE and its parametric form. Additionally. the results
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGMENT
The authors thank Julian Bernado for his input in the early stages this work. Ambuj Tewari acknowledges the support of NSF via grant DMS-2413089.
REFERENCES
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn imple- ment preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36:45614 45650, 2023.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algo- rithm is in-context learning? Investigations with linear models. arXiv preprint 2022.
Benedikt Alkin, Andreas Fürst, Simon Schmid, Lukas Gruber, Markus Holzleitner, and Johannes Brandstetter. Universal physics transformers: A framework for efficiently scaling neural opera- tors. Advances in Neural Information Processing Systems, 11-25152-25194 2024.
Kaushik Bhattacharya, Bamdad Hosseini, Nikola B Kovachki, and Andrew M Stuart. Model reduc- tion and neural networks for parametric PDEs. The SMAI journal of computational mathematics, 7:121-157, 2021.
Boris Bonev, Thorsten Kurth, Christian Hundt, Jaideep Pathak, Maximilian Baust, Karthik Kashinath, and Anima Anandkumar. Spherical fourier neural operators: Learning stable dy- namics on the sphere. arXiv preprint arXiv:2306.03888, 2023.
Edoardo Calvello, Nikola B Kovachki, Matthew E Levine, and Andrew M Stuart. Continuum atten- tion for neural operators. arXiv preprint arXiv:2406.06486 2024.
Shuhao Cao. Choose a transformer: Fourier or Galerkin. Advances in neural information processing systems, 34:24924 24444024-24940, 2021.
Shuhao Cao, Francesco Brarda, Ruipeng Li, and Yuanzhe Xi. Spectral-refiner: Accurate fine-tuning of spatiotemporal fourier neural operator for turbulent flows. In The Thirteenth International Conference on Learning Representations, 2025.
Yadi Cao, Yuxuan Liu, Liu Yang. Rose Yu, Hayden Schaeffer, and Stanley Osher. Vicon: Vi- sion in-context operator networks for multi-physics fluid dynamics prediction. arXiv preprint arXiv:2411.1666, 2024.
Ayan Chakraborty. Cosmin Anitescu, Xiaoying Zhuang, and Timon Rabezuk. Domain adaptation based transfer learning approach for solving PDEs on complex geometries. Engineering with Computers, 38(5):459944588 2022.
Xiang Cheng, Yuxin Chen, and Suvrit Sra. Transformers implement functional gradient descent to learn non-linear functions in context. arXiv preprint 1rXiv:2312.06628, 2023.
P.P.J.E. Clément. An introduction to gradient flows in metric spaces, 2009.
Frank Cole, Yulong Lu, Riley O' O'Neill, and Tianhao Zhang. Provable in-context learning of linear systems and linear elliptic PDEs with transformers. arXiv preprint arXiv:2409.1220 2024.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma. Zhifang Sui. and Furu Wei. Why can gpt learn in-context? Language models implicitly perform gradient descent as meta-optimizers. arXiv preprint arXiv:22 10559, 2022.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey in-context learning. arXiv preprint 2000::2:000000334 2022.
Yiheng Du, Nithin Chalapathi, and Aditi Krishnapriyan. Neural spectral methods: Self-supervised learning in the spectral domain. arXiv preprint 1rXiv:2312.05225, 2023.

---

Published as a conference paper at ICLR 2026
Sanchayan Dutta and Suvrit Sra. Memory-augmentedd Transformers can implement linear first-order optimization methods. arXiv preprint arXiv:2410.07263, 2024.
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? A case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598 2022.
Siavash Jafarzadeh, Stewart Silling, Ning Liu, Zhongqiang Zhang, and Yue Yu. Peridynamic neural operators: A data-driven nonlocal constitutive model for complex material responses. Computer Methods in Applied Mechanics and Engineering, 425:1169144 2024.
Palle ET Jorgensen and James Tian. Hilbert space valued gaussian processes, their kernels, factor- izations, and covariance structure. Advances in Operator Theory, 94):77, 2024.
Hachem Kadri, Emmanuel Duflos, Philippe Preux, Stéphane Canu, Alain Rakotomamonjy, and Julien Audiffren. Operator-valued kernels for learning from functional response data. Journal of Machine Learning Research, 17(20)11-54 2016a.
Hachem Kadri, Emmanuel Duflos, Philippe Preux, Stéphane Canu, Alain Rakotomamonjy, and Julien Audiffren. Operator-valued kernels for learning from functional response data. Journal of Machine Learning Research, 17(20):1-54, 2016b. URL nattp://jmle wttppppprra
v1 017/11-311 html
Alena Kopaničáková and George Em Karniadakis. DeepONet based preconditioning strategies for solving parametric linear systems of equations. SIAM Journal on Scientific Computing, 47(1): C151-C181, 2025.
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint (..iv:2108.0844: 2021.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprins arXiv:2010000885,, 2020a.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadeneshelii Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differ- ential equations. arXiv preprint arXiv:2003.03453 2020b.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhat- tacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differ- ential equations. Advances in Neural Information Processing Systems, 33:6755-6765 2020c.
Yuxuan Liu, Jingmin Sun, Xinjie He, Griffin Pinney, Zecheng Zhang, and Hayden Schaeffer. PROSE-FD: multimodal PDE foundation model for learning multiple operators for forecasting fluid dynamics. arXiv preprint arXiv:2409.09811, 2024.
Ziyuan Liu, Yuhang Wu, Daniel Zhengyu Huang, Hong Zhang, Xu Qian, and Songhe Song. Spfno: Spectral operator learning for pdes with dirichlet and neumann boundary conditions. arXiv preprint 2r1iv:23322.000 2,09800, 2023.
Lu LU, Pengzhan Jin, and George Em Karniadakis. DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprini arXiv: 1rXiv:1910.55 1910.0319999 2019.
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3):218-222 2021.
H. Luschgy. Linear estimators and radonifying operators. Theory of Probability & Its Applica- tions, 40(1):167-175, 1996. doi: 10.1137/1140011 URL attps: :ttpps////ee /ttp///...../111111/.111 1140017
12

---

Published as a conference paper at ICLR 2026
Alessandra Menafoglio and Giovanni Petris. Kriging for hilbert-space valued random fields: The op- eratorial point of view. Journal of Multivariate Analysis, 146:84 94, 2016. ISSN 0047-259X, doi: https://doi.org/ 10. 10w...//.....a........m..
URL https: www. sciencedirect. com/ Iscience/art at ic / 5001 578 Special Issue on Statistical Models and Methods for High Infinite Dimensional Spaces.
Tingwei Meng, Moritz Voss, Nils Detering, Giulio Farolfi. Stanley Osher, and Georg Menz. In- context operator learning for linear propagator models. arXiv preprint arXiv:2501.15100 2025.
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Am- atriain, and Jianfeng Gao. Large language models: A survey, arXiv preprint urXiv:2402.66190 2024.
Vivek Oommen, Khemraj Shukla, Saaketh Desai, Rémi Dingreville, and George Em Karniadakis. Rethinking materials simulations: Blending direct numerical simulations with neural operators. npj Computational Materials, 10(1):145, 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
George Stepaniants. Learning partial differential equations in reproducing kemel hilbert spaces. Journal of Machine Learning Research, 24(86):1-72, 2023. URL lattp://mlzzorg
papers/v2A/211133.
Unique Subedi and Ambuj Tewari. Operator learning: A statistical perspective. arXiv preprint arXiv:2504.03503 2025.
Jingmin Sun, Zecheng Zhang, and Hayden Schaeffer. LeMON: Learning to learn multi-operator networks. arXiv preprint arXiv:2408. 16168, 2024.
Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Hengjie Wang, Robert Planas, Aparna Chandramow/ishwwaran and Ramin Bostanabad. Mosaic flows: A transferable deep learning framework for solving PDEs on unseen domains. Computer Methods in Applied Mechanics and Engineering, 389:1 14424, 2022.
Sifan Wang. Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric par- tial differential equations with physies-informed DeepONets. Science advances, 7(40)=eabi8605, 2021.
Liu Yang and Stanley J Osher. PDE generalization of in-context operator networks: A study 1D scalar nonlinear conservation laws. Journal of Computational Physics, 519:1 19379, 2024.
Liu Yang, Siting Liu, Tingwei Meng. and Stanley J Osher. In-context operator learning with data prompts for differential equation problems. Proceedings of the National Academy of Sciences, 120(39):e2301442120 2023.
Huaiqian You, Quinn Zhang, Colton J Ross, Chung-Hao Lee, and Yue Yu. Learning deep implicit fourier neural operators (ifnos) with applications to heterogeneous material modeling. Computer Methods in Applied Mechanics and Engineering, 398: 198:115296,,0 2022.
Zecheng Zhang. Modno: Multi-operator learning with distributed neural operators. Computer Methods in Applied Mechanics and Engineering, 431:117229, 2024.
13

---

In Calvello al. (2024), , the setting of interest was extending the standard vision transformers that typically act mitte-dimensional images R#xW to instead act continuum, i.c. over function mapping Ω + R for Ω I?. doing so, they had to generalize the typical notions of patching that are introduced in vision transformers, by considering patches that decompose the domain, i.c. Q such that U,S, = Ω and Ω n Ω2 = 0. From here, each patch becomes a separate function tokens in their architecture, as Published as a conference paper at ICLR 2026
A CONTINUUM TRANSFORMER TOKEN DETAILS
fi Ω + R. which necessitated dealing with we assumed directly for our setting.
B TIME ROLLOUT META LEARNING
As discussed in the main text, the pairs (f() w()) generally time rollout pairs (f(), u())) (n(U-1))tt (()0)). We elaborate this common setting below. In the context of doing time- rollouts, a pre- praitinin dataset of the form D := ((U((: -(())) " is available, where U,(::T
(0) (AT) Ue (T-I))T) with u, (t) (22). Notably, such setup is equivalent having [u. 11
= T/AT training pairs {(x,tt (AT)))))))) ) It is further assumed that, for each sample i, there is true, deterministic solution operator 9i E O, where is space of operators, that maps from the spatial field at some time t to its state at some later + AT. The in-context learning goal, thus, is, given new sequenee of (0:T-1) generated by some unseen operator G. predict (F(T), i...
(!0)
((AI)
(((n-1)A()
((nAt))
Zo
((A))
(220))
((mL))
C RKHS FUNCTIONAL GRADIENT DESCENT THEOREMS
We provide here the precise statements of the relevant results from Cheng Proposition C.1. (Proposition 1 from Cheng et al. (2023) ) Let K. be an arbitrary kernel. Let H. denote the Reproducing Kernel Hilbert space induced by K. Let 2 (2x(9) yf()) for 1,..., be an arbitrary set of in-context examples. Denote the empirical loss functional by
(002)).
=(f)= = ==1
(17)
Let fo and let fe denote the gradient descent sequence of with respect to || l1, as defined in (3.1). Then there exist scalars stepsizes To. .......7.. Tm such that the following holds:
Let H be the function defined
H Wh
(18)
where U and WG) denote the ith column of U and W respectively. Let
0
V1
B = Iaxds
Ce Idxd-
(19)
Then for any x. = (n+1), the Transformer's prediction for ((n+1) at each layer & matches the prediction of the functional gradient sequence (3.1) at step E, i.e., for all l O....,k.
Te(x; (V, B. (1) ..... (m)) ff(xx).
(20)
Proposition C.2. (Proposition 2 from Cheng et al. (203)) Let
X= [1(1), [1))
r(a+1)]. Y (1)
y(m+1)).
(21)
Let K. Rd x Rdd + the attention activation
R be kernel. Assume that Y X is drawn from the K. Gaussian process. Let
H( = K(U()) w()).
(22)
and consider the functional gradient descent construction in Proposition 1. Then, as the number of layers L - 0R, the Transformer's 's prediction for y ( at layer E (2.4) approaches the Bayes (optimal) estimator that minimizes the in-context loss (2.5).
14

---

Published as a conference paper at ICLR 2026
D GRADIENT DESCENT IN OPERATOR SPACE
We start by some defining notation that we will use in the next sections. We denote by d' {a : Dx + IR and y {y Dy - R] the separable Hilbert spaces in which our input and output functions lie in respectively, We denote by C()))') the space of continuous operators from X' to y Let (()) denote the of bounded linear operators from to
We begin by defining gradient descent in operator space. Let O denote a space of bounded operators from X' y equipped with the operator norm || gradient descent of L is defined as the sequence
||o. Let L O - E denote a loss function. The
Oc = Oz -
(23)
where
d L(O + 1G)
VI(O)=
arg min GEO. dt
1==0
Suppose we have n onput-output function pairs as f()..... f(n) X' and a(), ..... u()) E y and we define L as the weighted empirical least-squares loss
(()) Σ 1 -
Then (Q( takes the form
d
VL(O)= = arg min GEO. Goll==1 di
[1(00 (0 (O+
(24)
(=
For simplification, suppose that we denote by G* the steepest descent direction. Then the method of Lagrange multipliers states that there exists some À for which the problem in Equation (4 is equivalent to
d
G = arg min GEO
1 (0 (G+1G)
+||G||B(x.))
(25)
dt
1=1
arg min
-f(0
+ Al|G|
(26)
GEO
The second line can calculated by thinking of the loss function as a composition of functions = L2 L1, L1 + which takes
Lttt = (1)) - (0++G)
and L2 : y + R where
L2(3) = (y, v). Then (s) () and (b( 2(y, h). We have
(I2 I. (s) = [L (t)) o (t)()(s)
- (0 (0 (0
-( (0 (0 cf(0%
Evaluating the derivative at t 0 gives the desired expression
D.1 GRADIENT DESCENT IN OPERATOR RKHS
We now introduce the RKHS framework on our space of operators by using an operator-valued kernel. The following definitions were posed in Kadrí et al. (2016b) (Section 4, Definitions 3 and 5).
Definition D.1. Operator-valued Kernel An operator- valued kernel is function & X x f()) such that
15

---

Published as a conference paper at ICLR 2026
(i) K is Hermitian, that is, for all f1 E (2)(f.. k(f2, fi) where denotes the adjoint operator.
(ii) K is positive definite on X if it is Hermitian and for every n E N and all (fi, u.) E X x J V i 1, 2,. ,,.,,, n, the matrix with (i, entry (MeSi, fi)us, uz) is a positive definite matrix.
Definition D.2. Operator RKHS Let O be a Hilbert space of operators X - y, equipped with inner product (. We call O an operator RKHS if there exists an operator-valued kernel x X - [()) such that
(i) The function g k(ffg)e for X' belongs to the space O for all f E N.
iii k satisfies the reproducing kernel property:
(0, = (Of, u)ff u)y
for all O E X
We now state the Representer Theorem for operator RKHS's, as stated in Theorem 11 of Stepani- ants (2023). Assume that O can be decomposed orthogonally into O Oo O where Og is umite-dimension Hilbert space spanned by the operators {Ex 1 and O is its orthogonal com- plement under the inner product (o,) We denote the inner product (,, to restricted Oo, O as
( )o, (:,)o, respectively.
Theorem D.3. Let 1 R + R be a strictly increasing real-valued function and let [(X + R be an arbitrary loss function. Then
= arg min L (100), u100,00000 )Em1 + @(lprojo, Ollo,) OEO
has the form
Ô
dg Ex +
Σ k(f(0)
Amm
im
for some E IR, Eg for all & E rI. and as E for all i E nI.
We this theorem simplify the expression for gradient descent in operator space.
Lemma D.A. Given any O O, let G denote the steepest descent direction of the weighted least- squares loss with respect to || llo as given in equation 25 Suppose O is an RKHS with kernel n. Then
GO()) = (( -
(27)
for some sealar e E R+
Proof. We apply theorem D.3 to equation Equation 25 with Og the trivial subspace and (ss)
2 s² Then our solution has the form
GO(,)
())
(28)
We also know that
| (a), ))ff))
(m(((() (U(F(), asly
1,1=1
where the last equality follows from the RKHS property. We observe that
(k( x(F(),
)a,)o Σ (as (f)))()))))))))))
16

---

Published as a conference paper at ICLR 2026
by the same RKHS property. Note that k(f() f()) E (())) that is, is a linear operator from y to D. Let U X", F )m be such that Uj u and F, 0f Then
2(u (9 - -f , , f0)))s))r + Mos, ,())))))))))))))))))))))))))))))
a" = arg min aE
f==1
(2(u - -f + )a;)) ;())))))))))))
arg min oE
1,=1
Taking the gradient of a zero, that is, V, 0 gives us U OF (here looking at as an element of ) We also note that since 11 1=0
(as
{()))))>))) =1
It follows that
1 12 i- (u()) Of(ii, x(f(), f(l)((li) ofolly (U-
αx"
Therefore
G*)+) Σij=i (u()) - - x(f - - s((()) - -0
This gives exact form of stated in equation Equation 27
E IN-CONTEXT LEARNING VIA GRADIENT DESCENT PROOF
We first recall some notation from section 3.2 We are given n demonstrations (f(1) u()) X for all E [n]. We set y X' for our purpose. The goal is to predict the output function for f(a+1), We stack these in matrix Zo that serves as the input to our transformer:
(11)
(
f(a)
f(+11)
Zu = [2 20), 2(++1]] =
(1)
u (n)
0
Ze denotes the output of layer E of the transformer as given in equation
Theorem E.1. Let X x X + L(X) be an arbitrary operator-valued kernel and O be the operator RKHS induced by k. Let {(f()) ( ) and L(O) = =10)==En1> -0 p 112 Let Og O and let Ot denote the operator obtained from the f-th operator-valued gradient descent sequence of L with respect to -loo as defined in Equation (3)) Then there exist scalar step sizes ro...., I'm such that if. for an m-layer continuam transformer T == Tm o... To. [H, W)le, k(u(), wnl)),
0
0
Wo,e
- x(T)
Wa, L, and Wa.e 1 for each 0, ..., m, then for any f E X.
Te(f: (W., WG, ()) ((() -Off.
Proof. From calculations in subsection Appendix has the form
D.I we know that the E-th step of gradient descent
O 0
(()) (z)(
From the dynamics of the transformer, it can be easily shown by induction that X; Xo for all L. We now prove that
" ()) + Te(s();; (W,, W., 22 (n))
(29)
17

---

Published as a conference paper at ICLR 2026
for all j :,...:74. n. In other words, "n()) Uz We prove this by induction. Let's explicitly write the dynamics for layer 1.
(i) is equal the predicted label for f. ( 5"
21 + + (H(W Ni.xXo)M(W..2200") T
ii(Xo: Xo)
[
0
0
(1o
0
0
f(
f(n) uln)
w(nt1)) )
(ott))
Zo- +
Xo)
0
0
-
-
(1
Zo +
i(Xo: Xo)
0
f(1))
k(f(), f(2))
k(f() ((1)
x(F() f)
0
(u(11) T
a(f(0),f(1)
Zo +
T
k(f.f())
x(f.f(n) x(f.))
0 !.. EL f(i
Zo - r' x(F())
Ei mlf, (())())
(()) [u()) - ro En A K(P(), )()
m(n) ra Ei= nil (((()) f(0) -6 nls, p
By definition, (f: (Wg. Wo, W4)l2(),,.. ,..., 2()) -r Ee n(f. ju() if we plug x a; for any column j. we recover the j-th column value in the bottom right. In other words, for the l-layer case, have
( - w( + (T Wo, W1)l= (!.., "1
For the rest of the proof, we use Te(f) denote Te(f); (W Wo, Ww)l ...., (m)) simplify notation. We now prove the induction case. Suppose that u, = ul + Te)(()). Then, by exactly the same calculation above, for the l + 1 layer:
Ze+1 - (H(W W M.. T
- (ii(Xo, (MXX...)).((MM..222))))
0 Ze r/é Enh k(F(), 1(0) E k((sn)) y
E m(S, )l) (9)
(30)
where the second equation follows from the fact that Wg and Wi. are identity operators and X, Xo. We now apply the induction hypothesis to the right hand side and get
(())
(1) + Te(s(1)) - re E. (((()) (() + T)(
f(m)
(n) + (a() ro Σi _ (0)000+++++000000
+ TE( Te(f) re Ei ((( ((l)) +TF((F()) (()
Now Te41(f) = Te(f) - En K(f, (1) + ()). Substituting f(i in place of f gives
us
(1) ())) + +Ter(((0))
18

---

Published as a conference paper at ICLR 2026
We now proceed to the proof of the theorem using induction. Atep 0, Zu := 0 Ou. Now assume Te(:: (Wy; Wo, Wk)l2(1) (()) -Oif holds up some layer f. For the next layer & + 1,
Te+1 (MW Wo, Wx)) = Te(f: (W W. Wa)l2
(1)) re
H (o0latt))
TT(J:WW,p W., W))lz()) ,..,,,))
(.)) -
Σ
22 =00ff - r/
K(S, {() O&f)
-0t1f
Here, the first line follows from plugging in Wg. We, We in Equation (30p. The second line follows from Equation (29) and the induction hypothesis.
F BEST LINEAR UNBIASED PREDICTOR
F.1 BLUP COINCIDES WITH BAYES OPTIMAL
We now provide the formal statement that the Best Linear Unbiased Predictor (BLUP) and Bayes Optimal predictors coincide in this Hilbert space kriging setting of interest.
Theorem F.1. (Theorem of Menafoglio & (etris Let Xa+1 E and X = (Xi,...,,X. Xn) X be zero-mean jointly Gaussian random variables. Assume that the covariance operator Cx is invertible. Then
E(Xn+1 = LX,
where the L is the linear operator given by Cxx+1xxCC xxxx Hence, the conditional expectation is an unbiased predictor and minimizes the mean squared prediction error
among all the measurable functions f: X" + X. The best predictor, in the mean squared norm sense, coincides with the Best Linear Unbiased Predictor.
F.2 IN-CONTEXT LEARNING BLUP PROOF
= ( ( Let K. X x X + C(X be an operator-valued kernel. Assume that UIF a k. Gaussian random variable per Definition 3.2 Let the activation function H of the attention layer be defined as (H, W)) : k(u()) w w())). Consider the operator gradient descent in Theorem Then as the number of layers m - the continuum transformer's prediction at layer m approaches the Best Linear Unbiased Predictor that minimizes the in-context loss in Equation 9
Proposition F.2. Let F =
Proof. The notion of Best Linear Unbiased Predictor (BLUP) for Gaussian random variable when all but one of the variables observed, has been studied extensively in the literature on 'kriging'. This problem was solved for random variables in a Banach space in Luschgy (1996). Hilbert-space valued random variables would be a special case of this. This was dealt with in Menatoglio & Petris (201)) We use these results find the BLUP for (=+1) conditional on (1)) ...., (x), First we partition the covariance operator IKF) into K. which represents the top-left block. Let E _(") denote the vector given by 1/3 = Ki,++1"
K J
K
Kntint1
We note that TK is the covariance operator of the random variable t = (u(1). ..... (n1)) E 2", The vector // is the cross-covariance operator between the random variable ( X and U. Following Theorem 2 in Luschgy (1996), assume that İ is injective. Then Theorem in Menafoglio &
19

---

Published as a conference paper at ICLR 2026
Petris (2016) (which is a Hilbert- space version of Theorem in Luschgy (1996)) states that the Best
Linear Unbiased Predictor with respect to the mean squared norm error
B(Ilu 1()))
among all measurable functions f: x + X, given by
v Ü x(f(n+1), ff f(0 (00))il')uu1)))
0
0 r
From the premise 3.1 know that Wa. 1, Wa,e 1, We,e
Set re
ô1 for
all where ó positive constant satisfying || - 5TK||| < 1, where the norm operator norm.
From previous caleulations done in Equation (30). we have:
(1)
M(F f(ojul)
Ug
We define the vector Ü2 = [ug
Ut (1))
Then vector notation,
U2+1 = (I-=
Using induction on gives us:
Úet1 (1 -1-8K)
Again, using Equation (30 we have:
(x+1)
(n+1) Σ
k( x(f(n+1),,( flilza
Ut1 ut
j=1
In vector notation, this gives us
(n+1) (n+1) ue+1 = =u =,(+t) -5
-8
Σ
Up -52
Σ (I S
k=0
k=0
Since 5 was chosen such that - oKK|| kー
Ekzo
5Tk)* Hence, T. 0c,
ut+1 + - U, which is the BLUP of (m+1)
(n+1)
G OPTIMIZATION CONVERGENCE PROOF
We begin by reviewing some basic notions of differentiation in Hilbert spaces, for which follow Clément 2009).
Definition G.1. Let X + R be a functional from a Hilbert space X' to real numbers. We say that L is Fréchet-diiferentiable a function á IR + R with lim o(s) = such that
fo X' if there exists a bounded linear operator A X - R and
L(fo + h) - L(fo) Ah + (([bll))).
In this case we define DL(f)) = A. Definition G.2. L is Fréchee-diiferrntiaale at fo E X' with DL(fo) E X then the X' X'-gradient VxI(fo) E X' is defined by
(Vx DL(f)((f)
for all f E X.
We state the assumptions required prove the results of the optimization landscape.
20

---

Published as a conference paper at ICLR 2026
Assumption G.3. (Rotational symmetry of distributions) Let PF denote the distribution of F (r(),... (.))...,(((.))) [(++1) and (K() = BujF [U @ U.. We assume that there exists self-adjoint, invertible L F and IKFF)
operator Σ X + X' such that for any unitary operator M, ((EV/MME-/33
Assumption G.4. For any Fi, P2 E x and any operator S : + with an inverse 5-1, the function satisfies H F2) = Fi,
Theorem G.5. Suppose Assumption G.3 and Assumption G.4 hold.
Let f(r, Wa. Wa)
Wo.e = {oo
0
J
re c-...... Wg Wa,e where L is as defined in Equation 99. Let S C
Omt11 x O denote the set of (Wa, Wa) operators with the property that (W Wo) E S if
and only for all l E - 1/2, 1-11 Then
m), there exist scalars be, R such that Wae b. 1/2 and
inf (w.
[(0m,)) + V.,,, Ilis + VV [iss] = 0.
(31)
t=
Here V W,,, and We. denote derivatives with respect to the Hilbert-Schmide norm -llsss
The key insight to generalizing the proof for functional gradient descent in Cheng et al. (2023) is that the finiteddimensional case, we are working in the Hilbert space gdxd equipped with the Frobenius norm. In the current setting of operators, we work with an arbitrary Hilbert space. We refer the reader to Clément (2009) for discussion on the existence and uniqueness of gradient flows in Hilbert spaces.
Proof. We define S-gradient flows as
r(tt)= (r,
dt
d
(t,((t) Be(l)
dl
d Wa,e(t) Coll) dt
where
(t) (Ww, w ((((f, (.(),, BB(t) = be(t)_11/2
(t) == (V,,, L(r(), ((r(t), (t).), (((t))))/)) Öe(t) = ce(t) 1/2
We observe that the definition of the functions B and C ensures that (t,)(t), Wk,e(t) S for all t.
(t),))
dt
3,,4,(0,,, (-0,0,000000000-00,040000,,0..00 (x())
(32)
+
Vw,, B((t)
(33)
0
Vw.
(34)
21

---

Published as a conference paper at ICLR 2026
Clearly, Equation (32
(10000 From Proposition G.6 note that
33
(,, (())))
Vw,,,
Similarly,
(3 S
1 10w,,,000,,,,,,
We have shown that at any time 4
dt ((((t)) Wa(t)) S (0,
t=
T(((), W2()) Wa
Σ (ww,, (t))l
Suppose Equation
does not hold. Then there exists positive constant c 0 such that for all t,
Σ Or, M,(),
Ww,, L(r(1), Wy(1) (c)
+
(,(t),
2 c.
This implies that de [r((t,, (t)., (a) < -c for all t, which is a contradiction since L(-) is bounded below by zero. Hence, we have proved that Equation I22 holds.
Proposition G.6. Suppose F, U satisfy Assumption G.3 and H satisfies Assumption G.4 Suppose Wg, Wi. S. Fix a layer j E (O...., m) For anr RE L(X), lei (R, denote the collection of operators where (y (R,j),, == Wa. + Rand (R, for all 8 + j. Take arbitrary R E L(X). Let
R Tr
where Ry denotes composition of operators and Tr is the trace of operator as defined in Definition G.10 Then for any j E {O.....m) m).
Lrr, (, 1= dt
S di
(), Wv) c=
(35)
and
d क L(r, W., Wr (a
Mrr, Wn, Wi
<
(36)
di
1=
Since the proofs of Equation 35 and Equation (36) are similar, we only present the proof of Equa-
tion 35
Suppose (R) be the collection of operators where (R))))) Wa R all l {o.....m). m). Since Proposition (.6) for any j. and
a Lr. W,(tt)) di
Σ d
wb)
(). Wa),
di
1=0
22

---

Published as a conference paper at ICLR 2026
an immediate consequence of Proposition G.6 is that
d
d L(r, Lrr (tR)., Wk) 1=0
de L(r (y, Wx) 1= dt
The rest of this section dedicated proving this proposition. As a first step, we re-write the in-context loss using an inner product.
Lemma G.7. (Re-writing the In-Context Loss) Let Zo be the input of the transformer where the last entry of Y has been masked (or zeroed out):
((a) (1))
f(n)
f(att1
Zo=
mm)) u(a+1)
Let Le be the output of the (f-1) 1)th layer of the transformer initialized Zo. Let Fe and Üt denote the first and second row of Zt. Then the in-context loss defined in Equation 99 has the equivalent form
L(W., Wa Wt) E K(I -MU m (I M)U m+ 1)1-
The Hilbert space corresponds to the direct sum of Hilbert spaces O X. where the inner product is given by
((1,,...,,+t1))) (wi, ..... Wn )
Σ 1=1
(vi, wx)xx
Proof. We deviate from the mathematical convention where an operator that acts on a function is always written the left of the function. We adopt the convention that fO, where O is an operator and ff is a function, also means that the operator acts on the function. This is how the equations henceforth should be interpreted.
Ae 0
When Wo.e
rel]. the output at each layer, given in Equation can be simplified as
follows:
Fet1
F2 Up
([A+
0
Fe
f(+t1) Aef
2
U2+1
=
=
rel
Un
T T
Fe
o o
([Artf
i( Fr, Wb.
U2
reUe
((((()) ((22)
m( {(a))
r(s Ss) k(f(?)) fii
k(f(22 ff p(1))
x(F12) {lai)
FP
(reUs) (AcFt)T
Un
0
0
((
s()e,
[F U2
AF F We rHU a WL.FF))
=
(37)
Suppose Zo is equal to Zo everywhere, except the (2, + 1)th entry, where it is e instead of O. We compare the dynamics of Ze and Ze. Since Fo = Fy. we see that Fr = Fe. We claim that U. U. = [0, ...... 0, c). We prove this by induction. This is trivially true for e 0. Suppose this holds for step E, then at step f +11:
UE+1
Up + TeU:MM Wa.eFi)
Up + T&U, MUUMMM Wo.
- [O..... .. + TUUMM Wr,eFi)
-Ue1 [0,,...0.c)
0.c,
23

---

Published as a conference paper at ICLR 2026
where the second equality follows from the fact that U&M has its (n 1)th entry zeroed out, so by the inductive hypothesis, U&M = U.M. The third equality follows from the inductive hypothesis again.
Replacing by u(++1) tells us that 12m+1lan41 12m+ilan41 + u(++1) Hence, the loss in Equation ( can be re-written as
L(W,, Wx) Ez.
[[Z 11]
Ezz [|Um+ (T- (M-M)
Ez, [((I M) (I A))m++i)))
(38)
where the inner product in the last line is the inner product on column vectors in + i=1 X.
Since A an immediate corollary of Equation (37 follows.
Corollary G.8. When A = for all i {0, ( m).
Fet1 = Fo.
Moreover,
U4+1 = U2 + T.U.M Fo, Wo.
Un II r,H Fo))
==
We now define the trace operator and the tensor product, which allows us use the covariance operator.
Definition G.9. Let L1, 22 be elements of the Hilbert spaces 21 and Xz. The tensor product operator (r1 @ x2) 2i + X2 is defined by:
(x) 2 22)2 (Z1, 2)
for any x E Xi. Definition G.10, Let X' be separable Hilbert space and let {er be a complete orthonormal system (CONS) in X. If e (1 X') be nuclear operator. Then we define
00
Tr T Σ (Te..)))
Henceforth, assume that our Hilbert space is separable.
Lemma G.11. Suppose I1, 22 X, be used construct the tensor product T2 T2. Then
Tr 21 ② D2
(I1, x22)
Proof. Let (e be CONS for the Hilbert space X. The inner product be written
(I1, x2) =
(1 ej
(2z es)e
Σ (1 (22
i=1
Similarly, the trace of the tensor product is given by
Tr & 12
(x) (2)eer))
Σ
( ei) ())
i=1
This completes the proof of the lemma.
24

---

Published as a conference paper at ICLR 2026
We also state some properties of these operators without proof:
1. Let A be a linear operator from A' + X' and E X' Then
(Axı @ Azi) = A @ x1)A",
where A" is the adjoint of A. This can be verified by writing out how each of the above operators acts on an arbitrary element E X.
2. The cyclic property of trace, that is, Tr(AB) Tr Tr(BA) also holds in infinite- dimensional spaces if A, B are both Hilbert-Schmidt operators. This can be verified using a CONS.
The rest of the proof for Proposition G.6 has the following outline:
1. We reformulate the in-context loss as the expectation of a trace operator.
2. We introduce function x O ((++1)x(a+1) which is used simplify the loss
equation.
3. We understand the dynamics of & over time.
4. We use all the identities have proved to complete the proof.
In- Tn-Context Loss as the Expectation of Trace-
Using Lemma G.11 we can write the in-context loss in Equation (38 as
L(W,, Wg, Wa) = Ez, M)
& ((I M)
Since (I M) is self-adjoint operator,
((I M)JU+ @ ((I M)U -I- & UU M).
Then
L(W., Wg. Wa) = Ez, [Tr(U--M((UT) 8 U (I M)]]
Simplifying the Loss using the Function d- We drop the bar to simplify notation and denote Fo by F. We also fix a E to...., m). We define the functional &i xn+1
O (ln+1)x(=-1) as
e (F,S) = II (I [
E
Again for the purpose of simplifying notation, we suppress the index j since the proof follows through for any index. We use é to denote Φ and (se)(S) denote (S,
The loss can be reformulated as
L(W,, Wg. Wa) = Ez, [Tr[(I - ()((U(((E)))) @ (L))(F - -MM]]
= Ezz [Tr[(T - (F, @ T (FF,S)) M)]]
= EF. [Tr[(I (K) S) -M)]]-
where the last equality follows from Assumption G.3 and the linearity and cyclic property of trace.
Let U be uniformly randomly sampled unitary operator. Let Let /// Using Assump-
tion
G.3
2/22ME-1 1/2 d F, we that
L(W, W, Wz) dl
-B Erg [T
(F)) M)]] d Tr((T- MM0 (P([T-M)0"FFo... (R)
1= di
1=0
2BF
o(Fo. 0) (T- M)]
dt
Tri(I - M) (2 ()F(F)) d Fllt2
2B
tR)
(I -I-10 M-
1=0
(39)
25

---

Published as a conference paper at ICLR 2026
where the last equality uses the assumption K())+Fo) ((F))) from Assumption G.3
Dynamics of & over Time-
We now prove the following identities-
@lUgFoo 0) = =(Fo, 0)
(40)
and
dt Fo tR)
(Foo, =FFo 1255 Rlly)
(41)
dt
1=0
We also recall that
Wg
1/2 u.
(42)
Similarly,
Wv,dls Wky
(43)
We now verify Equation 40))
oUUzF..0 0)
(I + MMM.MM
ビ=0 m
III
II (I (L (e.(())) (F.. ビ-
where the last equality follows from the rotational invariance of H from Assumption G.4
We verify the following identity that will be used later on-
H(WWee + Fo. We Fs
1=
==
_ iiW
tu Fu Wx.eFo)
(44)
where the first equality follows from Equation 42 and Equation lows from Assumption G.4
(43 and the second equality fol-
Using the chain rule, we get
o(UsFo, tR) dt
t=0
j-1
M d ((V tRY,FFo Wl, di
I
I MüN,, Wa, (o,
1=
II (I+ (Wt We, (=f+1
d
I ( + MHH(W Mantoo)
M (WW + ul RUsFo, Wa,eFo)
di
1=
II (I+ (. Wo, (=++1
d olls Fo, t Rly R)
1=0
where the second equality follows from Equation
42
Assumption
G.4 and Equation 44
and
26

---

Published as a conference paper at ICLR 2026
Putting it Together- Continuing from Equation 39
we get
L(W,, (tR),, Wa)
Fo
Tr((1 - M) (F)
J. (u ut Iltss)
(1- (1-M))
di
1=
1==0
Tr((T M) 0)K Fb)
dt (o.. u11 Rlls)
(T-M))
km0
(I- Min
2B
Tri(I - (Fo.
o thu (((R
dt
2EF
Tr((I (FG O)K(()) dt (Fo, tR)
(I- Ad)]
1=0
di LOW,, W; (ti),, Wv)
Here, the first equality follows from Equation 41 and the second equality follows from Equa-
tion (40). The third equality uses the linearity of zt o (S) in S and the fact that it is jointly continuously differentiable. This concludes the proof.
H EXPERIMENT KERNEL DEFINITIONS
We provide below the explicit definitions of the kernels studied in the experiments of Section 5.1
Kernels for f')
Kernels for (y, v)
Name
Definition
Name
Definition
Linear
( x')xx
Laplace
cxp
a vlllll d yll
lfffflx
Laplacian
exp
Gaussian
Exp
20
Gradient RBF
exp
20
[11 111lx-11,m ((1x)
Energy
exp
20
27

---

Published as a conference paper at ICLR 2026
I BLUP ADDITIONAL EXPERIMENTAL RESULTS
We present below the in-context learning predictions for each pair of the true data-generating kernel and choice of kg kernel, in the below visualizations. Paralleling the results seen in Figure | we find that, when hy matches the data-generan -ata-annn kernel, the resulting field prediction structurally matches the true Of Across these visualizations, fix Ky to be the Gaussian for simplicity of presentation.
Linear Gaussian (Truth)
Linear Gaussian
Laplacian Gaussian
Gradient Laplacian
Energy Laplacian
Figure 4: In-context predictions for (kzg, kyy being (Linear, Gaussian), (Laplacian, Gaussian), (Gra- dient RBF, Laplace), and (Energy, Laplace) with the data-generating kernel being (Linear, Gaus- sian).
Laplacian Gaussian (Truth)
Linear
Laplacian Gaussian
Gradient Laplacian
Energy Laplacian
Figure 5: In-context predictions for (kiz, ky) being (Linear, Gaussian), (Laplacian, Gaussian), (Gra- dient RBF, Laplace), and (Energy, Laplace) with the data-generating kernel being (Laplacian, Gaus- sian).
Gradient RBF Laplacian (Truth)
Linear Gaussian
Laplacian Gaussian
Gradient Laplacian
Energy + Laplacian
Figure 6: In-context predictions for (kz, kky) being (Linear, Gaussian), (Laplacian, Gaussian), (Gra- dient RBF, Laplace), and (Energy, Laplace) with the data-generating kernel being (Gradient RBF, Laplace).
28

---

Published as a conference paper at ICLR 2026
Energy + Laplacian (Truth)
Linear Gaussian
Laplacian Gaussian
Gradient RBF
Laplacian
Energy Laplacian
Figure 7: In-context predictions for kky being (Linear, Gaussian), (Laplacian, Gaussian), (Gra- dient RBF, Laplace), and (Energy, Laplace) with the getta-genraatng kernel being (Energy, Laplace).
J POISSON EQUATION SAMPLES
We provide visualizations of samples obtained by solving the Poisson equation below, used the experiments of Section 5.2.1
Poisson Equation Samples u)
U1 V2
1/3
Figure 8: Sample solution pairs generated for the Poisson equation in-context learning task. Samples of f were drawn from a GRF and u solved with an analytic FFT-based Poisson solver.
K OPTIMIZATION EXPERIMENT DETAILS
K.1
OPTIMIZATION KERNEL ASSUMPTIONS
We now provide the proof that Assumption 3.5 holds for the chosen kernel under the particular choice of k2 being Linear and ky being Gaussian, which follows trivially from properties of the inner product. In particular, for the assumed operator S X + X with an inverse 5-1 referenced in Assumption 3.5 notice
(())a)) kLinear (f ((2) * u)
= (1(), ((2) (KGanes * u)
F2)u = S
(() s01,,_110 (KGauss 4)
((()) (KiGausss u)
= -((1) (1))(22))))) (KiGauss * u)
(1(1), (())) ((2)) * u),
completing the proof desired.
29

---

Published as a conference paper at ICLR 2026
K.2
OPTIMIZATION EXPERIMENT HYPERPARAMETERS
The hyperparameters of the optimization experiment (results presented in Section 5.3) are given in T The continuum transformer implemented in PyTorch Paszke al. (2019). Training model required roughly 30 minutes to an hour using an Nvidia RTX 2080 Ti GPUs.
Table 1: Hyperparameters used in the continuum transformer training experiment.
Category
Hyperparameter (Value)
Number of Layers: 250 Image Size: 64 x 64
Model
2₂ a: 1.0
Initialization: -0.01
k, Kernel: Gaussian
Dataset
# In-Context Prompts:
# Operator Bases: 30
Optimizer: Adam
Learning Rate: 0.001
Training
Epochs: 10
# Samples: 128
Momentum: 0.0
30

---

Published as a conference paper at ICLR 2026
L
NOTATION REFERENCE
We provide below a comprehensive presentation of the notation from the paper.
Table 2: Notation for the data
Notation
Description
X
Hilbert space in which the input and output functions lie.
i-th pair of input and output functions.
Ze
A matrix in (n+11)+)) whose first row consists of input functions f();; i 1, ...... n+
1, and whose second row consists of output functions u(²),... ,..., u((), 0. The zero is
a placeholder for the predicted output of f(n+1) which will change as Z passes through the transformer.
X
The first row of Ze.
Space of operators in which the true operator lies. This is assumed to an RKHS.
K(P)
Covariance operator of the output functions U conditioned on the input functions F.
Table 3: Notation for the transformer architecture
Notation
Description
Te
The ε-th layer of the transformer.
Wa,e, Wae Wa,
Key. query and value operators at the ê-th layer respectively. Mask operator.
M
H())
Non-linear operator.
re
Scalars parametrizing the value operator at layer &.
Table 4: Notation for ICI., gradient descent, and the RKHS framework
Notation
Description
L(-)
In-context loss.
Oz
Operator the ê-th gradient descent step.
Operator-valued kernel which takes an element in X' x X' to a bounded linear operator on X.
M LLM USAGE
LLMs were not used in the writing of the manuscript; they (specifically Gemini) were used as a tool help with the debugging of some pieces of the code for the experiments.
31


Reviews:
[{"review_id": "Dx5swge5HA", "reviewer": "Reviewer_uu2i", "summary": "This work extends the line of work showing that Transformers perform gradient descent in-context during inference to continuum Transformers (defined on function spaces) and operator learning. The main theoretical contributions are that continuum Transformers can implement operator gradient descent in a reproducing kernel Hilbert space (RKHS) of operators and that in the infinite-depth limit, the model converges to the Bayes-optimal (kriging) predictor. The authors prove this by providing a weight construction implementing operator gradient descent. Furthermore, they prove that their choice of parameters also represents a stationary point of the training loss under gradient flow over the space of functionals. They empirically validate their theoretical claims on operator regression tasks.", "questions": "- As far as I understand, the theoretical results characterize the continuum transformer’s behavior in the infinite-depth setting, but do not provide any results or intuition about the rate of convergence. Would it be possible to extend the authors' analysis to provide convergence rate results (e.g. how many steps/layers are required for the in-context operator to approach the Bayes-optimal predictor)? Or alternatively, have the authors observed any empirical scaling trends relating depth to approximation quality?\n- In Figure 2, the authors attempt to provide empirical validation of their fixed-point claim by evaluating the cosine similarity of the key/query operators across layers during training as a proxy. I'm wondering if it would be possible to use a similar proxy to visualize the rate of convergence of the in-context operator prediction as the model evaluates layer-by-layer? For example, by evaluating the cosine similarity or relative error between the predicted operator at layer $\\ell$ and at layer $\\ell-1$, or between the layer $\\ell$ prediction and the Bayes-optimal operator? Such a diagnostic might more directly visualize the model prediction's evolution under the proposed functional GD.\n- Is it possible to extend this analysis of infinite-dimensional prefix-ICL to the infinite-dimensional causal-ICL setting, where given $n$ ICL examples, the model is supervised on all $n-1$ subproblems (given $i-1$ examples, predict the $i$th, for all $i \\leq n$) in parallel? (e.g. as in [1])\n\n1. Bai et al., 2023. Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is novel and investigates an important question, probing the fundamentals of continuum Transformers for in-context operator learning. This is a problem that may be of interest to folks in the in-context learning theory and operator learning/PDE foundation model communities.\n- The paper is well-written and clear. The theoretical results are interesting and well done, and the authors validate their theoretical results empirically.", "weaknesses": "As far as I understand, the work does not provide any analysis of how many functional gradient descent steps / number of continuum Transformer layers it would take to achieve convergence on the in-context operator learning problem. In my opinion, the work as it stands already provides enough results to recommend acceptance, but I would be curious to know if such results are possible (see questions).\n\nMinor:\n- The authors could more clearly emphasize the significance and limitation of the theoretical claim in Section 3.4 / Theorem 3.6. As stated, the theorem establishes that the proposed parameter configuration constitutes a stationary point (fixed point) of the training dynamics in the continuum limit, but it does not imply that this configuration is the unique or global minimizer of the loss, nor that it corresponds to the explicit solution of the gradient flow. I understand this type of result is standard in recent in-context learning theory (and I believe the result itself is valuable to the community), highlighting it explicitly would help readers interpret the scope of the claim.\nFor example, lines 61 and 273 could note that the parameters form a critical point, not necessarily a global minimum.\n- I think the writing would be slightly clearer if the authors allocated a specific subsection in the paper for discussion of the novelty/importance of the proof techniques, rather than scattering such comments throughout.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IiMBJS2NWe", "reviewer": "Reviewer_sPh8", "summary": "The paper lifts the in-context learning approach from finite dimensional transformers to continuum transformers acting on function spaces. Replacing softmax with a PSD operator valued kernel, each layer implements a gradient descent step for the in-context least squares objective in an Operator Valued Reproducing Kernel Hilbert space (OVRKHS) and depth corresponds to more steps. Under a $\\kappa$-Gaussian distribution and matched $\\kappa$ attention kernel, in the infinite depth limit the method equals a Bayes optimal predictor (BLUP/kriging). A pre-training analysis, under strong assumptions, shows that whitening Q/K and considering a residual injecting V forms  stationary solutions, suggesting that the model can learn parameters in a GD manner. The experiments validate the theory for a simple test.", "questions": "- How is the projection error bounded when considering a truncated spectral basis?\n\n- Is there a way to relax the whitening assumption by maybe considering a spectral normalisation of W_q, W_k and an adaptive step size? \n\n- How sensitive is the fixed point analysis to broken symmetries from non-periodic BCs, irregular meshes etc? \n\n- Please add at least one PDE example and compare to an FNO+meta learning or ICON approach.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper presents a non-trivial extension of in context leaning to an operator valued, continuous, setting. The math is elegant have technical substance and can be used for further analysis.\n\n- Considering operator valued kernels for this problem and their representer theorem is elegant and makes defining each layer as one gradient descent step clean. The analysis for the fixed point pre training and the connection to BLUP is nice despite the strong assumptions.\n\n- The attention, the values and how each part of the method is constructed is clearly stated. \n\nOverall, I find this paper to be mathematically elegant and insightful.", "weaknesses": "- The Bayes optimality/BLUP limit require the data distribution to be $\\kappa$-Gaussian and also an exact kernel match. In practice, the input functions come from non Gaussian distributions and the kernels might be misspecified. \n    \n-  Rotational symmetry and equivariance are strong assumptions. Correct me if I am wrong but I believe that a global invariance in whitened coordinates might not hold due to anisotropies or boundary conditions. \n\n- All the proofs are performed in the function space, however the implementations in practice are discretised. Maybe there should be an analysis linking the continuum to the discrete implementation and what errors this induces. \n\n- Operator valued kernels can be expensive. The paper bypasses that using a separable form trick, scalar similarity * cheap operator, and tiny prompts however a discussion should be presented on estimating rich kernels and larger contexts. \n\n- The theory requires a PSD and symmetric kernel which standard softmax is not. So, what the authors use is not technically an attention kernel because it is not row normalized. This is not that clear. I suggest to the authors to use the kernel construction considered in “Learning Operators with Coupled Attention” where they consider a PSD/symmetric/universal kernel that is also normalized. The authors there consider a scalar kernel I believe that this can be adapted to your method in a straightforward manner (see Micchelli “On learning vector-valued functions” and Caponnetto “Universal Multi-task Kernels”).\n\n - The empirical validation is very limited. You consider no PDE benchmarks, no baselines, ICON or FNO and a meta-learner, no ablations for kernel mismatch or symmetry braking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c9Pz19eR1M", "reviewer": "Reviewer_iXYP", "summary": "This paper presents a rigorous theoretical analysis of continuum transformers, the function-space analog of standard transformers, when applied to operator learning problems, such as those arising in PDE modeling. The authors show that:\n\n- If the query, key, and value operators are chosen appropriately, each layer of a continuum transformer performs one exact step of gradient descent in an operator-valued RKHS\n\n- As the number of layers tends to infinity, the transformer’s output converges to the Best Linear Unbiased Predictor, equivalent to the Bayes-optimal estimator of a Hilbert-space Gaussian process.\n\n- When the model is trained end-to-end, its parameters naturally converge to the same fixed-point configuration that realizes the gradient-descent dynamics\n\nEmpirical studies are limited but targeted:\n\n- The authors validate the gradient-descent interpretation when the attention kernel matches the data-generating kernel.\n\n- They train a 250-layer continuum transformer on synthetic Gaussian-process data to verify the convergence of learned query/key operators toward the predicted fixed-point structure.\n\nThe paper is primarily theoretical but provides convincing toy experiments supporting the analytical results.", "questions": "- How do you envision “tokens” being defined in PDE settings? Are they spatial patches, basis coefficients, or entire functional evaluations?\n\n- Could your analysis extend to models queried at arbitrary times (e.g., continuous-in-time diffusion or flow prediction)?\n\t​\n- Would the cosine-similarity convergence in §5.3 persist for shallower models? A quantitative study varying the number of layers could clarify practical limits.\n\n- Do you expect the same theoretical behavior when training on PDE datasets like Navier–Stokes or Darcy flow, where discretization and boundary conditions break some of the theoretical assumptions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Originality: Extends the “transformer = gradient descent” theory from vector spaces to operator-valued RKHSs, a nontrivial and conceptually elegant generalization.\n\n- The mathematical treatment is rigorous and self-contained, including a generalized Representer Theorem for operator-valued kernels.\n\n- The results provide a clear mechanistic interpretation of in-context learning for continuum transformers.\n\n- Toy experiments (BLUP matching and operator-alignment studies) convincingly illustrate the theorems.\n\n- The work strengthens the theoretical foundation of operator learning and may guide better architectures for PDEs", "weaknesses": "- Lack of practical tokenization discussion.\n\nIt remains unclear how tokenization is defined in PDE-learning settings. While a “next-token prediction” formulation is appropriate for fixed time-stepping tasks (e.g., 6-hour weather forecasts), such an in-context learning paradigm does not naturally extend to continuous-in-time scenarios where predictions are required at arbitrary temporal resolutions. Moreover, most PDE and computer-vision problems require some form of spatial tokenization to capture local correlations across the domain (e.g., patchifying a sample function). In this functional context, the notion of “next-patch prediction” becomes conceptually ambiguous and difficult to formalize.\n\n- Limited empirical scope.\n\nAll experiments are synthetic. The theory’s applicability to real PDE problems (e.g., weather trajectory generation or high-resolution dynamics) is not demonstrated.\n\n- Scalability concerns.\n\nIt is unclear how the proposed theory would extend to realistic, large-scale problems, such as generating full weather trajectories. In such settings, deploying a 250-layer network (as used in Section 5.3) would be computationally impractical. It would therefore be valuable to analyze how the average similarity score evolves as the model depth increases, from shallow configurations (1–10 layers) through intermediate (10–50) to deep regimes (50–250). My expectation is that the network would not reach the exact fixed point predicted by Theorem 3.6, but would likely approach it asymptotically.\n\n- Dense notation.\n\nThe presentation could benefit from more intuition and conceptual figures summarizing main theorems and propositions.\n\n- Lack of clarity on computational and memory complexity.\n\nThe continuum formulation replaces matrix multiplications with integral operators, but the paper does not discuss the resulting computational or memory scaling. It is unclear whether the operator-valued attention can be implemented efficiently in discretized settings or whether it incurs quadratic cost in the number of spatial degrees of freedom. A short discussion comparing computational requirements with standard transformer or neural-operator architectures would strengthen the paper’s practical relevance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gIwrvajnWN", "reviewer": "Reviewer_peRa", "summary": "This paper studies in context learning abilities of infinite-dimensional generalizations of the transformer architecture, called continuum transformer and previously introduced by Calvello et al. The authors extend previous finite-dimensional work by Cheng et al. showing that transformers implement gradient descent in-context to infinite dimensional settings for operator learning. The main result is that continuum transformers perform in-context learning by performing gradient descent in an RKHS, under a careful choice of the architecture. The authors conclude by providing numerical experiments on simple operator learning tasks that validate their theoretical findings.", "questions": "1. In Eq. (1), what measure do you take on the space of functions U?\n2. In Eq. (9), shouldn't one consider a relative loss by normalizing by $\\|u\\|_{n+1}$? Does the analysis extend to this case?\n3. I don't think $\\mathcal{O}$ is defined before line 209\n4. Theorem 3.1: I can't find the definition of $\\tilde{H}$ in the main text before Eq. (7). I see it defined in Eq. 17 but it would be helpful to include it in the main text given that it is crucial for the theorem.\n5. I guess Prop 3.3 implicitly assumes that the underlying operator is linear so that the conditional distribution of the output is Gaussian? Is it not a strong assumption to assume that the Gaussian kernel is exactly the same as the kernel induced by the architecture?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Despite the technicalities, the paper is well written and rigorous.\n2. The topic is timely and relevant, especially as transformer-based architectures are becoming increasingly popular in the field of operator learning with the apparition of foundation models.\n3. The main theoretical results are interesting and bring new insights on the in-context learning abilities of transformer architectures in infinite-dimensional settings, and may lead to more practical considerations regarding the choice of the architecture.\n4. The numerical experiments, although simple (linear operators), illustrate the theory.", "weaknesses": "1. Given the formulation of Thm 3.1, it is quite difficult to interpret the result in practical terms. I am wondering whether, instead of framing it as an existence result for the architecture, it would be more natural to introduce the kernel induced by the architecture. The result also makes a lot of assumptions on the weight matrices. Can the authors comment on these assumptions and whether they imply that one should fix these weights in practice during training?\n2. The two assumptions 3.4-3.5 seems quite specific, it would be helpful to have some discussion on how reasonable they are in practice.\n3. It seems counter-intuitive to me that one does not need to make any assumption on the operator $\\tilde{G}$ that one tries to approximate. Could the authors comment on this?\n4. The numerical experiments are performed for a very synthetic task (Laplacian kernel), and would benefit from a more challenging operator learning task (e.g. a time-dependent PDE, or nonlinear problem is the theory extends to nonlinear operators).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)