In Context Semi-Supervised Learning
Authors: Jiashuo Fan
Paul Rosu
Aaron T Wang
Lawrence Carin
Xiang Cheng
Venue: ICLR 2026 Poster (2026)
Keywords: semi-supervised learning
Transformer
In-context learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Semi-supervised learning is performed adaptively, in-context, via a Transformer with sound mechanistic understanding

Abstract: There has been significant recent interest on understanding the capacity of Transformers for in-context learning (ICL), yet most theory focuses on supervised settings with explicitly labeled pairs. In practice, Transformers often perform well even when labels are sparse or absent, suggesting crucial structure within unlabeled contextual demonstrations. We introduce and study in-context semi-supervised learning (IC-SSL), where a small set of labeled examples is accompanied by many unlabeled points, and show that Transformers can leverage the unlabeled context to learn a robust, context-dependent representation. This representation enables accurate predictions and markedly improves performance in low-label regimes, offering foundational insights into how Transformers exploit unlabeled context for representation learning within the ICL framework. Our code is available at https://github.com/Jason-fan20/ICL_Semi.

Under review as a conference paper at ICLR 2026
000
IN CONTEXT SEMI-SUPERVISE LEARNING
001
002
003
Jiashuo Fan
Paul Rosu
004
Department of Electrical and Computer Engineering
Department of Electrical and Computer Engineering Duke University prl 5514kee edu
005
Duke University
006
jf381 duuke edu
007
008
Aaron T. Wang
Lawrence Carin
009
Department of Electrical and Computer Engineering
Department of Electrical and Computer Engineering Duke University lcaringe Learingduke.edo auckkeedddd
010
Duke University
011
edu
012
013
Xiang Cheng
014
Department of Electrical and Computer Engineering
015
Duke University
016
x xiang. wwwnggduke.eee......ooo
017
018
019
ABSTRACT
020
021
There has been significant recent interest on understanding the capacity of Trans- formers for in-context learning (ICL), yet most theory focuses on supervised settings with explicitly labeled pairs. In practice, Transformers often perform well even when labels are sparse absent, suggesting crucial structure within unlabeled contextual demonstrations. We introduce and study in-context semi- supervised learning (IC-SSL), where a small set of labeled examples is accom- panied by many unlabeled points, and show that Transformers can leverage the unlabeled context learn a robust, context-dependent representation. This rep- resentation enables accurate predictions and markedly improves performance in low-label regimes, offering foundational insights into how Transformers exploit unlabeled context for representation learning within the ICL framework. Our code is available at htttps: / com/ Jason- fan20/ICL. Semi
022
023
024
025
026
027
028
029
030
031
032
033
034
INTRODUCTION
035
The Transformer (Vaswani al. 2017) has revolutionized natural language processing, achieving remarkable success in tasks like language generation Vaswani et al. (2017); Radford et al. (2019);
036
037
Devlin et al. ( Brown al. 2220 Touvron al. (2023); DeepSeek (2025); it has also been applied in many other areas, including vision Dosovitskiy et al. (2021). I). network analysis (Cheng et al.)
038
039
(025); Kreuzer et al. (2021) | |Rampasek (2022) and biology Jiang et al.) (2024). Motivated by the demonstrated effectiveness of Transformers for few-shot (or in-context) learning Brown et al.
040
041
2020 promising line of recent research has focused on functional data yon Oswald et al. (2023); Ahn et al. (2023): 3); Zhang et al. )2023;; [Schlag et al. (2021). In this setting one is given a set of contextual data pairs, where each pair characterized by the observed input and output of latent function. It has been demonstrated that a properly designed Transformer can in its forward pass perform functional gradient descent (GD) or close variations thereof for the function, and can use that inference to predict the expected functional outcome for specified query input to the function. The Transformer can perform this inference in-context, with no parameter fine-tuning.
042
043
044
045
046
047
Early work on Transformer-basee in-context learning (ICL. focused on linear functions von Oswald ct al. (2023); Ahnn et al. (2023); Zhang et al. (2023); Schlag et al. (2021), and that has been extended recently to nonlinear functions that reside in reproducing kernel Hilbert space (RKHS) (RKHS)CCheng et al. 2202)). These works focus on learning real- real-valued outcomes, and similar analysis was recently developed for categorical outcomes WWan al. (2025). Categorical observations are of interest when studying ICL for classification problems.
048
049
050
051
052
053
Eqqua contribution

---

Under review as a conference paper at ICLR 2026
054
Input
Leerr edl repreeentation ((xx0)
Predicted Labal Probabilitos P6y00 O context)
055
Un-isbelled Labelled (x1)) 1000- 0)
Add Back
056
Labellad (x19, (x0
Labels
057
Remove Labels
o
058
059
Transformer (In-context
060
Transformer
(Categorical ICL)
061
represeetatiion-leariigg
062
063
Figure I: Setup for semi-supervised in-context learning with a Transformer. Input Data consists of labeled pairs (, ... )..., (ax(m), , } where y(t) E {0, 1}, and unlahcled data {a(mm-1) "...,,..,,) xf")). In stage (A), a Transformer module takes input (ar(1), ..... ..()) and outputs o(x("), where Ø(-) is context- dependent feature representation. In stage (B), we augment (d(xx")), ).... o(x(")}} with the m observed labels. In stage (C), second Transformer module takes this augmented set as input. In stage (D), the second
064
065
066
067
Transformer module outputs prediction probabilities of y y ..... (x)) for each unlabeled 20)) in standard ICL fashion. The two Transformer modules combine into a single Transformer, trained end and-to-end As detailed in Section 3 the first module is motivated by learning eigenmap of the Laplacian, and the second module performs ICL. with categorical observations by implementing gradient descent at inference time.
068
069
070
071
Prior work on Transformer-based ICL largely treats the supervised case, using contextual labeled pairs. When unlabeled points appear, their labels are inferred independently, ignoring structure in the full unlabeled set. Yet in practice we have many unlabeled samples and few labels, and so is of interest for ICL to leverage the context provided by all of this data.
072
073
074
075
1.1 OUTLINE OF CONTRIBUTIONS
076
In-context semi-supervised learning setup. We extend Transformer-based in-context learning to a semi-supervised setting, broadening the concept of "context" to leverage both labeled and unlabeled data within a unified end-to-end Transformer architecture (Figure IT). We demonstrate how Transformers can effectively perform semi-supervised ICL, using scarce labeled data alongside abundant unlabeled examples, without the need for separate preprocessing of inputs.
077
078
079
080
081 A two-stage Transformer construction for representation learning ICL. We construct a two- stage Transformer (Section The first stage performs representation learning from unlabeled data by forming discrete Laplacian from local Euclidean distances, then computing yields context-dependent manifold structure in the forward pass (Section 3.I). The second stage (Section 3.2) is a construction for Transformer-based ICL module that can be shown to perform in-context learning for categorical observations by implementing functional gradient descent in its forward pass. Mechanistically, each stage is capable of implementing an explicit algorithm (1. Laplacian construction + power iteration; 2. kernelized GD for in-context inference), providing a first step toward transparent account of how attention and MLPs realize semi- supi-supe ICL com- putations. In practice, we instantiate both stages inside a single Transformer and train the model end-to-end with the IC-SSI. objective in Eq. 4 the modular construction serves as a strong mech- anistic inductive bias/initialization rather than hard constraint, and joint optimization can deviate from and improve upon the strict two-stage algorithm when beneficial. addition, the construction serves prototype for analyzing full end-to-end Transformers: it gives a modular reference rep- resentation against which we can compare baseline Transformers trained directly on data (e.g., via separation and alignment analyses in (..tion[F.6...
082
083
Eigenmap: this
084
085
086
087
088
089
090
091
092
093
094
095
096
Mechanistic understanding of IC-SSL. Section F.6 examines our method across manifolds of increasing complexity - low-dimensional manifolds in R3 (Figure 2). higher-dimensional product manifolds in 1R and high-dimensional image manifold from Stable Diffusion v1.5 (Figure 5 where our construction consistently outperforms strong baselines. We then use the construction as a reference model interpret how an unconstrained, end-to-end Transformer learns to perform in-context semi- semi-supervised learning on real data (ImageNet100), Our model serves as a reference point for what representations are learned by the standard Transformer (Table [.. as well as for understanding the transsttranstionns in the standard Transformer's 's learning curves (Figure(3).
097
098
099
100
101
102
103
104
Inductive bias and learning. Our two-stage design imposes concrete inductive bocs-local, sparse attention that can effectively implement spectral feature learning (first stage) and gradient- based learning (second stage) This makes IC-SSL markedly more sample-efficient in low-data regimes. As shown in Section across elementary manifolds in R (Figure bigher-dimensione product manifolds in IR 15 (Figure 4c), and high-dimensional image manifolds (Figure 5). this bias
105
106
107
2

---

Under review as a conference paper at ICLR 2026
108
translates into consistently higher aceuracy than strong baselines, including variants that rely on offline Laplacian Eigenmaps, even with few labeled examples. The same structural bias also supports robust generalization: the model retains strong in-distribution performance and transfers to previously unseen geometries (Figure 4b) and modalities (Figure 5). indicating that learns geometry-aware computations rather than overfitting a specific dataset coordinate system.
109
110
111
112
113
114
115
116
117
118
119
120
Figure 2: Synthetic manifolds we use. Left to right: Sphere (8"), Right Circular Cylinder (C), Right Circular Cone (Cone.). Archimedean Spiral (Swiss-Roll, SR), Flat Torus (T*). The colors reflect the binary labels.
121
122
123
1.2
RELATED WORK
124
semi-supervised learning (SSL) has been problem of long long-standing interest to the machine learning community van Engelen & Hoos (2020); Ouali et al. (2020); Chapelle et al. (2010). It is particularly valuable when labeled data are scarce, multiple unla unlabeled samples are observed at once, and there is an underlying structure associated with the covariates from both labeled and unlabeled data. Despite the recent surge in research on Transformer-bart
125
126
127
In-Context Learning (ICL) von Oswald et al.
128
2023); ; AAhn et al.] (2023); ZZhang et al.] (2023); |Schlag et al.] (202T); [Ceen et al. (2024): [aan al. (2025), standard implementations often overlook the potential of the unlabeled portion of the context.
129
130
131
Recent work has begun investigate this intersection, primarily focusing on scaling and data selec- tion. Agarwal et al. (2024) demonstrated that scaling to many-shot prompts yields significant gains and introduced Unsupervised ICL (using inputs only) to reduce dependence on curated labels. Sim- ilarly. CChnn et al. (2025) improved many-shot ICL by influence--eleccing unlabeled examples and pseudo-labeling them with L.M.. From a theoretical perspective, [i et al. (2025) proved that while single-layer linear-attention model cannot exploit unlabeled data, depth or looping constructs can implement polynomial estimators akin to the Expectation-Maxiiizze empirical gains tabular tasks.
132
133
134
135
136
137
(EM) algorithm, showing
138
139
These findings motivate our focus on the fundamental mechanism by which Transformers lever- age unlabeled context. Unlike previous approaches that rely adding pseudo-labeled shots Chen ctt al. (2025); Agarwal et al. (2024), we argue that Transformers can extract structure directly from unlabeled tokens. This relates to the field of spectral geometry and graph- based embeddings. Tra- ditionally, embedding vectors for unlabeled comples-ofter derived from the eigenvectors of the graph Laplacian Coifman & Lafon (2006) have been calculated offline and used as positional em- beddings (PEs) in graph Transformers Kreuzer et al. (2021); Rampasek et al. (2022). While While/Cheng al. (2025) recently showed that Transformers can compute certain PÉs during inference, we are unaware of any prior work that fully integrates this into a semi- cami-supervised ICL framework.
140
141
142
143
144
145
146
147
148
In this work, we bridge this gap by demonstrating that full Transformer extracts geometry- alligned representations in-context. By computing a representation from unlabeled tokens and using subse- quent attention layers to implement a gradient- based learning algorithm, our approach provides an explicit forward-pan mechanism that explains why depth is critical in semi-supervised ICL. This mechanism is supported by cross-domain evidence-ranging from synthetic manifolds to diffusion- generated images and ImageNet features- Seatures-showing that the Transformer learns representations consistent with the underlying geometry the data.
149
150
151
152
153
154
155
2
PROBLEM SETUP
156
157
Background: In-Context Supervised Learning. For in-context supervised learning von Os- wald et al. (2023); Cheng et al. (2024), the Transformer is presented with contextual data C (x y( )...., (x(m)) ((m) , where z(0) E Rdd and ( is label. The query is (mm-1), The al. (2024) was examination of the Transformer infer (m+))) based on the contextual data. In this setting, it was assumed that there is an un-
158
159
goal of prior work von Oswald al. (023):(Cheme
160
161
derlying function fxx responsible for the relationship between any input (+) and output (f) In
3

---

Under review as a conference paper at ICLR 2026
162
most prior work of this type, investigation was performed on the potential for Transformers to infer context-dependent (z) on the forward pass, and use it to estimate B(y(m+1)).
163
164
165 In most prior work von Oswald et al. (2023); Cheng et al. (2024) the outcomes y were real. Recently this has been extended to categorical y(") E {1,.... . C} Wang et al. (2025). and we will
166
167
consider that setting here. Given f(x), the relationship between x:1)) and y is modeled via the
168
softmax (like the output of language models Vaswani et al. (2017))
169
[w). yo0 f(zl)()))))
exp
170
TP (()0/a()))
(1)
171
Σ exp (w.. (x((t))
172
For each category (label type) e there is a learned embedding vector we E Rd and the function f() has d'-dimensional vector output. In-context learning (ICL) in this setting was first developed in Wang et al. (2025), and for reader convenience details are provided in Appendix B
173
174
175
176 A limitation of this approach is that the context that is exploited is only manifested in the form of the observed labeled data, and each query is analyzed in isolation. A contribution of this paper concerns extending this concept
177
remi-supervised ICL setting, discussed next.
178
179
In-context semi-supervised learning, Let n denote the total number of points, and let m < denote the number of labeled points. The input to the Transformer thus
180
181
= (a (1) )...,... (m)) yy ymm)) }
(m++1)
182
183
labeled points
unlabeled points
184
The goal predict the expected values of y (m++1) ..... .......()) (m), with this prediction placed within the context of all data, all n observed covariates and the m labels.
185
186
187
Define the matrix X (z(1)) ..... (())) E Rdx= and the function <(X) that outputs a matrix (の(1) ..... )),, with (i(()) feature vector for s((0), Importantly, ((X) function of all
188
189
observed covariates, and therefore features (( are not only dependent on 26()) but also account for
190
context (e.g., manifold information) provided by the columns of X. The model in (I) generalizes to
191
192
(()(0x)) exp (ppwooo (upoooo((o))) E exp (w. (w)T)))))
(3)
193
194
Note that the observation ( associated with any sample is now dependent on all covariates, via the function of (X). Further, the ICL prediction we develop in Section 3.2 is based on all observed labels
195
196
y ..... .y (m)) thus, cumulatively, the setup will utilize all contextual data, labeled and unlabeled.
197 2.1 IN-CONTEXT SEMI-SUPERVISED LOsS
198
We will define two Transformers, TFrep and TF Tuxup. which are respectively responsible for inferring @ (representation learning), and subsequently inferring f. Architectural details of TF and TF sup are discussed in Section below. Let {Orep: Osup) denote parameters for {TFreps TF sup) Let
199
200
201
= Orep Ouup {we)er lemI..... denote the union of all parameters. Then given a context 2 of encompassing labeled and unlabeled samples, the in-context semi sumii-speeeiied cross-entropy loss is:
202
203
exp (1
204
L(0:C) =
1
Σ log 11
(4)
205
DC exp [ {(((x()))
m
206
207
When training, this loss is fit to the n - m unlabeled samples (for which we have labels when train- ing). The training objective of the Transformer is thus ming L(0), where L(0) = Ee [L; C)), i.c., an average over the available training contextual datasets. Note that the optimization is performed wit the parameters f, not directly on the features (f()) and function f((()), which are manifested on the Transformer forward pass.
208
209
210
211
212
3
TRANSFORMER CONSTRUCTION
213
3.1
TRANSFORMER FOR IN-CONTEXT REPRESENTATION LEARNING
214
Given C. We define the affinity matrix A Rax with RBF bandwidth h as Aij
215
exp(-l|aa(0) ,(0))/((h)), and Ar; 0. Let D be the diagonal matrix defined as Dz.
4

---

Under review as a conference paper at ICLR 2026
216
Σj=1 The discrete Laplacian L := D-A, A, and its right normalized version 2 Inxn AD- OperatorCooffma
217
are approximations to the Laplace- Benttemmi operator CCoimann & Lafon (2006).
218
219
common approach to manifold learning semi-supervised learning based on the following two- step procedure (Belkin & Niyogi 2008 2006 [Begiio et al. 2013: Hein et al. 2205 Hein & Maier, 2006]: (I) approximate the Laplace-Belrrami Beltrami operator over the manifold, by the discrete Laplacian
220
221
matrix L; and (II) compute a representation ((x!()) of each input token z²(*) based on L. Popular examples include the diffusion map Coifman & Lafon (006)) and Laplacian Eigenmaps.
222
223
224
The design of our first Transformer module Teep motivated by the above two-step approach. We will further decompose Treep into two sub-modules: TF (for forming a discrete Laplacican as in (I) and TF (for computing the Eigenmap as in (II).. We prove by construction of TFrep = TF that they compute the Eigenmap of Lt, given x(1), ,...,. (()) inputs. In the following, we will consider two variants of standard attention, denoted by Attininnaar and Atim,by. These two variants replace the standard softmax activation by the linear and RBF functions respectively; their detailed definitions, along with other implementation details, are presented in Appendix A
225
226
227
228
229
230
231
Transformer For Computing Discrete Laplacian.
232 Let X E Rdxa denote the matrix whose t column is the raw coordinates of the jth token (0)) Let Zo E R(d+a))x denote the concatenation of X 0nxn. i.e. Zo = [X T O.xx]. This is equivalent to augmenting each (6) with Og- In Lemma I provided in Appendix A we show that there exists a simple Transformer TFC, which can compute the discrete Laplacian L, for any bandwidth h. Con- cretely, let TZ B(d+a)xn denote the output of the I-layer Transformer from Lemma 1. Then [TF |dddn = Lx. Notably, TF has a single-head, a single layer, and all its parameter matrices are zero everywhere except for diagonal block of (scaled) identity. Intuitively, such simple con- struction exists because the Attn,sf used in TF exactly computes the exp(-I1z<0) - x z06) (2())
233
234
235
236
237
238
239
240
weights in Lx In practice, however, we use multi-layer, multi-hean Transformer for TF for two reasons: first, multiple heads us to compute a combination of different Laplacians Lr. each with a different RBF bandwidth h; second, the higher expressivity that comes with more layers enables the TF compute potentially more complex functions during end-to-end training.
241
242
243
244
245
Transformer for computing Eigenmap.
246 Let Ż TTF (Zo)]d:4+> E Roxn denote the output of preceding Transformer stage. Given Ž as input, the next stage TFº takes as input 2 = [2 Oaxa; Laxa]- The architecture is modification of the construction in Cheng et al. (2024) for computing eigenvectors of graph Laplacians. Lemma 2 provided in Appendix we show that there exists a Transformer TF that takes in Zo as defined above, and outputs the Laplacian Eigenmap for tokens 1...n. Concretely, let the output of TF be
247
248
249
250
251
(TF) -Z .Zzue T .Zgut ].
252
253
where Zqutz 12xn Zo _= Zout E kxxx Lemma 2 shows that Zgnt ss [u]...Uk] where v is the 7th smallest eigenvector of E in the input to TF The 7th column Z constitutes an eigenmap of the ith token. Henceforth, we use (60) to denote the 7th column of 7 The parameter configuration of TF$ is once again simple; its Wºº, W WV matrices are all all diagonal. A key difference from TF is that TF uses the linear attention Attin/inear-. This is important because lin- attention is well-suited to implementing power- power-method-ie method- like algorithm that efficiently computes the eigenvectors U... We defer details to the proof of Lemma 2 to Appendix A.2
254
255
256
257
258
259
260
261
3.2
TRANSFORMER FOR IN-CONTEXT SUPERVISED LEARNING
262
From the above discussion, for input covariate (0),, the feature-repressnnaatio stage of our model yields associated features d()), which depend on all covariates (1),,, .n()) We now discuss how to utilize these features in the subsequent ICL stage with categorical observations.
263
264
265
Our derivation is based on the assumption that (()) resides in reproducing kernel Hilbert space (RKHS) Schölkopf & Smola (2002). ). but the setup extends to voftmax-based attention kernels as well Wang et al. (2025); Cheng et al. 22024). From the RKHS perspective, let f()) = (o))), with 268
266
267
269 (() a fixed mapping of features @ to a Hilbert space, and the parameters acting in that space are associated with matrix A.
5

---

Under review as a conference paper at ICLR 2026
270
Assuming have labeled samples in given contextual dataset, indexed = 1,...,m, m, the cross-entropy cost function for inferring the parameters A E F may be expressed as
271
272
273
explwif (Ab(())))
((A) = log
(5)
274
explw? (Ab()())
275
We emphasize that this loss is minimized anew for each contextual dataset, and it will be imple- mented in the Transformer forward pass, for Transformer Tus Performing gradient descent (GD)
276
277
wrt parameters A, and using k(δ(1 x00))= = +( ((((()) one yields the functional update rule
278
for = I...., n (all samples) (see Appendix B
279
280
1 1/0 + A
Af
Σ
m
Byy E(wlf( m(O(1) .001) (61)
(6)
281
282
283
expl explw" f6
284
B(wl (0)
~ MLP)((Ee)
c' exp [xppww?F
285
286
with a the GD learning rate. Importantly, in (6) the labeled data are employed within the sum over j = l...., m, and the relationship to the unlabeled data for i = + 1...., n is handled by the kernel k( ((0)). Hence, we see that the representation-leamiigg stage of the Transformer, discussed above, has the task of designing features gن(1) such that the kernel k(d)(t) ((1)) captures the appropriate interrelationship between the data samples for predicting associated labels.
287
288
289
290
291
The form of the update in (6) is the same that considered in previous Transformer ICL research von Oswald et al. (2023); Ahn et al. (2023); ; Zhang et al. (2023); Schlag et al. (202I): Cheng et al.
292
293
(024); Wang et al. (2025), ) and it naturally aligns with self-attention layer (which considered
294
veal-valued observations).
295
Once the vector f!() is updated to f = f( + Afll the expectation in 7 must be updated.
296
Expectation ? may be viewed as a nonlinear function of (now) ,(0) Therefore, the attention layer
297
298
(((()) is followed by an MLP layer, represented in 77) by (L (+1): with parameters 7. The MLP parameters 7 learned along with all other parameters of the Transformer. Details of the MLP
299
300
layer and the other elements of this ICL Transformer are provided in Appendix B
301
302
4
EXPERIMENTS
303
304
This section describes the benchmarks we devised to study the semi-supervised, in-context learning capabilities of our end-to-end Transformer. After presenting the baseline comparative benchmark models, and the dlata-generation pipelines, we discuss the evaluations we report. For all experiments, vary the percentage of labeled data among the fixed erwise, all results that employ ICL for classification use a one layer ICL Transformer (as in Section and utilize the RBF kernel as the core component of the GD ICL head. Further implementation details required for exact replication are collected in Appendices A and B
305
306
307
100 total samples. Unless stated oth-
308
309
3.2
310
311
Our evaluation focuses on three key aspects: (1) the ability of the Transformer learn meaning- ful representations from unlabeled data, (2) the effectiveness of the end-to-end approach compared to traditional methods, (3) the generalization capabilities across different manifolds and data dis- tributions, and (4) mechanistic understanding of in-context suneruupee learning (IC-SSL) on real-world data through an ImageNet100 experiment.
312
313
314
315
316
317
4.1
MODELS UNDER EVALUATION:
318
We consider the following models under the naming notation of (Data+Model), reflective of the form of input to the classifier (Data), and the model used to classify the unlabled samples (Model).
319
320
321
ORIG+E2E-ICL (OUR MODEL) The raw original coordinate matrix is fed directly into our end- to-end Transformer. See Section 3 and Appendix A for a full architectural breakdown. Crucially, all three constituent modules the Laplacian predictor, the eigenvector extractor, and the in-context GD head-are are trained jointly minimizing their combined loss.
322
323
6

---

Under review as a conference paper at ICLR 2026
324
EIG+ICL The ICL head uses ground-truth bottom-k Laplacian eigenvectors input, isolating the performance contribution of representation learning.
325
326
EIG+LR Standard fz-regularized logistic regression on ground-truth eigenvector features.
327
328
ORIG+RBF-LR Kernel logistic regression with Gaussian RBF kernel on the original coordinates.
329
ORIG+ICI The in-context learning head is applied directly on the original coordinates, without
330
Laplacian eigenvector features.
331
332
333
4.2
IMAGENETION SEMI-SUPERVISED ICSSL ON REAL DATA
334
335
Experimental Design. We evaluate on ImageNer100, a 100-class subset of ILSVRRC-202 Rus-
336
sakovsky et al. (2015) commonly used for efficient representation learning Caron et al. (0018: Chen et al. (2020). Here, the "dataset size" denotes the number of in-context datasets (lasks). Each dataset has 200 batches, each batch sampling two random classes (50 images per class, 100 total) with re- placement, and VGG-29 features (Sioonyan & Zisserman (2014) are used as inputs. The batch size fixed at 200, with 3% labeled samples (a 39% variant is reported in Appendix [14]. For compari- son, we use a scale-matched Transformer baseline (2-layer encoder, hidden dimension 128, ~1.5M parameters), where class labels are projected into the same embedding space as the inputs and un- labeled samples are assigned a special unknown token, following the MetalCL formulation Min ctt al. (2022) to ensure fairness. Ground- truth EIG embeddings are obtained via Laplacian eigen- decomposition of the affinity graph as a spectral reference.
337
338
339
340
341
342
343
344
345
We compare three systems: (i) our constructed two-stage model, (ii) a scale-matched 2-layer base- line Transformer trained end-to-end on the same episodes, and (iii) an ICL head raw VGG fea- tures (Orig+ICL). We report standard ICL accuracy and a simple separation score Caron et al.]
346
347
348
2018): Oord al. (2018); Padhan & Sahu (2024) (intra-class similarity minus inter-class similar- ity), which serves as a coarse proxy for representation quality (Figure 3}.
349
350
Accuracy and separation. Our model attains higher accuracy than the baseline Transformer, with the largest gaps appearing at small training-set sizes; both models outperform EIG+ICL, indicating that in-context representation learning crucial and that raw VGG features have poor separation. Separation and accuracy are correlated across data regimes: our model exhibits higher separation even with little data, and the accuracy advantage mirrors this difference (Figure (3)
351
352
353
354
355
Phase transition in the baseline Transformer. The baseline Transformer performs poorly when the training set smaller than 1000 and undergoes a sharp phase transition around 1200, after which accuracy rises quickly. Strikingly, this jump aligns exactly with an abrupt increase in separa- tion score, linking the performance transition to a change in representational structure (Figure
356
357
358
359
Representational alignment. To connect these trends to geometry, use mutual ANN (mNN) aligmment: for each sample, take its top-k neighbors in two embeddings and compute the fraction of overlap between the two neighbor sets; the mNN score is the dataset-average (higher more similar neigh- borhood structure) [suchiya
Table 1: Aligmment metrics at labeled ratio 3% and dataset size 5000, comparing three embeddings: (Our model), (Transformer baseline). and (EIG). We report three alignment metries (higher is better) introduced in the Platonic Representa- tion Hypothesis Tsuchiya et al. 22223)
360
361
362
363
364
365
Source Source-Target Pair
mNN 1
Cycle t
LCS 1
366
Our model Transformer bascline EIG Our model EIG Transformer baseline
0.302
0.289
0.193
367
0.304
0.295
0.195
368
0.269
0.262
0.177
369
370 et al.) (2023). We compare three embeddings- ours, the baseline Transformer, and EIG (top-4 Laplacian eigenvectors from the affinity graph). Alignment rises with training data. Our model starts highly aligned to EIG (spectral inductive bias), while the bascline progressively aligns with both; with enough data, baseline-ours can exceed ours-EIG, and baseline EIG improves but generally trails ours-EIG at moderate scales. Additional metrics (Cycle, LCS) show the same pattern; by 5000 episodes all three pairs exhibit high alignment (Table I.
371
372
373
374
375
376
Figure 3 and Table I show two-stage model is competitive on ImageNet- based artificial IC-SSL tasks, using real image features, provides mechanistic insight via a phase transition in the baseline, and serves as a reference model for how end-to-end Transformers acquire PF-like structure.
377
7

---

Under review as a conference paper at ICLR 2026
378
3% Labeled Data KLL Accuney
3% Labeled Dato Separatien Scom
3% Lakeled Data INN Compartions
379
380
381
2.
0.20
382
0.24
383
1.00
384
0.0
385
386
016
033 e022-44 va Tranrfonna Dnniiz
ICL
016
BID 0ED0 (DEXL
387
Spa
Datienet Size
Detoort Stae
388
Figure 3: In-context learning on ImageNet100 with 3% labels; "dataset size" number of con- structed in-context tasks. Columns: left-ICL accuracy; middle separation (intra- minus inter- class similarity); right-mNN neighborhhood-overlap similarity. Methods: Orig+E2E-ICI (red), Transformer baseline (blue), Orig+ICI. (green; VGG-29 features).
389
390
391
392
393
Data efficiency and spectral perspective. In a representative low-label regime (3% labels, N=5000), our model improves both accuracy and separation more rapidly than the baseline Trans- former, reflecting faster emergence of intra-class compactness and inter-class separation (Figure While both models eventually reach comparable performance at larger data scales, ours does so with substantially fewer samples. From spectral viewpoint, both representations appear approximate the same underlying geometry: alignment to the "EIG" (top Laplacian eigenvectors) strengthens with more data, with our model starting closer in low-data settings due its inductive bias, and the baseline progressively converging toward both our model and the spectral reference (Table
394
395
396
397
398
399
400
401
402
4.3
FAMILIES OF MANIFOLDS AND THEIR GEOMETRY
403
Let gM,g) denote smooth, connected Riemannian manifold with intrinsic distance dm- We
404
use five such manifolds, visualized in Figure 2 Each manifold admits closed-form geodesic and parametric map Φb U C RTT R For example, for the sphere S2, &ga (0, Φ) (sin 6 cos 6, sin @ sin o, cos f), and its geodesic is dga (p.g = arccos (p, 4)). Details of their parametric maps and geodesic derivations are in Appendix [E] Many of these manifolds have had
405
406
407
408
important applications in scientific machine learning (De Bortoli et al. 2022 Jing et al. 2022).
409
In our experiments, we sample n = 100 points uniformly from the manifold in IR3, followed by random rotations + translations (see Appendix E to generate family of perturbed manifolds. For each set, we select random center point and assign binary labels based on geodesic distance: points within certain manifold-distance form the positive class, while others form the negative class.
410
411
412
413
414
4.3.1
RESULTS ON INDIVIDUAL MANIFOLDS
415
416
In-distribution performance. For each elementary manifold we train and evaluate every model on data drawn from that same manifold-e.g. the "cylinder" curves in Figure (al are obtained by training exclusively on cylinder samples and measuring accuracy on a disjoint cylinder test split. The figure shows the mean + one standard deviation over three random initialisations, where each run uses an 90 %/10 % train-test split, identical label budgets, and fresh random draws of the unlabeled context points. The same protocol applied to the sphere, cone, torus and Swiss-roll manifolds yields curves with the same qualitative ordering; those results are deferred to Appendix F3
417
418
419
420
421
422
423
For the cylinder our ORIG+E2E-ICH model dominates all baselines across the entire label-budget spectrum, surpassing the strongest competitor (EIG+ICI.) by roughly 5-7 percentage points and seaching~990 % accuracy once 2 25% of the points are labelled. The advantage is pronounced the low-label regime (3% < p 15%), confirming that end-to-end contextual representation learning is particularly valuable when supervision is searce. Notably, direct classification on raw coordinates FORIG+RBF-LE performs markedly worse than any manifold-awar alternative, highlighting the benefit of exploiting geometric structure rather than relying on purely supervised fits.
424
425
426
427
428
429
430
Kernel choice in the ICL head. Replacing the RBF kernel with linear kernel changes the per- formance and occasionally the ordering, but the dominance of the end-to-end approach remains; detailed ablations appear in Appendix F2
431
8

---

Under review as a conference paper at ICLR 2026
432
10
433
E5
31
EXC-LR
434
CRIG-REFAR
435
64
436
437
64
DC-ICL
36
438
439
45
ESD1ER
63
02G--25-00
ORI U1
05
440
Labk Peceriace 19
Laaied Dors Recendage
Labeldd Cara Pareerage 151
(a) ID (cylinder)
(b) OOD (cylinder)
(c) ID (product manifold)
441
Figure 4: Accuracy vs. labeled-sample ratio manifold benchmarks. (a) In-distribution: train/test cylin- der. (b) OOD (cylinder): blue BEIG+EEEICLL and green ORIG+ICL are trained on (sphere, cone, torus, Swiss-roll), lested on cylinder; orange is the ID reference (train/test on cylinder). (c) In-distribution (product
442
443
444
manifold): training and lesting on the high-dimensional product manifold (S* x C Conea x SR x T
445
446
447
0.9
448
449
450
0.7
451
452
0105+E26-KI
453
0.5
010
454
455
Percentage (56)
456
Figure 5: Results on image manifolds. Left: zero-shot test accuracy, where blue ORIG+E2E-ICH and green EIG+ICL are trained only four synthetic manifolds {sphere, cone, Swiss-roll, cylinder} and tested image manifolds; red/orange denote ID runs (trained and tested images). Baselines MIG++R/ORIG++REF.
457
458
are omitted for large underperformance. Right: an example manifold obtained by sub-sampling every 5th of n 100 SLERP frames between two random latents; top two rows are label 1, bottom two rows are label 0, illustrating the smooth geodesic manifold structure.
459
460
461
Out-of-distributioo (OOD) evaluation. To test generalizability, we train on four source mani- folds- sphere, cone, torus, and Swiss-roll- and test on the held-out cylinder. Figure 4b contrasts this OOD setting with an in-distribution (ID) baseline trained and tested cylinder (Figure 4a Across label budgets, ORIG+E2E-ICL. matches or exceeds its performance and is 3-5 per- centage points higher in this case; other OOD trials (see Appendix F3 generally approach but do not surpass their ID counterparts. These results indicate that representations learned from diverse geometries transfer effectively to unseen manifolds. The logistic-regressii ORIG+RBF-LR) are refit on cylinder at test time and thus act as ID references rather than true OOD comparators; EIG+ICL underperforms the end-to-end model.
462
463
464
465
466
467
baselines (EIG+LR,
468
469
470
4.3.2
PRODUCT MANIFOLD RESULTS
471
472
The manifolds descrihed previously live 3R& and possess relatively simple decision boundaries. To further evaluate the representation learner, we form Cartesian product manifolds Pk MM x... Mr, where each factor M₁ is drawn from the five families of manifolds we individually tested against. The intrinsic dimension therefore grows additively with K, and likewise the complexity of the geodesic ball that defines the label boundary increases as well. As a consequence of our Theorem (see Appendix E.2.I. the geodesic distance on Pæ remains available in closed form, allowing us to generate exact labels in the exact same way as before even in the presence of the random scale, rotation, and permutation applied independently to each factor. Details on these transformations, the decision thresholds, and the construction provided in Appendix
473
474
475
476
477
478
479
480
481
Figure 4c reports accuracy on a representative 5-factor product manifold of all our individual manifold families. As expected, all the methods degrade as geometry becomes more intricate, yet the ORIG+ BRIG+E2E-ICL. Transformer retains an 8-10% margin over the strongest baseline (EIG+ICL) across the whole label spectrum. These results support that the end-to-end learner continues to extract useful Laplacian-brie based features and to propagate labels effectively even when the decision boundary is more complex and in space whose dimensionality exceeds that of any single chart.
482
483
484
485

---

Under review as a conference paper at ICLR 2026
486
4.4
IMAGE MANIFOLD EXPERIMENTS
487
488
To test whether the representation-lcarnigg routine transfers to higher dimensional, natural-looking data, build image manifold with Stable Diffusion v1.5 Podell al. (2023). Concretely, draw two latent noise vectors, treat them as endpoints, and generate a 100-step spherical linear
489
490
491
interpolation in latent space. Each interpolated latent is dichded-without classifier guidance-into a 500 x 500 RGB image. Repeating this over 500 prompt seeds yields 500 diverse sub-manifolds of images (each composed of 100 images). The intrinsic geometry of each sub-manifoll is inherited from the latent interpolation; full generation and labeling details appear in Appendix E3
492
493
494
495
The results demonstrate exceptional zero-shot OOD transfer capabilities: despite having been trained exclusively on simple synthetic manifolds from before. with prior exposure to the image data, our model achieves roughly 77% accuracy using only 15% labeled examples; and remarkably sustains performance above 62% accuracy even at extremely spare labeling (3/100 labeled images). This consistently surpasses baselines methods by margins of 20 percentages points across nearly model trained directly on the image manifold
496
497
498
499
the entire labeled data regime. For reference, achieves nearly 100% accuracy, far surpassing the other methods.
500
501
502
Perhaps more striking than the headline accuracies is the data efficiency: three labels guide the Transformer to a correctness level that baselines cannot match even after seeing 10x more labels. This suggests the network implementing efficient modality-agnostic algorithm for constructing a geometry-awant representation from raw coordinates at inference time. The sparse supervision is propagated through this learned geometry, leading to impressive label economy.
503
504
505
506
507
LIMITATIONS AND CONCLUSION
508
509
We introduced semi-supervised in-context learning with a single end-to-end Transformer that first builds ecometry-awant representation from all covariates and then predicts labels in- context-withore parameter updates. Across synthetic manifolds, product manifolds, and image manifolds, our model consistently outperforms strong baselines and shows robust out-of-distribution transfer. We use our Transformer model as a reference for studying the IC-SSL. capabiliie of a stan- dard Transformer. We observe correlations in the representations learned by model and by the standard Transformer, suggesting that representation learning may be an important direction in un- derstanding the Transformer's IC-SSL capabilities. While the construction is applicable in principle to sigh-dimensional data, our experiments use relatively small episode-style datasets, reflecting the novelty of the IC-SSL setting and the absence of established benchmarks.
510
511
512
513
514
515
516
517
518
519
520
ACKNOWLEDGEMENTS
521
522
We thank Zeyu Michael Li for his crucial contributions the image manifold experiments. These contributions warranted authorship: however, he was inadvertently omitted from the author list at the time of submission and could not be added retroactively under ICLR policy. Zeyu listed as co-author on the Xiv version of this paper.
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
10

---

Under review as a conference paper at ICLR 2026
540
7
ETHICS STATEMENT
541
542
This work uses (i) synthetic manifolds and product manifolds we generate procedurally, (ii) images produced by Stable Diffusion v1.5 without human subjects, and (iii) publicly available ImageNet100 features under their licenses. No personally identifiable information
543
544
sensitive attributes are col- lected or inferred. Potential risks are limited standard concerns in semi-supervised learning (e.g.. amplification of dataset bias); we discourage deployment without bias and robustness assessment on target data. No other foresecable dual-use concerns were identified.
545
546
547
548
549
8
REPRODUCIBILITY STATEMENT
550
551
An anonymized code archive, here is provided with scripts and configs to regenerate figures and tables. Data generation pipelines are specified for: classic manifolds and product manifolds
552
553
Theorem/derriation in Appendix E.2.I.. the Stable-Diffusion image manifolds (Ap- pendix E.3 and the ImageNet100 episodic setup (e.ction4.2)) Model architectures, loss weighting,
(Appendix E
554
555
optimization settings, and training schedules are listed in Section and Appendix [] derivations and construction details for the representation module and ICI. head appear in Appendices [B] Baselines and their hyperparameters seeds and report mean + sd over three runs; the repository provides code needed to reproduce our work and further details.
556
557
documented in Section F.7 and Appendix [D] We fix random
558
559
560
561
REFERENCES
562
563
Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. Many-shot in-context learning. Advances in Neural Information Processing Systems, 37:76930-76966, 2024.
564
565
566
K. Ahn, X. Cheng, H. Daneshmand, and S. Sra. Transformers learn to implement preconditioned gradient descent for in-context learning, 2023.
567
568
569
Mikhail Belkin and Partha Niyogi. Convergence of laplacian eigenmaps. Advances in neural infor- mation processing systems, 19, 2006.
570
571
Mikhail Belkin and Partha Niyogi. Towards a theoretical foundation for laplacian -base manifold methods. Journal of Computer and System Sciences, 74(8):1299-1388, 2008
572
573
574
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 1788)11788-1828, 2013.
575
576
577
T.B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D.M. Ziegler, J. Wu, CL. Winter, C. Hesse, M. Chen, Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language mod- els are few-shot learners. CoRR, abs/2005. 14165, 2020. URL https: http//aaxiv.v// org/ abs/ 7005.14165
578
579
580
581
582
583
584
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 132-149, 2018.
585
586
587
O. Chapelle, B. Scholkopf, and A. Zien. Supi-Super Learning. MIT Press, 2010.
588
589
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pp. 1597-1607,
590
591
592
593
Zihan Chen, Song Wang, Zhen Tan, Jundong Li, and Cong Shen. Maple: Many-shot adaptive pseudo-labeling for in-context learning. arXiv preprint 2rXXv:2505.16225, 2025.
11

---

Under review as a conference paper at ICLR 2026
594 X. Cheng, Y. Chen, and S. Sra. Transformers implement functional gradient descent to learn non- linear functions in context. arXiv.2312.06528, 2024.
595
596
597 Xiang Cheng, Lawrence Carin, and Suvrit Sra. Graph transformers dream of electric flow. In
598 The Thirteenth International Conference on Learning Representations, 2025. URL https:
//openrevi nwt/ wttts//ooommiddddrmmm www0rmmm22dd00000000003000
599
600
Ronald R Coifman and Stéphane Lafon. Diffusion maps. Applied and computational harmonic analysis, 21(1):5-30, 2006.
601
602
603
Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornoon, Yee Whye Teh, and
604
Arnaud Doucet. Riemannian score- score-based generative modelling. Advances in neural information processing systems, 35:2406-2422, 2022.
605
606
DeepSeck, Deepseek-Re RI: Incentivizing reasoning capability in LLMs via reinforcement learning. 1IX1::2501.12948v.. 2025.
607
608
609
Jacob Devlin, Ming-Wet Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. 10X00./810.04805, 2019.
610
611
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale, 2021.
612
613
614
615
616
Marco Fumero, Luca Cosmo, Simone Melzi, and Emanuele Rodolà. Learning disentangled repre- sentations via product manifold projection. International conference on machine learning, pp- PMLR, 2021.
617
618
619
Matthias Hein and Markus Maier. Manifold denoising. Advances neural information processing systems, 19, 2006.
620
621
622
Matthias Hein, Jean- Jean-Yves Audibert, and Ulrike Von Luxburg. From graphs manifolds-weak and strong pointwise consistency of graph laplacians. In International Conference on Computational Learning Theory, pp. 470-485. Springer, 2005.
623
624
625
Jian Jiang, Lu Ke, Long Chen, Bozheng Dou, Yueying Zhu, Jie Liu, Bengong Zhang, Tianshou Zhou, and Kei We. Transformer technology in molecular science, 2024.
626
627
628
Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional dif- fusion for molecular conformer generation. Advances in neural information processing systems, 35:24240-24253, 2022.
629
630
631
Devin Kreuzer, Dominique Beaini, William L. Hamilton, and Vincent Létourneau. Rethinking graph transformers with spectral attention. wwxiv:2/00.03393333 2021.
632
633
634
Yingcong Li, Xiangyu Chang, Muti Kara, Xiaofeng Liu, Amit Roy-Chowdhury, and Samet Oy- mak. When and how unlabeled data provably improve in-context learning. arXiv preprint 25X2522506.5339, 2025.
635
636
637
Sewon Min, Mike Lewis. Luke Zettlemoyer, and Hannanch Hajishirzi. Metaicl: context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pp. 2791-2809, 2022.
Learning to learn in
638
639
640
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. In Advances in Neural Information Processing Systems, volume 31, 2018.
641
642
643
Yassine Ouali, Céline Hudelot, and Myriam Tami. An overview of deep semi-supervised learning. arXiv.2006.0222 05278, 2020.
644
645
646
Subhranshu Padhan and Banshidhar Sahu. Introducing the cosine clustering index (cci): A balanced approach to evaluating deep clustering. SN Computer Science, 5(4):1-15, 2024. doi: 10.1007/
647
222799002-02270-11
12

---

Under review as a conference paper at ICLR 2026
648
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
649
650
651
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
652
653
654
Ladislav Rampasek, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. Neural Information Processing Sys- tems, 2022.
655
656
657
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695,,0 June 2022.
658
659
660
661
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211 252, 2015.
662
663
664
665 I. Schlag, K. Irie, and J. Schmidhuber. Linear transformers are secretly fast weight programmers. International Conference on Machine Learning, 2021.
666
667
668 B. Schölkopf and A.J. Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2002.
669
670
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
671
672
673
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. arXiv:2302.1397. 2023.
674
675
676
677
Takashi Tsuchiya, Jürgen Schmidhuber, and Cengiz Pehlevan. The platonic representation hypoth- esis. arXiv preprint arXiv:2310.02224, 2023.
678
679
J.E. van Engelen and H.H. Hoos. A survey on semi supervupe learning, 2020.
680
681 A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and Polo- sukhin. Attention is all you need. arXiv. 002117060037667 2017.
682
683
684 J. von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and
685 M. Vladymyrov. Transformers learn in-context by gradient descent. arXiv.2212.07677 2023.
686 A. Wang, W. Convertino, X. Cheng, R. Henao, and L. Carin. On understanding attention- based in-context learning for categorical data, (rXi::2405.1724822. 2025.
687
688
689 R. Zhang, S. Frei, and P.L. Bartlett. Trained transformers learn linear models in-context. arXiv:2306.09927 2023.
690
691
Sharon Zhang, Amit Moscovich, and Amit Singer. Product manifold learning. In International Conference on Artificial Intelligence and Statistics, pp. 3241-3249. PMLR, 2021.
692
693
694
695
696
697
698
699
700
701
13

---

Under review as a conference paper at ICLR 2026
702
APPENDIX
703
704
A TRANSFORMER FOR REPRESENTATION LEARNING (LEMMAS AND 2)
705
706
Given samples 2(2), ,...,, (() 2()) E Rd, let Z E RR denote the matrix whose 7th column z(() n(U, V) denote non-linear transformation. We define the k-attention module
Let
707
708
709
Attn, (Z:W%, W W&) W Zx
(8)
710
In standard softmax-activated attention, [s(UU V)l. exp (UwTVo) Vot) /i-n exp (U1) v())
711
712
For the purpose of this section, we will consider two other types of activations:
713
[Miinea: (U, Vl = U VG)
714
715
exp 1-100--0011
716
[Nist (U, V
(9)
k= exp (-11((k) VO
717
718
where (()) denotes the 5th column of U. Throughout this section, will consider the Transformer defined by the following update:
719
720
H
721
Ze+1 = (1 W +
Σ
Aunn W WO Ede W
(10)
722
1=1
723
where W is linear transformation, l is the layer number, H is the total number of heads, and (W w ed W.J. } are the value, query and key matrices, respectively.
724
725
726
727
A.1
TRANSFORMER FOR FORMING NORMALIZED GRAPH LAPLACIAN (LEMMA 1)
728
729
The Transformer that we use for learning a Laplacian-biig based representation is based on the one proposed in Section 14.2 of Cheng et al. (2025) with one important modification replacing the linear attention by RBF attention enables the same architecture to efficiently compute the Laplacian of the neighoss graph in-context. See (9) above for definitions of Attn.
730
731
732
733
Let (1) .().,,(()) denote the coordinate representation of token i. Let X E Rg denote the matrix whose jth column is given by (() = _(()) We will begin by forming the augmented token 2 (x; 0,] E ud++ Let Zo E IR xd denote the matrix whose columns are ( The input to the Transformer is thus Z [x Onxz]. We have used O,, and 0,xn denote all-zeros vectors and matrices, of dimensions indicated by the subscripts. In practice, we will also learn an additional initialization parameter matrix M E [Raxn, so that
734
735
736
737
738
739
z = - .M].
740
741
Adding an initialization M ensures that the architecture is identical to the Eigenmap Transformer in the subsequent section (except for choice of nonlinear activation), at no loss of expressivity.
742
743
744 Let Ze Rd++ denote the output of the 8h Transformer layer, as defined in (10), with Attn We impose the following constraints on the parameter matrices of the Transformer. To reduce no- tation, we focus the single-head setting; the case of multiple heads simply repeats the following construction parallel:
745
746
747
748
WY
aglaxd Onxd
Oaxn Ag
Wo
belaxd Onxd
Odxn Be
W.K
Telladd 0axn
749
Onxe
Ce
750
751
WS
seldxd
Onxd
Odxn Se
(11)
752
753
where AG, Be, Ce, St Raxn.
754
The total set Transformer parameters
755
(M) U (st, St, ag, Ae, be, Be, Ct Ce
14

---

Under review as a conference paper at ICLR 2026
756
Remark 1. In construction Lemma the matrices A. d'elaxn: Bt bgInxns Ce
757
C,Inxn, St = syInxn, where de. bg, dg, sy are scalar multiples of the identity matrix. In our im- plementation, deg 2/ S/ are learnable scalar parameters, and thus the parameter count does not
758
759
scale with n.
760
761
We show in the following lemma that a I-layer Transformer with computes the (RBF) Lapla- cian with bandwidth o². For ease of reference, we redefine several matrices of interest below:
762
763
The o babdddhid RBF adjacency matrix A E R=x= defined by
764
765
1 1lo00 2 Aij exp (-
(12)
766
767
768
2. The diagonal matrix D E Raxa is defined with diagonal elements Dyj
==
Σ exp (- - 2,00
769
770
3. The random walk matrix is defined AD-1 3 Rax" We verify that
771
exp 1/2 (1) x
772
[AD-
(13)
Σg exp [|x(()) -x05) 12)
773
774
4. The Laplacian L, symmetric- Laplacian and nightnnommlized Laplacian Σ are defined follows:
775
776
777
D= A, - - DD1122AD 1/2 == - AD-1
(14)
778
Lemma 1 (Construction of Transformer for Computing Laplacian). Les C Ruxn denote the right-normalized Laplacian with bandwidth o² as defined in (14) above. Consider the single-head
779
780
Transformer, parameterized as in [IT. with the following parameters: St Sg Onxn: 0, be = 1/0, At -luxn Be Ce = Onxn- Let [11ddn+1 E R= denote the last n. rows of
781
782 Zz Let M Laxa Then [Zi]detnn C.
783
784
Proof of Lemma For convenience, let z ddx= denote the first d rows of and Z denote the last rows of Zg. Let Ke, E R=** denote the self-similarity matrix. We begin by verifying that
785
786
for 0, the self-similarity matrix simplifies to
787
788
Z - - -
cxp
789
WK Ze, W Zz) )
RibE
790
1-Ilw,22 W 2o|0o
Σx exp
791
792
bece (10(00 ) xل.111
exp
793
794
Σ3 exp
becellz((h) (1) 112)
795
796
exp
1/2 |00(1) - -z(0)
797
798
Σx exp
(-12 ||o((k) -x(0) 112)
799
800
From the above, we verify that Kyor (W Zg, W. Zz)
AD-1, which is the random walk matrix,
801
defined in 12 and 13 By definition of M and Zo. we have
802
0 0dxx
803
WV Zo =
I Inxn
804
805
Consequently, for f f=0 0,
806
Atturbe (Zt) = WWY ZENEDE
(W! Ze, W. Ze
807
808
0dxn
0dxn
809
Inxn
AD-1=
-AD-1
15

---

As with the preceding section, the Transformer that we use for computing the Eigenmap is based on the Transformer from in Lemma 9 of Cheng et 2025 with one important modification. Here, algorithm for Under review as a conference paper at ICLR 2026
810
which is exactly equal to Aij. where A is the adjaceney matrix defined in Section 3.1. Finally, by definition of 85 = 0 and S = Inxns we have
811
812
Oaxn AD
X
0ax AD
X
813
Z1 Zo
=
I nx
814
815
where 2 is defined in
14
816
817
A.2
TRANSFORMER FOR COMPUTING EIGENMAP (LEMMA 2)
818
819
820
we retain the use of Attniizear as our goal is to implement block-power-neg computing the Eigenvectors of matrix. See above for definitions of Attn inear-
821
822
823
Let 2b(11) [1((),
(b(n) IR" denote some pre-computed representation of Let V
824
(5()) E Rax denote the matrix whose columns contain representations of each token. We are primarily interested in the setting when is the Laplacian matrix Ê, output by the Trans- former constructed in Lemma however, the construction we present below applies to general
825
826
827
V. This generalization practically important, as it enables the Transformer to learn to compute embeddings not based the RBF Laplacian.
828
829
Let Zo E R (nnn))xn denote the matrix whose columns are (0) The input the Transformer is thus Zo = v _nxx] . In practice, we will also learn an additional initialization matrix M R so that
830
831
832
2 = [D
Onok + M.
833
834
On a high level, the Transformer will perform a block-power-method algorithm using the V matrix compute the bottom l eigenvectors of Zo. and M enables the Transformer to learn a good initial guess of the k eigenvectors.
835
836
837 Let Ze E pd+= denote the output of the 9th Transformer layer, as defined in (10) (single head), but with Attn inear, and an additional normalization between layers:
838
839
Ze+1 (I+ w Ze + Attm (Ze; W. Wº w
840
Ze+1 norma 1 (e(+++))
(15)
841
842
where normal 1 (Z) applies row-wise normalization to Z. We impose the following constraints on the parameter matrices of the Transformer.
843
844
wY-
Uaxaxn ORXn
Onxk A
we
belnxn Oxxn
Onxk
w
Teelnxa Okxn
Onx*
845
B₁
Ce
846
stluxn
Onxk
847
W
0kxn
St
(16)
848
where A,, Br, Sr E Ri Thus the total set of Transformer parameters is
849
850
{M) Use, Se, ue, Ar, be, Be, ct, Cehemt......
851
Remark 2. In our construction of Lemma _ the matrices Ar, Br, Cr, St are x k parameter matrices which do not scale with n.
852
853
Lemma (Construction of Transformer for computing Eigenmap.). Let V E Raxn be some feature map. Let μs R+ denote constant that satisfies μI > V v.. Then there exists a Transformer of the form (1), whose parameter matrices satisfy the constraints of (16). which implements the block- power-iteration algorithm for pl v/ V. Consequently: let Zr. ( denote the output of L-layer Transformer, let th E IRkx denote the last rows of ZL. and let b denote the jth row of
854
856
857
858
4. Then
859
b Uj.
860
where Vj is the jth smallest eigenvector of 1 T V. When V is the Laplacian C, the bottom k eigenvec- tors of Vi V are the as the bottom l right-singulad vectors of L. When E has constant degree 862
861
(i.e. D x Inxah these are also equal to the bottom eigenvectors of the unnormalized Laplacian f. and the conmetric-normaliza Laplacian L.
16

---

Under review as a conference paper at ICLR 2026
864
Remark 3. Before stating the proof. we make number of remarks on how the above result relates
865 to Section 3.1.
866
867
1. Notice that the Laplacian Transformer TF from Lemmal has input/output dimension d + n and the Eigenmap Transformer TF from Lemma 2 has input/output dimension n + k. For consistency, we pad each Transformer accordingly, so that both Transformers have input/oupuu dimension + n n- k. Therefore, the "last l rows of Ze" " as described in this lemma is equivalent to Zgut defined in Section 3.1.
868
869
870
871
872
2. In practice, for both the Laplacian Transformer TFC and the Eigenmap Transformer TF we found it beneficial to use a maltiple parallel heads. One benefit, for instance, is that H heads can simultaneously orthogonalize H columns.
873
874
875
876
3. We loop each layer twice, and the entire Transformer twice in the forward pass, which we find to stabilize training.
877
878
879
Proof. Our proof is primarily based on Lemma 6, Corollary 7, and Lemma 9 of et al. 2025 For ease of reference, we reproduce the necessary details below.
880
881
We will demonstrate that the Transformer is capable of implementing the following block-power- method algorithm (also known as subspace iteration):
882
883
he+1 +be(ul -v TV)
884
885
1z+1 = QR(
(17)
886
The above is initialized at D E R with full column rank. In the above, QR(-): : R + R Rx
887
returns the Q matrix in the QR decomposition of the input matrix. Under the above, the rows of yp converge to the top l: eigenvectors of μI -v V V, which are equivalent to the bottom k eigenvectors of V v by definition of μ.
888
889
890
891
Concretely, let z E B2xa and Z E R denote the first n rows and last & rows of Z respec- tively:
892
893
2 = 2 1
894
895
Intuitively, b in the block power method algorithm in 17 is exactly Z
896
897 Our proof proceeds in two steps. First, we verify that there exists choice of parameters under which the Transformer orthogonalizes the 7th column of 2 with respect to all columns j > i. The construction is identical to Lemma 8 of Cheng et al. (2025).
898
899
900
Let us choose s St = be = 0. Let [Ad,, ;s = -1 and A₁j = 0 for j + i. Let [Bel,, 1 for j < and [Bel,, for j 2 í. Let C2= = B₂, so that B, C2 = Bg. Then we verify that
901
902
903
Ze+1 =7, + W. 7x W.K Wo Zu
904
-2 -
Onxn A Z BeZ
905
(18)
906
907
We verify that the jth row of the last term is
908
909
[az zt BiZz),
Σ
ZZ 2 z
910
911
912
Since each row of 2 has unit norm due to normalization in 15 the above expression is exactly the projection of the tth row onto the subspace spanned by rows i + 1,....ki: hb Consequently, (18) orthogonalizes the tth row of 2 wrt rows + I...., k. Repeating the same construction for each row i, over i layers, we exactly implement the QR decomposition of 2
913
914
915
916
917
Finally, we verify that there exists a construction under which the Transformer implements the first step of (17). This step is simple: first, in all constructions in this proof, we will pick az 8 0.
17

---

Under review as a conference paper at ICLR 2026
918
Consequently, z is never updated, and 2 v for all E. Let now choose az
88 0,
919 by = Cj = 1 and A = Ixkk Be = Ce = 0k and Se= = (p - 1) We verify that
920
Ze+1 Ze + W. ZeZe W w Ze
921
922
v/
Onxn
Onxn
923
7
7 vp
v
+ 1m-1)
924
T
925
2 (nI
v T v)
926
927
The above exactly the first step of 16p. We have thus verified Transformer constructions for implementing both steps of (16)
928
929
930
DERIVATION OF THE GD UPDATE EQUATION FOR ICL WITH B
931
CATEGORICAL OBSERVATIONS
932
933
Our derivation is based on the assumption that f()) resides in a reproducing kernel Hilbert space (RKHS) Schölkopf & Smola (2002), but the setup extends to softmax-basee attention kernels as well Wang et al. (2025). From the RKHS perspective, let f() (σ)(),, with (0)) a fixed mapping of features to a Hilbert space, and the parameters acting in that space are associated with matrix A.
934
935
936
937
Assuming we have m labeled samples, indexed = ....., m, the cross-entropy cost function for in- ferring the parameters E Rdd x (let K represent the output dimension (b)) may expressed
938
939
as
940
941
1
explwig (Ax())()))
((A) =
log
942
Σe explux !o? (Ab(G(())
943
944
1
W 01
(o)() log Σ exp(w= A
(19)
945
m
946
947
Let al E R represent the jth row of A. Taking the gradient of L wrt a();
948
949
V Vavl
1
Wy, (o)())))))))))
Ee Ed exp(w (Ab()(())
(Av(( (A
950
m
951
952
1
(yy((i (i)
Ee explw! ( (wa()))
(cb((()
953
Eezo exp(wx ((0)
i=1
954
(y)(()) - -B(w())) v((()))
955
(20)
956
m
1=1
957
The gradient update step for al() is
958
959
(+1
(1)
af
(3)
aV C
960
(5) a
U) (1) - ((W()))(1)) { ((()))
961
ag
uy
962
963
Using the GD update rules for (a(t) (160)))=1 J==1d'' we have
964
(a++1) ( ((61)) ((
(1)
965
966
1
967
(a (a
968
969
970
971
{(
+ a
Wy. - -E(w[f(()] 1((((1) ,66)).
(21)
m
18

---

Under review as a conference paper at ICLR 2026
972
C
TRANSFORMER FOR CATEGORICAL ICL
973
974
The Transformer design for ICL (referred to as susup in the main paper) is based on the GD update equations developed in Section The input the transformer at layer & is
975
976
977
978
(0
B(wlffe))
(22)
979
Way
o(0)
980
981
Within (i) the vector component ft" is iteratively updated with increasing layer index E, with the Cg"
982
update manifested by each self-attention layer. The expectation B(w]f(") is updated by each MLP layer. Vector components f! and [[w] (()) occupy what we term as computational scratch space. The features (+))) and embedding vector Wy, represent the encoding of the data
983
984
985
and the
(t) occupied by (g()) ,Wy.) remains fixed at all Transformer layers.
986
portion of ej
987
Each attention block consists of self-attention layer, composed of two attention heads; one of these attention heads implements (() + f8+1 like above (for which [[(w){() is needed), and the second
988
(())
989
attention head erases [[(] (()) preparing for its update by the subsequent MLP layer.
990
991
992
C.I
SELF-ATTENTION LAYER
993
In matrix form, the input at layer is
994
995
fe
(())
ft f(m+1)
996
B B(wlf())
ff ((m))
E(wl M( p(mm)1)) Of
(23)
997
Wy
Wym
998
Φ1
om
@mt1
999
We assume that there are m labeled samples, index = m, and sample m + 1 is unlabeled.
1000 For notational simplicity we only consider one unlabeled sample, corresponding to sample m In
1001 the semi-supervised setting, all unlabeled samples are treated the same was as sample + shown
1002
here.
1003
(t) The update equation for fe+1 is given by
1004
1005
M = 11(00 + +
(24)
1006
where
1007
1008
A (1)
Wgy E(wls( )((((()) ,001)
(25)
1009
1010
1011
C.I.1 SELF-ATTENTION HEAD
1012
We design (!)) w:()) and w such that
1013
1014
w ( e (0,r, Oz, Oze, (r)
1015
(26)
1016
1017
w()) (j) c
(0.r, Ow, Oue 55
(27)
1018
1019
1020
W (2) ( Iuy, m
- (E(w((( ). Oa, Ow, 0x)²
(28)
1021
1022
The output of this first attention head, at position j fl,...,m
1023
1024
1025
Way
((wl (b))),))))))..).....)))))))))))) Oc. Ow, 0x
(29)
171
19

---

Under review as a conference paper at ICLR 2026
1026
The output of this first attention head at this first attention layer (before adding the skip connection)
1027
1028
A ft (1)
A f())
ft ff(++))
1029
1030
O =
0a
Oa
0a
(30)
Og
Oar
0ar
1031
0g
O
0
1032
1033
C.1.2
SELF-ATTENTION HEAD
1034
1035
With the second attention head we want to add (0a, (Ba.-W(wF]F (f Or O2r)² from position j. so we clear out the prior expectation. This will provide "scratch space" into which, with the next attention
1036
1037
layer type, we will update the expectation, using |(4] To do this, we design W and w such
1038
that
1039
1040
1041
w K ct
A(0xr. 0z (
(31)
1042
1043
1044
() 8
c
A(Oar. Oa Our (()r)T
(32)
1045
1046
where À 1. With an RBF kernel, for example (similar things will happen with softmax), if is very large,
1047
1048
<W ci
(2) (2) w c()) = 81,5
(33)
1049
1050
where j.j = if = j. and it's zero otherwise
1051
The value matrix is designed
1052
1053
W (2) (6) ce = (0a, ((w)s(()) 0z
),0,, 0z)
(34)
1054
1055
The output of this head is
1056
1057
0g E(wlS( (1)
0a E f m)) Og 0
O₂ E(w f(m+1))
1058
0
(35)
1059
Ou
Or 0₂
1060
O
1061
1062
We then add p(2300 with p(1) and p(2) designed so as to yield the cumulative output
1063
of the attention
1064
1065
AS(1)
Af(m)
Aftmt)
1066
(total)
B(wlft)
B(wlft f!()) 0g
(36)
1067
0g
Og
1068
0&
0g
0g
1069
1070
This is now added to the skip connection, yielding the total output of this attention layer as
1071
1072
(1)
(1)))
(m++1)
1073
fett
ft-
1074
0,r
0.r
O.r
(37)
1075
Wy
WyN
Our
1076
Om
@m+1
1077
1078
With the first attention layer, with two heads, update the functions, and also erase the prior expectations. the next attention layer, we update the expectations, and place them in the locations of the prior expectations.
1079
20

---

Under review as a conference paper at ICLR 2026
1080
C.2
MULTI-LAYER PERCEPTRON (MLP) LAYER
1081
1082
The vectors connected to 7 above will go into the next layer, which will be characterized by a MLP.
1083
Ideally, the MLP should implement the function
1084
exp(w=
((0)
1085
E(wl 1
Σ
We
(38)
1086
exp(w (1)
1087
1088
1089 MLP acts on each of the vectors at positions i E+1 represent MLP with parameters 7. The same I,..., m, corresponding to the first m columns of T, from left. The components of that vector corresponding to f are input to (-), and the output
to be consistent with functional GD. Let g
1090
1091 is a d' di-dmmnnsional vector. The output is placed in the position of the zeros in T.
1092
1093 At each layer of the Transformer, the form of the function in (38) is the same. Consequently, within the Transformer implementation, we tie the MLP parameters across all Transformer layer. The MLP parameters y are learned via the cross-entropy loss connected to the query (0mm1 1m++1) While the MLP is motivated via functional GD to implement (38 this is not explicitly imposed when learning all Transformer parameters.
1094
1095
1096
1097
1098
1099
D EXPERIMENTAL SEEUP DETAILS
1100
1101
Parameter descriptions.
1102
Number of sample points (n = 100): The number of data points sampled from each manifold. These points form the vertices of our graph construction.
1103
1104
k-nearest neighbours (k = 6): The number of nearest neighbors used when constructing the adjacency graph for each point. This parameter directly influences the connectivity of the graph and the subsequent Laplacian matrix construction.
1105
1106
1107
RBF kernel width (a 10): The scaling parameter for the radial basis function kernel used to compute weights in the adjacency matrix. In the code, this appears as:
1108
1109
1110
A rbf torch. exp (-scale cbffdistt dist_sq)
scale rbf 10
1111
Eigenvectors exported (kieat 4): The number of eigenvectors of the Laplacian matrix that are extracted and used as features. Although the Laplacian has eigenvectors, only use the first 4 corresponding to the smallest eigenvalues, as these capture the most important structural information of the manifold.
1112
1113
1114
1115
Manifold transformations:
The isotropic scale factor, planar rotation angle, and planar translation are randomly applied to the embedded coordinates after sampling. These trans- forms be found in the implementation code, for example:
1116
1117
1118
# Apply random scaling
1119
sca lealee. e factor = np. randon. form (0.02, 0.1) aataaaaldedd
1120
factor
1121
1122
# Apply random rotation
theta np. random. uniform(0, 2 * mp.pi) rotation patrriioo_marrixiooee torch.tensor [[ [np.cos (theta), -np.sin (theta), 0), [np.sin (theta), (np.ain(theta), co.cos (theta), 0], [0, 0, 1]
1123
1124
1125
1126
1127
1, dtype=torch. float322)
1128
a_rotteed = torch. matmul (a_translated, rotation_matria
1129
1130
# Apply random translation translation_x = np. random. uniform (-1, 1)
1131
1132
translation ra.dammmm aniformm (-1, 1)
1133
translation = torch. tersor (anssor[[[[[tt [[[trannslation.x. translation_y, 0111)
translated a_rotated translation
21

---

Under review as a conference paper at ICLR 2026
1134
The random scale, rotation, and translation applied after sampling and therefore leave all intrin- sic geometric properties unchanged. The scale factor uniformly multiplies all coordinates, preserv- ing relative distances and angles. These transformations ensure that model learns the inherent structure of the manifold rather than relying on specific coordinate representations.
1135
1136
1137
1138
1139
E SYNTHETIC MANIFOLDS
1140
1141
E.
CLASSIC MANIFOLDS IN R
1142
1143
Formally. let (M,g) be any of the five smooth, connected Riemannian manifolds illustrated in
1144 Figure 2. Each manifold admits (i) an elementary chart & U C R - IR that we list below, and (ii) a closed-form intrinsic distance d.M. For every experiment we draw a 100 points { R by sampling the chart parameters uniformly and then applying an independent random rigid motion (isotropic scale, 3-D rotation, planar translation); this removes global cues while preserving
1145
1146
1147
intrinsic geometry that model must discover. A centre index i ~~ Unif(1..... ] is selected and
1148
binary labels are assigned via
1149
1150
1151
dm ( TM- ( y1 {
1152
otherwise,
1153
1154
so that the positive class is exactly the geodesic ball of radius TM about ã, The explicit charts, geodesic formulas, and manifold-specific thresholds TM are collected in the remainder this sec-
1155
1156
tion.
1157
1158
E.1.
SPHERICAL FAMILY S
1159
Parametrization:
1160
1161
Φ () (sin cos Ø, sin sin ø, cos
(0, ().d) (0, (0,2
1162
1163
Line element: ds? do2 + sin? 2 do?.
1164
1165
Geodesic distance: Great-circle length (s= = arccos(p, q)
1166
1167
Labeling threshold:
1168
n/3 radians. This corresponds to approximately 60 degrees of separation the
1169
sphere's surface, creating spherical cap that covers roughly 25% of the sphere's surface area.
1170
1171
1172
E.1.2
CYLINDER FAMILY
1173
Parametrization:
1174
1175
(0,, 2) = cr r rsin 0, z), r.
1176
Line element: ds² = d02 + dz2
1177
1178
Distance: L (r 20)² + (21 )2
1179
1180
1181
Labeling threshold:
1182
Teyl = 1.0. Given the unit radius, this threshold allows the geodesic distance to extend
approximately radian around the cylinder circumference and I unit along its height.
1183
1184
1185
E.1.3
CONE FAMILY
1186
Parametrization:
1187
((s, 0) (ssin cos ssin 0, scos E (0, 3)
22

---

Under review as a conference paper at ICLR 2026
1188
Line element:
1189
1190
dss² = ds² + sin² a d02.
1191
Distance:
1192
Unfolding to a sector gives
1193
1194
Loone = s + 28182 cos sin A0).
1195
1196
Labeling threshold:
1197
Toone = 0.5. This creates circular region the cone's surface, with the size of the region depending on the cone's apex angle a.
1198
1199
1200
E.1.4
ARCHIMEDEAN SPIRAL (SWISS ROLL)
1201
Parametrization:
1202
(1) = (C² cos 40l, l sin 4wl, 1), E [],
1203
Line element:
1204
Speed [[([ 21 4x2 hence
1205
1206
ds = 2t 1 47 dt.
1207
1208
Distance:
1209
Integrating and taking absolute differences yields
1210
12) (1 12)333/3 (1 2
1211
1212
6R²
1213
Labeling threshold:
1214
The class boundary defined at median((), where points with parameter less than the median are assigned to class 1, and others to class O. This creates natural division of the Swiss roll into inner and outer regions.
1215
1216
1217
1218
E.1.5
FLAT TORUS T
1219
Parametrization:
1220
(0, Φ) = (0, o0 0), (0, e (0,27)?
1221
1222
Line element: ds² d0 do².
1223
1224
Distance (wrapped Euclidean):
1225
1226
c = A0 + A
AO = min((0) 02), 2x -
1227
Labeling threshold:
1228
Tra = 0.5. Given the parameter range (0, 2x) this threshold creates small circular
1229
region the flat torus, corresponding to approximately 1/8 of the torus surface area.
1230
1231
E.2
PRODUCT MANIFOLDS
1232
1233
The five classical manifolds introduced in Section E.1 are all low-dimensional and possess rela- tively simple label boundaries. In practical applications, however, data often reside on much higher- dimensional nonlinear spaces whose intrinsic geometry cannot recovered from any single ele- mentary chart. By taking the Cartesian product of elementary manifolds, we can construct highly complex manifolds in arbitrary dimensions. These product manifolds play an important role in practical applications ranging from computational chemistry (Zhang et al. 2021) and image and 3D structure analysis (Fumero et al. 2021). Evaluating on these products therefore probes the algorith-
1234
1235
1236
1237
1238
1239
1240
mic generalisability of our Transformer namely.. whether it can still recover the manifold structure and correct labels when confronted with far higher ambient dimension and intricate decision bound-
1241
aries.
23

---

Under review as a conference paper at ICLR 2026
1242
For any integer K 2 2 and any ordered tuple (M Mr) chosen from base set (Sphere, Cylinder, Cone, Spiral, T we consider the Cartesian product P M1 x... Mg.
Definition.
1243
1244 Each factor inherits its Riemannian metric MM and P carries the standard product metric
1245
Ek' T. leading to the distance
1246
1247
K
dm (Pk. 1/2
dx(p,Q)= =
P (PI)...,PP), Q =
(39)
1248
1249
1250
E.2.1
STATEMENT AND PROOF OF THEOREM
1251
1252
Theorem (Geodesic distance on a product manifold). Let (M,, be complete Riemannian man-
1253
ifolds with intrinsic distances das,- Endow M = M1 x.... x Mx with the product metric described
1254
above. Then for every p.9 9 E M, (p. = Σ das. (T:p,
1255
1256
Although the formula (39 is intuitive, we were unable to locate an explicit proof in the literature. For completeness, provide the following proof.
1257
1258
Restatement:
1259
1260
Let p (P1, ( pk) and (91...., : 9x) be points of M. Then
1261
dm (p.g) = / dm. (11,
+ dag (pk: 9x)3.
1262
1263
Proof. Set d2 == das, (p:: 9:) for s z k. We prove the two opposite inequalities.
1264
1265
Lower bound. Letc: : [0, 1] + be any piecewise C curve with c(0) = and c(1) = q. Write and denote
1266
c(t)
1267
1268
ax(() = (2)(1) 0).
1269
Because g(c,t) -
a;()),
1270
1271
I(c) ('' Σ (1)) dl.
1272
1273
1274
Using the triangle inequality in R
1275
1276
1 (aI(t),... ak()) dt
S I(c).
1277
Rd
1278
1279
But lo at(t) dt = I(ci) 2 de for each i, so
1280
1281
[(c) 2
1282
1283
1284
Taking the infimum over all yields
1285
1286
(M 2
Σ dz.
1287
1288
1289
Upper bound. Fix >> O. For each choose piecewise C' curve [0, + M, joining to
1290
1291 l with constant speed and
L(x)) d, +
1292
Define e: [0. - M by c(t) = (o.tt)..,,e) Since each % has constant speed a; := I(77),
1293
1294
1295
g(é(t),c(t)) = Σ a² (independent of t),
24

---

Under review as a conference paper at ICLR 2026
1296
1297
1298
L(c)
=
<
Σ 1=1
di + E)².
1299
1300
Letting &. O gives the reverse inequality
1301
1302
1303
d -
1304
1305
1306
1307
Remark 4. If every (M, g.) is complete, the Hopf-Riie RinfoR theorem guarantees the existence of
minimizing geodesics; in that case the curve constructed in the upper bound can be chosen geodesic in every factor, giving an explicit length minimizer in the product manifold.
1308
1309
1310
1311
E.3
IMAGE MANIFOLD
1312
To evaluate our approach on digeesinenion we created image manifolds using Stable Diffusion v1.5 RRombach et al. 2002)). These manifolds represent significant increase in complexity and dimensionality compared our synthetic manifolds, testing the transfer capabilities of our semi- supervised in-context learning method.
1313
1314
1315
1316
1317
E.
IMAGE MANIFOLD GENERATION
1318
1319
Our image manifolds are constructed using spherical linear interpolation (slerp) in the latent space of Stable Diffusion v1.5. For each manifold:
1320
1321
1. We sample two independent random latent vectors from standard normal distribution.
1322
2. We perform spherical linear interpolation between these two vectors, generating 100 evenly spaced intermediate points.
1323
1324
1325
3. Each interpolated latent vector is decoded into 500x500 pixel RGB image using the Stable Diffusion model without classifier guidance.
1326
1327
4. This process repeated 950 different random seed pairs, creating 950 diverse image manifolds.
1328
1329
As in our other experiments, there are 100 total samples connected with given manifold, here corresponding to 100 images. The smooth transitions between images in each manifold confirm that the latent space of the diffusion model indeed has manifold structure. This allows us to test our methods on natural image data while maintaining control over the underlying manifold geometry.
1330
1331
1332
1333
1334
E.3.2
DATA PROCESSING
1335
1336
The image data processing involves several specific steps:
1337
1338
All 500x500 images are downsampled to 32x32 pixels. This reduces the dimensionality from 250,000 to 1,024 dimensions per image while preserving essential visual features.
Downsampling:
1339
1340
1341
Grayscale conversion: All images converted to grayscale simplify processing.
1342
Normalization: Pixel values are first normalized to the range [0,1] by dividing by 255.
1343
Pixel scaling: A pixel scaling factor 0.001 applied, reducing the normalized pixel val- ues the range [0, 0.001]. This scaling is specifically designed match the distance scale image manifolds with our synthetic manifolds, enabling effective transfer models trained synthetic data to image data. Without this scaling factor, the distances between image points would have a significantly different distribution compared to synthetic mani- folds, hindering generalization.
1344
1345
1346
1347
1348
1349
The relevant code for image processing is:
25

---

Under review as a conference paper at ICLR 2026
1350
Downsample from 500x500 to 32x32 img = ing. resize ( (32, 32), Image LANC20S)
1351
1352
1353
Convert to tensor and normalize to [0, 1]
1354
img_tenser torch. mercc..fro__ummaee (np. (img) float () / 255.0
1355
1356
Apply pixel scale factor 0.001
1357
ing_tenscr = img tensor / 1000.0
1358
1359
E.3.3
TRAIN-TEST SPLIT
1360
We divide the 950 image manifolds between training and testing:
1361
1362
Training manifolds:
Prompts with IDs 1-500 (first 500 manifolds)
1363
Test manifolds: Prompts with IDs 501+ (remaining manifolds)
1364
1365
This split ensures that models evaluated previously unseen interpolation sequences, testing true generalization rather than memorization.
1366
1367
1368
E.3.4
GRAPH CONSTRUCTION
1369
1370
For each image manifold:
1371
1372
We compute pairwise Euclidean distances between flattened image vectors.
1373
These distances can be (optionally) scaled using the parameter -- --image 0. 001.
1374
An RBF kernel is applied convert distances similarities: Aij exp(-a dft). a has the same scale 10 here as for our other manifolds.
1375
1376
A 4-nearest neighbors graph is constructed for each manifold.
1377
1378
The normalized Laplacian is computed from this graph as for our other manifolds.
1379
1380
E.3.5
LABELING
1381
For image manifolds, we use a position-based labeling scheme:
1382
1383
The first 50 images in each interpolation sequence are assigned class
1384
1385
The last 50 images are assigned to class
1386
This creates natural separation based on position in interpolation sequence, dividing each manifold into two equal parts. Only m < n of the images are labeled as given to our Transformer.
1387
1388
1389
E.3.6
VISUAL REPRESENTATION
1390
1391 To visualize the manifold structure of our image datasets, we present two examples of complete 10x10 image grids, each showing all n 100 interpolated frames from different manifolds, These
1392
1393
grids demonstrate the smooth transitions created by the spherical linear interpolation in Stable Dif- fusion's 's latent space.
1394
1395
These examples illustrate several important aspects of our image manifolds:
1396
1397
1. Continuous transitions: Each row shows gradual changes without abrupt shifts, support- ing the assumed smooth manifold structure.
1398
1399
2. Semantic coherence: The interpolations maintain thematic consistency while traversing the latent space, with changes in perspective, detail, composition, and style.
1400
1401
1402
3. Natural class division:
The first 50 images (top 5 rows) and last 50 images (bottom 5 rows) form our two classes, with the most significant changes often occurring near the midpoint of the interpolation.
1403
26

---

Under review as a conference paper at ICLR 2026
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
Figure 6: Example image manifold showing interpolation of a deer in atmospheric lighting. The transformation progresses from distant view of antlers (class 0, top 5 rows) close-up facial details (class 1, bottom 5 rows), demonstrating how the manifold captures continuous changes in perspective.
1439
1440
1441
1442
1443
1444
4. Diversity across manifolds: The three examples demonstrate how our methodology cre- varied manifolds across different themes and visual styles, providing challenging and diverse test set.
1445
1446
1447
Our semi-supervised in-context learning method must navigate these complex visual manifolds. identifying the underlying geometric structure from just few labeled examples. The ability to transfer knowledge from synthetic 3D manifolds to these nigh-dimensional image manifolds demon- strates the robust generalization capabilities of our approach.
1448
1449
1450
1451
1452
1453
F ADDITIONAL EXPERIMENTAL RESULTS
1454
1455
F.I
DEPENDENCE ON ICL TRANSFORMER DEPTH
1456
1457 (see Cection affects performance by comparing 1-layer and 2-layer ICI. Transformer architectures
We investigate how the depth of the in-context learning (ICL) Transformer for categorical outcomes
27

---

Under review as a conference paper at ICLR 2026
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
Figure 7: Example image manifold showing interpolation between anime-style character concepts.
1493 The progression moves from an action pose with bright colors (class top rows) increasingly detailed portrait shots (class 1, bottom rows), highlighting the manifold's ' ability to represent both style and compositional changes.
1494
1495
1496
1497
1498
1499
1500
1501
1502
(in general, we observed little change in performance for deeper models). Figure shows the results of this comparison on the product manifold.
1503
1504
The results demonstrate that the 2-layer ICL architecture consistently outperforms the I l-layer archi- tecture across all labeled data percentages. This supports the theoretical framework in Wang et al. (025), showing that increased depth allows the model to learn more sophisticated manifold repre- sentations. The performance improvement is more pronounced for the RBF kernel, which further emphasizes the kernel's ability to capture non-linear relationships in the data manifold. We observe that for both kernel types, the 2-layer ICL architecture provides a consistent advantage, with RBF kernels outperforming linear counterparts both depths. This advantage is most notable in the low-label regime (5-15% labeled data), highlighting the importance architectural capacity when labeled data is scarce.
1505
1506
1507
1508
1509
1510
1511
28

---

Under review as a conference paper at ICLR 2026
1512
Network Depth Comparison with FF Configuration
manifold) Id
1513
1.0
1514
1515
0.9
1516
1517
1518
0.8
1519
Accuracy
1520
1521
0.7
1522
1523
1524
0.6
LINEAR 1-Layer
1525
LINEAR 2-Layer
1526
RBF 1-Layer
0.5
1527
RBF 2-Layer
1528
10
15
20
25
30
35
40
1529
Labeled Data Percentage (%)
1530
1531
Figure 8: Comparison of 1-layer vs. 2-layer ICL Transformer architectures on the product mani- fold. The 2-layer architecture shows improved performance, particularly in the low-label regime, suggesting that deeper networks can learn more sophisticated manifold representations.
1532
1533
1534
1535
1536
F.2
COMPARISON OF KERNEL TYPES ACROSS MANIFOLDS
1537
While the paper focused primarily results with the RBF kernel on selected manifolds due space constraints, here we present a comprehensive analysis comparing linear and RBF kernels across all individual manifolds well as product manifolds.
1538
1539
1540
The results in Figure / demonstrate that our approach generalizes well across different manifold types and kernel choices. For each manifold, observe similar patterns where end-to-end Transformer consistently outperforms the baselines. While the RBF kernel typically yields better results for most manifolds, the relative performance varies depending on the geometric complexity of the manifold. These results further demonstrate that the semi-supervised Transformer framework we have developed is robust across a variety of manifold structures.
1541
1542
1543
1544
1545
1546
1547
F.3
IN-DISTRIBUTION vs.
PERFORMANCE
1548
1549
We examine the generalization capabilities of our model by evaluating its performance in both in- distribution (ID) and out-of-distribution (OOD) settings. For ID, the manifold types seen at test were observed when the Transformer
1550
1551
trained, while for OOD the Transformer did not the class of test manifolds when being trained. For OOD evaluation, we train the model on four manifold types and test on the held-out fifth manifold. Tables 2 and present detailed results for different context sizes.
1552
1553
1554
1555
Our results reveal interesting patterns in generalization capabilities. For simpler manifolds like cylinder, sphere, and torus, we observe that OOD performance is competitive with or sometimes even exceeds ID performance by a large margin. This suggests that training on diverse set of manifolds provides rich geometric knowledge that transfers well to these simpler structures. The simpler manifolds share similar geometric features, making knowledge transfer more effective.
1556
1557
1558
1559
1560 In contrast, for more complex manifolds like Swiss Roll and Cone, OOD performance is typically lower than ID performance. This is particularly pronounced for the Swiss Roll manifold with the RBF kernel, where OOD performance is significantly lower than ID. This performance gap likely stems from the unique geometric properties of the Swiss Roll that are not well captured by the kerne bandwidth parameters leamed from other manifolds. The RBF kemnel's bandwidth is particularly sensitive to the intrinsic curvature and density variations present in the Swiss Roll but absent in simpler manifolds.
1561
1562
1563
1564
1565
29

---

Under review as a conference paper at ICLR 2026
1566
1567
1568
0.9
0.9
1569
1570
Acqurety
1571
0.7
0.7
1572
1573
ORIG+EZE
ORIGHEZE
1574
EIG+ICL
EIG+ICL
1575
0.5
0.5
ORIG+ROF-LR
1576
10
15
Parcentage (56)
Percentage
1577
1578
(a) Sphere (Linear kernel)
(h) Sphere (RBF kernel)
1579
1.0
ORGG+E2E-CL
1.0
1580
EIG+LR
1581
ORIGI RBF-LR
1582
0.8
0.8
1583
Accurait
Recuracy
1584
0.7
1585
0.6
0.6
ORIGHEZE-CCL
1586
EIG+ICL
1587
0.5
ORIG+R3F
EIG+LR
1588
25
35
40
5
10
15
20
25
30
35
1589
Labeled Data Percentage (56
Labeled Data Percentage 1%)
1590
(c) Torus (Linear kernel)
(d) Torus (RBF kernel)
1591
1592
1593
0.9
0.9
1594
1595
Regurey
1596
0.7
0.7
1597
1598
ORIG
ORGHEZE
LIG+ICL
LIG+ICL
1599
1600
0.5
GRIG+RBF-LR
0.5
ORI++ROFLRR
1601
Parcentage (%6)
30
15
20
Percentage
1602
Cone (Linear kernel)
(f) Cone (RBF kernel)
1603
1.0
1.0
1604
1605
1606
1607
Acqurary
0.8
0.0
1608
0.7
1609
1610
0.6
ORIGHEZEICLL
0.6
ORIGHEZE ICL
1611
EIG+ICL
EIGHICL
BIG+LR
BIG+LR
1612
0.5
1613
10
15
20
25
35
40
5
30
1.5
20
25
30
35
40
Labeled Data Percentage (56)
Labeled Data Percentage (%)
1614
(g) Swiss Roll (Linear kernel)
(h) Swiss Roll (RBF kernel)
1615
1616
Figure 9: Comparison of model performance across all individual manifolds with linear and RBF kernels. Our end-to-end Transformer (ORIG+E2E-ICL) consistently outperforms baseline methods across all manifolds and kernel types, with the advantage being particularly pronounced in the low- label regime.
1617
1618
1619
30

---

Under review as a conference paper at ICLR 2026
1620
1621
1622
1623
Table 2: Performance Comparison at Context Size 3
1624
1625
Kernel-Targe
EIG+ICL (ID)
EIG+ICL (OOD)
E2E ICL (ID)
E2E ICL (OOD)
1626
Linear-Cone
0.778 0.007
0.799 0.007
0.859 0.008
0.757 0.018
1627
Linear- Cylear-yllinerr
0.734 + 0.007
0.790 = 0.010
0.772 =
0.923 = 0.007
1628
Linear-Sphere Sphere
0,708 = 0.009
0.788 = 0.007
0.767 = 0.008
0.919 # 0.005
Linear-Tore
0,605 = 0.010
0.857 = 0,009
0.627 = 0.008
0.968 = 0.003
1629
Linear-Swiss Roll
0.789 + 0.009
0.788 0.008
0.854 + 0.006
0.826 0.015
1630
RBF-Cone
0.731 0.008
0.826 = 0.013
0.790 + 0.013
0.753 = 0.017
1631
RBF-Cylinder
0.096
0.918 0.006
0.725
0.927 0.005
1632
RBF-Sphere
0.658
0,912 0,006
0.679
0.920 = 0.006
1633
Ror-Torus
0.010
0.963 0.004
0.590
0.969 + 0.003
1634
RBF-Swiss Roll
0,811 0.010
0.785 = 0.009
0.805 0.010
0.694 + 0.014
1635
1636
1637
1638
1639
1640
1641
Table 3: Performance Comparison Context Size 21
1642
1643
Kernel-Taree
EIG+ICL (ID)
EIG+ICL (OOD)
E2E ICL (ID)
E2E ICL (OOD)
1644
Linear-Cor Cone
0,931 + 0.004
0.955 = 0.004
0.933 = 0.003
0.798 = 0.015
1645
Ainear-Cylinder
0.861 + 0.005
0.929 0.004
0.871 + 0.006
0.935 + 0.002
1646
Linear-Sphere
+ 0
0.925 0.003
0.882 = 0.004
0.930
1647
Linear-Torus
0.703 +
0.964 0.003
0.716 + 0.006
0.970
Lincar-Swiss Roll
0.951 = 0.003
0.952 = 0.002
0.946 + 0.003
0.919 # 0.009
1648
1649
RBF-Cone
0,937 + 0.004
0.877 = 0.009
0,939 + 0.003
0
1650
RBF-Cylinder
0,866 = 0.005
0.934 # 0.002
0.885 + 0.005
0.936 # 0.002
RBF-Sphere
0.839 = 0.003
0.929 0.002
0.865 0.004
0.929 # 0.003
1651
RBF-Torus
0.771 + 0.007
0.970 0.001
0.768 0.006
0.970 0.001
1652
RBF-Swiss Roll
0,950 + 0.002
0.860 0.007
0.940 + 0.004
0.764
1653
1654
1655
1656
1657
1658
1659
Table 4: Performance Comparison at Context Size 39
1660
1661
Hernel-Target
EIG+CCL (ID)
EIG+ICL (OOD)
E2E ICL (ID)
E2E ICL (OOD)
1662
Linear-Cone
0,947 + 0.003
0,966 = 0.002
0,947 : 0.003
0.803 : 0.014
1663
Linear-Cylinder
0.876 + 0.004
0.935 # 0.002
0.885 + 0.004
0.935 # 0.002
1664
Linear-Sphere
0.857 + 0.004
0.930 = 0.003
0.901 # 0.004
0.929 # 0.002
1665
Linear- Torus
0.724 0.007
0.969 0.002
0.730 0.006
0.970 0.001
1666
Linear-Swiss Roll
0.964 0.002
0.964 0.003
0.964 + 0.002
0.947 # 0.009
1667
RBF-Cone
0.949 + 0.002
0.877 0.007
0.952 0.003
0.804 0.016
1668
RBF-Cylinder
0.882 + 0.004
0.935 0.002
0.902 + 0.005
0.935 = 0.002
1669
RBF-Sphere
0.860 + 0.004
0.930 + 0.002
0.892 + 0.004
0.930 # 0.002
1670
RBF-Torus
0,800 + 0.006
0.970 + 0.002
0.804 + 0.004
0.970 + 0.001
RBF-Swiss Roll
0.960 + 0.002
0.868 0.007
0.954 + 0.003
0.775 + 0.009
1671
1672
1673
31

---

Under review as a conference paper at ICLR 2026
1674
When training exclusively manifold type, the model may specialize excessively to the spec- tral decomposition of that manifold's ' Laplacian matrix. This results spectral features with lower bias but significantly higher variance upon simte--ample re-estimation essimation-charrcteristttt characteristic of overfitting. Exposure to multiple manifold types appears to promote more robust spectral feature learning that generalizes better.
1675
1676
1677
1678
1679
1680
F.4
IMAGE MANIFOLD TRANSFER PERFORMANCE
1681
1682
The ability transfer knowledge from synthetic manifolds nigh-dimensional image manifolds is
1683 a crucial test of our approach's robustness. Table 3 presents results for different combinations of training manifolds when evaluating on image manifolds.
1684
1685
Table 5: Performance Comparison for RBF Kernel on Image Manifolds at Different Context Sizes
1686
1687
Context
Training Manifolds
EIG+ICL (ID)
(CG+IC((OO))
E2E ICL (ID)
E2E ICL (OOD)
1688
1689
wewreeyylinder.sheee.ssweee swiss rool
0.700 0.008
0.555 + 0.005
0.730 + 0.007
0.557 0.007
1690
0.700 # 0.006
0.555 + 0.005
0.730 + 0.009
0.600 + 0.007
3
cone,sphere,
hww.spphere..ous.swe... swiss roll
0.700 = 0.009
0,555 + 0.005
0.730 + 0.006
0.596 : 0.006
1691
cylinder.spherr worurreteorrsssste swiss roll
0.700 # 0.007
0.557 + 0,009
0.730 # 0.008
0.543 e 0.005
1692
cone,cylinderr.pperr...
0.700 0.005
0.555 0.007
0.730 + 0.010
0.524 0.004
1693
cone,cylindee www.eeyyiiddrrpeeeee... .swss swiss roll
0,850 + 0.007
0.657 + 0.008
0.880 # 0.006
0.686 = 0.005
1694
wir.eccyllidder.rrusssssss. roll
0.850 0.009
0.657 + 0.008
0,880 + 0.005
0.744 # 0.004
1695
paaeesspheeeeeuu...oo cylinder,spherr. torus, swiss_roll roll
0.850 + 0.006
0.880 + 0.008
0.745 + 0.003
1696
0.850 + 0.005
0.652 + 0.006
0.880 + 0.009
0.650 = 0.005
1697
cone,cy coneecylindeer.sphere.oo
0.850 + 0.008
0.612 + 0.008
0.880 + 0.007
0,584 + 0.004
1698
cone,cy cyr.ciiierrrrsherreee. ,siiss roll
0,900 # 0.006
0.689 + 0.005
0.930 + 0.008
0.740 + 0.006
1699
fime..ylinderrtorus... swwss roll
0.900 + 0.005
0.689 + 0.005
0.930 + 0.007
0.781 + 0.004
1700
39
cone.sphere, torus, woll
0.900 + 0.008
0.689 + 0.005
0.930 + 0.005
0.786 0.004
sylinder.sphere. torus swiss_ross _roll
0,900 + 0.009
0.688 0.006
0.930 + 0.006
0.710 + 0.008
1701
cone,cylinder.sphere...
0.900 + 0.007
0.626 + 0.006
0.930 + 0.009
0.616 0.005
1702
1703
The results demonstrate that our end-to-end semi-supervised Transformer achieves remarkable zero- shot transfer image manifolds despite being trained solely on synthetic manifolds. Several impor-
1704
1705
tant observations emerge:
1706
1707
The combination of training manifolds significantly impacts transfer performance. Including both Swiss Roll and torus in the training set appears to consistently yield the best transfer results, likely because these manifolds capture certain intrinsic properties present in image manifolds better than other combinations.
1708
1709
1710
1711
Secondly, there is clear relationship between context size and transfer performance. As the text size increases from 3 to 39, the transfer performance improves substantially across all training configurations. This suggests that the model's ability to leverage contextual information generalizes well from synthetic to image manifolds.
1712
1713
1714
1715
Finally, our end-to-end approach (E2E ICL) consistently outperforms the baseline EIG+ICL model in both ID and OOD settings, confirming that the learned representation is more effective than using ground-tre -tunh eigenvectors directly in diffeult problems.
1716
1717
1718
1719
F.5
IMAGENET 100 EXPERIMENTS WITH 39% LABELS
1720
1721
We additionally report results on ImageNet100 with a higher label ratio of 39%.
1722
Remark (Separation Score), The separation score defined as the difference between average intra-class similarity and inter-class similarity Caron et al. (2018); Oord et (2018):
1723
1724
1725
Sep =
Σ
Ex,zzec [sim(x,, - Erge, wee', egte [sim(r, y)],
1726
ICI
OEC
1727
where sim().) denotes cosine similarity.
32

---

Under review as a conference paper at ICLR 2026
1728
100
31% Labeled Datn RLL Sccurccy
30% Labrled Data Sepecation Score
30% Labeled Data mNN Comperhonn
1729
1730
140
0.20
1731
1732
1733
1734
8:20
1735
Treufrree Boscie
ECC Ttankkrner Deviao
1736
1737
1738
Figure 10: In-context learning on ImageNet100 with 39% labels and dataset size 5000. Each dataset consists of 200 batches, with each batch containing 100 samples (50 images per class) represented by
1739
1740 VGG-29 features. The left column reports ICL accuracy, and the right column reports the separation score (intra-class minus inter-clas: similarity, following Caron et al. (2018): Oord et al. (2018)) We compare three methods: Ori++E2E-ICI, E2E-ICL (red, PE from our model), a Transformer baseline (blue, PE from the final transformer layer Min al. (2022)), and Eig+ICL (green, features pre-extracted
1741
1742
1743
1744
from VGG-29).
1745
1746
F.6
ADDITIONAL EXPERIMENTAL DETAILS
1747
1748
All models were implemented in PyTorch and trained on Tesla V100 PCle 16 GB GPU. The complete set of experiments required roughly 2 hours of computation time. All source code is available here: GitHub
1749
1750
1751
1752
F.7
MODELS UNDER EVALUATION
1753
1754
We denote models by Modea)Model], where Data specifies the form of input features, and Model specifies the classifier applied to the unlabeled samples.
1755
1756
ERIGEE2E-ICL (OuRS) The raw original coordinate matrix is fed directly into our end-to-end Transformer. All three modules-the the Laplacian predictor, eigenvector extractor, and in- context GD head- are trained jointly with a combined loss. (See Appendix A for archi- tectural details.)
1757
1758
1759
1760
EIG+ICL
The ICL head is supplied with the ground-truth bottom-k Laplacian eigenvectors, com- puted offline. This removes representation learning, so the gap ORIG+E2E-ICL directly quantifies the benefit of learned representations.
1761
1762
1763
EIG+LR A standard fz-regularized logistic regression refit on the ground-truth eigenvectors at each label ratio.
1764
1765
1766
ORIG+RBF-LR Kernel logistic regression the original coordinates using the Gaussian kernel
KRBF(xx,2) = cxp(->llæ x' [| Schölkopf & Smola (2002). The decision function is af <EE; (x., with an l2 penalty on a. providing a nonlinear classifier refit at each label ratio.
1767
1768
1769
1770
ORIG+ICL The original coordinate matrix is fed directly into in-context GD head, with- out any additional learned embedding module. This serves as control to measure the contribution of the learned PE module ORIG+E2E-ICL.
1771
1772
1773
1774
F.8
ABLATION ON LAP (LAPLACIAN PREDICTOR) AND PE (EIGENVECTOR EXTRACTOR)
1775
Experimental Setup:
We replace the RBF activation in the Laplacian- formiing module (dtsp) with linear mapping (no activation) and additionally compare a 1-layer variant of our Laplacian and po- sitional embedding (PE) modules. All experiments
1776
1777
conducted the 5-factor product manifold task (train and test on the same manifold family), averaged over 10 runs per context size.
1778
1779
1780
Note "1-layer" configuration: The "I-layer" Lap+PE variant refers non-iterative version of our spectral representation module, which performs only single forward pass to compute the Laplacian and positional embeddings, without iterative refinement. By contrast, the full in-context
1781
33

---

Under review as a conference paper at ICLR 2026
1782
Transformer iteratively refines the Laplacian and PE features through multiple forward passes within each Transformer block. As shown below, the single-pas configuration produces weaker manifold representations, confirming the importance of iterative refinement.
1783
1784
1785
1786
1787
Table Ablation on Laplacian and positional embedding modules using product-manine data (5
1788
factors).
1789
1790
Context Size
Lap+PE (1 Layer)
Our Model
Lap (Linear)
1791
0.534 0.008
0.663 + 0.031
0.479 + 0.009
1792
0.548 = 0.016
0.688 = 0.029
0,480 + 0.012
1793
10
0.567 0.015
0.720 = 0.016
0.465 0.010
1794
15
0.577 0.007
0.739 + 0.009
0.475 + 0.009
1795
20
0.584 0.013
0.735 + 0.011
0.508 + 0.009
1796
40
0.608 0.009
0.744 + 0.007
0.536 0.005
1797
80
0.619 0.007
0.744 + 0.007
0.555 1 0.009
1798
1799
1800
Observation: Both the linear Laplacian and the 1-layer (non-iterative) Lap+PE configurations substantially reduce performance. The full in-context Transformer benefits from iterative Laplacian updates and non-linear kernels, which yield improved manifold alignment and label propagation.
1801
1802
1803
1804
1805
1806
G TRANSFORMER BASELINE ARCHITECTURE
1807
1808
1809
SmallScale SmallScaleTransfommm Transformer (adapted from von Oswald et al. 2023). This serves as an in- context Transformer pipeline, where raw features and tokenized labels jointly embedded, pro- cessed by Transformer encoder, and predictions are made in-context.
Model:
1810
1811
1812
Pipeline Overview:
1813
1814
Input construction:
Raw features [B, N. DJ and label IDs [B.]] are embedded via input projection(raw) label embedding(labels) - ,, Context tokens use true labels; query tokens use special unknown label id.
1815
1816
1817
Encoder: Two Transformer layers (4 heads, dmmdt==40) Each layer uses Multi- Head Self-Attention and feed-forward block
1818
with LayerNorm and residual
1819
connections.
1820
Output heads: Feature head Linear(240-044 (positional encoding)
Classification
head Linear(4-22) (binary prediction)
1821
demonstra- tions. Cross-entropy on query logits vs. true labels; all parameters optimized end-to-end.
Training: Loss computed only query tokens; context tokens
1822
1823
1824
1825
Total Parameters: 1,442,178 ( (1.4M). This baseline acts as a standard in-context Transformer for comparison with our structured semi-supervised version.
1826
1827
1828
1829
H MODEL PARAMETER COUNTS
1830
1831
1832
The parameter counts reported below refer to the total number of trainable model parame- ters that is, all learnable weights (including attention projections, feed-forward layers, and embed- ding matrices) within each model configuration. These values indicate the relative model capacity rather than computational cost, and allow for fair comparison between our proposed in-context semi- supervised Transformer and standard baselines.
1833
1834
1835
34

---

Under review as a conference paper at ICLR 2026
1836
Table 7: Parameter counts and architectural summary of all compared models.
1837
1838
Model
Parameters
Architecture Configuration
1839
ORIG + LR
627
Linear logistic regression
1840
ORIG + RBF-LR
628
RBF kernel logistic regression
1841
EIG LR
627
Regression on eigenfeatures
1842
ORIG + ICL
576
Our in-context learning Transformer (on raw inputs)
1843
EIG ICL
576
Our in-context learning Transformer (on eigenfeatures)
1844
ORIG + Transformer
1,442,178
In-context Transformer baseline
1845
ORIG E2E-ICL (ours)
10,852
Lap (2,592) PE (7,684) ICL head (576)
1846
1847
1848
Parameter Composition (ours):
1849
1850
Lap module 2,,59 parameters
1851
PE module - 7,684 parameters
1852
ICL Transformer 576 parameters (in-context classification head)
1853
Remark 6. Unless otherwise stated, all ICL-based models single-layer ICL Transformer with
1854
the RBF kernel as the core component the the GD ICL head. For all experiments, we vary the labeled- data ratio among fixed 100 total samples.
1855
1856
1857
I
ADDITIONAL RELATED WORK
1858
1859
In the following, describe several recent work which investigate the use of unlahelled data for in-context learning. Agarwal et al. (2024) show that scaling to many-shot prompts yields large gains across diverse tasks, and further introduce Reinforced ICL (using model-generated rationales) and Unsupervised ICL (inputs only) to reduce dependence on curated labels. Chen et al. (2025) improve many-shot ICL by ceattencc--sllectnn unlabeled examples, pseudo-labeling them with an LLM, and adaptively choosing demonstrations per query. [i et al. (2025) prove that single-layer model cannot exploit unlabeled data, while depth/looping constructs polynomial estimators akin to EM; they demonstrate looping gains on tabular tasks.
1860
1861
1862
1863
1864
1865
1866
1867
These results motivate our focus on how transformers leverage unlabeled context: rather than adding Aoseudo-Jlablle shots, we extract structure directly from unlabeled tokens by computing a repre- sentation in-context and then using additional attention layers to implement gradient-based learn- ing algorithm. Our approach complements maayy-sho/pseuu---abeiiin by providing an explicit forward-pard pass mechanism that explains why depth helps in SS- ICL,, and is supported by cross-domain evidence from synthetic manifolds to diffusion images and ImageNet100 features that full trans- formers learn coometr--aligned representations consistent with this mechanism.
1868
1869
1870
1871
1872
1873
1874
1875
1876
1877
1878
1879
1880
1881
1882
1883
1884
1885
1886
1887
1888
1889
35


Reviews:
[{"review_id": "ZC4U6GsDK9", "reviewer": "Reviewer_jojL", "summary": "This paper introduces In-Context Semi-Supervised Learning (IC-SSL), extending Transformer-based in-context learning to settings with few labeled and many unlabeled samples. It proposes a two-stage, end-to-end Transformer: the first stage learns representations from unlabeled data by constructing a Laplacian and computing an eigenmap, and the second stage performs in-context supervised learning to infer labels for unlabeled samples. This enables both representation learning and label propagation within a single forward pass. Experiments on synthetic manifolds and images show it outperforms baselines, leveraging unlabeled data to improve performance with few labels, and generalizes well in and out of distribution.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Strengths:\n\n1. This paper presents a novel extension of in-context learning to the semi-supervised setting, enabling Transformers to jointly exploit labeled and unlabeled examples within a unified framework. \n\n2. The proposed two-stage architecture—combining Laplacian-based representation learning with an in-context gradient descent head—offers rare mechanistic interpretability and provides insight into how Transformers may implement semi-supervised reasoning. \n\n3. Empirically, the method demonstrates strong label efficiency and outperforms competitive baselines across synthetic manifolds, product manifolds, and ImageNet100, showing promising generalization and transfer, especially under low-label regimes.", "weaknesses": "Weaknesses:\n\n1. Despite its contributions, the experimental evaluation is limited in scale and focuses primarily on small episodic settings, leaving open questions about robustness in large, noisy, or highly diverse real-world scenarios. \n\n2. The model relies on a strong geometry-driven inductive bias, which may restrict flexibility when the data does not exhibit clear manifold structure, and the architecture requires non-standard attention variants and customized modules that could hinder practical adoption. \n\n3. Additionally, the overall complexity introduces training sensitivity and may limit applicability beyond the studied domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G3O2c3VXRH", "reviewer": "Reviewer_TXiC", "summary": "The paper studies a setting of semi-supervised in-context learning (termed IC-SSL), where the input consists of many examples, a few of which are accompanied by gold labels. From a theory of ICL perspective, the authors construct and train small transformers on various classification tasks with synthetic data drawn from manifolds, and on one more realistic task using interpolations between images in ImageNet. They propose a specific model for this setting that outperforms baseline transformers, based on an argument about how semi-supervised learning might be performed in context mechanistically.", "questions": "Q1. Can you elaborate on line 72, the claim that the model \"provably\" performs ICL? \n\nQ2. In Figure 1, you add the labels for the few labeled examples back into the representation in step B. Why not allow representations to be learned with awareness of these labels? \n\nQ3. What is the transformer baseline in Figure 3/10? It's not in the list of models in 4.1.\n\nQ4. Can you provide the parameter counts for all model types trained? It's a bit hard to tell from the descriptions alone whether some methods might have more learnable expressivity.\n\nMy work is more in the empirical direction, so some of these clarifications may be sufficiently clear already in the text for a more theory audience. \n\nOther comments:\n- Please fix citations to use \\citep{} where appropriate, especially in the introduction. \n- for the sentence on line 31 about functional gradient descent, would be nice to cite back to the prior work in this area \n- I found the contributions section 1.1 to be quite long and a bit confusing before reading the rest of the paper; I think this might be better positioned near the end of the work, with just a short summary of contributions in the introduction. \n- 4.1 and F7 are very similar text; F7 could perhaps be merged into 4.1.\n- it would be nice if the line in Fig 4(b) that's a directly comparable result from 4(a) was the same color in both, though I understand the color scheme is meant to indicate the different model types.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The paper thoroughly explores semi-supervised ICL, presenting experiments on carefully designed synthetic data, using multiple random seeds, and reporting interesting ablations. \n\nS2. The comparison of performance on in-domain and out-of-domain data when training on data generated from different subsets of the classification manifolds was quite interesting. \n\nS3. The proposed E2E-ICL is both mathematically well-justified and empirically effective at semi-supervised ICL.", "weaknesses": "W1. It would be nice to at least mention recent [empirical work](https://arxiv.org/abs/2404.11018) on unsupervised ICL. \n\nW2. The claim in line 343 that the model is competitive on real data seems a bit ill-supported, as even the ImageNet task is a fairly synthetic classification task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kBp9KvxWAM", "reviewer": "Reviewer_NFR7", "summary": "This paper introduces IC-SSL, a novel Transformer architecture explicitly constructed to perform in-context semi-supervised learning. The core contribution is a two-stage model: 1) A representation-learning stage ($TF^{\\phi}$) that is designed to leverage unlabeled data by computing Laplacian Eigenmaps, a classic spectral method for manifold learning. 2) An ICL stage ($TF_{sup}$) that uses these learned representations, along with the few available labels, to perform classification by provably mimicking functional gradient descent. The authors provide theoretical analysis (in the appendices) to demonstrate that their specialized attention modules (e.g., $Attn_{rbf}$, $Attn_{linear}$) can indeed be parameterized to execute these specific algorithmic steps. Empirical results on synthetic manifolds and real-world data demonstrate the method's superiority over baselines, especially in low-label regimes.", "questions": "1. **On the Necessity of Iteration**: Your model relies on iterative algorithms (power iteration, GD) implemented via depth/loops. What are the theoretical trade-offs of this explicit iteration versus a hypothetical non-iterative (e.g., single-layer) but perhaps very wide model? Is there a theoretical justification for why an iterative approach is fundamentally necessary for IC-SSL, thus motivating your design?\n\n2. **On the Choice of Spectral Method (Linear vs. Non-linear)**: Your $TF^{\\phi}$ module is complex: it first uses $Attn_{rbf}$ to compute the Laplacian $\\hat{\\mathcal{L}}$, and then uses $Attn_{linear}$ to compute its eigenmap via power iteration. A simpler approach might skip the first step. Given that stacked $Attn_{linear}$ layers are mathematically known to compute matrix powers, a standard Transformer might implicitly learn to compute powers of the data covariance matrix ($X^\\top X$), a linear spectral method (i.e., PCA).\n  * (a) Can you theoretically justify the necessity of your explicit, two-step, non-linear spectral method ($\\hat{\\mathcal{L}}$) over this simpler, linear spectral alternative ($X^\\top X$)?\n  * (b) Did you empirically compare your $TF^{\\phi}$ module against a simpler $TF^{\\phi}$ baseline that only uses stacked $Attn_{linear}$ to (implicitly) learn a linear spectral embedding?\n\n3. **On the Algorithmic Structure (Decoupled vs. Coupled)**: You chose a 'decoupled' two-stage design (Learn Reps -> Classify), which is highly interpretable. However, many successful SSL algorithms (like EM) are 'coupled,' iteratively refining label-estimates and model parameters. What are the theoretical limitations of your decoupled approach? Did you explore or consider a more coupled, end-to-end model where the $TF^{\\phi}$ and $TF_{sup}$ stages are interleaved at each layer?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. **Theoretical Quality**: The appendices provide a rigorous \"proof-by-construction.\" The mathematical links between specific attention mechanisms and algorithmic steps (like $Attn_{linear}$ for power iteration) are non-trivial and well-executed.\n\n2. **Empirical Effectiveness**: The method is shown to be highly effective, outperforming strong baselines, which validates the authors' algorithmic choices.", "weaknesses": "1. **Lack of Justification for Algorithmic Complexity**: The paper jumps directly to a sophisticated, iterative, non-linear SSL algorithm (Laplacian Eigenmaps). It doesn't first establish why this complexity is needed. The theoretical argument would be much stronger if it included an analysis (or even a citation to existing work) demonstrating that simpler, non-iterative (e.g., single-layer) or linear models fundamentally fail at this task.\n\n2. **Specialization of the Spectral Method**: The choice of the Laplacian $\\hat{\\mathcal{L}}$ (via $Attn_{rbf}$) is a very specific one, tailored to non-linear manifolds. This is a strong design choice, but it's presented without robustly comparing it to simpler alternatives. A major question is whether a linear spectral method (e.g., PCA) would be sufficient. Given that the $TF^{\\phi}$ module itself uses $Attn_{linear}$  (which is known to be related to matrix power computation), a standard model might more naturally learn to compute powers of the data covariance matrix ($X^\\top X$) rather than the complex $\\hat{\\mathcal{L}}$. The paper fails to discuss the trade-offs of its specific, non-linear choice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RSYMMKcazS", "reviewer": "Reviewer_w9xM", "summary": "This paper studies semisupervised in-context learning via two-stage Transformer framework. The first stage is to implement representation learning via computing discrete Laplacian and Eigenmaps. While the second is for in-context learning with additional prediction head for classification problems. Experimental results have shown that this two stage framework achieves the best performance compared to the others.", "questions": "1. What is $f$ in Eq(1)? It seems that the representation of each raw feature $x^{(i)}$ is independent of others $x$'s, which contradicts the nature of attention and ICL, where representation $f$ at token $i$ should depend on prior examples $(x^{(j)},y^{(j)})_{j<i}$. Otherwise, it is not aligned with standard ICL.\n2. Why in Eq(4), the loss is calculated on unlabeled data given that $y^{(i)}$ is unknown? Also, could the authors clarify on this and the Line 175 \"When training, this loss is fit to the n−m unlabeled samples (for which we have labels when training).\"?\n3. Could the author provide more details on the meaning of $w_y-\\mathbb{E}(w|f_\\ell)$ in Eq (6)?\n4. In Eq (7), the left contains an expectation symbol but its expression does not include any expectation operation. Could you clarify on this?\n5. Is the $\\gamma$ in Eq  (7) equal to the parameter set {$w_c,c=1,2,\\cdots C$}?\n6. What does $\\ell$ represent in Section 3.2?\n7. Since E2E is treated as feature exactor, have the authors considered replacing it with other feature extraction methods, such as using pretrained DNN models?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper focuses on a realistic problem, semi-supervised learning, where a large amount of data may lack labels.\n2. This paper proposed method that assigned different functional roles to different Transformer blocks, eg, representation extraction and ICL+categorial mapping. Such separation may provide insights into understanding the roles of different Transformer layers.", "weaknesses": "1. In Line115, the authors claimed \"we are unaware of any prior semi-supervised implementations.\" However, there are numerous prior works that have explored semi-supervision ICL. For examples,\n> [Agarwal et al. 2024] Many-shot in-context learning\n>\n> [Li et al., 2025) When and How Unlabeled Data Provably Improve In-Context Learning\n>\n> [Chen et al., 2025] Maple: Many-shot adaptive pseudo-labeling for in-context learning.\n\nwhile there are many more relevant works. Therefore, I am surprised that the authors are unaware of any prior work on this topic.\n\n2. In Lines 200-212, the authors construct the input $X$ from original dimension $d\\times n$ to $(d+n)\\times n$ where $d$ is the input feature dimension and $n$ is the sequence length. It is impractical and raises several issues:\n- In sequence models, the input length $n$ is typically changing over time;\n- It is computationally inefficient, especially when $n\\gg d$. However, modern large models are typically capacity of long input sequences, and $n\\gg d$ is common in practice. \n- According to Appendix A.1, due to such input construction, the number of model parameters have increased from $d\\times d$ to $n\\times n$, which greatly increases the computational and memory cost.\n3. Some notations lack clarity. For examples, what are the dimensions of $\\phi^{(i)}$ in Line 233, $\\varphi(\\phi)$ and $A$ in Line 237?\n4. Due to the Weaknesses 2, the authors could only test on the fixed in-context sample size $n=100$.\n5. While the authors have shown that OROIG+E2E-ICL achieves the best performance, this method require more computation and larger model parameters to train. Such comparisons may be unfair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)