When and How Unlabeled Data Provably Improve In-Context Learning
Authors: Yingcong Li
Xiangyu Chang
Muti Kara
Xiaofeng Liu
Amit Roy-Chowdhury
Samet Oymak
Venue: NeurIPS 2025 poster ()
Keywords: In-context learning
attention
semi-supervision
Gaussian mixture model
Primary area: general_machine_learning

Abstract: Recent research shows that in-context learning (ICL) can be effective even when demonstrations have missing or incorrect labels. To shed light on this capability, we examine a canonical setting where the demonstrations are drawn according to a binary Gaussian mixture model (GMM) and a certain fraction of the demonstrations have missing labels. We provide a comprehensive theoretical study to show that: (1) The loss landscape of one-layer linear attention models recover the optimal fully-supervised estimator but completely fail to exploit unlabeled data; (2) In contrast, multilayer or looped transformers can effectively leverage unlabeled data by implicitly constructing estimators of the form $\sum_{i\ge 0} a_i (X^\top X)^iX^\top y$ with $X$ and $y$ denoting features and partially-observed labels (with missing entries set to zero). We characterize the class of polynomials that can be expressed as a function of depth and draw connections to Expectation Maximization, an iterative pseudo-labeling algorithm commonly used in semi-supervised learning. Importantly, the leading polynomial power is exponential in depth, so mild amount of depth/looping suffices. As an application of theory, we propose looping off-the-shelf tabular foundation models to enhance their semi-supervision capabilities. Extensive evaluations on real-world datasets show that our method significantly improves the semisupervised tabular learning performance over the standard single pass inference.

When and How Unlabeled Data Provably Improve In-Context Learning
Yingcong Li
Xiangyu Chang?
Muti Kara²
Xiaofeng Liu'
Amit Roy-Chowdhury"
Samet Oymak'
UUniversity of Michigan University of California, Riverside
Bilkent University
ANJIT
Abstract
Recent research shows that in-context learning (ICL) can be effective even when demonstrations have missing or incorrect labels. To shed light this capability, we examine a canonical setting where the demonstrations are drawn according to a binary Gaussian mixture model (GMM) and certain fraction of the demonstrations have missing labels. We provide a comprehensive theoretical study to show that: (1) The loss landscape of one-layer linear attention models recover the optimal fully-supervised estimator but completely fail to exploit unlabeled data; (2) In contrast, multilayer or looped transformers can effectively leverage unlabeled data by implicitly constructing estimators of the form Σi (X"T)) with X and y denoting features and partially-obsere
labels (with missing entries set to zero). We characterize the class of polynomials that can be expressed as a function of depth and draw connections Expectation Maximization, an iterative pseudo-labeling algorithm commonly used in remi-supervised learning. Importantly, the leading polynomial power exponential in depth, so mild amount of depth/looping suffices. As an application of theory, we propose looping off-the-shelf tabular foundation models to enhance their semi-supervision capabilities. Extensive evaluations on real-world datasets show that our method significantly improves the semisupervised tabular learning performance over the standard single pass inference.
Introduction
In-context learning (ICL) is intriguing capability of modern language models and has enjoyed remarkable empirical success (Brown et al., 2020; Min et al., 2022). This success is also being extended multimodal scenarios (Zhou al., 2024) well as other modalities such as tabular data (Hollmann et al.. 2022). The push toward test-time scaling and long-contex, models (Snell al., 2024; Guo al., 2025) has further boosted the benefits of ICL by allowing the model to ingest a large number of demonstrations. For instance, in "Many-shot in-context learning" paper, Agarwal et (2024) demonstrate that pushing more examples into context window can substantially boost the accuracy. MAPLE (Chen et al., 2025) improves many-shot ICL by pseudo-labeling high- impact unlabeled examples and incorporating them into the prompt. The many-shot ICL setting naturally raises the question of when and how ICL can succeed with weaker supervision. As we can harness longer context models to boost predictive accuracy, we may indeed run out high-quality demonstrations with verified www.wrrschin--o--hooogeoo and may want to utilize weaker data sources. This motivates our central question:
Q: When and how can transformers learn context from unlabeled data?
We primarily investigate this question under a semisupervised ICL (SS-ICL) setting with Gaussian mixture models (GMMs). Formally, given a prompt containing a dataset of feature-label pairs (x) E Rd x R as demonstrations and query feature x (see Eq. (3)), a model trained for ICL
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

learns to predict the corresponding output y given prompt. For ICL with a supervised binary GMM model, we have x; y I and y, é [-1,1), [n]] and the component means μ+1 that parameterize the classification task are sampled from prior task distribution. This prompt model well studied under various fully-supervised settings (Garg al., 2022; Von Oswald al., 2023; Ahn et al., 2023; Akyürek et al., 2023; Mahankali et al., 2024; Collins et al., 2024; Shen et al., 2024) where each demonstration includes a clearly labeled output. In our SS-ICL setting, only m out of total samples have correct labels (m S n) either -1 or 1, and remaining labels are unknown and fed to the model as y; = O.
In this work, we provide a comprehensive theoretical and empirical study of attention models with varying depths when trained with SS-ICL. Our analysis reveals the importance of depth: Despite being able to implement the optimal fullly-uperrvsed estimator, single-layer linear attention completely fails to leverage unlabeled examples. In contrast, deeper or looped transformer architectures can emulate strong semi-supervision algorithms, approaching the performance of the Bayes-optimal classifier as depth increases. Informed by the importance of depth/looping, we also devise semisupervision strategies for tabular foundation models. Our specific contributions are:
Landscape of one-layer linear attention ($3): We study the optimization landscape of single- layer linear attention for the SS-ICL problem under an isotropic task prior. We prove that the global minimum of the loss function returns the plug-in estimator (see Eq. (SPI), i.e., $ = sgn(x' A) with ft XT y, where X nn represents features and y E R" denotes partially-observed labels (with missing entries set to zero) of the ICL demonstrations. This implies that 1-layer model learns Bayes-optimal classifier in the sully--upe setting, but completely fails to make use of unlabeled data.
Depth is crucial but shallow can suffice ($4): We show that multilayer linear attention can emulate semisupervised learners by implementing polynomial estimators of the form
a XXXx)'xx> 1=0
(1)
Crucially, an L-layer (or looped) attention can express up to K = 0(3²) powers, highlighting that logarithmic depth suffices represent high-degree monomials. We provide characterizations of the set of expressible polynomials through different constructions (where each layer gets to update the features or labels of the previous layer). Corroborating these, experiments reveal that shallow models with L 2 2 already achieve strong results and their performance can be approximately predicted through an eigen-estimator combining i 0 and (see (SSPI-K)).
What learner attention emulates? In Section 4.3, we describe how each attention block can update the label estimates by emulating pnpecctationmmxxmmiztne propagation (for softmax attention). For instance (1) can be interpreted the model implicitly conducting an aopeetation-Maximiannn XTy, each term XXTXX""> can be viewed as a sequence of pseudo-labeling (expectation) 9; Xp:-1 and training (maximization) û;; = XT;; steps. Corroborating this, we show that softmax- attention and feftmax-transformen models similarly benefit from increasing depth and can emulate semisupervised learners competitive with Bayes limit (see Fig. 2c).
(for linear attention) or belicf'
algorithm: Starting with the supervised estimator f.o
Applications to Tabular FMs ($5): Tabular foundation models such as TabPEN (Hollmann et al., 2022, 2025), TabICL (Qu et al., 2025) and TabDPT (Ma et al., 2025) represent suitable application of theory as they also model the ICL examples with a single token. To harness unlabeled examples, propose novel strategy that iteratively creates soft pseudo-labels by explicitly looping the tabular FM while controlling validation risk. Focusing on the shw-shot learning setting where TabPFN-v2 (Hollmann et al., 2025) excels, we demonstrate that our approach can significantly improve predictive performance on various real-world datasets.
1.1 Related Work
Theoretical Analysis of In-Context Learning Recent work has developed theoretical frameworks for understanding in-context learning in transformers. Akyürek et al. (2023), Von Oswald et al. (2023) and Dai et al. (2023) demonstrated that transformers emulate gradient descent during ICI. Xie et al. (2022) offered Bayesian perspective, while Zhang et al. (2024) showed transformers learn linear models in-context. Ahn et al. (2023) established they implement preconditioned gradient

---

descent, and Mahankali et al. (2024) proved one- gradient descent is optimal for single-layer linear attention. Multiple works (Li et al., 2023; Yang et al., 2024; Li et al., 2024; Bai al., 2023; Shen et al., 2024) studied the generalization capability of transformers. However, these exclusively focus on fully-supervised settings, leaving a critical gap in understanding how transformers handle partially labeled data a common real-world scenario. Our work addresses this gap by providing the first theoretical characterization of semi-supervised in-context learning. Wang et al. (2024) considers a setting where the model observes demonstrations of the form (query, response,, reward,) and aims to correct its response based on the reward sequence, Our work has a different focus as highlights that the model can correct/impute the missing labels using implicit feedback from labeled demonstrations.
Semi-SSpeevised Learning Traditional semi-supervised learning (SSL) aims to leverage unlabeled data to improve classifier performance. For linear classifiers, Oymak & Gulcu (2021) character- ized self-training iterations and demonstrated rejecting low-confidence samples; further theoretical analyses of anl--rainiingpssudo-llabbbe
cover deep networks (Wei et al., 2020). For Gaussian Mixture Models (GMMs), Lelarge & Miolane (2019) quantified maximal improvement from un- labeled data, while Krishnapuram et al. (2004) developed graph-based priors. Learning GMMs via Expectation Soppecttton-Maximizzll Maxxmization (EM) or pseudo-labeling
especially with few labels, is well-studied. Ratsaby & Venkatesh (1995) provided early PAC-style bounds for GMMs learned from few labeled and many unlabeled points. Balakrishnan et al. (2017) offered further statistical guarantees for EM. Nigam et al. (2000) demonstrated empirically that EM (viewable as iterative pseudo labul-ig Xu et al. (2024)) with pseudo-labels significantly reduces text classification error using unlabeled documents. These foundational works, with ongoing research areas like agnostic learning (Kwon Caramanis, 2020) underpin many SSL concepts. While these works established fundamental principles, they did not consider how these concepts apply to in-context learning with transformers. A most recent concurrent work (Liu & Yang, 2026) makes a similar observation to ours, showing that softmax attention approximates an EM estimator in sem-supervised ICL setting, but with a different focus on the underlying model and data regime. Our contribution bridges this gap by showing how transformer depth enables effective utilization of unlabeled examples within the prompt, essentially implementing semi-supervised learning without parameter updates.
2 Problem Setup and Preliminaries
We study ICL in the setting of semi- semi-supervised classification, where the in-context demonstrations are drawn from a binary Gaussian mixture model (GMM). We begin by introducing the following core notation: Denote the set (1, 2,...... ...) as [n] and use bold letters, such as and X, to represent vectors and matrices, respectively. Let Q(-) function return the right tail of the standard normal distribution.
[1. x 0
We use sgn(.) denote the sign function which is defined as follows: sgn(x)
(-i,
x x<0
2.1 Semi-supervised Data Model
Consider a d-dimensional semi-supervised binary GMM with n examples (x,, where x, e Rd denotes the feature vector and y; E (-1,0. IJ represents the corresponding observed label, with y; = indicating a missing label, and each label revealed independently with probability p E [0, 1]. Specifically, the data is generated follows (for each i e [n]):
(y 00
w.p. P w.p. 1 1-p
w.p. 1/2
x; = yi + Y1
and
y - w.p. 1/2
(2)
-1,
Here a U denotes the task mean, which is sampled uniformly from the unit sphere, and &: NNO, is the random noise with 2 O being the noise level that controls the variability of x; around its mean. yf denotes the true class label that is uniform over (-1, 1). Observe that p = corresponds to fully-supervised learning and 0 corresponds to fully-nnsupervised learning.
2.2 In-context Learning and Linear Attention
We build on the setting of (Garg et al., 2022; Mahankali et al., 2024; Zhang et al., 2024; Li et al., 2024) and construct the in-context prompts with examples drawn from the model (2) follows.
3

---

Prompt Generation Given task vector f ) we sample (n+ 1) in-context demonstra- tions )1 | according to (2) and construct the prompt
X1 y1
X yn
x
E ER
RuHlxd++1)
y2
0
We will investigate training a transformer such that given Z as prompt, it correctly predicts the label y = Ynei of the query x := .x+t1 through ICI.
Model Architecture Our work primarily focuses on training of linear attention models. Given any prompt Z R(+lxx(++1 which can be treated sequence of (d + 1)-dimensional tokens, the linear attention mechanism outputs
at a)) (ZW, WT Z" MZWWW
(4)
where W == (Ws, W, W, E R( denotes the set of the key, query and value weight matrices. Therefore, given the prompt matrix E ((++1xid+1) as input, the attention mechanism outputs a (n 1) leegth sequence (i.e., at) E R(n+lxdd+)) Note that the label for the query is excluded from the prompt Z. Similar to Ahn et al. (2023), we consider a training objective with mask
I 0
M = examples treated equally and that the model remains invariant to their order/position, do not apply a causal mask following Ahn et al. (2023). In contrast, Li et al. (2025) explores the use of causal masking in multi-layer linear attention and analyzes its impact on the final prediction.
0 0
to prevent input tokens from attending to the queries. To ensure that all in-context
Building upon the single-layer linear attention mechanism of (4), we can extend our model to multiple layers to capture more complex patterns. Consider optimizing an L-layer linear attention model and let Ze be the input of /th layer, & ε [L]. Additionally, let W. := (War, Wae, Wie E R(d+1x(++) be the corresponding weight matrices of 7th layer. Then, recalling the attention mechanism (4), the input prompt of ?th layer is defined by
Z, = Z + att(Zz-i:
for 2
(5)
and Z = Z. We focus on the next-token prediction setting, where the model makes prediction based on the final query token [x" OJ Let IRd+1 denote the linear prediction head. We define the output of the 1-llaeer linear attention model at the last (query) token
(Z) h² att Zz; WL)
Recalling the sign function, the predicted label for x given by Yauz sgn(fau-L())
Model Training With our attention-basee based architecture established, we now turn to the training procedure and evaluation metrics. Consider the ICL setting where each input prompt Z (cf. (3)) corresponds to randomly sampled task vector μ Unif(3 and let E(-) R - R be the loss function. Additionally, define the set of attention weights W U We (Rid+1)dd+1)1 The objective of L-laaer linear atention takes the following form:
min ((WLL) h) W
where
Lai-(WW h) B [[ fmuz())
Here, y yf- and the expectation subsumes the randomness of μ and (E. The search space for Whl is (Rid+Ixxdd+ and for h is Rd+1
3 Loss Landscape of One-layer Linear Attention under SS-ICL
Previous work (Ahn et al., 2023; Li et al., 2024; Mahankali et al., 2024) has shown that an optimized single-layer linear attention implements form of preconditioned gradient descent over the linear in-context demonstrations provided within the prompt. However, to the best of our knowledge, prior studies have not addressed the semi-supervised setting, where some in-context labels are missing. In this section, we analyze the optimization behavior single-layer linear attention under the semi- supervised binary GMM setting described in Section 2, and demonstrate that the single-layer model learns the optimal fully-supervised learner, but fails to utilize the unlabeled data.
We begin with the following optimal supervised label estimator under our problem setting.

---

Supervised Plug-in (SPI) Estimator The plug-in method is classical approach for supervised classification problems, aiming find linear combination of features that separates different categories. Under our problem setting. it also serves as the asymptotically Bayes-optimal estimator given only labeled data (Hastie et al., 2009; Devroye et al., 2013). Consider the binary cemi-super GMM problem described in (2) with dataset y (x,,)E and let I C [n]] represent the indices of labeled samples, c.g., Yi * for E The SPI estimator returns the task mean
A
Y.
(SPI)
We next present the following theorem establishes that, under isotropic task prior, optimal single-layer linear attention is equivalent to the SPI estimation.
Theorem Let the prompt (ef. (3)) be generated described in Section 2.2. Consider the objective (cf. (7)) with L = and squared loss function (ly. (y 9)² and denote the optimal prediction as Yau (2). Let ft, represent the SPI estimator defined in (SPI). Then, for any Z from (3), we have
Yan- (Z) sgn(x' pts).
(8)
Additionally, its classification ervor obeys
1 Egg
P(y an-1 (2) + y) = =
(9)
o ((1 + Eg8)
+ &
10de
$ e
+ e + e1160
where we define = 0 upp and X3 defines chi-squared distribution with degrees of freedom.
The proof of Theorem deferred to Appendix Eq. (8) shows that one-layer linear attention model indeed implements the optimal supervised predictor, assuming access to np labeled examples. Therefore, the classification error corresponds exactly to that of the SPI estimator. The supervised classification problem has been extensively studied (Bartlett et al., 2006; Belkin et al., 2018; Mon- tanari et al., 2019; Thrampoulidis et al., 2020; Chatterji & Long, 2021; Cao et al., 2021; Wang & Thrampoulidis, 2022; Deng et al., 2022), with most existing work focusing task in asymptotic data or overparameterized regimes. contrast, within the ICL framework consid- ered in our setting, the task mean μ is randomly sampled, and the classification error is computed by averaging over random draws of Z, y, and μ. Accordingly, in (9), we express the error in a simplified form as expectation.
single classification
The experimental results in Figure 1 support Theorem 1, where dark blue circular markers represent the performance of the single-layer linear attention model, blue curves show the classification accuracy of the SPI estimator, and the red dotted curves depict the accuracy - P(y art_ (Z) # y) as computed from (9). The alignments of these curves empirically validate Theorem 1. Implementation details
and further discussion are provided Section 5. Based on these results, we reach the following conclusion:
I-layer linear attention learns optimal supervised estimator but doesn't 't benefit from unlabeled data.
As shown in Figs and Ic, when the number labeled samples (np 10) fixed, increasing the number of unlabeled examples (even up to 10000) has effect performance, as the dark blue markers remain at the same level.
At first glance. this may seem counterintuitie- the data is unlabeled, it still contains infor- mation about the classification feature. For instance, the mean of the data points carries relevant information, and one might expect the model to extract and leverage this for better predictions. This expectation particularly reasonable when a large amount of unlabeled data is available, as the sample covariance matrix approximates the population covariance, i.e., E[XT X/n) 14p" + o where X = x2,, .x,]"' Raxd The key insight into why single-layer attention fails to leverage unlabeled data lies in the expectation structure. In our isotropic GMM setting where Unif(gd-I), the sample covariance matrix converges E|XT X/n] = Biuq²" + o² (1/d+ R which contains no task-specific information. The expectation across multiple tasks loses the signal from μ. This
5

---

0.84
0.80
Accuracy
0.75
L-ayer
1-4ays
24ayur
2layerr
2ayy
0.80
Slayer
0.70
layer
Flyyer
5R
SH
5R
5311-1
SSP13
0.78
55R--
Thermem
0.76
Tagorem
10 20
30
40
50
20
40
60
80
100
10
10-
10a
10
of labeled samples
of cn-nontext samples
# of in-context samples
(a) 50, p (0. 1J.
(b) np 10. (10. 100 (c) np 10. (10, 10000).
Figure 1: Experimental results support our theoretical findings presented in Sections 3 and 4. all three subfigures, blue, green, and orange markers represent the results of 1-, and 5-layer linear attention models, respectively. The SPI estimator (cf. (SPI)), SSPI-1, and SSPI-oo (ef. (SSPI-k)) solid, and green dotted curves, respectively. layer/SPI results described in Eq. (9) of Theorem I, while the black dotted line in Fig. Ic corresponds to Eq. (13) of Theorem 2. Additional details and discussion can be found in Sections 3, 4, and 5.
shown blue solid, green dotted curves in all subfigures correspond
single-
explains why single-layer attention, operating in a meta-learning framework across many tasks rather than optimizing for a single fixed task, cannot extract useful information from unlabeled data.
In the following section, study multi-layer linear attention and demonstrate that it has the ability to propagate X into deeper layers, thereby enabling the model to utilize the unlabeled data.
Multi-layer Attention and the Benefits of Depth
In this section, explore how deeper attention models can effectively utilize the unlabeled data. Let xn E axd
X x=
and
[y
yn E R..
(10)
4.1 I-laaer Linear Attention can Implement Degree-O(34) Polynomials in x
We first present the following propositions show that multi-layer well looped linear attention can expressed as a polynomial function of X X. This structure allows the models to leverage unlabeled data to improve the estimation of the task mean μ.
Proposition 1 Given an L-layer linear atlention model described in Section 2.2 with inpul prompt Z defined in (3), can construct the key, query, value weight matrices and the linear prediction head such that the model outputs (ef. (6))
x
(11)
Then, the following A matrices are achievable via label and feature updutes:
Label propagation: = c [[ (I CCXTX) for arbitrary constants (C,
Feature propagation: A xX 3-11-1 3 for an arbitrary constant c.
Proposition 2 Consider the same setting in Proposition There exists a single-layer linear attention model whose parameters can be constructed such that, when looped I. times, its output reproduces that of (11), with Ct = c' for some arbitrary constant e'.
The proofs of Proposition and are deferred to Appendix C.1 and C.2. In the following, we provide further clarification on the label and feature propagation.
The final prediction of the label propagation process can be rewritten as
(Z) ex X y where ye+1 (I + cxxXXXX yy: for LE [E[--
with yi y. Here, Yr can be interpreted as the soft pseudo-labels input to the éth layer, and each is parameterized by the attention mechanism in the corresponding layer. Although not

---

exactly equivalent, the L-layer linear attention process shares similarities with the Expectation- Maximization (EM) algorithm for sup-rsuiee learning, with L iterations of pseudo-labeling and different label update strategy.
2. In contrast, the feature propagation process yields the final prediction
(Z) = XLy where Xc.1 XX; and X/+1 = (X, X for E [L
with X1 = X and x1 = x. Here, xx) can be viewed as the input features at the 4th layer, encoding exponentially higher-order powers of X X. This result highlights that linear attention model requires only Ollog K) layers to represent polynomial functions of degree K.
Our construction for label propagation inherently related to the gradient descent emulation capability of linear attention Ahn et al. (2023). However, the feature propagation construction fundamentally different and underscores the transformer's capability implement rapid power iteration over the empirical covariance X X. In the above constructions, each attention block with residual connections updates features or labels using one parameter, namely mappings of the form X + X + xxX X y y -BXX"". The lemma below shows that, even if the multilayer model can express polynomials of X X with exponential degrees depth, the expressible manifold of polynomials has dimensionality linear in depth.
Lemma (Label + Feature Propagation) For an L-layer linear attention model, the resulting eventual prediction corresponds to the matrix A in Proposition I of the form
31/2
A
Σ 7a0
axxx"xx
(12)
The coefficients := [Ao di 433-33/2 lie on manifold of dimension at most 21. as can be expressed as g(e) for some smooth function g :R2L R33-31/2 with representing the parameters of individual layers.
4.2 Which Semi-supervised Algorithm Does Multi-layer Attention Approximate?
Recall the SPI estimator A, from (SPI), and that y denotes the visible labels defined in Section 2.1 and (10). We have ùú = | X"y. Motivated by Proposition 1 that multi-layer linear attention can implement higher-degree polynomials of XT we introduce the following SSPI estimator, which makes predictions based on the supervised estimate û, combined with higher-order debiased term of the form (X (X"X/n- I)
Semisupervised Plug-in (SSPI) Estimator Observe that the feature covariance satisfies E|XT X]/n = =1pT" 1 and the top eigenvector of the centered covariance matrix (X"X/rr X/n o² r) asymptotically aligns with either -u. Therefore, with substantial amount of unlabeled data, we propose the semisupervised plug-in (SSPI) estimator follows:
A at; + (1 (X X/n- Xjn-o' -'D
(SSPI-k)
where ft the SPI estimator (cf. (SPI)), and [0, 1] controls the trade-off between the fully- estimators. The optimal choice of depends on the problem
supervised and semi-supervise parameters n, /l and p. Note that as & - co, the term X/n- converges (up to scaling) to a rank-one projection onto the top eigenvector of the debiased covariance matrix, effectively serving as an estimator for (up to sign).
In Figure I, we present the prediction accuracies of 2-layer and 5-layer linear attention models, shown by green and orange markers, respectively. We also evaluate the SSPI algorithm with varying values, where the green solid curve corresponds to SSPI-1, and the green dotted represents SSPI-co, both using their respective optimal choices of Details on selecting the optimal a values are provided in Section A. and illustrated in Figure 2. The results reveal close alignment between multi-layer linear attention and SSPI estimators. Notably, the 2-layer model outperforms SSPI-1, due to its ability to implement higher-degree polynomials of X X (cf. Proposition and Equation (12)). When the sample size is sufficiently large (e.g., n 50 Figure 1b), the top eigenvector provides a more accurate estimate of the task mean, enabling SSPI-co to achieve higher accuracy. Furthermore, since
7

---

the 5-layer model is capable of representing higher- order functions than the 2-layer model, it can better estimate the top eigenvector, resulting performance that closely matches that SSPI-co.
In the following, we analyze the optimal classifier of the form sgn(x' Ap.) for a GMM, and provide insights into its behavior in the asymptotic regime as
Theorem Consider binary GMM defined in Section 2.1 and suppose that (x,,))) is generated using a fixed following (2). Given matrix A Bexd, define prediction
AA = sgn(x' Aî.))
where As is the SPI estimator defined in (SPI). Let म* := min PÔA # y) be its optimal solution Then, un" E A.. Additionally, obeys
PÔpn # y) Q(1/0) + Q vпp/o) -2(())))))))))) vnplor).
(13)
Rayes error
Note that, P0 # y) depends on np and only, regardless of μ and d.
Theorem Let the prompt Z be generated as described in Section 2.2. and consider an L-layer linear attention model with L 2 2 and n = co. Additionally, let fg be the SPI estimator defined in (SPI). There exist model constructions such that for any Z following (3), its prediction satisfies
Yau- = sgg(xx' pp fts).
The proof follows directly from Proposition (label propagation), which shows that multi-layer linear attention can output x (XT X/n Da As n its expectation, i.e., X X/n 021 - μup The results Figure I validate Theorem 3, showing that becomes large enough (i.., attention models, as well as the SSPI-1 and SSPI-co estimators, closely align with the classification error characterized in Theorem 2, depicted by the black dotted line.
the empirical covariance converges to
10000), the predictions from both 2-layer and 5-layer linear
Theorem establishes that, with infinitely many unlabeled samples, an L-layer linear attention model (for L. 2 2) can implement the predictor characterized in Theorem 2 using the optimal choice of A, thereby achieving the classification error specified in (13). the following, we shift to the non-asymptotic setting where is finite and analyze the model's performance in this regime.
Theorem Let the prompt Z. be generated described Section 2.2. Consider an L-layer linear attention model with L 2 2 and denote its optimal prediction as Yau- (Z). Additionally, let fas be the SPI estimator defined in (SPI), Suppose that the number of labeled samples satisfies np 2 8do2 and > O(d) is sufficiently large. Then, there exists universal constant C such that the classification error satisfies
Ply (2) # y) e
Cydnn
+ eedd
The proof is deferred to Appendix C.5. Note that when > d. the classification error approaches the Bayes error, i.e., P(y y) x @(1or).
4.3 Multi-layer Attention Expectation Maximization and Belief Propagation
In Section 4.1, we discussed how multi-layer linear attention can express polynomial functions of X. Here, we further explore the connection between multi-layer attention and the Expectation Maximization (EM) algorithm for semi-supervised learning. Beyond linear attention, we also highlight key differences between linear and softmax-based attention mechanisms, particularly in how they implement labeling strategies analogous to those in the EM algorithm.
Consider the following construction of the (-th layer attention weights:
la
0 0
[o W,
W W
and
0

---

We examine both linear and softmax attention mechanisms. Let S(-) denotes the softmax operation that applies on the rows of a matrix. With this, the data update defined in (5) becomes:
Linear attention:
Ye+t = yc + yt
Softmax attention:
Me+1 = yr )
In the case of linear attention, given the pseudo-labels yr 11y 16. th..,, estimates the task mean using the SPI algorithm (cf. (SPI)) as fDr = y,. each pseudo-label through the residual rule:
y%1 at layer e, the model
yr. The attention then updates
y cxx Ar.
where C₁ is a layer-specific coefficient. Owing to the linearity of this mechanism, the resulting pseudo-labeling strategy aligns with linear EM-style update.
In contrast, softmax attention computes pairwise similarities via the softmax of dot products. Define then each pseudo-label is updated via:
A' sij Ene 87zj
f yf CE Σ siNT
This update is nonlinear, memilarity-weightee pseudo-labeling strategy which can also be viewed as belief propagation. The nonlinear nature highlights a key distinction between softmax and linear attention in how they emulate EM-like updates (Sij xx x, for linear attention).
5 Experiments
In Sections 3 and 4, we introduced Figure 1 and demonstrated its consistency with our theoretical results. In this section, we describe the experimental setup and implementation details. Motivated by Proposition 2, which suggests that looping can help leverage unlabeled data, Section 5.1 introduces an algorithm based on the TabPFN, showing how it can enhance prediction performance by incorporating a small amount of unlabeled data and iterative pseudo-labeling through model looping. Additionally, we present further empirical findings to investigate additional questions of interest in Section A.
Experimental Setup Following Section 2, set d 10 and noise level o = 1. All models are trained using Adam optimizer with learning rate of 10-3 for 40,000 epochs, with a batch size of 512. We use logistic loss in our experiments. Since our study focuses on the optimization landscape and model expressivity, and experiments are implemented via gradient descent, we repeat 10 trainings from random initialization and results are presented as the maximal test accuracy among those 10 trails.
5.1 Tabular Experiments
To investigate how model looping (Proposition can improve label prediction, we propose the LoopTabFM algorithm that addresses unlabeled data by iteratively assigning pseudo-labels. More details of the algorithm are deferred to Section A.2 and Algorithm 1.!
We evaluated the effectiveness of our proposed looping strategy by iteratively applying TabPFN-v2 on real-worke binary classification benchmarks used in Hollmann al. (2025). The results summarized in Table 1, where each entry represents an average over 100 random splits of the dataset, with 80% of the data used as the test set in each split.
For each experiment, we randomly sample 10 labeled and unlabeled examples, ensuring that the labeled set includes at least one example from each class. As baseline (Loop-O), we apply TabPFN- v2 using only the labeled data. The corresponding test accuracies are reported the "Loop-0" column of Table 1. We compare this to models updated through up to k < 5 iterations of pscudo-label update, with results shown in the "Loop-k" columns. The final column reports the relative improvement (Rel. Imp.) over the baseline. Our results demonstrate that the looping strategy can significantly improve test accuracy. For instance, on OpenML datasets 1049, 1464, 40701, and 40983, accuracy improves by more than 10% over the baseline using only 10 additional unlabeled samples. The last row of
OOu code is available at https: :/gi thub
water/loooarrrrr

---

Table I: Comparison of test accuracy (%) between the baseline (Loop-O) and LooPTabPM (Algorithm after I iterations using TabPFN-v2 Each result is averaged over 100 random trials. The highest test accuracy for each dataset is highlighted in bold. The final column reports the relative improvement (%) of Loop-5 over the baseline, computed as (Loop-S 5 Loop-0) (oop-0)/Laa (LLooooxx00000 Positive signs indicate performance improvement over the baseline, while negative signs indicate performance drop.
OpenML
features # of samples
Class imbalance
Loop-0
Loop-1
Loop-2
Loop-3
Loop-4
Loop-5
Rel. Imp. (%)
3196
1.09
58.6
58.97
0.60 (+)
31
20
1000
2.33
66.18
65.95
66.05
65.58
65,52
65.07
1.68 (-)
1049
37
1458
7.19
72.00
75.62
79.48
80.31
81.49
81.40
13.06 (+)
1067
2109
5.47
73.12
76.59
77.94
77.92
78.57
78.60
7.50 (+)
1464
4
748
3.20
60.46
63.96
70.20
71.29
72.26
72.18
19.38 (+)
1487
82.54
(+)
1489
5
5404
2.41
66.40
67.62
68_30
68.14
68,21
68. 18
2.69 (+)
1494
1055
1.96
62.24
63.05
64.62
94
66.05
6.12
40701
20
5000
6.07
66.45
70.65
75.99
78,18
78.00
77.70
16.93 (+)
40900
36
5100
67
98.53
98.41
98.39
98.39
98.27
98.26
0.28 (-)
40981
690
1.25
73.56
74.41
74.67
74,93
74.94
188
40983
5
4839
17.34
79.71
85.04
89.36
92.94
92,90
92.75
16.35 (+)
64.64
0.76
41144
259
3140
1.01
50.70
50.63
50.68
50.67
50,71
50.77
0.14
41145
308
5832
56.16
56.28
56.21
56,24
56.19
56.22
0.12
41146
20
5124
71.26
73.90
75.39
75.84
76.02
77.07
8.51 (+)
41155
48
4147
3.03
67.74
69.78
70.64
71.82
71.72
71.74
5.90 (+)
Average
68.84
70.76
72.35
72.96
73.24
73.21
6.35 (+)
the table reports average performance across datasets, revealing that the majority of performance gains occur the first two iterations. This observation aligns with our synthetic experiments using multi-layer models (Figure 1), where the improvement from 1-layer to 2-layer is substantially greater than the improvement from 2-layer to 5-layer. These findings highlight that explicitly looping the tabular foundation model to iteratively refine soft pscudo-labels of unlabeled data using only few iterations can substantially enhance performance.
As shown and discussed, our LoopTabFM algorithm enhances model performance. However, this improvement not consistent across all datasets. For example, performance drops on the OpenML datasets with IDs 31 and 40900. This may be attributed to factors such as noise levels in the raw data, class imbalance, or other dataset-specific characteristics. In contrast to our synthetic experimental setting, where the model is pretrained in a meta-leaming fashion on the distribution of the given dataset, TabPFN used as general-por general-purpose pretrained foundation model and applied directly to target datasets in single-shot inference setting. Prior work (Ye et al., 2025) has also shown that TabPFN can be sensitive to input length, which may further affect performance consistency. Despite these limitations, experiments with TabPFN offer an initial insight into how unlabeled data and iterative looping can be leveraged to improve predictive performance. These findings suggest promising future directions, such as designing data-aware looping algorithms that adapt to dataset-specific properties.
Discussion and Limitations
Our paper introduces theoretical study of semisupervised in-context learning and characterizes how transformer, specifically linear attention, models can harness unlabeled data in their context window make inference. We show that depth is crucial to go beyond supervised estimation and utilize unlabeled data, and the latter is achieved by constructing estimators of the form f = ΣK aX' Xy'XTy. y. log K depth suffices express a Kthh order polynomial which is in line with synthetic and real experiments that corroborate that mild amount of depth/looping already achieves most of the benefit. Our core theoretical results are limited to linear attention models and it is important to understand the capabilities of the full transformer architecture. Indeed, transformer (MLP+softmax) empirically outperforms a linear attention model with equal number of layers, well approximating the Bayes optimal semisupervised estimator. It would also be exciting to go beyond the classification setting and examine how self- eeff-generated CoT rationales, as in (Wu et al., 2023), can enhance ICL capabilities for tasks that require proposed LoopTabbMM algorithm demonstrates that iteratively pseudo- -abelln unlabeled data can indeed enhance predictive performance for tabular tasks. However, there remains significant potential for developing more intelligent, data-specific algorithms that more effectively leverage unlabeled data to further improve model performance.
Additionally, our
10

---

Acknowledgements
This work was supported in part by the National Science Foundation grants CCF-2046816, CCF- 2403075, CCF-2008020, the Office of Naval Research grant N000142412289 and by gifts/awards from Open Philanthropy, Amazon Research, and Google Research.
References
Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, et al. Many-shot in-context learning. arXiv preprint ar wwxv::244..10011 2024.
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36, 2023.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL WRLLhtts///ppeeeeiieee www.:rwwettefoormmm
0g0X4H8yN4I.
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems, 36:57125-5 255-57211, 2023.
Sivaraman Balakrishnan, Martin J Wainwright, and Bin Yu. Statistical guarantees for the em algorithm:
From population to sample-based analysis. The Annals of Statistics, 45(1):77-120, 2017.
Peter L. Bartlett, Michael Jordan, and Jon McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-156, 2006.
Mikhail Belkin, Daniel J Hsu, and Partha Mitra. Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. Advances in Neural Information Processing Systems, 31, 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yuan Cao, Quanquan Gu, and Mikhail Belkin. Risk bounds for over ver-parameterized maximum margin classification on sub- aab-gaussian mixtures. Advances in Neural Information Processing Systems, 34: 8407-8418, 2021.
Niladri S Chatterji and Philip M Long. Finite-sample analysis of interpolating linear classifiers in the overparameterized regime. Journal of Machine Learning Research, 22(129)):-30, 2021.
Zihan Chen, Song Wang, Zhen Tan, Jundong Li, and Cong Shen. Maple: Many-shot adaptive
pseudo-labeling for in-context learning. arXiv preprint arXiv:2505. 16225, 2025.
Liam Collins, Advait Parulekar, Aryan Mokhtari, Sujay Sanghavi, and Sanjay Shakkottai. In-context learning with transformers: Softmax attention adapts to function lipschitzness. arXiv preprint
arXiv:2402.11639, 2024.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can GPT learn in-context? language models secretly perform gradient descent meta-optimizers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Findings of the Associa- fion for Computational Linguistics: ACI. 2023, pp. 4005-401 Toronto, Canada, July 2023. URL
Association for Computational Linguisties. doi: 10.1 10.18653/v1/200 (665)//1/023.fnninns-aees org/2023. findings- acl 247.
Zeyu Deng, Abla Kammoun, and Christos Thrampoulidis. A model of double descent for high- dimensional binary linear classification. Information and Inference: A Journal of the IMA, 11(2): 435-495, 2022.
11

---

Luc Devroye, László Györfi, and Gábor Lugosi. A probabilistic theory of pattern recognition. volume 31. Springer Science & Business Media, 2013.
Shivam Garg, Dimitris Tsipras, Perey Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Trevor Hastie, Robert Tibshirani, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume Springer, 2009.
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: transformer that solves small tabular classification problems in second. arXiv preprint arXiv:2207.014 0184 2022.
Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045)3119333)) 2025.
Balaji Krishnapuram, David Williams, Ya Xue, Lawrence Carin, Mário Figueiredo, and Alexander Hartemink. On sumi-suee classification. Advances in neural information processing systems, 17, 2004.
Jeongyeol Kwon and Constantine Caramanis. The em algorithm gives sample-optimality for learning mixtures of well-separated gaussians. In Conference on Learning Theory, pp. 2425-2487, PMLR, 2020.
Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selection. Annals of statistics, pp. 1302-1338, 2000.
Marc Lelarge and Léo Miolane. Asymptotic bayes risk for gaussian mixture in semi-supervised setting. In 2019 IEEE 8th International Workshop on Computational Advances in Multi- -tetii-Senoor Adaptive Processing (CAMSAP), pp. 639-643. IEEE, 2019.
Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning, pp. 19565-19594. PMLR, 2023.
Yingcong Li, Ankit S Rawat, and Samet Oymak. Fine-grained analysis of in-context linear estimation: Data, architecture, and beyond. Advances in Neural Information Processing Systems, 37:138324- 138364, 2024.
Yingcong Li, Davoud Ataee Tarzanagh, Ankit Singh Rawat, Maryam Fazel, and Samet Oymak. Gating is weighting: Understanding gated linear attention through in-context learning. arXiv preprint arXiv:2504.04330 2025.
Renpu Liu and Jing Yang. Unlabeled data can provably enhance in-context learning of transformers. arXiv preprint arXiv:2601.1005. 2026.
Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Alex Labach, Jesse C Cresswell, Keyvan Golestan, Guangwei Yu, Anthony L Caterini, and Maksims Volkovs. Tabdpt: Scaling tabular foundation models real data. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Arvind V. Mahankali, Tatsunori Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. In The Twelfth International Conference on Learning Representations, 2024. URL https: httttt//////eeee https/////p/pe/e///e/e/ee/ www.:ewwwwwe.oorre/eee 8p3fu561Kc.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint 22X2 2022.
12

---

Andrea Montanari, Feng Ruan, Youngtak Sohn, and Jun Yan. The generalization error of max mar-in linear classifiers: nigh-dimensional asymptotics the overparametrized regime. arXiv preprint arXiv:1911.0154, 7, 2019.
Ojash Neopane. Lecture notes on high- digh-dimensional statistics. https://www.stt stat. cmu edu/ -arinallobTeachingg /270//66799/9 / S19/Scribed / Lecturreeeetttetseeeeeeeee Ojash pdf,, 2018.
Kamal Nigam, Andrew Kachites McCallum, Sebastian Thrun, and Tom Mitchell. Text classification from labeled and unlabeled documents using em. Machine Learning, 39(2-3):103-134, 2000. doi:
10.1023/A:1000692211333 13085.
Samet Oymak and Talha Cihad Guleu. A theoretical characterization of semi-supervised learning with self-training for gaussian mixture models. In International Conference on Artificial Intelligence and Statistics, pp. 3601-3609. PMLR, 2021.
Jingang Qu, David Holzmüller, Gaäl Varoquaux, and Marine Le Morvan. Tabicl: A tabular foundation model for in-context learning on large data. arXiv preprint arXiv:2502.05564 2025.
Joel Ratsaby and Santosh S. Venkatesh. Learning from a mixture of labeled and unlabeled examples with parametric side information. In Proceedings of the Eighth Anmual Conference Computa- tional Learning Theory (COLT '95), pp. 412-417. 1 ACM, 1995. doi: 0111145/22529.2255533
Wei Shen, Ruida Zhou, Jing Yang, and Cong Shen. On the training convergence of transformers for in-context classification. arXiv preprint 240.114100 2024.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 1lm test-time compute optimally can more effective than scaling model parameters. arXiv preprint 20ti:2400.0031144. 2024.
Christos Thrampoulidis, Samet Oymak, and Mahdi Soltanolkotabi. Theoretical insights into mul- ticlass classification: A high-dimensional asymptotic view. Advances in Neural Information Processing Systems, 33:8907-8920, 2020.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev. Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pp- 35151-35174. PMLR, 2023.
Ke Wang and Christos Thrampoulidis. Binary classification of gaussian mixtures: Abundance of support vectors, benign overfitting, and regularization. SIAM Journal on Mathematics of Data Science, 4(1):260-284, 2022.
Yifei Wang, Yuyang Wu, Zeming Wei, Stefanie Jegelka, and Yisen Wang. A theoretical understanding of self-correction through in-context alignment. In Advances in Neural Information Processing Systems, volume 37, pp. 89869-8991 2024.
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. arXiv preprint arXiv:2010.03622 2020.
Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? arXiv preprint arXiv:2310.08391, 2023.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning implicit bayesian inference. In International Conference Learning Representations, 2022. URL https: /ttp:///////////////e/ee ew. net www.tttoorurnidddddeeeet.e.
Moucheng Xu, Yukun Zhou, Chen Jin, Marius de Groot, Daniel C. Alexander, Neil P. Oxtoby, Yipeng Hu, and Joseph Jacob. Expectation maximisation pseudo labels. Medical Image Analysis, 94:103125, 2024. ISSN 1361-8415. doi: https://www http://wwww.ccinneieretettt htt.c/ideeeccoeeeeeeeee/eaeaaaaa
103 URL
Tong Yang, Yu Huang, Yingbin Liang, and Yuejie Chi. In-context learning with representations: Contextual generalization of trained transformers. arXiv preprint arXiv:2408. 10147, 2024.
Han-Jia Ye, Yi-g Liu, and Wei-Lun Chao. A closer look at tabpfn v2: Strength, limitation, and extension. arXiv preprint arXiv:2502. 2025.
13

---

Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49):1-55, 2024.
Yucheng Zhou, Xiang Li, Qianning Wang, and Jianbing Shen. Visual in-context learning for large vision-language models. In Findings of the Association for Computational Linguistics ACL 2024, pp. 15890-15902, 2024.
14

---

Appendix
Table of Contents
A Additional Experiments and Algorithmic Details
15
A.I Additional Observations
15
A.2 Algorithmic Details of Tabular Experiments
17
B Analysis of Single-layer Linear Attention
17
B.1 Supporting Lemmas
17
B.2 Proof of Theorem
20
C Analysis of Multi-layer Linear Attention
22
C.I Proof of Proposition
22
C.2 Proof of Proposition 2
24
C.3 Proof of Lemma
24
C44 Proof of Theorem 2
25
C.5 Proof of Theorem 4
27
0.80
0.8
0.8
0.75
0.6
SSR-1
0.6
S5H-1
0.70
I-kyer,
1..* SSR-x
0.4
La SSR
0.65
Ideyer, suttmax
S-laya, linaa
0.2
0.2
S-lyer, sottmex
0.55
Asymptotic Bayes
10 labeled samples
20
30
40
50
10'
10a
20 30
40
50
in-context samples
# of labeled samples
(a) 50, (0. 1].
(b) np 10, (10, 10000).
(c) 50, (0, 1].
Figure 2: Additional experimental results. (b): Analysis of the optimal ar values for the SSPI estimator (cf. (SSPI-k)) under varying (n, k). Green solid and dotted curves represent optimal values for SSPI-1 and SSPI-co, respectively. The SSPI results shown in Figure use the corresponding values from Figs. 2a and 2b. (c): Comparison of different model architectures for the SS-ICL problem. Dark blue and orange curves show results I-layer and 5-layer attention models,
solid dashed curves representing linear softmax attention, respectively. Cyan curves correspond to S-layer Transformers. The black dotted curve shows the asymptotic Bayes-optimal error (cf. Lelarge & Miolane (2019)). Results suggest the performance ordering: Transformer linear attention softmax attention. Further details provided Section 5.
A Additional Experiments and Algorithmic Details
A.1 Additional Observations
Exploration of Optimal a Values In Section 4, we introduced the SSPI-k estimator (ef. (SSPI-K)), but did not discuss the choice the mixing parameter a, which plays a crucial role in balancing the contribution of the supervised estimator fú, Specifically, a controls how much weight is given the purely supervised signal. the fully supervised case, the optimal choice is a 1, as f., corresponds to the optimal estimator.
In Figures 2a and 2b, we empirically examine the optimal values of a. Given μ Unif(Sd-1), we define the optimal as the minimizer of the following cosine similarity-based objective:
a := min L(a) where L(a) Efcoosine lari ty m)). [c[0,]]
15

---

Algorithm/ LoopTTabM:: Looping Tabular FM with Soft Pseudo-labels and Risk-aware Updates
Require: Dataset Diah Dunlah, looping iterations K
1: procedure LOoPING(DIa.. Dumaal K)
2: FMn - TabPFN-v2(Diaa)
+ FM, corresponds to model of Loop-k. Assign pseudo labels via fron FM Dunlab).
3: Dunlab < FMo(2unlab)
4: FMbest - FMo
5: Rval Gal__RiskDDunlal
6:
for Looping iteration & I.....K do
7:
FM, TabPFN-v2(Drab unlab)
8:
Dunlab - (2, unlab)
Update pseudo labels via groft - FM,(x E D Dunlab).
9:
if Val @al_RiskkDDnlaall < Ryal then
10:
FMhest c- FM,
11:
Ryl Val Val__Rssk(Dunbal
12: end if
13: end for
14: return FMoest
15: end procedure
16: procedure VAL (on__RssK(Dullll unlab)
17: return Dunal Σ, min( - 1/, 15,00 + 1l)
18:
groft corresponds to the assigned soft label for feature in Dunlab-
19: end procedure
For each setting, we optimize a using the Adam optimizer for 10,000 epochs with a batch size of 128 and learning rate of 0.01. The results are shown in Figs 2a and 2b.
In Figure 2a, for both SSPI-1 and SSPI-oo, the optimal a starts near zero when the number of labeled examples is small, reflecting the limited utility of fú, in low-supervision regimes. As the number of labeled samples increases, a grows approximately linearly and approaches I when the problem becomes fully supervised. In Figure 2b, when labeled), the optimal a begins at As n increases and the fraction of unlabeled data grows, a decreases significantly. This trend indicates that as the volume of unlabeled data increases, the SSPI estimator adaptively reduces reliance on the supervised component ft, and increases reliance on the semi-supervisee component, which leverages the structure of the unlabeled data through X X.
10 and p = 1 (i.e., all examples are
Comparison Across Different Model Architectures Beyond linear attention, investigate additional model architectures under our SS-ICL, setting. The comparison results are presented in Fig. 2c. The softmax attention model uses the same structure described in Section 2.2, with the only difference being the addition of softmax operation in Eq. (4). The Transformer model introduces further nonlinearity and capacity by incorporating multi-layer perceptrons (MLPs) and layer normalization. The Transformer experiments are conducted with 5-layer models.
When comparing weaker models-suce as 1 1-layer linear (dark blue solid) and softmax (dark blue dashed) attention we observe that softmax attention consistently underperforms linear attention. Notably, softmax attention fails to match the performance of the optimal supervised estimator, even when all labels are observed (i.e., when the number of labeled samples equals n = 50). Furthermore, increasing the depth of softmax attention (orange dashed curve for 5-layer softmax) still does not surpass the performance of 5-layer linear attention (orange solid curve). Among all architectures, the Transformer achieves the best performance due to increased model capacity and expressiveness. Compared with Fig. la, where the orange and dark blue markers (linear attention) are identical, the Transformer significantly improves accuracy. This improvement highlights that SSPI, while effective, is not the optimal semi-supervised estimator. Although our semi- rami-suervviedd setting assumes isotropic data, the characterization of its optimal algorithm remains an open and foundational problem for future exploration. In the figure, we also include the asymptotic Bayes-optimal curve (black dotted; derived from Lelarge & Miolane (2019)) As the number of samples increases, the results from linear attention, softmax attention, and Transformer all converge toward this optimal curve. We attribute the initial performance gap, particularly at low values along x-axis (e.g., np 1), to the scarcity of labeled data.
16

---

A.2 Algorithmic Details of Tabular Experiments
In this section, we provide additional details regarding the tabular experiments discussed in Section 5.1. We propose the LoopTabFM algorithm with its details outlined in Algorithm 1. Suppose that we are given labeled Lab and unlabeled Dnllah datasets. The overall workflow of the algorithm proceeds as follows:
Perform ICL using TabPEN on the labeled dataset Dlsb and treat the resulting model as the base model (Loop-0). The corresponding test accuracies are reported in Table I.
Base Model:
2. Pseudo-Label Assignment: Using the current model (e.g., Loop-k) to generate predictions for the unlabeled data Dunlab Assign soft pseudo-labcl based on these predictions. Note that the model outputs scalars (i.e., elements of R) and can be interpreted as soft labels.
3. Model Update:
prompt by combining the labeled examples with their true labels and the unlabeled examples with their assigned soft pseudo-labels. Perform ICL using TabPEN on this combined prompt to obtain updated model (Loop-(k 1)). Repeat this process from Step 2 until the maximum number of looping iterations reached.
Construct
Model Validation: To improve the stability of the looping process, we introduce an additional validation step and retain the model with the lowest validation risk the final (best) model. Specifically, after assigning soft pseudo-labed labels to the unlabeled data, i.e., Duntal (xx,, Dt0 Y21), compute the validation risk over these pseudo- labeled examples as follows:
Val Val_Risk(Dunall
_ Σ min (1s
1/, |st0n + 11).
n (e[M)
which penalizes predictions that deviate from confident binary labels +1.
B Analysis of Single-layer Linear Attention
B.1 Supporting Lemmas
Recap the SPI estimator from (SPI). Given semi-supervised dataset (x,,)); described in Section 2.1, let I denote the token indices set corresponding to the labeled demonstrations, that is, have
yr iEi otherwise.
y1
(14)
0,
Then, the SPI estimates the task mean via
Σ
A
7 y
TET
Let W Redd be the preconditioning matrix. We define the following objective:
W
arg min L(W) where L(W) WeRdla
E
x
W
y
(15)
Je.
Here, we (x. y) be the query feature and its corresponding true label. The expectation subsumes the randomness in (r), y) as described in Section 2.
In the following, we provide lemma that establishes equivalence between optimizing (WW() (), h) (cf. (7) and choosing L 1) and Z(W).
Lemma Consider ICL problem described in Section 2.2 with prompt defined in (3). Consider training single-layer linear attention with squared loss, thut is, L= 1 and (ly = (y Recall the objectives from (7) and (15), and let L* and L L(W*) ) be their corresponding optimal losses where W is defined in (15). Then, we have
L* I
(16)
17

---

Additionally, let fae -Rin-lpidd1)) - R denote the optimal prediction (associated with the optimal loss L ). We have that fan-1 is unique and for any prompt Z (cf. (3))
fon- (Z) = x W Σ
(17)
IET
Proof.
Recap the single-layer linear attention model and its prediction from (4) and (6). We have
fan-1 (Z) ht aat((: where att(Z:W) (ZW, W Z MZW,, with w == (... Ws. W.J being the set of the query, key and value matrices of the attention. Since W and h are tunable parameters, without loss of generality and for simplicity, let
(18)
W and := W.h.
Following the proof of Li et al., 2024, Proposition I, similarly, we denote
W
W1 and h= h
W
W
w
where We E Radd W1, W2, h E Rd, and w. h R.
Additionally, let I denote the token indices set corresponding to the labeled demonstrations (cf. (14)).
Recall the prompt Z from (3), and X = [x] ... xnl RRadd and Then get
lyi ... ya]" E R. from (10).
Z =
X2 Y2
Xx
X y
E
R
Ix+1))
(19)
Yn
Combining (18) and (19) together,
rewrite the one-layer linear prediction
fan-1
[x"
OJwz MZh
= [xx 0]
W
w
X
I
0
X
x
h
W
w
y
0
y
h
X
X
h
= [x W x [wi]
yTX y
h
XT y yT +
[x W x x[wil]
= WXX" Xh; + +hx x) + Xh; + hy y)
= -xT +
Xh; + hyyyyy yw1)
-xTXX y x (WXX Xh; + +mhw;
where W == hW + w and we define := [Z].
Next, recall the loss from (7) and consider the squared loss function, (6. = (y 893. We have
L hh) = E (Sam r>²
E (x WXy + x (WX Xh; m/w ->))
Bix [xxTWXy WXy yx (Wx Xh; + mhwi) 1)²
For simplicity and without loss of generality, we omit y and use to represent yx. Note that the distribution of (updated) x is not conditioned on its class and given mean vector it follows N(H, Of), Similarly, let x; represent y We can then write
x + xx (WX" Xh; + mhwi) -1]] )
( 1)) = B
(20)
TeT
+ E (x" (WX' Xh; + m//w ))²
Xi -
+ 2B
x W Σ
(x
(wx" Xh; + mmhw)))
iEl
18

---

We start with showing that for any given parameters W e R(d+l)dd h E Rd*1, the component
BI(x' W EiEr x; 1)(x (WXT Xh, + +mlww))) ))] O. To prove it, we first expand
(x W
xi 1)(x' (WX" Xh m/w
IeT
= (x W
xx)xx WX Xhi)
x' Xh (x W
Σ Ie
x Xmmxxx'' wi) mhxx w1.
(6)
(d)
(a)
(c)
In the following, we consider the expectations of (a), (b), (c), (d) sequentially, all of which take the value zero, First note that since Unif(3d-1) and (Ei) ~ NNO, the odd moments of μ. 5 and E., i E [n] are all zeros.
(a):
E
(x
W
Σ
xx) T W Xhi)
iEI
E
(1 5) W Σ
(1 + ED( +) W (t EN E:)Th)
IET
( + 5) W(H ED(H + E) W(μ + Ej(u + + B
ET Mnn
F WHr W(Hp +5,5 + E WHE W(pp +5,5) hi
IE Jall
0,
(b): E
T
W Xhi]
B
5) W Σ
( EN + 5.) h1
E
W
mh + 5.5. hh
sc[m]
0.
(c): E (x W Σ (mhxx wi)
=mhB
(1 + 5) W Σ ( E:4H + &)wwi
mh
B (14 E) WHip 5) wi
mh
B I Waee wi + E WHe wi]
le7
=0,
(d) E mhxx w1]
Therefore, loss in (20) returns
(WW) h)
W Σ
x,
+ [x (WX Xh; + +mhw )
TeT
Z(W)
19

---

Here, the first term B[(x' W EiEr x; 1)²] L(W where L(W) is defined in (15).
Recall that W hw wi h Then for any W Raxd, setting h1 Wi 0g and h returns
(x" [WX Xh; + m/w ))²
O, and then
Σ x |
Lan- L h) =B
Therefore, optimizing L (WAD) h) returns the same minima as optimizing L(W), which completes the proof of (16). Note that optimal loss Lan.11 depends the labeled data E I only.
Furthermore, since L(W) is strongly convex (see (21)), W exists and is unique. Therefore, (16) and uniqueness of W* leads to the conclusion (17).
Lemma 3 Consider the objective defined in (15) with semi-supervised data following Section 2. Then the optimal solution W* satisfies
W =c
for some c > O.
Proof. Recap the Objective (15) and its optimal solution W Let be the index set corresponding the labeled in-context examples, and [Z] m. Note that, m is also random variable, independent of x, yf. y.
As in the proof of Lemma 2, we use x to represent yx and x; to represent yf for simplicity, where (updated) x, ~ N(u, o"), Letting & E. o be independent, we obtain
((W) B
(x W
x; 1)²
(21)
ief
(11 E) > (1 Ei) 1)2
B
((H &) W(mu + Vmn') 1)?
E pF WH) m(H Wg') m² Wp) + m(5 w 1| 2B my Wy
B[m²] d(d + (tr(W)2 tr(ww tr(W2)) B[m m²)
o'tr(WWT)
d
+B + 2 2 tr(W).
Differentiating it results in
V Vw-L(W)= 2 [mm] d(d + 2)
+ 2 2 +
m²
W W 2.B(ml] _ I.
Setting Vw-[(W)= 0, obtain the optimal W
W
(I (1+o') Eiml + t
which leads to the conclusion that W cl, for (l Bimii > O. It completes the proof.
B.2
Proof of Theorem 1
Proof. Note that (8) can be easily proven using Lemmas and 3. Then, we focus on proving (9),
Given that (8) holds, we can rewrite its classification error as
P(y* m (2) # y) P(sgn(x") (f,) # y) (SsgnyyxTAA 1)
(22)
20

---

where f, = [7] Eier y;x; defined in (SPI) and I is the index set of labeled samples. Let m = [7]
Recall from Section 2.1 where u I2, We can rewrite
yx yx= where
81 ~ N(O.D.
Then for any given μ. A. we get
P F(sgn(xx' (.)) (..(i)) = + n, <0 u.
=P Ps <ggAss p..is)
0 o lliallle
(23)
Here O-function the tail distribution function of the standard normal distribution. Next, similarly, given that x; ~ N(Y: - u1, 12, for i I, we can rewrite
A
Σ tEI
yixi yax== + ++
82 Vm
where N)
Then combining (22) and (23), we have
f
P(y a (Z) # y)
By.s:
Q
1 ile.
1 (1 vm g2)
Bp.
Q
|
a
82
vm
1/2
1
g2
Vin
Bpst
Q
1 2 Vm
g2
g2
Note that for any a with ||pll = 1, we have n g2 N(O, 1). Therefore, can write
4=g2= g where N(O, I),
and let U e Radd be unitary matrix with first row being μ. We can write
lgaall.. = ||0 |Uggalf; = g² h where h h~ X3-1.
Here, X3-1 denotes chi-squared distribution with (d degrees of freedom. Then, get
+ g
Vmn
P(y an (Z) (Z)) # y) E Q
2 or
g
(g2
h)
Vmn
m
1
8
EgA
σ v (1
a g)
von
1 6v
Egg
(1
8) +
where o= Vm. It completes the proof of (9).
21

---

Next, we derive an upper bound for P(y (2) x)). c 6,'. Then we have
c +
P(y (2) * y)
(c + g)² +h
c+8
c+g g
=Ba--s f
0
B ggc f 2
Vc + g) +h)
o Lovic g)2
c+8
s B--- 1
e
+ Q((222
Vcc + g) +h
+ Q(c22),
(24)
or v h/(c g)2
where the inequality comes from the fact that P(g s -c/2) Q(c/2) and Q(x) s for any x E R. Next, have
h
2h
2 1
2 1
VI + h/(c + 8)²
(c g)2
Here the first inequality comes from that
_ 1 - fxx and the second utilizes that 2 - vi+s
Since h ~ d-1" from the Laurent-Massart inequality (Laurent & Massart, 2000), we have that
P p(h h d 1 ( 1)ri 21) s e
Therefore, we have that with probability at least e
2(d - 1 + 2 v(d
D) + 25))
h/(c g)²
²²
Setting d, get with probability at least 1 - e-d
10d
VT + h/(c g)
c²
Combining the result with (24), since Q(x) for xe and Q(x) s e'112 for x 1, we get that
10d
Ply att (Z) ()) ((cl((c2
1
-2
0
s sed - 0
(1- 10de?)
completes the proof.
C Analysis of Multi-layer Linear Attention
C.1 Proof of Proposition 1
Proof. We consider the following model constructions for the attention matrices in the /th layer, f e [L] and the final linear prediction head:
1
0
La
0
Eth layer:
W WT
and
W
0
0
b
(25)
0g)
Prediction head:
h
Suppose the input to 9th layer is
2,
X x
ye
E B (++1)xd++1
where 2,
Z
X
x
22

---

Recapping the model construction from (25), the 7th layer output returns
M) W.
X
la
0
X
X y1
M
X
yr
acls
0
x/
yr
yr
X
ye
bt
X,XX
Xixe]
I,
0
a
baye
x
x/
0
a
beye
aeX,XX y xx Xt b,x x
(26)
Therefore, following (5), the input of (f+ 1)th layer is
y axX,X; X
Ze+1 Ze
X b
X1 XXXXXX
ye+ b,X,X, Ye b,x
(27)
x + axx X
Label propagation:
We first focus on deriving label propagation results. Suppose that we have
d 0 for E [L].
Then following (26), the output of l'th layer takes the following form:
0
b,X,X;
W.. of Z M) W,
b X
Here, the first d coordinates of each token's 's output are zeros, and therefore, the corresponding input coordinates remain unchanged, and we have
X and x x for E [L].
The prediction (based on the last token output and after applying prediction head) is given by
fan- cb,x' x x yt.
(28)
We next focus on obtaining y2. From (27), we have
Yet1 xxX = (I + b,XX"
Therefore,
L-1 (1+ )y
y1
Combining with (28) results in
L-1 [ (1 b,xX cb,x'
= =Z)
X
I (I+ X)XTx.
completes the proof.
Feature propagation: We now focus on the feature propagation setting. In contrast to the label propagation, let us assume that
and b - 0* for f [L]]
The prediction (following (26), based on the last token output and after applying prediction head) is given by
(Z) Ab..x..T.va
(29)
We first obtain yı. From (27) (since b; 0), we have
Ye+1 yt Xx yı yt.
23

---

Therefore,
= for E [L].
Next, we focus on X2,x2 From (27), as - oo, we have
X(+1 X; + az XxX X; = X, a axx X
x/-1 = x + a X X =
axx;
Therefore,
X = 4 4 (X
/-1 XL-1)
= = X1 X1 (-2) t21
21 =aLia X1 (X 4 x 1
= - X(X X) _
and
x = -1-1 (X X1,-1)
= =aL-10 x L-2 (12) the
=a 1 x1-33 +
= - x x1 (Xxx) x²
Combining all together with (29), we have that
fan = cbixIX[N1
4-1
cbi.
x x
completes the proof.
C.2 Proof of Proposition
Proof. The proof follows directly by adopting the same model construction and proof strategy in Proposition 1, under the additional assumption that
at a and b, b for le [Z].
C.3 Proof of Lemma
Proof. the proof of Proposition 1, we showed how to derive the label and feature propagation results by restricting the construction either a; O (for label propagation) or (a: by 0) (for feature propagation). Here, we consider propagation process without imposing restrictions on the choices of (a, b,), and study the form of the final prediction returned by the model.
To avoid the notation conflict, we express the matrix in (12)
K
A =
(x x
and let Ico e2 e0-31/2 RK
24

---

Recall the same model construction used in the proof of Proposition 1, defined in (25). From (26), we have that
fat-1 = =cbx
where following (27), we have
Xe+1 X,(I + axx; Xt),
x/41 x x;
yes1 (I b,
At each layer, the operations performed are linear combinations and multiplications involving X; X and identity matrices scaled by the parameters (ar, bı). Thus, each coefficient of (X" Xy depends smoothly on the scalar parameters (ar, bt). From (26) and (27), we have that
fau-L(2) 1bbxx,XX,,0
(30)
= =cb xi a XL-
That is, in the final (Zn-L(() expression, the coefficients corresponding different degrees of (X", X)* depend on the model parameters cbę and (at, boy= which together have at most 2L - degrees of freedom. Let
[cb2 ai dt-1 bj b[-1]". This means there exists a smooth function 2L-1 - R such that: e g(c).
It remains to show that L-layer linear attention model can produce terms involving powers of X X up to degree (3 - 3)/2.
Let f(Z) be function that contains terms of the form x =x"X for various powers k. Define P((()) as the projection that extracts the highest degree present in f(Z). For example, P(x (I (X x)))) = 2. Then from (30), we have
P( (2)) = x(yx)
-X XL-I
(X x (X (_2x
=P(E X1-2 X (1-2))22)
=P(xT X1
(X (x X
=P(x[,(X x,
+3 (333)
=...
=P(x (X" X) 4...33433 x))
31
3/-1 +... + + 3
2
It completes the proof.
C.4 Proof of Theorem 2
Proof. Let ~ N(O, D) and rewrite yx = μ + of. For any matrix A E Ridd the prediction error of fA sgn(x" Ap,) given A returns
PfA #
(1)) (.) P(yx' Ap, <0 <0(pp) (.))
= =P((H + o&)T T Ap, < <0 (.))
= e μ¹ Ap,
(31)
||
25

---

For any A E B we can decompose it as
A 1.N.y
where 41 = = Ilulll, = 1 and u, = for any i j. Let di > O. Then, we get
H Ap, = 1
d
Σ uupl is
Arp uivi
d Ps..
(32)
Now consider ||AA,ll,, where we have
Aβ,
Σ Auv Auv.A. f
= AAu + Σ AuvT f..
Since u;, i # is orthogonal μ, Aij A, is orthogonal to Σ &u,,, a. Therefore, given |ludle for all i E [d], it obeys
IAp.ll = =lAiiurT
kuv Relle, (1rTA,))
1 a1 ApT (t.).
(33)
1=2
For simplicity, define
A(O.) > (2 2,'
1n2
where A(-) is function of 2j and (2,, v/)'s for 2 2, and we have
A(P,) 2 0 and A(- = A(p,).
Recall that a is the SPI estimator (cf. (SPD). Let [ m. We can write fa; = # 5 vm where N(0, I2
Using (31), (32) and (33), the classification error becomes
P(A # y) Bp.
AApales
r.A.
Ep, Q
(TT..)2 A(P.)
TTA.
v i
e
E.Ta. <0 0
+ Byjn,0
o (v PT. A(P.)
(oT + A(Pa)
First, note that for any x @(x) 0.5 Q(-x). Therefore, the optimal choice of v Rd that minimizes P(A # y) is contained within the set vi values that maximize P(v ft > 0). Let
"" := arg max,, P(r f. > 0). Given that A N(μ, /2/ml), we have that V cu for e 0. Let
c= and therefore, V" lt without loss of generality (since dj can be any positive scalar). Then we
obtain
min P(A # min Bp, AER
A
Viu fA.) A(P..)
26

---

Let f(ū,) be the probability density function of ft, Since f N(H, o(,,)²/ml) then it satisfies f(i,) 2 f((-)) for any A, 0.
(34)
Therefore, the classification error becomes
P(PA # PI = =)= S 1(
1 i
dît
((rT))² + A(pss)
[c
1 HT,
-p' p. ((T p2.) A(P.)) 1
+
da,
VHTHTA A.) + A(P,))
[
n:
J JRRD0
(s(a.)
((-A)-(-Aa))) VH i,,) A(pe))
R200
for μ p' > 0, which can easily done by choosing 2; for 2 2. Then we get A(2,) = O. Therefore, the optimal solution set A* defined in Theorem 2 satisfies:
Following (34), to minimize the enror, we need minimize 2
न (211p"" |21>
Combining all together, obtain
Sr
(f((,)) f(-)) - + f(-A,)dps
min P(AA # y) AcR
=
f
(1- (1-
f(a
f(ia)dfis
A220
AD
Vm
2
Q
2
e
Vmi
Vm
2
Vm
vm
+
Q
2@
completes the proof.
C.5
Proof of Theorem
Proof. Recap from Proposition 1. For any L-layer attention model with L 2 2, it can output
(Z) = x (X X
I'
(35)
Let
y sgn( (ant)()())
with (Z) defined (35). Then we have
Ply (Z) # y) P(O # y).
Therefore, in the following, we focus on upper- poper-bounding the classification error P(ý y) cor- responding to (35). Given that the optimal prediction under the form sgn(x" is given by Sep" == sgn(x" "hp" ft,)) (ef. Theorem 2), with its corresponding error presented in (13). To analyze the performance of 9, we study its difference from the prediction 9pp
To begin with, let gi = Eijo ~ N(O,I) and g = Σ Ei/o vn- N(O, D). For simplicity, let == X X/n o We get
A - X X - 0²1
Σ uh
+ Eip'" E. - T²1
uh
σ
[ug
+ gy ) + σ²
Σi= 8:8
vn
27

---

Recall (31) from the proof of Theorem 2. Our goal is to bound
Ep o P Ap,
PÕ # y)
Define
E
gig.
A A-A4p u1
(ug VnI
+ gp
+ o²
1
(36)
n
From the Laurent-Massart inequality (Laurent & Massart, 2000), we have that with probability at least 1 - e (assuming 2 d). the first term of (36) can be bounded by
glgll
ug gH"
6
(37)
Additionally, from Neopane (2018), we have that with probability at least - e (assuming f2 2 d), the second term of A (ef. (36)) is bounded by (with universal constant
Σ gig
1
2
V
(38)
Combining (37) and (38), we get with probability at least 1 - 2 (for d)
|| s Ci V where := 60 + Co
We also bound [|,,||| as follows. Let fa, = o vmg' N(u, ommI, similar (37), with probability at least 1 - e (assuming 2d s 13 S m/40²), we can bound
f3
IA.11 1+
gg'll S 30
s 3.
m
Then consider a significantly large (to ensure that ||||| = 1/12, e.g., n 2 (12C1)²1). With probability at least 3e min(t.s)) and suppose that 4 n'As "As 0.5, we can bound
μ¹ Aps ||A llepe Pulle,
H 1"pp" a.
H up IA, [I(A + un sAllle
1 up P.s lleyee pulles
H TTA,
min(I(A + -ape" Dille, lleee Pulles)
||-|| |ل
#r pt. ||A|I MAAll
5 A|| I|A
C2 where C=:= 12C,.
Now, we are ready to bound the classification error, where we get
1
AΦ,
Bp
σ
1 Aps
"Tu paue" its
Bp
Q
||Ap,lls
papafe Pulles
C2 vijn
$ P(4
ft 0.5)
e
+ 3e min((.s))
+P f., 0.5)
C2 Vi/n
+ o Vm
3e min(tss)
20
28

---

Choosing /= 13 2d, since m/40² 2 2d, we obtain
C2 v2d/n
P( # 5) @
3e
+ 0.5e
C2 v2d/n
$ o
t
completes the proof.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Our assumptions underlying the algorithm and their necessity are discussed. Also, the limitation is discussed in the supplementary material.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiiatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
30

---

Answer: [Yes]
Justification: Our assumptions underlying the analysis and algorithm are discussed. Detailed proofs can be found in the supplementary material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All the information needed to reproduce the main experimental results of the paper are provided, either in the main paper in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
31

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have attached the code for implementing the algorithm and reproducing the experiments in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ cctppccic/pguuuucccccccccuccccce
/ocee//CCiuubbissee ss sssssnollici for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the information needed reproduce the main experimental results of the paper are provided, either in the main paper or in the supplementary material.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Detailed experiment results with errors included in the supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
32

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details can be found in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurrig: hctppucllcc/ttthccciidddaaaa
Answer: [Yes]
Justification: We have reviewed and confirmed that the research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
33

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cited the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwoee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
34

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code//modll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
35

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: This research does not involve LLMs any important components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy hatpss///nerris...oo /eearrppp.cc/ommfeeeeeee///2222 for what should or should not be described.
36


Reviews:
[{"review_id": "ONGLzPslaj", "reviewer": "Reviewer_9Hjq", "summary": "This paper provides a theoretical proof to understand how linear attention models leverage unlabeled data in in-context learning. The theory is demonstrated under the tabular foundation model setting.", "questions": "* Does looping TabPFN-v2 mean that using the output of one model run as input to the next run? I have a concern about the reusage of parameters. I believe this will deteriorate the performance as compared to running the same number but with different layers.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n* This paper provides a concrete proof of why unlabeled data benefits semi-supervised ICL and points out the key factor of model depth.\n* The experiment on tabular models demonstrates the effectiveness of the proposed looping strategy.\n\n**Weaknesses**:\n* The analyzed attention module is simplified as w/o softmax version (eq. (3)), differing from mainstream model settings. This may limit the application of the theory.\n* The experimented dimension is only set to 10 (Line 285). Why not try a high dimensionality?\n* There is a gap between the discussed task setting and the real datasets. For example, it's been widely proven that the attention with softmax outperforms the attention without softmax. However, Figure 2(c) reaches an opposite conclusion. Therefore, the content of this paper may not directly apply to standard Transformer-structured models.", "was_revised": true, "final_justification": "The rebuttal is clear and convincing. It addresses my concerns and questions. Therefore, I have decided to raise my rating.", "rebuttal": "We thank the reviewer for recognizing our theoretical contributions. We address the comments below.\n\n> *W1:  The analyzed attention module is simplified as w/o softmax version (eq. (3)), differing from mainstream model settings. This may limit the application of the theory.*\n\n**Response:** In this work, our goal is to analyze the exact optimization behavior of attention mechanisms. Note that 1) attention captures token dependencies through pairwise similarities; and 2) even linear attention induces a non-convex landscape. Therefore, its theoretical analysis is already a challenging and meaningful problem. Prior works (Ahn et al.(2023), Mahankali et al.(2024), Li et al.(2024)) and ours begin with linear attention as a tractable starting point. In contrast, our work considers the semi-supervised setting and examines how model depth affects prediction performance. To the best of our knowledge, this perspective has not been theoretically explored.\n\nWhile our analysis does not directly cover softmax attention, we empirically compare linear attention, softmax attention, and full Transformers (softmax attention + MLP) in Fig. 2(c). Under our GMM setting, softmax attention without the nonlinear MLP does not outperform linear attention. We provide further discussion and explanation in our **response to W3**, where we describe how **attention mechanisms can generally be formalized as a pseudo-labeling mechanism, applicable to both softmax and linear**. However emphasis is given to linear attention as we can establish concrete and nontrivial theoretical guarantees.\n\nLastly, although our theoretical results are based on linear attention, they motivate the idea of looping, which we successfully validate in realistic scenarios using real data and TabPFN. These results demonstrate that the theoretical insights carry over and provide practical benefits in more complex settings.\n\n> *W2:  The experimented dimension is only set to 10 (Line 285). Why not try a high dimensionality?*\n\n**Response:** First, since the GMM is a standard and generalizable data model, the dimensionality does not affect our theoretical results. To clearly validate our theoretical findings while keeping the computational cost low, we chose a representative dimension of $d = 10$. Notably, even with such a low dimension, it requires up to 10,000 in-context samples to empirically approach Theorem 2 (see Fig. 1(c), where the green curves approach the black dotted line).\n\nTo further address the reviewer's concern, we also conducted additional experiments at higher dimensions (presented in the table). The results are consistent with those in Fig. 1, where we observe that 1-layer=SPI=Theorem 1, and SSPI-1<2-layer<5-layer$\\approx$SSPI-$\\infty$. (Note: due to time constraints, each setting is evaluated based on a single run, which may introduce variance.)\n|Dimension ($d$)| # labeled ($np$) | # in-context ($n$) | 1-layer| 2-layer | 5-layer | SPI | SSPI-1 | SSPI-$\\infty$|Theorem 1\n|-------------------------|-------------------------|-------------------------|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|\n| 50 | 50 | 50 | 0.7602 | 0.7603 | 0.7602 | 0.7602 | 0.7602 | 0.7601 | 0.7602 |\n|  | | 100 | 0.7602 | 0.7696 | 0.7698 | 0.7603 | 0.7668 | 0.7698 |  |\n|  | | 500 | 0.7604 | 0.8167 | 0.8173 | 0.7602 | 0.8129 | 0.8188 |  |\n| 100 | 100 | 100 | 0.7599 | 0.7600 | 0.7601 | 0.7602 | 0.7602 | 0.7601| 0.7602 |\n|  | | 500 | 0.7602 | 0.7979 | 0.7980 | 0.7604 | 0.7954 | 0.7993 |  |\n\nIn addition, for the Looping TabPFN experiments, the model employs an embedding dimension of 192.\n\n> *W3: There is a gap between the discussed task setting and the real datasets. For example, it's been widely proven that the attention with softmax outperforms the attention without softmax. However, Figure 2(c) reaches an opposite conclusion. Therefore, the content of this paper may not directly apply to standard Transformer-structured models.*\n\n**Response:** \nWe thank the reviewer for raising this important point. While linear and softmax attention differ in their operational functions, they share similarities from a label propagation perspective due to their attention-based structure, and both can be interpreted as a pseudo-labeling algorithm. \n\nA generic attention mechanism with suitable value weights, regardless of linear or softmax, updates the labels via:\n$$y_i^\\ell\\to y_i^\\ell+c_\\ell\\sum_js_{ij}y_j^\\ell.$$\nHere, $s$ is the attention map: $s_{ij}=x_i^\\top W_\\ell x_j$ for linear attention, and is the softmax similarity between $i$th and $j$th token for softmax attention, both determined by the attention weights at layer $\\ell$, where $W_\\ell=W_{k\\ell}W_{q\\ell}^\\top$. From this, it can be seen that the first attention layer will impute the missing labels using labeled data based on attention similarities between labeled and unlabeled features. The future layers act as a label propagation mechanism that updates the labels by creating their data-dependent mixtures. We will revise the paper with this exposition to highlight how attention generically implements pseudo-labeling.\n\nWe agree that softmax attention is often better than linear attention for **real datasets** but it doesn’t have to be true for **all datasets**. Specifically, for the supervised GMM setting, a single linear attention emulates the Bayes optimal estimator as proven in the paper. This mostly explains why linear attention outperforms softmax attention in Fig 2(c). Although softmax attention underperforms linear attention in isolation, the full Transformer model (softmax attention + MLP) outperforms all other models. (See Fig. 2c). \n\nWe also emphasize the following \n\n1. Linear attention is widely used in theoretical study of transformers (see Ahn et al.(2023), VonOswaldetal.(2023), Mahankali et al.(2024)) and also underlies state-of-the-art architectures like gated-linear attention and Mamba2 (Li et al.(2024), Li et al.(2025)). \n2. GMM is a classical and established dataset model for the study of classification problems/algorithms.\n3. The benefit of increased depth for both attention types is evident, supporting our main theoretical claim that depth enhances semi-supervised performance.\n4. Our theory highlights the utility of looping, a mechanism we validate in real-world settings using TabPFN. These results demonstrate that the insights from our theory translate into practical benefits beyond the simplified setup.\n\nWe will include a subsection in our revised manuscript to incorporate this discussion.\n\n\n\n\n> *Q: Does looping TabPFN-v2 mean that using the output of one model run as input to the next run? I have a concern about the reusage of parameters. I believe this will deteriorate the performance as compared to running the same number but with different layers.*\n\n**Response:** Model looping is not a new concept and it has been widely explored and applied in various contexts (see citations below). It offers several advantages: it enables smaller models to handle more complex tasks, allows models to self-correct and refine outputs over multiple passes, and supports adaptability to varying input complexities by adjusting the number of iterations. \n\nThe primary benefit of looping is its flexibility and efficiency: In our TabPFN experiments, we did not retrain or redesign a deeper model. Instead, we used the same pretrained model and applied it iteratively in-context. This looping strategy improved performance without requiring additional model capacity or retraining from scratch. At a high-level, an autoregressive LLM can also be viewed as text generation by looping the same parameters. That said, we agree with the reviewer that our results can be enhanced by either (1) training a deeper architecture or (2) optimizing looped TabPFN for semisupervised learning.\n\n- Mohtashami, Amirkeivan, et al. \"Cotformer: A chain-of-thought driven architecture with budget-adaptive computation cost at inference.\" arXiv preprint arXiv:2310.10845 (2023).\n\n- Giannou, Angeliki, et al. \"Looped transformers as programmable computers.\" International Conference on Machine Learning. PMLR, 2023.\n\n- Yu, Qifan, et al. \"Enhancing auto-regressive chain-of-thought through loop-aligned reasoning.\" arXiv preprint arXiv:2502.08482 (2025)."}, {"review_id": "fTGJAdwHkW", "reviewer": "Reviewer_GrWH", "summary": "This paper investigates how transformers can leverage unlabeled data during in-context learning (ICL), a setting the authors term semi-supervised ICL (SS-ICL). The analysis is centered on a binary Gaussian Mixture Model (GMM) where some in-context examples have missing labels.\n\nThe core findings are: (1) A  single-layer linear attention model, while optimal in the fully-supervised case, completely fails to use unlabeled data. The optimization landscape leads it to an estimator that only considers labeled examples. (2) Deeper architectures are essential for semi-supervised learning. Multi-layer or looped transformers can utilize unlabeled data by implicitly constructing polynomial estimators. (3) Based on these insights, the authors propose a looping strategy for off-the-shelf tabular foundation models (like TabPFN) to improve their performance on tasks with few labeled examples.", "questions": "1. **Role of Softmax and MLPs**: Your experiments show that a full transformer outperforms the linear attention models. What is your hypothesis on the specific roles that softmax attention and MLP layers play in improving semi-supervised performance? Do they allow the model to learn a better polynomial estimator of $X^\\top X$ or do they enable a different, more powerful learning mechanism altogether?\n\n2. **Beyond Isotropic Priors**: Your proof for the failure of 1-layer attention hinges on the isotropic task prior, which makes $\\mathbb E[X^\\top X /n ]$ uninformative. How would your results change for an anisotropic prior (e.g., $\\mu \\sim \\mathcal N(0, \\Sigma)$)?  Could a single-layer model then leverage unlabeled data, or would depth still be required?\n\n3. **Connection to Pseudo-Labeling**: You compellingly argue that the label propagation process resembles iterative pseudo-labeling. Could you make this connection more formal?", "limitations": "Yes, the authors have adequately addressed the limitations of their work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength\n\n1. The question of how transformers use weak or unlabeled supervision is highly relevant, especially as context windows grow and high-quality labeled data becomes a bottleneck. This paper provides the first theoretical characterization of this important setting.\n\n2. The paper presents a sharp, insightful contrast between single-layer and multi-layer attention. The proof that single-layer models cannot use unlabeled data in this setting is a strong negative result, which makes the positive result—that depth enables semi-supervised learning—much more compelling.\n\n3. The theoretical insights directly motivate a practical algorithm. The proposed looping strategy for TabPFN is a clever and direct application of the theory that depth/iteration is key.\n\n4. The paper proposes a concrete mechanism for how depth helps: by constructing polynomial functions of the data covariance matrix. \n\n### Weakness\n\n1. The core theoretical results are derived for linear attention models. \n\n2. The analysis relies on an isotropic Gaussian Mixture Model where the task mean is sampled from a uniform sphere. This assumption is crucial for the argument that single-layer attention fails because the expected covariance $\\mathbb E[X^\\top X /n ]$ becomes uninformative. It's unclear how the results would change under more structured or anisotropic data priors.", "was_revised": true, "final_justification": "The paper is good and the rebuttal is clear. I vote for an acceptance.", "rebuttal": "We thank the reviewer for the insightful comments and suggestions, which we found valuable and have incorporated to improve the clarity and quality of the manuscript.\n\n> *W1: The core theoretical results are derived for linear attention models.*\n\n**Response:** We kindly refer the reviewer to our responses to **W1 and W3 of Reviewer 9Hjq** as well as **our response to Q3** below (where we establish a connection between linear and softmax attention) for a detailed discussion addressing this concern.\n\n> *W2: The analysis relies on an isotropic Gaussian Mixture Model where the task mean is sampled from a uniform sphere. This assumption is crucial for the argument that single-layer attention fails because the expected covariance $\\mathbb{E}[X^\\top X/n]$ becomes uninformative. It's unclear how the results would change under more structured or anisotropic data priors.*\n\n**Response:** We provide a detailed response and additional experimental results in our **response to Q2** below.\n\n> *Q1: Role of Softmax and MLPs: Your experiments show that a full transformer outperforms the linear attention models. What is your hypothesis on the specific roles that softmax attention and MLP layers play in improving semi-supervised performance? Do they allow the model to learn a better polynomial estimator of $X^\\top X$ or do they enable a different, more powerful learning mechanism altogether?*\n\n**Response:** Great question. We believe the improved performance of the full Transformer model comes from mechanisms beyond simply learning higher-order polynomials of $X^\\top X$.\n1. As noted in our Lemma 1, a 5-layer linear attention model can already express polynomial functions of $X^\\top X$ up to degree 120. Yet, full Transformers still outperform it (see Fig. 2(c)), suggesting that their advantage is not merely due to increased expressive power over $X^\\top X$.\n2. Attention mechanisms compute output based on pairwise similarities, and given input $X$, they operate as $att(X)=f(X^\\top X)$. In contrast, MLPs act directly on the input $MLP(X)=f’(X)$, and therefore do not improve the polynomial estimator of $X^\\top X$, but instead provide more flexible and nonlinear feature transformations.\n\nTherefore, we hypothesize that the MLP layers contribute through a more complex, nonlinear data projection mechanism that complements the attention module by capturing feature-level representations beyond pairwise relationships. Understanding the exact theoretical role of MLPs in this context remains an open and promising direction for future work.\n\n\n> *Q2: Beyond Isotropic Priors: Your proof for the failure of 1-layer attention hinges on the isotropic task prior, which makes $\\mathbb{E}[X^\\top X/n]$ uninformative. How would your results change for an anisotropic prior (e.g., $\\mu\\sim \\mathcal{N}(0,\\Sigma))$? Could a single-layer model then leverage unlabeled data, or would depth still be required?*\n\n**Response:** We thank the reviewer for this insightful question. First, when the task prior is changed to $\\mu\\sim\\mathcal{N}(0,\\Sigma)$, single-layer linear attention still fails to benefit from unlabeled data. This is because attention computes pairwise similarity as $x_i^\\top W x_j$ where $W=W_kW_q^\\top$. When $\\mu$ has covariance $\\Sigma$, the input data satisfies $\\mathbb{E}[xx^\\top]=\\Sigma+\\sigma^2 I:=\\bar\\Sigma$. This can be transformed into the equivalent isotropic case by: \n$$x_i^\\top W x_j=\\bar x_i \\bar\\Sigma^{1/2}W\\bar\\Sigma^{1/2}\\bar x_j^\\top,$$ \nwhere $\\bar x$ has identity (up to a scalar) covariance. Thus, the anisotropic setting effectively reduces to the isotropic case under a linear transformation, and the conclusion that single-layer attention fails to leverage unlabeled data still holds. The first row of the table below supports this where we fix $d=m=10$ and generate non-identity covariance using $\\Sigma=A^\\top A/trace(A^\\top A)$ with a random matrix $A$. Results show that performance remains unchanged with increasing numbers of unlabeled samples.\n\nMotivated by the reviewer’s comment, we further explored when single-layer models can benefit from unlabeled data. From our analysis in the proof of Lemma 2 (Supplementary Material), the prediction from a single-layer model depends on $x^\\top X^\\top X$ (assuming $W=I$). If $\\mathbb{E}[\\mu]=0$, then $\\mathbb{E}[x^\\top X^\\top X]=0$, and the unlabeled data does not help. However, if we use a prior with nonzero mean (e.g., $\\mu\\sim\\mathcal{N}(\\nu,I)$ where $\\nu\\neq 0$), the model can extract useful signal from $\\mathbb{E}[x^\\top X^\\top X]\\neq 0$. The second row in the table shows that under this setting, increasing unlabeled data improves single-layer model performance: (we set $d=m=10$)\n\n| | $n=10$| $n=50$ | $n=100$ |\n|-------------------------|-------------------------|-------------------------|-------------------------|\n|$\\mu\\sim\\mathcal{N}(0,\\Sigma)$| 0.7842 | 0.7842 | 0.7843 |\n|$\\mu\\sim\\mathcal{N}(\\nu,I)$| 0.8781 | 0.8928 | 0.8998 | \n\nThis is a valuable discussion, and we will incorporate these findings into our revised manuscript.\n\n> *Q3: Connection to Pseudo-Labeling: You compellingly argue that the label propagation process resembles iterative pseudo-labeling. Could you make this connection more formal?*\n\n**Response:** As discussed in Proposition 1 (Label Propagation) and the discussion that follows (Lines 216–221),   multi-layer linear attention can be viewed as an EM-like algorithm, where each layer corresponds to one iteration of pseudo-label refinement. Here, we provide a formal discussion as suggested by the reviewer, and we also refer to our response to **W3 of Reviewer 9Hjq** for further details. We will incorporate it into our revised manuscript.\n\nSpecifically, for attention mechanism (subsuming both linear and softmax), at the $\\ell$th layer, the pseudo-label assigned to the $i$th example is updated as follows\n$$y_i^\\ell\\to y_i^\\ell+c_\\ell\\sum_js_{ij}y_j^\\ell$$\nwhere $s_{ij}$ is the weighting strategy that determines how much different tokens influence the token $i$. For linear attention, $s_{ij}=x_i^\\top W_\\ell x_j$; for softmax attention, $s_{ij}=e^{x_i^\\top W_\\ell x_j}/\\sum_k e^{x_i^\\top W_\\ell x_k}$. This form resembles the update step in EM/label propagation algorithms (but with different update rules for different models), where the model iteratively updates predictions based on current pseudo-labels."}, {"review_id": "TJPNT9M9ee", "reviewer": "Reviewer_HQka", "summary": "This paper studies semi‑supervised in‑context learning, where only some in‑context examples have labels. Under a binary Gaussian‑mixture setting, the authors show that a single‑layer linear‑attention transformer reduces to the supervised plug‑in estimator and therefore ignores unlabeled data. By contrast, deeper or looped linear attention can implement polynomial functions of the sample covariance, effectively performing iterative pseudo‑labeling; only logarithmic depth is required to capture high‑degree terms. They introduce a semi‑supervised plug‑in (SSPI) estimator that blends the supervised estimate with higher‑order covariance corrections and show that multi‑layer attention approaches the optimum as unlabeled samples grow.", "questions": "1. Your analysis assumes a shared isotropic task prior for both training and test tasks. How sensitive are the results to this assumption\n\n2. Have you considered tasks drawn from non‑spherical distributions, or mismatched priors between training and inference?\n\n3. In Proposition 1 and Proposition 2, you construct weight matrices to implement polynomial functions of the sample covariance. In practice, are such weights discoverable by gradient‑based training?\n\n4. Can you comment on whether your depth‑induced pseudo‑labeling leads to the kind of “weak‑to‑strong generalization” seen in other semi‑supervised settings, where pseudo‑labels gradually correct errors and expand coverage as more iterations are applied? If so, can the SSPI framework help explain this behavior?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The paper addresses a realistic and relevant gap. Previous analyses of ICL focused on fully‑supervised tasks; this work is one of the first (to the best of my knowledge) to provide a formal framework for semi‑supervised ICL, where only a subset of demonstrations has labels. This setting arises naturally in many ICL scenarios with scarce labeled data.\n- Theorem 1 shows that a one‑layer linear attention model reduces to the supervised plug‑in estimator and thus cannot benefit from unlabeled data. The multi‑layer constructions (Proposition 1, Proposition 2, and Lemma 1) demonstrate that deeper attention can represent polynomial functions of the sample covariance and that only logarithmic depth is needed for high‑degree polynomials.\n- Experiments on synthetic mixtures verify that one‑layer attention does not improve with more unlabeled data, whereas adding a few layers substantially boosts accuracy and matches the SSPI predictions. The tabular experiments demonstrate a practical application of looping pseudo‑labeling.\n- By relating multi‑layer attention to iterative pseudo‑labeling and EM, the paper theoretically connects ICL and classical semi‑supervised learning, explaining why depth matters.\n\n---\n\n## Weaknesses\n\n- The theoretical analysis relies on an idealized setting: binary Gaussian mixture with isotropic covariance and task means drawn uniformly from the unit sphere. The assumption of a shared task prior across training and test tasks may not hold in more complex scenarios.\n- Although the paper shows that deep linear attention can implement polynomial estimators, the constructions rely on hand‑crafted weight matrices rather than learned weights. There is no analysis of whether gradient‑based training can reliably learn these polynomials, nor of how softmax attention or full transformers achieve similar behavior.", "was_revised": true, "final_justification": "The rebuttal and follow-up discussion addressed my earlier questions on prior misspecification, gradient-based learnability of the constructed estimators, and the connection between depth-induced pseudo-labeling and weak-to-strong generalization. The authors provided thoughtful theoretical clarifications, empirical checks for non-isotropic priors and OOD settings, and a clear mapping of their SSPI framework to observed behavior in iterative pseudo-labeling.\n\n### Resolved issues\n- Clarified robustness to non-isotropic priors and OOD settings, with supporting empirical evidence.\n- Explained the role and limits of gradient-based training in discovering the constructed weight configurations.\n- Linked depth-induced refinement to the weak-to-strong trajectory, with both theoretical and empirical support.\n\n### Remaining considerations\n- Analysis is based on an idealized Gaussian mixture setting, and broader applicability to more complex or non-linear regimes is not directly shown.\n- Empirical results focus on synthetic and tabular settings; it would be valuable to see extensions to richer domains.\n\nOverall, the paper makes a clear theoretical contribution to understanding semi-supervised ICL, backed by well-motivated experiments. I consider it a solid and relevant contribution to the field.", "rebuttal": "After reviewing the comments, we suspect that the reviewer may have inadvertently submitted a review targeted at a different paper (also on the topic of ICL),  as the content does not appear to relate to our submission. As a result, we were unable to prepare a rebuttal.\n\n\nIf possible, we kindly ask the reviewer to share any key concerns specific to our paper so that we can address them during the discussion period. We look forward to your feedback."}, {"review_id": "fFxGkUU8BZ", "reviewer": "Reviewer_qhJr", "summary": "This paper investigates how transformer-based models perform ICL in a semi-supervised setting where some demonstration labels are missing. The authors focus on analyzing the effectiveness of linear attention transformers at different depths. It finds that single-layer linear attention achieves the optimal supervised solution but fails to leverage unlabeled data while multi-layer or loop linear attention can exploit unlabeled data by approximating estimators of the form. Consequently, the paper proposes looping-based methods for tabular foundation models like TabPFN, which significantly improve semi-supervised classification accuracy.", "questions": "1.\tWhat is w.p in eq(1).\n\n2.\tEq (3): is Z the demonstration example or instruction without query x or with query x but without x’s label?\n\n3.\tIine 138, why input of layer l only contains attention part without ffn part?\n\n4.\tLine 142, I do not understand this question? The prediction of y is decided by res(attnZlast)+ffn(attnZlast+res(Zlast-1)) according to transformer block. \n\n5.\tHow sensitive is the performance of the looping-based TabPFN method to the accuracy of initial pseudo-labels? Would noisy pseudo-labels mislead the iteration? Does the portion between label, unlabel, psedu-label sensitive?\n\n6.\tHow should one determine the optimal number of loops in practice? Does it correlate with data properties like noise level or number of unlabeled samples?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n1.\tTheoretical Contribution: Provides the comprehensive theoretical characterization of semi-supervised in-context learning (SS-ICL) using linear attention models.\n\n2.\tPolynomial Estimator Characterization: Establishes that deeper models can compute higher-order polynomials of the empirical covariance, and thus emulate powerful iterative methods like EM or pseudo-labeling.\n\n3.\tUncovers the limitations of shallow models under isotropic priors, tying it to how expectation washes out task-specific signal.\n\nWeaknesses\n\n1.\tLimited to Model structure: theoretical results focus only on linear attention models. While the paper does include empirical comparison to softmax-based Transformers, no theoretical treatment is provided for these more widely used architectures.\n\n2.\tAssumption of Isotropic Priors: The analysis assumes task means are uniformly sampled from the unit sphere, which may not reflect realistic downstream data distributions.\n\n3.\tWhile effective, the looping approach introduces additional compute overhead, and the paper does not explore how to optimize this trade-off.\n\n4.\tLine 296-298 How to decide α is confused. does it need require domain knowledge or validation data to set optimally?\n\n5.\tSee Questions.", "was_revised": true, "final_justification": "Author's clarification resolved my questions. I keep my positive review for this paper.", "rebuttal": "We thank the reviewer for the detailed feedback. Below, we provide our responses to the comments.\n\n> *W1: Limited to Model structure: theoretical results focus only on linear attention models. While the paper does include empirical comparison to softmax-based Transformers, no theoretical treatment is provided for these more widely used architectures.*\n\n**Response:** We kindly refer the reviewer to our **responses to W1 and W3 of Reviewer 9Hjq, and W1 of Reviewer GrWH**  for detailed discussions addressing this concern.\n\n> *W2: Assumption of Isotropic Priors: The analysis assumes task means are uniformly sampled from the unit sphere, which may not reflect realistic downstream data distributions.*\n\n**Response:** We use the isotropic GMM to ensure theoretical tractability and to investigate the effects of model depth and unlabeled data in a controlled setting. While it may not capture the full complexity of real-world data, it serves as a principled and analytically tractable starting point for studying algorithms designed to improve semi-supervised performance.\n\nBuilding on the core conclusion of our work that deeper or looped models enhance semi-supervised learning, we proposed the Looping TabPFN algorithm, which effectively leverages unlabeled data to improve performance in practical settings. Thus, while our theoretical framework assumes isotropic priors for analytical clarity, the key insights extend to broader scenarios. We agree that further exploration under more realistic or structured priors is valuable and constitutes a promising direction for future work.\n\n\n> *W3: While effective, the looping approach introduces additional compute overhead, and the paper does not explore how to optimize this trade-off.*\n\n**Response:** We agree that the trade-off between accuracy and computational cost is an important consideration. While this work does not focus primarily on optimizing that trade-off, both our synthetic and TabPFN experiments show that only a small number of iterations (typically fewer than 5) are sufficient to achieve the best performance. In fact, the first few iterations contribute the most to performance gains, suggesting that the additional compute overhead is relatively modest given the improvement achieved.\n\nWe also refer the reviewer to our **response to Question of Reviewer 9Hjq**, where we further discuss the effectiveness and practicality of the looping strategy.\n\n\n> *W4: Line 296-298 How to decide α is confused. does it need require domain knowledge or validation data to set optimally?*\n\n**Response:** In our SSPI algorithm (Lines 246–247), the parameter $\\alpha$ is used under the assumption that its optimal value is known. In our experiments, $\\alpha$ is selected based on domain knowledge of the data distribution. However, determining the optimal value of $\\alpha$ in practice given only hyperparameters such as the distribution, $d$, $n$, and $p$ is a fundamental mathematical problem. We view this as an important direction for future theoretical investigation.\n\n\n> *Q1: What is w.p in eq(1).* \n\n**Response:** \"w.p.\" means \"with probability\".\n\n> *Q2: Eq (3): is Z the demonstration example or instruction without query x or with query x but without x’s label?*\n \n**Response:** In Eq (3), $Z$ includes the query $x$ (as the last token in $Z$) but without $x$’s label. This setting is consistent with prior work in the linear attention literature (Ahn et al.(2023), Mahankali et al.(2024), Zhang et al.(2023), Li et al.(2024)). \n\n>*Q3: Iine 138, why input of layer l only contains attention part without ffn part?*\n\n> *Q4: Line 142, I do not understand this question? The prediction of y is decided by res(attnZlast)+ffn(attnZlast+res(Zlast-1)) according to transformer block.*\n\n**Response (Q3&Q4):** In this work, our goal is to analyze the optimization behavior of the attention mechanism, and FFN will introduce additional nonlinearity and complicate theoretical analysis. To that end, we focus only on the attention block, excluding the FFN component and taking the output of the final attention layer as the model output for prediction. This setting is consistent with prior work on multi-layer linear attention (Ahn et al.(2023), Li et al.(2025)). We agree that studying the FFN is also important. As shown in our experiments (Fig. 2(c)), the full Transformer model (attention + MLP) achieves the best performance. However, we believe that a focused study on the attention mechanism, and in particular, the role of depth in attention, remains underexplored and is a necessary step toward a deeper understanding of the full Transformer architecture.\n\nNotably, this restriction applies only to our theoretical study. In our Transformer (Fig 2c, cyan curve) and TabPFN experiments, we do not impose any architectural constraints. This demonstrates that the theory-inspired looping strategy remains effective in practice, even when applied to more complex models.\n\n>*Q5:How sensitive is the performance of the looping-based TabPFN method to the accuracy of initial pseudo-labels? Would noisy pseudo-labels mislead the iteration? Does the portion between label, unlabel, psedu-label sensitive?* \n\n**Response:** Good point. To evaluate this, we examined the performance of TabPFN under varying levels of initial pseudo-label accuracy. The results, shown below, confirm that the quality of pseudo-labeled data plays an important role and higher-quality pseudo-labels lead to greater performance improvements. (Note: Due to time constraints, we conducted the experiment using a single data setting with a fixed labeled set. We randomly sample 100 groups of pseudo-labeled data and report the average relative improvement.)\n\n|Acc of pseudo-labels | 0.2~0.4| 0.4~0.6 | 0.6~0.8 | 0.8~1 |\n|-------------------------|-------------------------|-------------------------|-------------------------|-------------------------|\n|Relative improvement|  5.24% | 6.70% | 7.05% | 10.88% |\n\nRegarding the proportion of labeled, unlabeled, and pseudo-labeled data, our synthetic experiments (e.g., Fig. 1) show that increasing both labeled and unlabeled data consistently improves prediction performance. Motivated by the reviewer's question, we believe that more effective looping strategies could be developed. For example, by gradually assigning pseudo-labels to high-confidence unlabeled examples rather than labeling the entire unlabeled set. Such adaptive approaches may further enhance the performance of the looping method and represent a promising direction for future exploration.\n\n>*Q6: How should one determine the optimal number of loops in practice? Does it correlate with data properties like noise level or number of unlabeled samples?*\n \n**Response:** In our TabPFN experiments, we implement a validation risk criterion to monitor the variance of assigned pseudo-labels (see Lines 710–715 in the Supplementary Material), computed as\n$$Validation~Risk=\\frac{1}{n}\\sum_i \\min(|y^{pseudo}_i-1|,|y^{pseudo}_i+1|).$$\nIt is used to determine when to stop looping.\n\nWe also evaluated the optimal number of looping iterations with respect to different levels of initial accuracy, and observed an inverse correlation: lower initial accuracy tends to require more loops. However, due to the limited number of datasets and looping iterations, we are cautious about making a definitive conclusion. A more rigorous investigation is needed to establish this connection.\n\nAdditionally, we emphasize that based on both our synthetic results (Fig. 1) and TabPFN experiments, the number of loops (or layers) required for strong performance is relatively small (typically fewer than 5) with the initial iterations/layers contributing the most significant improvements."}]

Author rebuttal:

Meta-review: The paper provides a theoretical framework explaining how linear attention models leverage unlabeled data in a semi-supervised in-context learning setting. All reviewers were positive, recognizing the paper's theoretical contributions, particularly the insightful contrast between single-layer and multi-layer attention models. The authors' rebuttal successfully addressed all major concerns, including the limitations of the linear attention assumption, the role of deeper architectures, and the effectiveness of the proposed looping strategy on practical datasets like TabPFN. The consensus among reviewers is that the paper makes a strong, clear, and relevant contribution to the field, warranting acceptance.
Decision: Accept (poster)