Transformers with Endogenous In-Context Learning: Bias Characterization and Mitigation
Authors: Haotian Wang
Hao Zou
Haoxuan Li
Haoang Chi
Yang Shi
Yuanxing Zhang
Wenjing Yang
Xinwang Liu
Zhouchen Lin
Venue: ICLR 2026 Poster (2026)
Keywords: In-Context Learning
Hidden Confounder
Debiasing
Primary area: learning theory
TL;DR: This paper presents a pioneer theoretical analysis on the endogenous bias occurred in ICL prediction, with a higly-efficient debiasing method.

Abstract: In-context learning (ICL) enables pre-trained transformers (TFs) to perform few-shot learning across diverse tasks, fostering growing research into its underlying mechanisms. However, existing studies typically assume a causally-sufficient regime, overlooking spurious correlations and prediction bias introduced by hidden confounders (HCs). As HC commonly exists in real-world cases, current ICL understandings may not align with actual data structures. To fill this gap, we contribute the pioneer theoretical analysis towards a novel problem setup termed as ICL-HC, which offers understanding the effect of HC on the pre-training of TFs and the following ICL prediction. Our theoretical results entail that pre-trained TFs exhibits certain prediction bias with proportional to the confounding strength. To migrate such prediction bias, we further propose a gradient-free debiasing method named Double-Debiasing (DDbias) by collecting and prompting with extremely few unconfounded examples, correcting pre-trained TFs with unbiased ICL predictions. Extensive experiments on regression tasks across diverse designs of the TF architectures and data generation protocols verify both our theoretical results and the effectiveness of the proposed DDbias method.

Published as a conference paper at ICLR 2026
TRANSFORMERS WITH ENDOGENOUS IN-CONTEXT LEARNING: BIAS CHARACTERIZATION AND MITIGA-
TION
Haotian Wang', Hao Zou", Haoxuan Li Haoang Chi', Yang Shi', Yuanxing Zhang', Wenjing Yang', Xinwang Liu', Zhouchen 5.6.
"College of Computer, National University of Defense Technology Tsinghua University "Center for Data Science, Peking University *School of Computer Science, Peking University
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University 6 IInstitute for Artificial Intelligence, Peking University
ABSTRACT
In-context learning (ICL) enables pre-trained transformers (TFs) to perform few- shot learning across diverse tasks. fostering growing research into its underlying mechanisms. However, existing studies typically assume a causally-suffiiie regime, overlooking spurious correlations and endogenous prediction bias intro- duced by hidden confounders (HCs). As HC commonly exists real-world cases, current ICL understandings may not align with actual data structures. To fill this gap, we contribute the pioneer theoretical analysis towards a novel problem setup termed as Endogenous ICL (EICL), which offers understanding the effect of HC on the pre- pre-training of TFs and the following ICL prediction. Our theoretical results entail that pre-traue pre-trained TFs exhibits certain prediction bias with proportional the confounding strength. To mitigate such prediction bias, we further propose a gradient-fre free debiasing method named Double- Debbas-Dei (DDbias) by prompting the biased pre- pre-trained TFs with few unconfounded examples twice-once with the original label and once with residual, to yield unbiased ICL predictions. Extensive experiments on regression tasks across diverse designs of the TF architectures and data generation protocols verify both our theoretical results and the effectiveness of the proposed DDbias method.
INTRODUCTION
In the of Transformers (TFs)--uiddd) foundation models (Vaswani 2017, Dosovitskiy 2020
Gulati et al.] 2020 Achiam et al. 2023), In-context learning (ICL) (Brown et al.) 2020) offers gradient-free learning approach by prompting pre-trained TFs with few-shot samples, e.g., predicting on novel inputs x* given the sequences of pairs (xr, y) Garg et al. 2022). As ICL enables pre-trained TFs generalize well on novel inputs without tuning models, it becomes critical understand intrinsic mechanisms of ICL for developing more powerful and interpretable AI systems (Akyürek et al. 2022
Xie et al. 2021 Allen-Zhu & 2023).
Recent research has explored ICL through theoretical analyses (Ahn et al. 2023 Mahankali et al. 2023) Akyürek et al., 2022 Von Oswald et al. 2023) and empirical studies (Garg et al. 2022 Raventós al.] 2024: Panwar et al. 2023), often with stylized setups to examine its various facets (Garg et al. 2022; Panwar et al. 2023 Wang et al. 2023b). Especially, a prominent line work informs that the gradient-free inference of ICL is theoretically equivalent to performing implicit gradient descent (GD), i.c.. the feed-forwarding output of TFs with ICL prompting equals to the output of TFs updated by GD (Akyürek et al. 2022) Von Oswald et al. 2023 Cheng et al. 2023). Consequently, ICI. implicitly learns the input-output pairs exhibited in the prompt and predict on the novel input. Further studies have investigated the pre-conditions behind such equivalence by analyzing pre-trained TFs et al.] (2023) Mahankali et al. 20233 Cheng et al. 2023 Zhhng et al. 2024b), where most of such advances are built on the assumption of linear data generation.
1 Corresponding authors.

---

Published as a conference paper at ICLR 2026
However, the recent researches of ICL regression overlook the potential endogenous training & inference of ICL, i.c., the existence of hidden confounders between y and x (as pointed out by a concurrent work (Liang et al. 2024 ). Specifically, previous studies understanding ICL often assume the independence of the additive noise e and the predictors x: (when generating the label y -wTx x + (Ahn et al. 2023 Von Oswald al., 2023 ZZhang et al. 2024b By contrast, emerging empirical evidence reveals that the violations of this assumption (i.e., endogeneity) is prevalent across a variety of tasks(Cheng et 2022a Kumar al. 2019 Landeiro Culotta 2016 Wang et al. 2021 Huang et al., 2022] Wang et al. 2022 2223a Li et al. 2022; b), leading to the concurrent understanding of ICL becoming rather restrictive. Meanwhile, remains unexplored that the effect of the endogeneity on the pre-training process of TF's and the following ICL predictions. Therefore, two natural questions are intrigued below:
Will transformers pre-traines pre-trained with endogeneity achieve biased ICL predictions? 2. If such bias exists, can we design a cost-effective strategy (e.g., few prompting samples) to correct the predictions without fine-tuning the pre-trained TFs?
In response to these questions, we study the problem setup termed as Endogenous ICL (EICL) by allowing the (unobserved) noise depend perform theoretical analysis on the pre-training dynamics of transformers with hidden confounders, revealing that the ICL prediction of TFs exhibits bias proportional the confounding strength. To answer our second question, we then propose a double-debiasing method, i.e., prompting the pre pre-trained transformer (biased) with extremely few unconfounded examples. Our theoretical results justify that: (1) the proposed DDbias method can correct biased, pre- t--irrined TFs to produce unbiased ICL predictions; (2) our DDbias can achieve robust predictions even when the prompted ICL samples are partially confounded. We summarize our contributions as follows:
M. x. To answer the first question, first
To the best of our knowledge, we conduct the first theoretical analysis characterizing the endogenous bias induced by hidden confounders in both the pre-training and inference dynamics of ICL. Notably, due to the attention parameters embodied in TFs, our theoretical analysis exhibits fundamentally difference compared typical OLS theory such as the anisotropy feature- wise confounding results in bias cancellation.
We propose innovative ICL method, Suable-Debiasing (DDbias), which requires only extremely few unconfounded examples, offering a lightweight and gradient-free approach to mitigate the bias embodied in frozen transformers. Moreover, we prove that our DDBias remains robust towards either weakly confounded ICL. samples or mixed examples.
Extensive experiments validate our theoretical findings and demonstrate the effectiveness of the proposed DDbias method.
2 PRELIMINARIES AND PROBLEM SETTING
We denote random variables by uppercase letters (e.g., X and Y) and their realizations by lowercase letters (e.g.. I and y). The notation marked in bold refers to vectors/matrices (e.g., X). To index terms in sample matrices (e.g., x), we use the superscript to index i-th sample (e.g., (+) and [j] to index j-th feature (e.g., x[j]). In addition, On and o(n) denote constant terms and vanishing terms with oo, with Op and (p) denoting boundness and convergence in probability, respectively.
2.1 IN-CONTEXT LEARNING
We focus on the linear ICL. regression setup in (Von Oswald et al. 2023: Mahankali et al. 2023 Garg
ct al. 2022 Akyürek et al. 2022) in this paper: predicting ( for novel inputs x (++1) given
sequence of x2)g1)))
Data generation: linear regression instances. We let x E Rd denote feature vector with dimension
d, where labels / responses are defined as y Rd (x Px and X' is the domain of x). We
Thii assumption spans diverse perspectives such as implicit Bayesian explanations (Panarr et al. 2023 Wang et al. 2023b) heyond GD-based explanations.
2

---

Published as a conference paper at ICLR 2026
model the linear relationship between x and as y² (xx() w.) + c(,), with referring to the noise variable (Ahn et al., 2023; Mahankali et al. 2023). and w. E R drawn from Pw. We let D = (x(+), J= and D'e = {x(*, () Nee denote paired samples during the pre-training and ICL inference stages, respectively, with Ner and Noe denoting the sample numbers. The input samples are organized into the input matrix M E R (((++))((++) (n denotes the sample number):
x ((
x(2)
x(1)
x ((+11)
M
y
y(m)
0
where m E Rd+1 refers to the pair (x(9) v)). Intuitively, each column in M refers to data sample, i.e., token sequence with response.
Linear Transformer. To ease the analysis, we focus on TFs equipped with linear self- attention [Vaswani 2017; Von Oswald et al. 2023: Ahn et al. 2023 Schlag et al. 2021). With the value, key and query weight matrices denoted as Wv. Wk. k. W. [atttix(at) single block of
TF can be expressed as follows ( (Von Oswald et al. 2023 Ahn et al. 2023):
TF(M) Attention Wv. Wq. Wk. M)
W,MCsof [ Wa M),
(2)
where (-) refers to the softmax operator operating on each column of M equation the mask
In
matrix C
0 E p(++1)x(++)) reflects that the asymmetry that x (n+1) will not affects x(6
0
with j S n (Ahn et al. 2023). With S = We E g(4+1)x(4+1) and = W T Wg E (dd+1)x(d+1)
denoting attention weights, equation 2 can be reformulated with residual connections:
1 SMC(M TM)
TFs. M +
Ntr
where Nor is the scaling factor without influencing the expressive power of TF (Ahn et al. 2023
Remark 1. Clarified by previous advances (Ahn al. 2023 Akyürek et al. 2022), the omission of the softmax operation is somehow over-simplified, while the linear attention eases the theoretical analysis.
In-context Prediction and Training Dynamics. ICL first prompting the TF with the first n pairs (0) and then predicting the response for the final sample (n+1) with the grounding (n++)) We note that the predicted value of y(x+1) can be read out from the (d+ 1, n + 1)-th entry (M) The minus signal in TF,T (M) follows the fact that the
TF (M) [TFs,, (M)] predicted outcome can be read out from predictions of TF by multiplying - at the corresponding
entry of the output matrix (see derivations (Ahn et al. 2023 Von Oswald et al. (023)). During the training stage, the ICL objective (Ahn al. 2023) guides the optimization of Tis:
( (TTS: (M) + yy n 21
Lie (S, T) = E(a....)
(4)
Meanwhile, the linear coefficient vector w. shares the same distribution Pw between training and testing data (Ahn et al. 2023 Von Oswald et al. 2023 Akyürek et al.) 2022).
Weig-wi We now introduce previous theoretical results (Ahn et al.) 2023 Von Oswald et al.
2023 Dai et al. 2022) that pre-trained TFs is performing implicit gradient descent (GD) by updating
the meta-weights during the ICL inference. A formal definition presented in below:
Definition 1 (Meta-weights weights). linear prompting with n sample pairs (x()) y y0) 1 accom-
plishes implicit GD on the meta-weight w with the objection L(w) == 1. Σx= (w Toxxl x(() (())
In
Then the TF predicts on x(++1) with the "learned" meta-weight w: g(n+1)= = (x()+1) w).
2.2 ENDOGENOUS ICI.
As pointed out by (Liang et al., 2024), ) previous ICL theories (Ahn al. 2023 Schlag al. 2021 Von Oswald et al. 2023 Mahankali et al. 2023) conveys an implicit assumption
Assumption 1 (Causal Sufficiency). The pre-training data D admits the assumption that no hidden confounders exist, ie., is exogenous such that X.
3

---

Published as a conference paper at ICLR 2026
However, such assumption violates wide range of tasks in diverse areas (Cheng et al. 2022a Kumar et al. 2019 Wang et al. [222]] Huang et al., (022), and we offer detailed numerical evidence in the Many-aspec aspect Online Review (MSOR) task running example in Appendix [ By allowing the endogeneity in the pre-training data, we thus consider the following EICI. setup:
Problem 1 (Endogenous ICL (EICL)). We allow the existence of hidden confounders e c c X in the pre-training data Dtr, In other words, simultuneously affects x(() and yyt) (Pearl 2009) (see illustrative causal graph in the middle of Fig. in Appendix). Our goal is unbiased prediction on y i.e., w. x! x given a sequence of prompting data D (x(1) ( during the ICL inference. We let ;- = [e] denote the confounding strength.
2.3 PREVIOUS ENDOGENEOUS REGRESSION THEORY CANNOT ANALYZR EICI.
We note that a promising concurrent work (Liang et al. 2024 is aware of similar problem by leveraging Instrumental Variables (IVs) to mitigate the hidden confounding. Unfortunately, we note that in the EICL setup, traditional linear regression theories (Angrist et al. 1996) cannot be directly adopted to analyze the effects of hidden confounders (Angrist et al. 1996 Chen et al. 2024c), due to the newly emerging features of ICL.
Unlike linear regression, ICL prediction emerges from the interaction of attention blocks and the training dynamics of K, Q, V matrices after projection. As summarized in Tab. ICL regression differs from classical regression in two essential aspects:
Different pre-training loss and representation dynamics. Transformers are trained by sequentially feeding (Ty, Øt) and predicting Hn+1, whereas OLS directly solves w" from all samples jointly.
Different inference mechanism (few-shot prompting). ICL performs regression through attention over in-context examples, not by solving parameter. This few-shot inference has no analogue in OLSS.
Because of these mismatches, one cannot transfer the OLS endogenous bias derivation the EICL setting. In particular:
(Challenge The analysis must identify which attention parameters correspond to unbiased ICL prediction; there is no direct analogue of "using w*" as in OLS.
(Challenge 2) These attention parameters do not admit closed-form expressions, unlike the OLS bias term E[ XX -
Table 1: Comparison between traditional linear regression and ICL regression.
Tasks
Training
Inference
Linear Regression
Lmac (w) T3 (w))) (wTx -)
y w ((+1)
ICL Regression
ICL Loss (see equation
TFST (M) = - (TFs.
WHETHER AND HOW HIDDEN CONFOUNDERS CAUSE BIAS FOR ICL?
Theory Sketch. We outline the overall analysis into three steps (see Fig. in Appendix): (1) Subsection 3.1. In the presence of hidden confounders, we construct specific weight parameters of TFs, termed as "U "L__eights"" With U_ l_weights served as pre-conditioned parameters. the induced meta-weights yields unbiased ICL prediction results in EICL: (2) Subsection 3.2. We then prove that the convergence of TFs in confounded data will deviate from the constructed U. weights, i... the grounding TF parameters. (3) Subsection 3.3. Based on such deviation pre-training stage, we finally prove that the downstream ICI. inference will incur estimation bias, which is proportional to the strength of the confounding effect.
3.1 UNBIASED ATTENTION WEIGHTS WITHOUT HIDDEN CONFOUNDERS
4

---

Published as a conference paper at ICLR 2026
As considering hidden confounders might inevitably bias the TF parameters during pre-training, first have to construct some "grounding-truth parameters, termed as S", T", inducing meta-weights w with unbiased ICL predictions, such that one justify which TF parameter is "unbiased" and then quantify the bias happened in the pre-training and inference stages. To this end, we find some grounding TF parameters with unbiased ICL prediction, when the ICL prompting data is also confounded, i.e., c cl x(1) for (x(), y()) >==
Lemma (Unbiased Attention Weights (U weightt)))) Constructing the TF with parameterized by S", T in below:
Taxd w S" = [
0 T
Taxd
(5)
2 20
0
Lef 1f be predicted y such that j(xt1) S=, (M). Then it holds that gln+1) (x(m+11, w") where the Meta-weights (w") is optimized by standard gradient descent w.r.t the loss [(w")= = 1. (i=i(((")) ((w")("xd x(1) w. (.)
We also note that pre-trained weights in the unconfounded case (Ahn 2023 Mahankali et al., 2023) will induce biased inference results in the ICL stage with confounded testing examples (see Appendix By contrast, Lemma informs that in the case of EICL. the equivalence between forwarding process of ICL and implicit unbiased gradient w.r.t squared loss on learning w..
3.2 EFFECT OF ENDOGENOUS BIAS DURING THE PRE-TRAINING PHASE
Since the "grounding parameters" are constructed in Sec 3.1, we then are capable of characterizing how parameters of TF pre-trained endogenous data deviate from such grounding parameters, i.e., offering the theory of the pre-traine pre-training stage in our ECIL problem (see Fig.5)
To be specific, denote the covariance matrix of the input X as Σ Udiag( A A....., UT) with w. randomly sampled from (/ Ia). We use Sh To to denote the parameters T's pre-trained the confounded data. Following previous protocols (Ahn et al. (023),
first equivalently
reduce the form of parameters from (S², T
more sparse version { T Th ".j (the same as (S", T (see detailed derivation in Appendix R22.. Subsequently, to prepare for our first main result, extra regularity assumptions are required:
Assumption 2.
We assume that X₃- TyE Kj, where refers to the hidden confounder, T is the confounding strength, Kj refers to the noise term with E[[sy] 0 and E[x3] 1.
Assumption (No interference). For any i in, ((11) 1L (2) and vice versa.
Our Assumption 2 aims simplify our main results, while removing will not affect our final conclusion of the blas characterization. Mcanwhile, the Assumption 3 is commonly adopted in causal inference area, i.e., the Stable Unit Treatment Value Assumption (Zhang Wang 2024 Tchetgen
Weele 2012). Then we present our first main theorem in the pre-training phase below:
Theorem (Deviated Parameters During the Pre-training Phase). Under Assumptions with in our EICL problem, we derive the following result for the single-layer TF with Px ( E):
A
S. T
=
K K R U
(6)
Con CStreeStteggth
where Aire (S.T) characterizes the bias of the simplifeedtreduced TF parameters Sh To compared (unbiased) U weights St. To (see Sec. 3.1 the j-th feature dimension, the matrices R and K consists of some constants w.r.t the moments of the underlying weight w. and X.
Our Theorem informs that: (a) The (biased) TF parameters $6 To deviate from the ground- ing (induced unbiased ICL predictions) parameters Su, T with the gap proportional to the confounding strength r, (see proof in Appendix F3.1
"S refers the last of sh and Th (1 S S d) refers the first j-th column of T
5

---

further Published as a conference paper at ICLR 2026
3.3 EFFECT OF ENDOGENOUS BIAS DURING THE ICL PREDICTION PHASE
When characterized the effect of endogenous bias in pre-training phase, it is natural to doubt whether such bias will propagate into the ICL prediction stage. To answer this question, then derive the endogenous prediction bias through the meta-weight induced by deviated pre-training in Theorem 1 O) More specifically, definition 2 in below informs that our ICL Gradient Divergence equivalently quantifies the estimation bias y
Definition (ICL Gradient Divergence). The j-th coordinate on the meta-weight (see Deff 7 decides the prediction bias, as shown below:
A est = w w
Av Σ (w"-w"") w") ("="="")
(7)
Difference in Induced Meto-Weights
Prediction Bias
We then present second main theorem by utilizing results from Theorem 1:
Theorem 2 (Estimation Bias), With regularity assumptions: (1) Bounded cample-covariane matrix Z m (0) (m()) as On minigiksa ZuI S az where m'-) (x(() yr)); ); (2) Finite second- moments of Z: maxkt Var S kz, the following inequality holds with probability in 1 Σ 91:
est [il
rj
On
Σ
Σ
w.((l] 0²
+ +OKKz E
KZ
(8)
Q
Conf. Strength
Constant wort. increasing n2
Constant W.F.f. q
Remark 2 (New Features of Our Bias Characterization). Notably, in our Theorem 2 A t Dealil caused by hidden confounders exist proportional to the confounding strength r, which shares similar expression in endogenous OLS regression (Angrist et al.) 1996). To be specific, under the rypical theory of OLS regression, the bias can be derived via a flomed-form estimation of w.:
El w = EXX -
and its prediction bias
(=)
[riox, oe.............. Faxx,, Tdox,o.]
which depends only the confounding correlations rj. However, these derivations fundamentally rely on properties that do not hold in trunsformer-bused ICI. By contrast, our bias in Theorem differs qualitatively from OLS: (1) Bias cancellation via E. The global bias depends on the sum of confounding strengths across dimensions, enabling cancellation when E, Ti O. Classical OLS regression does not exhibit this property; (2) Dependence on attention geometry. Additional bias terms arise from how confounders interact with attention, a behavior entirely absent in OLS/IV regression.
4 DOUBLY- BEBBL-DDEIIAING:: PROMPTING WITH UNBIASED DATA COLLECTION
Current ICI--based de-confounding methods are limited to instrumental-variiale (IV) ap- proaches (Liang al. 2024). Moreover, data-fusion de-confounding techniques (Kallus al. 2018; Li et al. (024) require fine-tuning transformers on unbiased data, which contradicts the inference-one nature of ICL. Motivated by these gaps, we propose a novel gradient-free debias- ing framework, Noubly-Debiasion (DDbias), which operates without any additional fine-tume fine-tuning, auxiliary labels, or IV construction.
Collecting small number of unbiased prompting data (i.e., samples with independence JL. x) ml as D (x/2 yre Ji=i without hidden confounders existing between yre and (ff) (4) me with predicted (biased)
(1) We first prompt the pre-trained TF with t {x(2) yre outcome as (0)
(2) We then prompt the TF again with residual prompting (xre, Ure 0b m ns and the prediction
(6)
for y (m+1) will be unbiased.

---

Published as a conference paper at ICLR 2026
We first prove that our DDbias method implicitly performs gradient descent w.r.t learning the residual term y)(+) ((6 using a squared loss in below:
Theorem 3 (Implicit Implementation of Debiasing Algorithm). Consider the TF with parameterized by biased 8 Th, and let g(n+1) be predicted y such that gf TF To (M). Then it holds
that g(++1) 1) = (xre ()-1), wi w.DEB) where the Meta-weights {wP } is defined as wo some constant and as follows for 0:
DEB equaling to
n
DEB wT DEB - (/WL(WDEB) where [deb (w) W
(
16 w x(1))
(9)
2n
where 0/ refers to the biased prediction from pre-trained TF equipped with Sh T
We then prove that optimizing equation Theorem yields unbiased ICL. prediction over y ((n-1);
Proposition 1. Assuming that: (1) TF equipped with s and T" is a consistent estimator con- founded pre-trainined data; (2) X, Y, Y, have finite fourth moments over the unbiased prompting
data, optimizing over _deb (d)) =
2n Ei=((1 05 w x yields unbiased ICL prediction.
Theorem 3 and Proposition Tinforms that with our proposed DDbias, TFs pre-trained on biased data can achieve unbiased ICL predictions without requiring tuning the model parameters or constructing IVs as in (Liang et al. 2024) (see proof in Appendix F3.3)
4.1 ROBUSTNESS OF DIBIAS ON PARTIALLY CONFOUNDED DATA
In weal-world in in-context learning (ICL) setups, the residual correction set is rarely perfectly unbiased. Confounding may occur due to imperfect interventions, correlated noise, or heterogeneous data sources, resulting in non-zero feature- noise correlation E[ A 0. We consider forms of such bias: (1) Weakly confounded samples: All samples exhibit mild but bounded with small 6, reflecting globally weak bias, where each example slightly deviates from the ideal unconfounded assumption; (2) Partially confounded samples: Only fraction p of samples are contaminated while the remaining (1 - p) are unbiased. This case captures the realistic situation where most context examples are clean, but small portion introduces systematic bias. Analyzing these two cases is critical for understanding our DDebias:
representative
confounding effects, i.e., [[[zz]]|
Proposition (Weakly Confounded ICI. Samples). the confounded ICL. samples f(con)) contain nı unbiased samples with
Amia 1 (com (X(con)))) 2 A. > 00,
and let the corresponding noise e satisfy B[o,ed = 0 and E[E?] < 0². Then, for the DDbias estimator 0, there exists constant C' 0 such that
10 - 01I2 S C( ((
EEyer - UDEA S C'
Proposition (Mixed ICL Samples). Assume a fraction p of the "unconfounded" batch is con- taminated: for those contaminated samples B[rre] rj # 0 (denoted by X(oont)), ), while the remaining (1 p)ms samples unbiased (denoted by X (clleau)). Suppose the clean subset is
well-conditioned: Xmin (1-)),,
((clean) (cllaa)) A > 0. Let the mean confounding strength
be ;:= _ Eieentt [ra]. Then there exist constants C' > 0 such that
pTb
Elyot - ŪDEA] S C
((1-)) + pF
Hence, the bias remains asymptotically negligible n, +0 p O. and a sufficiett condition for asymptotic unbiasedness is
1
p = o(1), and
v(1 (11-))n6d. o(1)
7

---

Published as a conference paper at ICLR 2026
Table 2: Brief comparison between IV (proxy-based) and DDbias (unbiased-sampp) correction) in ICL settings.
sample
Aoumpptioe
TV (Proxg)
Vilid intronestis).
Not sogries.
Miliii Fannii
Goarie instuments jol.
Tovald weak instu saoms.
ky shocks.
DDbias
Noc
Few uncorionnded
Net
Rusdemieed tuffic
Puniall confennded pla.
cenreceder AIB.
Table 3: Experimental results of DDbias on the Yelp Sentiment-Review Prediction tasks.
Datasst Vanilla L.MMA DMCER cl
Changesa 2022
Dfltow (15)
Diliines (30 ICI.) DiOtiis (60 ICS.)
EXPERIMENTS
Linear Data Generation. We follow the function regression setting in (Garg et al., 2022 Ahn
et al.) 2023 Von Oswald et al.] 2023) by considering the ICL. loss for linear regression. Our
simulation protocols contains two parts, L.c. unconfounded x, y and confounded æ, y respectively: (1)
Unconfounded Case. We simulate 2x( Mn Σ) and w, (0, where 5, f4r 10
Vhe 20 (context length), Σ = U DU, U is a uniformly random orthogonal matrix, and D is diagonal matrix with entries (1, 0.25,, 0.0625, 1). Meanwhile, we generate the exogenous (2) Confounded Case. We first generate the exogenous variable ~ N 0.5,, and then sample the linear x - e relationship, i.c., the confounding effect ry U(0, 1) (uniform). Then the features are simulated as xx{² = rre r + Kig with K2 ~ N (0, 1). Finally, (0) is generated by an analog. By controlling |ryl. the strength of confounding effect can be tuned, i.c., the larger r corresponds to larger effect. To check our Theorem [ and 2 that the deviation and bias increasing w.r.t. increasing ry, we tune the value of ry by multiplying factos in [0.5, 1.0, 1.5, 2.0). with corresponding data denoted as Conf_ e.g., Conf 2.0.
0.5), and then simulate y(ب) = (w+, x()
~
VV-oriented Data Generation. To compare DDbias with an IV-based approach, we extend our original confounded DGP by introducing instruments Z. We first draw Z ~ (0, Im). latent confounder U ~ ((,,,)) and noise e(0) ~ NO0 o²)- Features are generated by combining instrument relevance and confounding: x ( (T=("); ) + a,,U66 x where I controls instru- ment strength and aj matches the heterogeneous confounding coefficients ry. Outcomes follow ( = ( w. + (U) + (6), with w, (O --) By construction, Z T e) (instrument exogeneity) and + X via (relevance), yielding a clean comparison: IV exploits large confounded observational data with valid instruments, whereas DDbias requires only small unconfounded batch.
Non-linear and Partially Confounded Data Generation. We further test the robustness of theoreti- cal conclusions on non-linear models partially confounded ICL. samples. We leave the protocols of non-linear DGP in Appendix G.7 and G.8
Real-world NLP Datasets. Finally, we conduct experiments on larger models pre- praitedined on two NLP datasets, i.e., two sentiment review tasks, collected from the Yelp website (Cheng et al. c022b]c). We leave detailed setup of compared baselines with in Appendix G.6) We refer to the adopted two datasets as (1) Restaurant Popularity Index (RPI) and (2) Restaurant Overall Rating (ROR).
Implementation of Transformers. Throughout our experiments, we set the number of layer of the TF as (TF@3) and I (TF@ 1), respectively, with each weight is initialized as i.i.d. Gaussian matrices. The optimizer is set to the Adam optimizer (Kingmal 2014) w.r.t. the ICL loss in equation 4 with each gradient step computed from a minibatch of size 20,000, each minibatch resampled every 100 steps, and gradients clipped to 0.01. All results are averaged over five runs, with different U (and thus Σ) sampled for each run.
design three metrics to test whether the constructed U_ weights in Lemmal match GD on ('(w) 1 Zw (0)
Justifying the U_ Weights. Following (Von Oswald et al.
2023
(
Let 8" denote our constructed TF parameters and 8" those optimized on L' (Appendix G.2p. We
compare: (a) prediction divergence (("² g'=(8") |: (b) L2 sensitivity divergence; (c) cosine
sensitivity divergence, averaged over 500 trials. Fig. b) confirms that U. weights closely track implicit GD on L'
8

---

Published as a conference paper at ICLR 2026
Para Dff of 3 Dim
0,40
800-
Con DV
Div
2b1
Pred/
Cos Div
0.c-
Gox Biv
99.5
2.00
Cont 0.5
1--
Cont 0.5
Conf
0.-
+10-1
82-
(8)
Cond 2.0
Cont 1.5
05-
Cont 1.5
C
2000 50c0
5050
2000
5000
w000 lleraton
Iteration
Iteration
lteration
3-th Layer of TF@3.
(b) 1-th Layer of TF@1.
(c) 3-th Layer of TF@3
(d) 1-th Layer of TF@
Figure 1: (a-b): Verification of Lemma
(c-d): Deviation of Pretrained Weights in Theorem
Conf@2.0
Conf@110
Conf@2.0
Conf@2.0
10a
Pred Bias (log)
10
Blased
3Dbias-Ours
Pred Blas (log)
10
107
DOblas-Ours
Blased
10-1-
Cos Div
0.8
107
DDbias Biased
10
10a
10-2_
2.
0.4
Blased
10
DDblas Ours
1.6 2.4
3.2
6.4
1.6 2.4
3/2
6.4
1.6
2.4
3/2
6.4
1.6
2.4
3
6.4
Ratio (xle-))
Ratio (xle
Ratio (x (x1e-3)
Ratio (xle-3)
Figure 2: The left two figures unveil the prediction bias comparision between our DDbias method and vanilla pre-trained biased model. The right two figures show the L2 Div and Cosine similarity with grounding w. of the regressed wi before (red) and after debiasing (blue).
Table 4: f = Ploy, Prediction error deeper, non-linear Transformers (ReLU, L=3 / LayerNorm, Non-linee Non-linear MLPs, Softmax, Ratio=1.61e ")) under different confounding strengths r.
ICL Samples L
Co
Conf@
Conf@
Conf@ 2.0
L=5
Biased (Vanilla TF)
0.280
0.370
0.475
0.600
L=5
DDbias (Ours)
0.115
0.150
0.200
0,260
L=7
Biased (Vanilla TF)
0.320 0.135
0.450
0.600
0.750
L=7
DDbias (Ours)
0.175
0.225
0,290
Table 5: Effect on the Prompting Sample Ratio (Weak Conf@0.
ICI. prediction error
0..900
0.092
0.095
0.100
Sample ratio for DDbias (oracle) (x10-3)
6.4
3.2
2.4
1.6
Sample ratio DDbias (weak) (x10-3)
8.6
7.2
6.3
3.5
Deviation of the Pre-training. We further measure the deviation A Arre (S, T) between confounded pre-trained (St Th) and U. weights ("") across feature dimensions. Fig. [ c-dd shows that pre-trained weights diverge from U weights (Thcorem TT. and that larger r, leads to larger deviation, matching our theoretical prediction. Additional visualizations appear Appendix Figs. 10
Estimation Bias during ICL. We compare ICL estimation using unconfounded (St T") and confounded (S", Th) weights by regressing their predictions on confounded x via OLSS 7daniuk 2024). 1.2 and cosine divergences between w,, and w are computed. As shown in Fig. 6 aligns with w., whereas w diverges as Tý increases, validating Theorem 2 Additional comparisons under
confounded vs. unconfounded data are provided Appendix Figs. 7
Verifying Our DDbias Method. Fig. 2 demonstrates the effectiveness of our proposed DDbias sample ratio increases, prediction bias decreases and the OLS- reggesssdd w aligns with the ground truth (Appendix G.5); (2) only very few unbiased samples suffice to debias, showing efficiency. Additional experiments (Appendix Figs. 20) and tests under Oon-Gaussian data (Appendix G.10) further confirm robustness.
method, highlighting two advantages: (1) as the prompt/pre-eeeee

---

Published as a conference paper at ICLR 2026
Results Real-world Datasets. Table reports results RPI and ROR. whose MAE scales differ due to distinct outcome ranges (RPI: foot-traffic popularity ~0-100; ROR: 1-5 star ratings). DDbias consistently reduces MAE as the number of ICL examples grows (15 + 30 60), eventually outperforms strong causal baselines. These results show that DDbias mitigates hidden-confouunerr effects.
Generalized Analysis on Non-linear, deeper Models. We conduct extensive studies to prove that our bias characterization analysis can be generalized into non-linear regime, including the non-linear TF' with non-linear DGP in Tab. [ (see more detailed results in Appendix G.7p.
Robustness Analysis on partially confounded ICL samples. Finally, shown in Tab. [5] we perform robustness to validate our Proposition 2 and 3, where the prompted ICL samples are either weakly confounded or a mixture of unbiased/biased examples. Results further support our theoretical robustness analysis that our DDbias remains robust even with imperfect prompting samples. We leave detailed experimental results with analysis in Appendix G.8
Comparison with IV-based ICL Method. We evaluate DDbias against IV-based in-context debias- ing (Liang et al. 2024) under three controlled setups: (1) both unbiased ICL samples and valid IVs can be queried (see details in Tab. 26 in Appendix); (2) only valid IVs can be queried and partially confounded ICL. samples are accessible (see details in Tab, 27 Appendix); and (3) only unbiased
ICL samples are available but confounded IVs exist (see Tab. 28 in Appendix).
6 RELATED WORK
Intrinsic Mechanisms inside ICL. Recent works seek explain ICL through implicit Bayesian
inference (Xie et al. 2021 Raventós et al. 2024 Wang et al.
(024), pre-training data properties (Chan
al. 2022). and implicit gradient descent (GD) (Ahn et al. 2023 |023 [kküüek al, 2022) In function extrapolation, Akyürek et al. (2022) first analyzed ICL d dynamics of a pre- trainan linear TF, later extended by Von Oswald al. (2023); Cheng et al. (2023) who proved equivalence between ICL and implicit one-step GD under tailored weights. Further, [Ahn et al. (2023); Mahankali et al. (2023) linked pre-training dynamics to converged weights inducing implicit GD. In contrast, Liang ct al. (2024) highlight that prior studies neglect hidden confounders, showing TFs realize IV regression for bias correction. Yet, theory for endogeneity in ICL remains undeveloped.
Exploring Causality in Prompting. A bunch of causally-inspired prompting methods emerge to enhance the causal inference performance of LLMs from diverse perspectives (Chi et al. Chen et al.
2024a b Liang al. 2023 2025), including the Causal Prompt (Zhang et al.) (024a). Intervented
Prompt Tan 2023), Peta-CausalPrompp (Ohtani et al., 2024). and Casual Chain-of-thought (Jin
al.
2023 In recent, [Liang al. (2024) reveals that the TFs highly-efficient IV estimator
during ICL process. Unfortunately, the above-mentioned work is lack of theoretical understanding and insights whether and how hidden confounders affects the ICL prediction.
Empowering Transformers for Causal Tasks. The zero-shot or the few-shot properties of TF's have inspired diverse causal tasks, including the zero-shot treatment effect estimation (Zhang et al. 2023
Mahajan et al. 2024] Nichani et al. 2024), and causal discovery (Nichani et al.) (2024) Besides, the
LLMs are also directly prompted for complex causal tasks (Liu et al. 2024 Jiang et al.).
7 CONCLUSION
In this paper, we propose a pioneering analysis reveals the presence of prediction bias proportional to the strength of the confounding effect, highlighting the importance of addressing hidden confounders in data preparation. Alternatively, our proposed DDbias method demonstrates that unbiased predic- tions can be achieved using small number of unconfounded prompting examples. Future work includes: (a) extending the framework to nonlinear settings; and (b) exploring the generalization capabilities of ICL, such as -fid-iiiibtittn generalization (Liu et al., 202T), by investigating shifts in across training and testing regimes, providing broader insights for the ML community.
ACKNOWLEDGEMENT
This work was jointly supported by the National Natural Science Foundation of China (Nos. 62276004, 62325604, 62525213 and 62372459), the Beijing Natural Science Foundation (No. L257007), the Beijing Major Science and Technology Project (No. 125100000))2500) and the NUDT Youth Independent Innovation Science Fund (No. ZK25-20).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint S1.i:::3030087744, 2023.
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning, Advances in Neural Information Processing Systems, 36:45614 01144614445655 45650, 2023.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algo- rithm in-context learnin?? investigations with linear models. arXiv preprint arXiv:221 2. I5661, 2022.
Zeyuan Allen- Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint arXiv:2309. 14316, 2023.
Joshua D Angrist, Guido W Imbens, and Donald B Rubin. Identification of causal effects using instrumental variables. Journal of the American statistical Association, 91(434):444-455 1996.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:18777-901,, 2020.
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. Advances Neural Information Processing Systems, 35: 1878-188811 2022.
Meiqi Chen, Yixin Cao. Yan Zhang, and Chaochao Lu. Quantifying and mitigating unimodal biases in multimodal large language models: A causal perspective. arXiv preprint arXiv:2403. 18346, 2024a.
Sirui Chen, Bo Peng. Meiqi Chen, Ruiqi Wang, Mengying Xu, Xingyu Zeng, Rui Zhao, Shengjie Zhao, Yu Qiao, and Chaochao Lu. Causal evaluation of language models. arXiv preprint arXiv:2405.00622, 2024b.
Xuxing Chen, Abhishek Roy, Yifan Hu, and Krishnakumar Balasubramanian. Stochastic opti- mization algorithms for instrumental variable regression with streaming data. arXiv preprins arXiv:2405. 19463, 2024c.
Lu Cheng, Ruocheng Guo, Kasim Candan, and Huan Liu. Effects of multi-aspect online reviews with unobserved confounders: estimation and implication. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pp. 67-78, 2022a.
Lu Cheng, Ruocheng Guo, and Huan Liu. Estimating causal effects of multi-aspect online reviews with multi-moda proxies. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining (WSDM), pp. 108 116. ACM, 2022b.
Lu Cheng et al. Effects of multi- aspect online reviews with unobserved confounders: Estimation and implication. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM), volume 16, pp. 122-133. AAAI Press, 2022c.
Xiang Cheng, Yuxin Chen, and Suvrit Sra. Transformers implement functional gradient descent to learn non-linear functions in context. arXiv preprint arXin:2312.066228 2023.
Haoang Chi, He Li, Wenjing Yang, Feng Liu, Long Lan, Xiaoguang Ren, Tongliang Liu, and Bo Han. Unveiling causal reasoning in large language models: Reality or mirage? In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt lear in-context? language models implicitly perform gradient descent as meta-optimizers. arXiv preprint ar arXiv:22/2.1055 2022.
11

---

Published as a conference paper at ICLR 2026
Alexey Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1192) 2020.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang. Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convoluiionaaggmente transformer for speech recognition. arXiv preprint TrXiv:2005.08800 2020.
Jason Hartford, Greg Lewis, Kevin Leyton-Brown, and Matt Taddy. Deep iv: A flexible approach for counterfactual prediction. In International Conference on Machine Learning, pp. 1414-1423.
PMLR, 2017.
Jianqiang Huang, Yu Qin, Jiaxin Qi, Qianru Sun, and Hanwang Zhang. Deconfounded visual grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 998-1006, 2022.
Haitao Jiang, Lin Ge, Yuhe Gao, Jianian Wang, and Rui Song. LIm4causal: Democratized causal tools for everyone via large language model. In First Conference on Language Modeling.
Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, LYU Zhiheng, Kevin Blin, Fer- nando Gonzalez Adauto, Max Kleiman- Mrinmaya Sachan, et al. Cladder: Assessing causal reasoning in language models. In seirth- conference on neural information processing systems, 2023.
Nathan Kallus, Aahlad Manas Puli, and Uri Shalit. Removing hidden confounding by experimental grounding. Advances in neural information processing systems, 31, 2018.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Sachin Kumar, Shuly Wintner, Noah A Smith, and Yulia Tsvetkov. Topics to avoid: Demoting latent confounds in text classification. arXiv preprint arXiv:1909.0043: 2019.
Virgile Landeiro and Aron Culotta. Robust text classification in the presence of confounding bias. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016
Haoxuan Li, Chunyuan Zheng, and Peng Wu. Stabledr: Stabilized doubly robust learning for recommendation on data missing not at random. arXiv preprint 22w.::::2200007701,, 2022.
Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, and Peng Wu. Balancing unobserved confounding with few unbiased ratings in debiased recommendations. In Proceedings of the ACM web conference 2023, pp. 1305-1313, 2023a.
Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, Peng Wu, and Peng Cui. Propensity matters: Mea- suring and enhancing balancing for recommendation. In International conference on machine learning, pp. 20182-20194. PMLR, 2023b.
Haoxuan Li, Kunhan Wu, Chunyuan Zheng, Yanghao Xiao, Hao Wang, Zhi Geng, Fuli Feng, Xiangnan He, and Peng Wu. Removing hidden confounding in recommendation: a unified multi-task learning approach. Advances in Neural Information Processing Systems, 36, 2024.
Haodong Liang. Krishnakumar Balasubramanian, and Lifeng Lai. Transformers handle endogeneity in in-context linear regression. arXiv preprint arXiv:2410.0126. 2024.
Ke Liang, Lingyuan Meng. Meng Liu. Yue Liu, Wenxuan Tu. Siwei Wang, Sihang Zhou, and Xinwang Liu. Learn from relational correlations and periodic events for temporal knowledge graph reasoning. In Proceedings of the 46th international ACM SIGIR conference on research and development in information retrieval, pp. 1559-15663 2023.
Ke Liang, Lingyuan Meng. Hao Li, Jun Wang. Long Lan, Miaomiao Li, Xinwang Liu, and Huaimin Wang. From concrete to abstract: multi-view clustering on relational knowledge. IEEE Transac- fions Pattern Analysis and Machine Intelligence, 2025.
12

---

Published as a conference paper at ICLR 2026
Chenxi Liu, Yongqiang Chen, Tongliang Liu, Mingming Gong, James Cheng, Bo Han, and Kun Zhang Discovery the hidden world with large language models. arXiv preprint arXiv:2402.0332 2024.
Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of distribbitiiitionnnn generalization: survey. arXiv preprint arXiv:2 13244, 2021.
Divyat Mahajan, Jannes Gladrow, Agrin Hilmkil, Cheng Zhang, and Meyer Scetbon. Zero-shot learning of causal models. arXiv preprint arXiv:2410.06/28, 2024.
Arvind Mahankali, Tatsunori B Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer linear self-attention. arXiv preprint arXiv:2307.03576. 2023.
Eshaan Nichani, Alex Damian, and Jason D Lee. How transformers learn causal structure with gradient descent. arXiv preprint arXiv:2402. 14735, 2024.
Ryusei Ohtani, Yuko Sakurai, and Satoshi Oyama. Does metacognitive prompting improve causal inference in large language models? In 2024 IEEE Conference on Artificial Intelligence (CAI), pp. 458 459. IEEE, 2024.
Madhur Panwar, Kabir Ahuja, and Navin Goyal. In-context learning through the bayesian prism. arXiv preprint arXiv:2306.045: 0489/, 2023.
Judea Pearl. Causality. Cambridge university press, 2009.
Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. Advances in Neural Information Processing Systems, 36, 2024.
Lynn C Rogers. Derivatives of cigenvalues and eigenvectors. AIAA journal, 8(5):943-944) 1970.
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, pp. 9355-9366. PMLR, 2021.
Juanhe TJ Tan. Causal abstraction for chain-of-thought reasoning in arithmetic word problems. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 155-168, 2023.
Eric J Tchetgen Tehetgen and Tyler J VanderWeele. On causal inference in the presence of interference. Statistical methods in medical research, 21(():55-75, 2012.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference Machine Learning, pp. 35151-35174, PMLR, 2023.
Haotian Wang, Wenjing Yang. Longqi Yang, Anpeng Wu, Liyang Xu, Jing Ren, Fei Wu, and Kun Kuang. Estimating individualized causal effect with confounded instruments. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 1857-1867, 2022.
Haotian Wang, Kun Kuang, Haoang Chi, Longqi Yang, Mingyang Geng, Wanrong Huang, and Wenjing Yang. Treatment effect estimation with adjustment feature selection. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2290-2301, 2023a.
Tan Wang, Chang Zhou, Qianru Sun, and Hanwang Zhang. Causal attention for unbiased visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3091-3100, 2021.
Xinyi Wang, Wanrong Zhu, and William Yang Wang. Large language models implicitly topic models: Explaining and finding good demonstrations for in-context learning. arXiv preprint arXiv:2301. /1966, pp. 2023b.
13

---

Published as a conference paper at ICLR 2026
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. Advances in Neural Information Processing Systems, 36, 2024.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint 1eXiv:2111002000 2021.
Bozena Zdaniuk. Ordinary least-squares (ols) model. In Encyclopedia quality of life and well-being research, pp. 4867-4861 Springer, 2024.
Congzhi Zhang, Linhai Zhang, Jialong Wu, Deyu Zhou, and Yulan He. Causal prompting: Debiasing large language model prompting based on front-door adjustment. arXiv preprint arXiv:2403.02738, 2024a.
Jiaqi Zhang, Joel Jennings, Agrin Hilmkil, Nick Pawlowski, Cheng Zhang, and Chao Ma. Towards causal foundation model: on duality between causal inference and attention. arXiv preprint arXiv:2310.008800 2023.
Ruiqi Zhang. Spencer Frei, and Peter Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49):1-55, 2024b.
Zhiheng Zhang and Zichen Wang. Online experimental design with estimation-regret trade-off under network interference. arXiv preprint arXiv:2412.03777, 2024.
APPENDIX
We provide supplementary documents to support our research. The details of Large Language Model usage are presented in Section We provide concrete case to illustrate the commonly existence of hidden confounders for ICL prediction in Section C with detailed notations in Section Moreover, we first state the failure of previous analysis in constructing unbiased meta- weeth- in Section P and offer the proof details Section Finally, we put experimental details, including the baseline setups, data preparations, with more experimental results in Section G
A LARGE LANGUAGE MODEL USAGE
In this paper, clarify that large language models (LLMs) are employed solely to support and refine the writing process. Specifically, we use L.LMs to provide sentence-level suggestions and to enhance the overall fluency of the text.
B FRAMEWORK ILLUSTRATION OF OUR METHOD
C CASE OF HIDDEN CONFOUNDERS IN MSOR EXAMPLE
This case study illustrates how the consideration of hidden confounder (HC) can alter the relationship between sentiment aspects and outcomes like Popularity and Rating. For Popularity, the inclusion of HC led significant reversals the effects of both Price Neg and Mise Pos, changing negative associations to positive ones. For Rating, while some effects remained unchanged, the influence of Misc Neg was neutralized, highlighting the critical role the confounder adjusting the analysis of sentiment's impact these two metrics (Cheng et al. 2022a). For readers, we refer further details in (Cheng et al. 2022a).
D NOTATION TABLE
We summarize various notations used our problem formulation, method design and algorithm in
Tab.
14

---

Published as a conference paper at ICLR 2026
Unbiased ICL
Hidden Confounders
ICL-Debiasing
Pretraining Stage
Causal Graph
Pre-training Bios Characterization
Statements
Training Sampls y Sentiments y
Location
Rating
Training
Samples
Prefraining
Stage
Preteaned Midel
Biasol Ford @ Piemmeteer
TenBble didies!
Negative
Good meal
ICL Inference Bins Characterization
Trandfrmer
Hised
goon
V Positive
P(X.)) * P(O)CG)
Parameters
Prompting Samples
Inference Stage
Double Debiasing
Testing Samples Die
Example
Samples
Prompted Sumples
Loestoo
Kating
Pogelariy
V Infersuee Wiased
Prompting yill gooss)
Biaael
Inference Stage v Bosed Binee A.e
Bitte
Negative
Pretmined Mockl Ficed Parsmenee
Yax Vua
The service Ria
By
Pax
littl darr
Pusilive
Unbissed
The illustration of (1) Traditional (unbiased) ICL framework; (2) Our proposed EICL problem; (3) The bias characterization contributed in this paper with the proposed DDbias method. As shown in the Figure, the review "The service is little slow' may be perceived relaxed and positive in Italy, but inefficient and negative in the UK.
Figure 3:
Table 6:
The table compiles all data related to hidden confounders extracted from (Cheng et al. 2022a). The table illustrates the changes the correlation coefficients of the effects that MAS has on popularity and rating before and after accounting for hidden confounders. Nearly every sentiment aspect's influence coefficient has changed. However, for several sentiment aspects, the coefficients have reversed signs dramatically, indicating a pronounced effect from hidden confounders, which is shown in Table in below.
Popularity
Rating
Sentiment Aspect
With HC
Without HC
With HC
Without HC
Ambience Neg
-0.24
-0.22
-0.02
-0.01
Food Pos
0.39
0.34
0.25
0.23
Food Neg
0.09
0.05
-0.06
-0.08
Price Pos
-0.04
-0.03
0.06
0.03
Price Neg
0.03
-0.00
-0.05
-0.03
Service Pos
0.10
0.03
0.22
0.20
Service Neg
0.08
0.09
-0.37
-0.31
Misc Pos
0.03
-0.04
0.05
0.03
Misc Neg
-0.03
-0.08
-0.03
0.00
Table 7: The table presents the mean values for Popularity and Rating across different Sentiment Aspects, with and without the consideration of a hidden confounder (HC). The primary observation from this table is the notable reversal the influence of certain sentiment aspects on Popularity and Rating after accounting for the hidden confounder.
Popularity
Rating
Sentiment Aspect
With HC
Without HC
With HC
Without HC
Price Neg
0.03
-0.00
-0.05
-0.03
Misc Pos
0.03
-0.04
0.05
0.03
Misc Neg
-0.03
-0.08
-0.03
0.00
15

---

Published as a conference paper at ICLR 2026
Rating
Rating
Locattion
Review
Location
Popularity
Popularito
Figure 4: Illustative Causal Graph Our Running Example on MSOR.
Table 8: Summary of notions with their definitions.
Notation
Delinition
Features of s-th Sample
Label the i-th Sample
600
Noise
Sample strength confounding herween 5-th
5 R[X,r]
The cuogenous noise of X w.s.l.
Dimension of
D" and
Training and Testing Datsets.
D' and D
Training Testing Dutsets.
x
x(n=1
M (y
Inpur format of data when feeding the munsformers,
m (xe9)
Input Pairs.
Distributions input features The covariance matris of input Ieature X undertying parameter The projection paramcters transfomers when computing the sel-attentions
E= Uding An..... T
w.
o
W., Wa. W. R(ATINEE
C= Ri-
The mask matrix to reflect the asymeety in ICL, proeess.
S = W.
W. W. (00+00004))
The aggregated purameters of TF from W., W,, W..
S
The U weights, the constructed weights of TPs inducing unbiased prodictions. Comessonding unbiased
relationship updated by implicit steps.
S
The biased weights, i..., the converged weights of TT's confounded training data.
Conreponding hiased
relationship updared by implicir stepe.
SE R and T ROFIXo
The last row of S and the first columns of T. The j-th column T
T. C. R
G -5-55 T.
The comgosition of T. and Derivative Divergence Def
A/.. (8,T)
Constant Matrieesscalars Thoorem
Prediction bias of v/ in Theorem 2
D 12
Unbissed (unconfounded) Daraset.
Dew
Induced implicit gradicnt by our proposed method.
K.0
Regulaniy Constants in Thooee
E FAILURE OF PREVIOUS CONSTRUCTED WEIGHTS
Lemma [Biased Pre-trained Attention Weights] Attention weights in previous studies (Ahn et al.
2023 Akyürek et al. 2022 Von Oswald et 2023) suffers from biased (confounded) prompting
Weights Prediction
Our Theory Sketch
Without HC
With HC
P00FOC)
Finding Grounding (Unhiased) Parameters
Pretruined TF
Pretrained TP s
5.7
Weights)
Unbiased Meta-Weighta
ICL Prediction (w12()
Unconfounded
Confounded Prompting (200 glast
Grouncing Parameter
Prompting
has
See Theorem
Endogenous Pre-training Bias
Weights
Meia-Weights
(w
y0
y00
Grounding TF 5
Compare
Pre rre-training Deviation (Proportional r,)
Pretrained st.re
ICL Prediction (w. (e)
ICL Prediction (
3.3
Endogenous ICL, Prediction Bias
Unbisssd
Blaaed Our Cermua in Apowootie
Prediction you-3) - (wr), 20011)
Pre-training (Propoctional tn
Lamonn 1 (107
(Proportional to
Figure 5: Outline of Our Theory Sketch
16

---

Published as a conference paper at ICLR 2026
examples D
Oaxd 0
S
1
{
)mm)
(10)
T
Udiag
n n- Act
(ExAz) Ax)
U
1
suffers from biased ICL inference, and let g(ntl) be predicted y such that g(++1) T,ST (M). Then it holds that gln=1) (x(1+1), wGD) where {w?D} is defined as wo 0 with either exogenous or endogenous E:
WGD =wo wGD VL(WGD)
(11)
where
[(w) = 2n
w x - (0
(12)
Proof. We only consider the special case that U with A d2 Ad Then our derivation
follows the same protocols as in (Von Oswald et al. 2023):
gin+t) =IF
TFored (M)
M+ SMC(MTTMM n
-(x(ett)
ddx
(x(1)
Laxd
0 (x(0)
y( ))
Laxd
o'
(x(+11)
+ 2n
-1 y(1))
0 0
0 0 y
(xlett))
(13)
where the optimized loss function is biased, as the y( is confounded by e(
F THEORETICAL ANALYSIS
In the following, we perform all theoretical analysis in the case of Px = WO0,, diag(A),... Ad)). and then turn back the correlated that Px NNO,E).
F.1 U_ WEIGHTS CONSTRUCTION
Lemma (Construction of Unbiased Implicit Gradient). Consider the TF with parameterized by S T in below:
Odxd w.
S
(14)
T Lixe 0
0
0
and let g(n+1) be predicted y such that f( (xle+1), wGD) where (wGD) is defined WGD 0 with biased (confounded) prompting exam- ples Die:
T. (M). Then holds that gjn+1)
w GD w
VL(W GD)
(15)
where L(w)
1
w (1)
-w.x
(16)
2n
1=1
17

---

Published as a conference paper at ICLR 2026
Proof. The extension follows the same protocols in (Von Oswald et al. 2023):
g(m+l)
T, (M)
S,T
= + 1 SMMC(M
(xl(ati)
Σ
0 Odx
(x))
x(1)) y
(Fo
Laxd
x(1)) y(
Laxd
(x(++1)
+ 2n
@
y(++1)
(x
(-
C(wOG ( (x(
(17)
where the underlying label is w, Tx(1), serving unbiased guide for optimization during ICL inference.
F.2
SIMPLIHICATION ON TRANSFORMER PARAMETERS
F.2.1
DETAILS ON PARAMETER SIMPLIFICATION
To ease the theoretical analysis, introduce the Gaussian assumption, which has been commonly
adopted (Ahn et al.) 2023 Mahankali et al. 2023) when analyzing into the dynamics of pre-training
optimization of TFs:
Assumption 4 (Gaussianity). We assume that Px E), with E Udiag(AI,.... UT) and w, is sampled from N00 Ta).
We use Sh To to denote the parameters of TF's pre-trained on the confounded data. We first show that quantifying the difference between the converged parameters S To in EICL and the grounding parameters S", T" can be reduced to only small subset of each parameter matrix:
Lemma 2 (Parameter Simplification). During the pre-training phase, analyzing the variation of {S Th == equals to analyzing that of Sh Th, where refers to the last row Sh, and Tb (1 S j S refers to the first j-th column of T
Unfortunately, find that directly quantify difference between the unbiased TF parameters {Sh, Th and the converged parameters (potential biased) {S", , Tm) exhibits challenges, since the spurious correlation rj E X prevents the derivation of closed-form expression of Sh T as in equa- tion 5 (see (Ahn et al. 2)23)). We therefore compare the two groups of TF parameters from the perspective of derivative instead, that is, quantifying how S To and S T contribute to the pre-training objective Liet differently:
Definition (Derivative Divergence). We measure the deviation Sh Th from S", T as follows:
dLial
dLiel dG -5
A
(S.T)
dG
G/
To
Intuitively, A/.. quantifies the divergence between effects of Sh To and St, T in the j-th feature- dim on Liet during pre-training. We note that the derivatives of Lis w.r.t. S5, To equals to 0, as 5 T refers to the global minima points with endogenous e.
F.2.2 PROOFS OF PARAMETER SIMPLIFICATION
To simplify our proof of Lemma 2
we first introduce several usefuu lemmas:
Lemma 3 Ahn al. 2023 the optimization of Lict, only S E R (d++1) and T Ridttpdd are updated, where S441, refers to the last now of S and T Taed refers to the first d columns T.
18

---

Published as a conference paper at ICLR 2026
We use S, T interleavely with notations S, T without ambiguity. We then present another lemma for decomposition of S and T into d pairs of weight parameters: (S, (T) where T. E Rd+1 refers to the j-th column of T:
Lemma 3 (Decomposition along T. Px = (00 diag().....,.au)) Finding global minimum, i.e., S, T of Lid can be equivalently turned into finding the global minimum of each component Lid.j W.E.L T
Liel Σ W..... (Z.G'.) + w. w.bl]] E[x 1[x00) + Σ EM.... (Z, By, (18)
where Gy==5 TT,T 33 refers to the correlation between Xj and e: ry = E[Ex,] and (A, B) = Tr(AB T for two matrices A and B here and below.
Proof. We first simplify the original ICL objective:
(1at))) o]
Lid (S, T) w..)).
S, (M) + y
( (TPTTT (M) + w, T (x(+1) + clott)) 27 9
= ((,...)
((M+ Nor
= (....)
SMC(M TM)
+w, (x + c(ut1)
(d+1,u+1)
27
[x(++17
1 n
T [x(nti]
w.)
+
m ()
w. x((at)) (m+1)
0
n
(d+1)
(SZTx(n+1) (x+1) + w. (x(+1) + +(( 27
= E(M...))
E (SZT +w. x (m+1)) + 2E (
(SZT w. T) + [(n+1)]
Optimizable
(19)
where we further expand the first term as follows:
(SZT + w. 7) x
Σn, 2, ( (Z.S"T..) TTJ 7) w.Ll]]
(20)
j=1
as (1xx+++)) (ip(at = for j j' and E|x(n+1) = X,  The second term expanded as
follows in analog:
E (n+1) (SZT + w, x(x+1)
Σ
B Z T.j T...)) |
By R
(21)
Hence, the overall expression of Lie can written as follows:
d Lie = Σn X 2.5 (2.S'T. + w.bl]] + 23, E
[ (Z.5'T..
T }
(22)
1=1
With the decomposed expression shown above, we further prove our lemma by performing induction on When 1, the objective becomes:
Lid,1 A1 E.. [Z, (Z,Gg) w.]² E..... ( m
(23)
19

---

Published as a conference paper at ICLR 2026
By taking G an integration, observe that Liet, exhibits the quadratic-forn w.r.t G ie., convex formulation w.r.t. Ga Furthermore, when taking derivative of Liet,1 w.r.t. G ats we obtain the following expression:
Viet,1 2λ1 B,,,,, [((Z, Gz,)) + w.) Z + m E VG
(24)
To achieve vanished derivative, we just have to show that there exists G such that:
S ZT: w4 7/1
(25)
We now decompose Z due its symmetry:
Z diag (21, (U*)-1
(26)
and turn the above expression into:
S dig (Vj X3) (U3)-
(27)
= S U diag (Λ,, X3) ((UR)-
(28)
and we construct S and T. such that S U*) A5).
(
(ww+ and ((U*)-IT dA1())))
We assume that the our lemma holds for the case d = d'. For the case that d = + 1, given the the existence of ( Tisz=1 that achieves global minima ا Licijj we only have to show that there exists + such that the pair T S can achieves the global minima of Lield'++1- Be awaring of the fact that did, is also convex w.r.. G we just have construct T, with given S such that:
2 FM.. (d
[(
w. (d' + 11) z 77 E E
(29)
where we further have to construct the vector T:,d' such that:
w=(d' + +1] 77++1
S T
(30)
2A8+11
which is obviously feasible. Then our claims follows by induction.
Remark 3. Previous studies has been explored similar conclusion in (Ahn al. 2023). However, the conclusion relies on the explicit closed-form expressions of S, T with shared S across each T. by coincidence when hidden confounder is missing.
F.3 PROOF
F.3.1 PROOF OF THBOREM 1
Theorem 1. (Deviated Weights, Px endogenous we derive that:
Under Assumption 4 with
K K
K2 K3
WiliiDiag (1 (1==l Xnds + 1(k = -11
dje; On ({ا)))= (31)
Aire S, T = rj
Xjez
where each sub- maa-matrix are expressed as follows:
K1 = w
K2 = On ((> w.|ul o²
K₃ = On(
Xw))
where ej denotes the base vectors with j-th element as and others as 0, Aa = diiagAi,,...... Aa). E Rd and r Rd are vectors consisting of X and pd J=1' respectively, o² 4 denotes the 2-th
and 4-th moments ofe, respectively.
20

---

Published as a conference paper at ICLR 2026
Proof. Before our formal proof, we further clarify the decomposition of unbiased weight matrices, Lc., S", T in our Lemma 4 as follows:
S w. .w.
(32)
T .. Cj.
where we remove the scaling factor 1/2 without loss of generality. Recalling the expression of Lial w.r.t. G as follows:
Lid.j Ag W..w..
(2, + w.1ll]]
EM....
(2, n/2
(33)
with derivative follows:
VLield
22, EM.... (Z, Z
BM [w. 1i + 3 E3 ZZ
(34)
VG
(2)
(3)
(1)
Meanwhile, we also recall the expression of decomposed derivative follows:
Lidt
Lidd
A pre (T.
(35)
VG
ST.T!
VG
ST.
and we compute two terms one by one. To be first, as S in corresponds to the global minimum of Lid.i with endogenous e, we have:
Lidl
(36)
VG S
We then calculate
Viig VG
by expanding Z as follows:
LE 1 6 x(1)
(x())
Z
(T
(37)
We then derive term (2) in equation 35
by further decomposition:
(x(0) ==
! i= 8(00) x100
n
W.....r [w = EM.
w.bi
#
( (x(1)
1=1 (2(
Oaxd Lim (x(0))
" 1= x(1)
........
w-[il
> (1))
EM....r
w.bb
Oaxd
+
(w, T xx(i) +c(0))
]
1. E (w, Tx(() + t((() (x(1)T
EE (0)3
Ossxd
tye,
Xje]
1 E......
w.bil (w, T x(() + )
Odxd
Xyc; 23,-
Aye]
(38)
where the second equality is due to the fact that w, is symmetric, and the third equality is due to [i²
x
x (0) [K]] Ay 1 and the final equality due to the derivation that:
w. T xx()) + ) = E w.[i] (w, Tx(1)) + +w.bl (<:))² 2 + 2w.bl (w, xx(() c(i)
EM.w..F
w.l
Σ
6=1
2
> i=1
EE [wlli] (w. T x) c66)
= 2
[
= 2/,
(39)
21

---

Published as a conference paper at ICLR 2026
We then derive term (3) in equation 35 by further decomposition:
(x()
-ー x00
E = 3y
E
(x())
Ad
E
xx(x)( + c( x
3
Di=l (w. Tx(x) + e (x(1))
E
(w, xx(1
() + { /
(40)
Ad
= 33
rT
E
- (w, T x() + c(0) {(0))
A
33
o² +
E
where the final equality due to the fact that:
W xx()) + <(1))
EE (w, xx(1) 2 + (c(0)) + 20 (w, Tx(1))
E
(w.. (w. =x(0)) 27
2 +
0² + d
(41)
E w.lll (x(0 7
3=1
A
Finally, we derive term (1) in equation 35 in three sub-blocks:
12
Z
Z
Z
(42)
(1)
%())
Z
Then we derive the following equations for u S d:
WM. (Zz())EE.
(0)
E
x
((
(1x)))))))
1k
n
n
[o)
(1}
n
E
[x
E. [
n²
11
1 n
1(-=j=1=
Σ
10= k))a1
n
(43)
where we utilize the fact that zero correlation in joint Gaussian implies independence. Hence, conclude the expression as follows for = j:
n (n
1)
0
E,,,,, (Z Egs) 2
nnn
1 1) A2
0
0
n(n - 1) + n2
n(n A)
(44)
22

---

Published as a conference paper at ICLR 2026
where EM.,., (Z((), [(n)
0 when +
We then calculate the second term as follows:
W.....c (z() (Z() Bu.s)) ((6)
[o)x (w,Te Tx((x) + +c(02)) x x
n²
IR
n²
[x] E [(
$1
[1)
[u|x 11c0
n
(45)
where 1 can be derived as follows:
( [1]
E [c()))e(2)1]]]]
1(v -) 1 =)) (n(n 1)mo(r,)
E
(46)
$17
and ② can be derived as follows:
n² Σ x (61)
1/]
n² Σ (00 (rye( + + (rec + + }
1
(ryrorio* + Errodidd + r, + n10jvul)
n
1 (Iyroo + +o j)r, + 1(v j)r + 1(v = 1)v==))r
(47)
where we invokes linear confounding effect derive the expectation. Finally, we calculate the third term, i.e., W..... [(Z()), E,.)) [ as follows:
W...,, (z(o).E.)) Eus) [
E
x(( (o)e((((111) (c(tz)) 27
E (ope(()))) [s] (w. xx( w.
(48)
where we further expand each term in below:
((16))) [[]((
n=1 1(u j) 11== 1 (r = j)o
10
if i1 + ig.
For Term I
n
if i = 12
1==i-==1
(49)
where we invokes linear confounding effect to derive the expectation. For the term 2, we perform the following derivation:
[0)x(()) (1) (w, T x (22) (iz) w.)
E
2
E w. (x(()) (x()))))((((>>)))>>>>>))>>
w.]
n
1=11==
(50)
n
Σ
( (2]x) 11x [2)
n
n= 12=1
[Ne) No 1)
= =1
1 mm
{
+ - 1)1,
= =0 -1(1)55)1
kgj
k
23

---

Published as a conference paper at ICLR 2026
Now every components calculated, and we ready to summarize main results by summing over v [d] with weight [,[
For the upper-left area of A pre i... (A (Aprelaxd. we have the following equations by com- bining equation 40 and equation
nt
n
Aydg + د n 2
n
)adi) ) + 3yAd
w. Wiag
A1A1:
Diag
n 1
n
n 231j x n n- Aadi + w. walilrjo2
(51)
The upper-right area (A pre dxx1 1dx1 can be derived as follows by combining equation 46 equa-
tion equation 38 and equation
(A (Apre)xx n
1 ri
Wa w
Σ (.(o)))2nn 1)rrro?) + dyez
((
=r,On
walll
0² r +
(52)
The lower-right area can be derived as follows by combining equation 50
equation 49
(A (eteei On ((4))==1, +r,0, Σ %o)
(53)
On (MW3)/ + +r, + E
(n).
Prediction Weight w with endogenous noise c. We show that inside the forward pass of ICL inference, deviated parameters Sh, To still induces implicit gradient weights w Consider the fact that G G + &g,, and we have:
g(nttl S,T (M)lj] = Z, G. x((+1)()]
Z. G.d + +822) x((t)))
(54)
=x(ntllljwwCD GD] + (z. (2.0))x(1)) x(++1)
where we let w"[]] = w (GD + (z. 82
F.3.2 PROOF OF THEOREM 2
Theorem 2. (Estimation Bias) By assuming the lower-boundness of the sample-covariance matrix Z = m (m()), i.e. a1 & minisiksa Zu S a2, and the finite second- moment of Z:
max Var Var[Z] s sz, have the following inequality holds with probability in Σ1 qi
A 2 min
K K(n, K(n,2.r,o",00
Σ
Kz
K(n, xrr,
Σ
Riz
daz
doj
(55)
Proof. We observe the following equations during the forward process:
9b
(n+1)
[ = Tp-red T
(M)b) = (Z. Gby x (x+1)
Biased
(56)
24

---

Published as a conference paper at ICLR 2026
Yu
1 TF S",T" (M)l) (z. G% x(
Unbiased
(57)
Then we only have to compare (2 Z. 2 and ( G?: ) where their difference are already shown in equation 34 the proof of Theorem 1:
W.. (z. G. Z - E.. (Z, G z EM w. ess Li]2,
(58)
where the quantity w... A [i]z] has already been computed by combining. Hence, by invoking Chebyshev's Inequality on the region of Z (lower-left area equation 42). we have the following inequality holds with probability 1d+1,1 for 1 sIs d:
Varm A Aillii d+1.j
A A w.....
(z A)E+11)
(Zn
z"] S
Kz
qd+1,
9d+1,1
(59)
where we further recall the expression of the lower-left region (in equation 52) as follows:
(0) = (Apre)axi - X,e; rn ((_ w. 0²
W.,
Es+1.) (
we further perform the union bound of equation 59 over 1, we further have the following inequality holding with probability - >; 9d+1.;
Σ A b D Za - WM, ( (z B(, A)E,1)) =00 S
Σ V
Kz
(60)
9d+1d"
where the left side can be derived as follows:
m 1/ [Z4z.. EM. [ B(,,,)(n)B))) A) z
(61)
A.r [i) E.....
(z X A) 21(0
with:
Σ A w [i] Za| EM....r
(62)
A [i] Σ -
(E w.(
d2
)
r1
Then substitute the above quantity back to equation 60
obtain:
(r:On (EE, w. (u)) o²) Ein
rnOn (EE, wu(u))
Eire
A we [i] 2 min
Σ
Kz
Rz ga+1,day
daz
qa+1,1da2
day
(63)
We then turn everything back to the correlated case, i.e., X exhibits covariance matrix Σ (aiiag using the same trick in (Ahn et al. 2023
Proof for Theorem I when Px = No0 E). We deduce the case with Px = (0, Σ) to the previous
case. By defining g(() == U (6), we conclude that ( A₂ due to the orthogonal
properties U. With (0) = (p back to the loss decomposition in equation 18 we have:
B SZZ + w. x E (SZT w. T) Uz(n+1)
(64)
Σ A (SZT w. T) U[i]
25

---

Published as a conference paper at ICLR 2026
and the vector (SZT + W.
U can be transformed follows:
[SZT + w. ) 11
1
Σ 5
x(1) (x(1), W.)
@2
TU W. TU
+ c
- 71
+> T
(UZ w.) UU
UT(
82
TU + W.
TU
(65)
ト Σ S
U
0
200
192
UT
TU w.
U
0
(I10) w.) (6)
0
U
U
0
where we use 02 to denote the operation x xx
Letting S
and
TU be
0
1
the new weight parameters, respectively, with the fact that w. U stills follows (0, Ia), the case is deduced to previous cases, and our final claim of Theorem follows.
Proof for Theorem 2 when Px = (0, 2.). We note that Z, G. and (z. G//) remain invari- ant under the transformation z(() U (0) due to the fact that UU I, and our claim on Z, G
(z. G²/ follows the same.
F.3.3 PROOF OF THEOREM 3
Theorem (Implicit Implementation Debiasing Algorithm) Consider the TF with parameter- ized by biased Sh T and let gf++1) be predicted y such that gln+1) = TF To (M). Then it holds that g(n+1) (x(n+1), DEB, where {wGD is defined as wDB equaling some constant w.r.t.
imput data and pre-trained weights:
W
DEB DEB VV
Wi
(66)
where
deb (w) = 2n
(67)
where yi),, refers the biased prediction from pre-trained TF equipped with Sh Tbb
Proof. We first observations as follows:
1. The parameters 3,T T can be decomposed using the feature-wise expression:
S" T (Gudid J3=1 S¹. S T
(G 3==1
(68)
and note that G &2 + Gbd.
2. The pre-trained parameters of TF is fixed, while inputing different data matrix will invoke different unbiased implicit gradient for the same unbiased parameters G shown in Lemma
With the above observations, first derive the behaviors of the biased parameters {Ghin 1 when meeting the residual input matrix Z". By treating the residual term ( yr the new label (0), we invoke Lemma 4 and conclude that Su, T achieves unbiased implicit gradient w.r.t.
26

---

Published as a conference paper at ICLR 2026
the loss _drb(w implemented in the forward process. By re-writing G recalling equation 54 we have:
G 8 and
g(at))= (wDEB) T (x++1) Z" E.>0
8 (x
(69)
j=1
= =(WPEB) T x ((Z 82)) : 822) ,M (Z", (21))) (n+1)
Recalling Lemma 4, the starting point of S", T when performing unbiased implicit gradi- ent descent is Wo
0. Being aware of the fact that (/)) (Z", 8 ............................... (Z", (Z",84)) remains fixed w.r.t imput prompts and dar embodied in fixed equivalently treat (Zr, 82,)) (ZF, o2.) ..... (Zr, 8d shifted starting
as the imput matrix sets y(==1)
pre-trained TF, point of gradient descent comparing to intrinsic starting point Wo = O. As the squared function is convex and no local minima exists when performing GD, our claim follows.
Proposition 3. Assuming that: (1) TF equipped with 5e and T is consistent estimator on con- founded pre-trainined data, (2) X, Y and Yo has finite fourth moments over the unbiased prompling data, optimizing over _deb (w) = 2n for y 1))+))
1 E 1y g. w x will leads to consistent estimation
Proof. We extend techniques in (Kallus al. 2018 in the case of high- dimennsionll regression. First, we denote the residual term y() 0% as g(,) with VoT = w. x(()) is underlying ground truth.
We denote the expected (biased) prediction by biased TF with 5 T as y Meanwhile, we have g = Yb. Letting X,c = Xre
(m) the estimated residual parameter for 0, i.e.,
Xyc
can be derived follows:
0=
(X,
Xreg
(70)
then decompose g in the following three terms:
VGT
yb
UGT
yb
(71)
(0)
8
By noting the linear relationship between and y. with the linear architecture of TF, we obtain that the residual term between expected truth and expected bias prediction is linear w.r.t. Ver yb 0 Xre, and then expand g(1) follows:
(2) (6)
B()) = ((g()
g(d) TT
XyC
(1)
Xre
(2)
(n) XPc
= 0 X
(72)
We then derive the following equations:
i - 0 = (X re)
X -0 -1
X XraXTe
XregT - -8
Tye
Nre
X mC
X T + 8(2)
((1)
+82) + +g(3) -
Tye
Nre
X
X
X Xx.g(1s) W.
Xrc
Xmogia) +
Tye
Nro
Tyy
Thre
Thre
The
(73)
where we further derive each term the last equation in below. Letting Anr.
X
we have:
X
-1
2
rC
(74)
Drr
Wvc
min, o, (An)e)
27

---

Published as a conference paper at ICLR 2026
where o; refers to the i-th eigenvalue of An,c. Meanwhile, we denote E[X A, it is obvious that An 0.5, A. Mcanwhile, as A is symmetric and real-valued, we have its eigenvalues are distinct (Rogers 1970), such that o; is continuous mapping of A. By invoking the continuous mapping theorem, we immediately obtain that:
min 0i (A
a.S.
min (A):
and we furthe have:
- (75)
A (IA - A A122)
- A 11) = o(1)
2
0.
where og refers minimal eigenvalue of A. We then consider the term
Fiee X22 as follows:
1
1 Tle
Rrc
Xr)
y
V&T
(76)
Tlye
where we further note that Eyy Cauchy-Schwartz incquality and the finite moment assumption, we have:
VGT = Ele] = O. By invoking the Chebyshev's Inequality,
(6)
X rc (Y
Yor)*)
S [
E (Y - Yar) S o0:
(77)
and
Tic
(1()) v Op
(78)
Trre
Hence, we have that:
XreXre
1
Xro)(2) ( (011)) +
1
=
1 (
Op
Op
(79)
Tre
Nre
σx
Nrc
Tire
In similar, we further check the term
x X
Xr))(3), where we first recall the consistency
Are
assumption that [i) yb)) = o(1). ,. We then expand the term 1 Xx)) as follows:
1 X, g(3)
Rec
yb (+)
yb
(80)
Thy
Tly
where we invoke Chebyshev's 's Incquality, Sauchy-Schwarre incquality and finite moment assumptions again:
E
[1[(16 X |2
< E
(J6
E
X 1121 = o(1),
(81)
and
[1 - Tb) )b)
Xrc 12 2
< [V6
E X | s co,
(82)
We then obtain that:
(001)=11))0) -00 -0)))(0)==0000) o1()
X,) =
(83)
Tre
Tre
Hence, the overall gap between and @ can be expressed as follows:
0-0=
(1).
(84)
Since the final estimation equals to Ø
xre
(2) we combine the consistency assumption that
yb
(4)
yb (i) 2 o11 and obtain the following final conclusion that:
E
VGT
(4) Ø
x
yb
(8)
E
(i) UCT
0
x
x
xxx(( 0 x + yi
(1)
(85)
E
Ub
(1)
Uh
(
TT (6)
0
(i) x
=0,(1).
Then our claims follows.
28

---

Published as a conference paper at ICLR 2026
Remark 4 (Feasibility of Unbiased Samples in Real Applications). Obtaining small (few-shot- scale) set of approximately unconfounded samples is feasible in multiple realistic settings. Below we outline representative examples.
First, in sentiment aaatiment-analyiis or review modeling, as shown in Appendix C and Figure 1, hidden confounders such regional culture context reverse the effect of sentiment aspects metrics like Popularity or Rating. In this case, unconfounded samples can be collected via controlled labeling. where independent human annotations are obtained without product or regional information; context randomization, by presenting the same text in varied neutral contexts; and temporal control, by selecting reviews from stable, non-pronne prmppoomooion periods. These controlled subsets, rypically containing only tens of samples, serve as the unconfounded set "D" used by DDbias.
Second, in recommender or ranking systems, large-scale logs are often confounded by exposure and position bias. However, most industrial systems already maintain small unbiased traffic bucket through A/B random exposure for evaluation purposes. Interactions from this random traffic, or from cold-start users, naturally satisfy approximate I x and can directly provide the required D
Finally, to verify or filter candidate unconfounded samples before applying DDbias, lightweight verification and diagnostic strategies can be employed. One approach is perform residual- fiiiual-featrre independence tests, such as HSIC or MMD, between residuals (ūs - y) and the features x. Another approach is sensitivity analysis, reporting the smallest confounding strength (or I'boond) that would alter the conclusions. These diagnostics provide a practical and low-cost pipeline for building validating D
Proposition (Weakly Confounded ICL Samples). Let the confounded sample X (cconf) contain Y&b unbiased samples with
Amin X (com (x xlcon())]] (x(on)(x)(om)))) 2 Ao 00
and let the corresponding noise satisfy [[msed] and B(s?] S 0². Then, for the DDbias estimator 0, there exists a constant such that
10 01l2 S C( (
Elyor @DEBB s C¹
(Tnss
Proof. From the DEB update rule,
0-0 0 = A ((con()eT)
An = 1 ((conf) [x(con((((coo))) (ccons)))
Thus,
10 10-011- 0112 - ||, 1 [((en)))
By the eigenvalue condition Amin Amin(An)) (An) 2 X., we have ||A, S 1/>.. Standard random matrix concentration yields
Xleonl) -= 1
n2
and the claim follows.
Proposition 3. (Mixed ICL Samples) Assume a fraction p of the "unconfounded" batch is contami- nated: for those contaminated samples E[x,e]] = rj & (denoted by X (cout)), ), while the remaining (1-p)ng samples are unbiased (denoted by ((elaan)). Suppose the clean subset is well-conditioned:
Amin (1-p)mg
(clenn) ((ceaan)T) 2 Ao 0.
Let the mean confounding strength be
Irsl.
prb
#Econt
Then there exist constants C, C' O Ouch that
0 - 001222 s C
(
ns + +
Elyer - dDEBl SC' (v7a-mm + +pF)
29

---

Published as a conference paper at ICLR 2026
Hence, the bias remains asymptoticallly negligible as n. -000 p-+ O, and a sufficient condition for asymptotic unbiasedness is
1 = o1).
p = o(), and
v(1 10m-p)
In particular, the practical bound
P TVPOA.
ensures the contamination term is dominated by the sampling term.
Proof. From the DEB estimator, we have:
i --,=-,"" X.ee )
An == 1 XroxTe
= y Do.
ny
Hence, the following upper-bound is derived:
||0 011 S |A 112
ni Xreg"
By the eigenvalue condition the clean block, ||A 1|2 S and contaminated parts:
Then split g into clean
g= (clean) + g (cont) = g(leeu)) c(oont) 4 p(
and we separately bound the three terms in below: (1) Clean noise aggregation:
(cleam) (clean))
O ((T))m)))
Og(
((L-))nsd.
(2) Random contaminated noise:
X (cont) (cont)) =0((V))
which is absorbed into constants for small
p.
(3) Systematic confounding bias:
(cont)
SO(PP)) dpr) after scaling by A 1|2.
rar
Combining all terms in (I) yields
00 1 011 s ( (Vi+mm + +F)
and the same order applies to the expected prediction error |E[yGT ØDEB] up to constant.
G EXPERIMENTAL RESULTS
G.1 COMPUTATIONAL RECOURSES
We perform all experiments using a Nvidia A 100 GPU, and all codes are implemented using Python.
G.2 VERIFICATION OF CONSTRUCTED WEIGHTS
We detail how we learn 8" by optimizing ('(w) =
2n Σn (w (x( w. 0) With the same
simulation protocols in generating the confounded data, generate w, (1) by analog, while generate ( (w, ( without considering By training the TF@ I and TF@3 such data, obtain the grounding 0* that is indeed obtained by optimizing over L'.
30

---

Published as a conference paper at ICLR 2026
L2-Div (log)
100
L2-Div (log)
100
U-Weights
U-Weights
Conf_ 0.5
Conf 2.0
3000
6000
9000
3000
6000 9000
Iteration
Iteration
1.0-
1.0-
Cos Div
Cos Div
0.5-
U-Weights Conf_
0.5-
U-Weights Conf 2.0
3000
6000
9000
3000
6000 9000
Iteration
Iteration
Figure 6: Comparison between U weights and Biased weights obtained from Pre-training TFs.
L.2 Div (log)
L2 Div (log)
L2 Div (log)
L2 Div (log)
10a.
102_
10a
10a
U-Weights
U-Weights
U-Weights
U-Weights
Conf _.5
Conf_1.0
Con
Conf_20
10-14
3000
6000
9000
3000
6obo
9000
3000
6000
9000
3000
60b0 9000
Iteration
Iteration
lteration
Iteration
1.04
1.0-
1.0-
Cos Div
0.8
Cos Div
0.8
Cos Div
0.6-
0.8
0.6-
0.4
0.4-
0.2
U-Weights
0.2-
U-Weights
0.2-
U-Weights
0.2-
U-Weights
0.0
Conf 0.5
0.0
Conf 1.0
0.0-
Conf 1.5
0.0
Conf 2.0
-0.2
-0.2
-0.2
3000 6000
3000 600e
3d00 6000 9000
3000
6000 9000
Iteration
Iteration
Iteration
Iteration
Figure 7: Comparison between estimation with biased weights pre-trained from confounded data and U weights we construct the main paper, with X
N(0,, I) and TF@3.
L2-Div (log)
L2-Div (log)
L2-Div (log)
L2-Div (log)
10a
10a_
100.
MW
109.
U-Weights
U-Weights
U-Weights
U-Weights
Conf_ 0.5
Conf
Conf_1.0
Conf Co5f_1.5
Conf_2.0
3000
6000
90b0
3000
6000
9000
3000
6000 9000
3000
6000 9000
Iteration
Iteration
Iteration
Iteration
1.0
10-
Cos Div
Cos Div
Cos Div
Cos Div
0.5
0.5
0.0
U-Weights
0.0-
U-Weights
U-Weights
0.0
U-Weights
Conf 0.5
Conf 1.0
Conf_1.1
-0.5-
Conf 2.0
3000 6000 9000
6
3000 6000 9000
3000 6000 9000
-1.0
3000 6000 9000 Iteration
Iteration
Iteration
Iteration
Figure 8: Comparison between estimation with biased weights pre-trained from confounded data and U weights we construct in the main paper, with X ~ (0, Σ) and TF@3,
31

---

Published as a conference paper at ICLR 2026
L2-Div (log)
L2-Div (log)
L2-Div (log)
L2-Div (log)
10.-
MW
10a-
100.
109
U-Weights
U-Weights
U-Weights
U-Weights
Conf_0.3
0.5
Conf_
Conf 2.0
3000
6000
9ob0
3000
6000
9000
3000
6000
9000
3000 6000 9000 Iteration
Iteration
Iteration
Iteration
1.04
1.0-
1.0-
Cos Div
0.5
Cos Div
0.5-
Cos Div
0.5-
Cos DW
0.5
0.0
U-Weights
0.0-
U-Weights
0.0-
U-Weights Conf 1.5
0.0
U-Weights
Conf_0.c 0.5
-0.5-
Conf_
_100
-0.5
-0.5
Conf 2.0
3000 6000 90'0 Iteration
0
3000 6000 9060
0
3000 6000 9000
3000 6000 9000
Iteration
Iteration
Iteration
Figure 9: Comparison between estimation with biased weights pre-trained from confounded data and U_ U_weights we construct in the main paper, with X ~ N Σ) and TF@1.
Cont.0.5
Cont.0.5
Con
Cont. 1.0
Cont_1.0
Cont
Cont_1.0
Cont_11 1.5
Cont, 1.5
1.5
Cont_1
Cont _or
Conl
Conl_2.
2.0
Conl
Corf_2.0 2
1250
Iberation
Iteration
Iteratio
Iteration
Iteration
Figure 10: Deviation of Pre-trained Weights along each feature dimension for the 1-th layer of TF@) 1 with I).
G.3 VERIFICATION OF DEVIATION IN PRE-TRAINING
The computational details of computing the weight divergence between , T" and Sh T exactly follow the protocols Def. To be specific, the derivative divergence defined as follows:
dLial
dLiell
S
(86)
dG st G ST
dG at
where we report the tensor-norm (L2) of A (s, T) the divergence between S T and So T along each feature dim (j).
G.4 VERIFICATION OF INFERENCE BIAS
When performing ICL inference, checking whether S TM aligns with Sh Th is complicated, as judging the distance between two high-dimensional tensors challenging. Alternatively, recalling
Parn biffal of 672 oo
pare Oiffal dugh Dim
L0009
20ene
scee-
of
reos
Joce
aces
Cont
Cont.0.5
Cont_3.0
Cort_1.
1.5
Cand
2000
C..
Corf 2.0
Curd 2.0
Cond 2.0
Corf 2.0
Curf 2.0
2009 4050 sibo 2008 Iteration
3009 4850 eioo 1550
2008 4550 r8bo 9906
2009 4283 #533 3006
2009 4850 4tbo 3350
Itraaion
iteration
Iterabion
Figure II: Deviation of Pre-trained Weights along each feature dimension for the 1-th layer of TF@3 with X ~ N(O, E).
32

---

Published as a conference paper at ICLR 2026
Sax-
Corl 0.5 Conf 1/0
Dff
Corl 0.5
Card 0.5
Conf u.5
Corf 0.5.
Cont 10
Cont
Conf 1.0
Cont.15
Pand
Gont 1.0
Corf 1.0 Cont.15
Con
Cont.2.0
Cont
4050 6093 6000 iteration
2003 600
4093 Iteration
0050
Iteration
Iteration
iterationn
Figure 12: Deviation of Pre-trained Weights along each feature dimension for the 2-th layer of TF@3
with X ~ NO0, E).
9231
0.80
0.25
0.29
9325
0.20
03330
0.15
Conf. 0.5
Conf 0.5
Corl 0.5
Conf 0.5
Corf 0.5
Conf 1.0
5.40
Conf 1.0
Conf 1.0
0.00
Cont 1/0
Conf 1.0
Conf.1.5
0.00
Cont.1.5
Cont 1.5
Con
Cont_2.0
Cont_2.0
Cont,2.0
033
4050 6000
<90 Iteration
0009
Acco 530c
Iteration
Iteration
Figure 13: Deviation of Pre-trained Weights along each feature dimension for the 3-th layer of TF@3 with X ~ NO,
the linearity of our experimental setup, we observe that it is sufficient to compare the grounding w. with the regressed w with predicted y and original x. To be specific, first perform OLS regression on predicted pairs (x("), y(1.,) j from S" 7 and obtain w By analog, we obtain w from 5 Th. Finally, we compare the 122similarity and Cos-similarity between w wb and w.. respectively.
G.5 VERIFICATION OF OUR PROPOSED DDBIAS METHON
We verify our proposed DDebias method using three metrics, including the prediction bias (comparing grounding (( and predicted (n++))), and the L2-Div and Cos-Div. To be specific, the L2-Div and Cos-Div are computed as the similarity between the grounding w, and OLS regressed w with the same protocols before.
G.6 VALIDATION ON REAL- WORLD DATASETS
We evaluate our method on two real-world datasets constructed from the publicly available Yelp restaurant review corpus, following the experimental settings in Cheng et al. (2022b/c). We refer to these two datasets as the Restaurant Popularity Index (RPI) dataset and the Restaurant Overall Rating (ROR) dataset.
Each dataset contains rich textual reviews, a 10-dimensional Multi-Aspeci Sentiment (MAS) repre- sentation (e.g., Food, Service), and restaurant metadata such category and location. Both datasets naturally exhibit hidden confounding, as previously reported in Cheng et al. (2022b)c). c). In the RPI dataset, the outcome variable the popularity index, defined as the real-worle foot-traffic score at Saturday 7-8 PM. The input feature is the 10-dimensional MAS vector, while the hidden confounder corresponds to socio-cconomic status or food preference (as illustrated in Fig. 6 in our revised Appendix). In the ROR dataset, the outcome is the aggregated restaurant star rating, with the same MAS input and the same type of hidden confounder (also illustrated in Fig.
In the experimental evaluation, the raw dataset to obtain a model exposed to real-world confounding. During the in-context learning (ICL.) stage, provide prompting examples that contain no hidden-confounder information. A set
LLaMA 3.2-3B unified base model and fine-tune it on
33

---

Published as a conference paper at ICLR 2026
of relatively unbiased samples selected by three human evaluators serves as the ICL prompting set. Specifically:
(I) The hidden confounder (socio-cconomic status food preference) cannot be directly queried from the dataset (each dataset contains approximately 6,000 instances);
(2) Domain knowledge in sentiment-review modeling indicates that the observable variable "location" is a reliable proxy for socio-economic status or food preference;
(3) With the assistance of GPT-5 reasoning, test the correlation between "location" and the MAS inputs, followed by verification by three human evaluators;
(4) We then select and rank the top 60 samples exhibiting verified independence between "location" and the input features, which constitute our unbiased ICL set;
(5) Finally, fine-tune the base LLaMA model on the remaining data and apply our DDbias method using the selected ICL examples.
For empirical comparison, we include three baselines:
Baseline uses the vanilla LLaMA model (fine-tuned on the full dataset without debiasing).
Baseline 2 uses DMCEE (Cheng al. 2022b), which employs observable metadata (e.g..
location, category) as proxy variables to estimate causal effects under hidden confounding.
Baseline 3 uses AAAI-SC (Cheng al. 2022c) which learns lowddimeniional surrogate confounders from the MAS representation via probabilistic factor modeling and adjusts for them during estimation.
We report the Mean Absolute Error (MAE) as the primary evaluation metric for both datasets (lower better).
The MAE values differ substantially in magnitude across the two datasets because the outcome vari- ables measured different scales. The RPI dataset uses popularity score ranging approximately from 0 to 100, whereas the ROR dataset uses the average star rating ranging from 1.0 to 5.0.
G.7 GENERALIZATION OF OUR CONCLUSION DEEPER, NON-LINEAR TFs
Moreover, we generate non-linear data in various cases with deeper, non-linear transformers (Softmax, Layemorm, Non-linear Activation with MLPs), further show the generalization of our theoretical conclusions on the bias characterization. In the nonlinear confounded data generating process, we adopt the additive model y = (x) e with ~ N(0,0.5) The confounder influences the covariates through a nonlinear transformation 2y ry h(e) Kýj. where ry ~ UO, 1), K3 ~ N(0, 1). and h(c) denotes the quadratic mapping. To examine robustness under different structural nonlincarities, we instantiate using three common forms:
(i) Polynomial mapping: [Pay(() = aTr + b||x||12 The coefficients are sampled as a (0, I).), b N00 of), where oa, σb E [0.5, 1.0] control the linear and quadratic contributions to the output. This mapping allows simple additive non-linear effects for benchmarking.
(ii) MLP-ReLU: fMLP(*) W. ReLU(W - by) + by The weights are sam- pled from Gaussian distributions: W, o(), W2 NN bj ~ (0, 0.12/), b2 (O0,0.13) where 01, [0.5, 1.0] control the signal strength. This ensures diverse non-linear mappings for our confounded DGP.
(iii) Softmax-activated: fsM(x) w T softmax(Ux= + by) The weights are sampled as ~ (O, 1), w ~ N (0, I) by ~ (0, 0.121), with oU, ow E [0.5, 1.0) to control output amplitude. This allows systematic control of signal-to-noise ratio in non- linear confounded data.
By scaling each confounding coefficient ry by factors in [0.5. 1.0, 1.5, 2.0], obtain datasets Nonlin-Conf,, that mirror the linear case and enable a controlled study how nonlinearity amplifies confounding bias.
34

---

Published as a conference paper at ICLR 2026
Analysis. By combining conclusions from Tab. 12 Tab. Tab. Tab. T0 Tab. 13 Tab. 14 Tab.
15
Tab. [16] Tab. [7] Tab. 18 Tab. and Tab. 20 we observe that our bias characterization remains consistent across wide spectrum of architectures and data-generating mechanisms. Specifically, across additive polynomial mappings, single-layer non-linear Transformers with ELU or ReI.U activations, ML.--ReI.U networks, and Softmax-activater mappings, the predicted increase of ICL prediction bias with rising confounding strength ry is consistently confirmed. Furthermore, deeper Transformers (L=3/5/7) exhibit the same qualitative trends, demonstrating that the theoretical lower bounds and deviation analyses derived in our paper are not limited to shallow or linear models. Overall, these extended experiments empirically validate that DDbias effectively corrects for heterogeneous confounding effects across non-linear and deeper architectures, reinforcing the robustness and scalability of our approach beyond the standard linear ICL setting.
Table 9: f Ploy, Prediction error of non-linear Transformers (single-layer MLP + softmax + ELU) under different confounding strengths Columns denote confounding strengths, rows denote ICL sample sizes and the prompiing/peetraining sample ratio. Lower is better.
ICL Samples Ratio
Conf@@05
Conf@ 1.0
Conf@1.5 1.5
Conf@2.0 2.0
Biased (Vanilla TF)
0.182
0.236
0.314
0.417
DDbias (Ours): Ratio (1.6 x 10 10-3)
0.072
0.098
0.130
0.178
DDbias (Ours): Ratio (2.4 10-3)
0.070
0.095
0.126
0.173
DDbias (Ours): Ratio (3.2 x 10
0.069
0.093
0.124
0.171
DDbias (Ours): Ratio (6.4 x 10-3)
0.068
0.091
0.120
0.167
Table 10: f Ploy, Prediction error of non-linear Transformers (single-layer MLP softmax +
ReLU) under different confounding strengths r. Columns denote confounding strengths, rows denote ICL sample sizes and the prompting/prerraiinng sample ratio. Lower better.
ICL Samples Ratio
C.
Conf@ 1.0
Conf@ 1.5
Conf@2.0
Biased (Vanilla TF)
0.195
0,254
0.336
0.452
DDbias (Ours): Ratio (1.6 10-3)
0.083
0.111
0.144
0.195
DDbias (Ours): Ratio (2.4 x 10-3)
0.081
0.109
0.141
0.193
DDbias (Ours): Ratio (3.2 x 10-3)
0.080
0.107
0.138
0.190
DDbias (Ours): Ratio (6.4 x 10-3)
0.078
0.105
0.134
0.187
Table 11: f == Ploy, Prediction error of deeper, non-linear Transformers (ReLU, 5 7, Lay-
erNorm, Non-linear MIPs, Softmax, Ratio=1.61e-9) under different confounding strengths r. Columns denote confounding strengths, rows show ICI. sample sizes and different model depths L. Lower better.
ICL Samples L
Conf@0.5
Conf@ 1.0
Conf@ 1.5
Conf@2.0
L=3
Biased (Vanilla TF)
0.205
0.290
0.380
0.500
L=3 DDbias (Ours)
0.090
0.130
0.170
0.220
I=5 Biased (Vanilla TF)
0.280
0.370
0.475
0.600
L=5 DDbias (Ours)
0.115
0.150
0.200
0.260
L=7 Biased (Vanilla TF)
0.320 0.135
0.450
0.600
0.750
L=7 DDbias (Ours)
0.175
0.225
0.290
G.8 ROBUSTNESS ANALYSIS ON PARTIALLY CONFOUNDED DATA
Moreover, we also performed experiments in our synthetic data setup, by simulating two cases the partially confounded data, i.e., the weakly confounded data
35

---

Published as a conference paper at ICLR 2026
Table 12: f Ploy, Prediction error of deeper, non-linear Transformers L-3/5/1 LayerNorm, Non-linear MLPs, Softmax, Ratio=1.61k 1.)e-)))))))) under different confounding strengths Columns denote confounding strengths, rows show ICL sample sizes and different model depths Lower better.
ICL Samples L
C.5f@0.5
Conf@ 1.0
Conf@ 1.5
Conf@2.0
L=3
Biased (Vanilla TF)
0.190
0.270
0.360
0,470
L=3 DDbias (Ours)
0.085
0.125
0.165
0.215
L=5 Biased (Vanilla TF)
0.250
0.335
0.440
0.580
1 DDbias (Ours)
0.110
0.145
0.190
0.250
1 Biased (Vanilla TF)
0.280
0,410
0.550
0.700
1 1 DDbias (Ours)
0.120
0.160
0.210
0.280
Table 13: f MLP-Relu, Prediction error of non-linear Transformers (single-layer MLP softmax
+ ELU) under different confounding strengths r. Columns denote confounding strengths, rows denote ICL sample sizes and the prompting/prerraninng sample ratio. Lower better.
ICL Samples Ratio
Conf@0.5
Conf@ 1.0
Conf@ 1.5
Conf@2.0
Biased (Vanilla TF)
0.230
0.310
0.405
0.540
DDbias (Ours): Ratio (1.6 10 10-3)
0.112
0.148
0.188
0.248
DDbias (Ours): Ratio (2.4 x 10 10-3)
0.109
0.144
0.184
0.242
DDbias (Ours): Ratio (3.2 x 10-3)
0.107
0.142
0.181
0.238
DDbias (Ours): Ratio (6.4 10 10-3)
0.104
0.138
0.176
0.232
Table 14: f MLP-Relu, Prediction error non-linear Transformers (single-layer + MLP softmax + ReLU) under different confounding strengths Columns denote confounding strengths, rows denote ICL sample sizes and the prompting/peerraniing sample ratio. Lower is better.
Model (L) Method
Conf@0.5
Conf@ 1.0
Conf@ 1.5
Conf@22 2.0
L=3 Biased (Vanilla TF)
0.260
0.345
0,455
0.610
L=3 DDbias (Ours)
0.118
0.155
0.198
0.265
L=5 Biased (Vanilla TF)
0.310
0.420
0.545
0.710
L=5 DDbias (Ours)
0.135
0.175
0.225
0.300
1=7 Biased (Vanilla TF)
0.360
0.505
0.660
0.850
1=7 DDbias (Ours)
0.150
0.195
0.250
0.330
Table 15: f MI.P-Relu, Prediction error of deeper, non-linear Transformers (ReI.U, L=3 7, LayerNorm, Non-linear MIPs, Softmax, Ratio=1.61e-3)) under different confounding strengths Columns denote confounding strengths, rows show ICL sample sizes and different model depths L. Lower better.
ICL Samples I.
Conf@0.5
Conf@ 1.0
Conf@1.5
Conf@ 2.0
L=3 Biased (Vanilla TF)
0.260
0.345
0.455
0.610
L=3 DDbias (Ours)
0.118
0.155
0.198
0.265
L=5 Biased (Vanilla TF)
0.310
0.420
0.545
0.710
L=5 DDbias (Ours)
0.135
0.175
0.225
0.300
L=7 Biased (Vanilla TF)
0.360
0,505
0.660
0.850
L=7 DDbias (Ours)
0.150
0.195
0.250
0.330
To show more generlized conclusions, we adopt the non-linear, deep TF model (L=5, LayerNorm, relu activation, Softmax) the base model;
36

---

Published as a conference paper at ICLR 2026
Table 16: f MLP-Relu, Prediction error of deeper, non-linear Transformers (ELU, L=3 5 LayerNorm, Non-linear MLPs, Softmax, Ratio=1.6le-) under different confounding strengths Columns denote confounding strengths, rows show ICL sample sizes and different model depths L. Lower is better.
ICL Samples L
Conf@0.5
Conf@ 1.0
Conf@ 1.5
Conf@ 2.0
L=3 Biased (Vanilla TF)
0.245
0.330
0,425
0.560
L=3 DDbias (Ours)
0.112
0.150
0.195
0.255
L=5 Biased (Vanilla TF)
0.295
0.395
0.520
0.670
1=5 DDbias (Ours)
0.130
0.170
0.220
0.290
L=7 Biased (Vanilla TF)
0.335
0.465
0.610
0.780
L=7 DDbias (Ours)
0.145
0.188
0.240
0.315
Table 17: f Softmax, Prediction error of non-linear Transformers (single-layer MLP softmax + ELU) under different confounding strengths r. Columns denote confounding strengths, rows denote ICL sample sizes and the prompiing/pretrainnng sample ratio. Lower is better.
ICL ICL Samples / Ratio
Conf@0.5
Conf@ 1.0
Conf@ 1.5
Conf@ 2.0
Biased (Vanilla TF)
0.205
0.280
0.375
0.500
DDbias (Ours): Ratio (1.6 x 10
0.092
0.122
0.162
0.218
DDbias (Ours): Ratio (2.4 x 10
0.089
0.119
0.158
0.212
DDbias (Ours): Ratio (3.2 x 10-3)
0.087
0.117
0.155
0.208
DDbias (Ours): Ratio (6.4 x 10-3)
0.085
0.113
0.150
0.202
Table 18: f Softmax, Prediction error of non-linear Transformers (single-layer MLP softmax + ReL.U) under different confounding strengths r. Columns denote confounding strengths, rows denote ICL sample sizes and the prompting/preerraning sample ratio. Lower is better.
Model (L) Method
Conf@0.5
Conf@ 1.0
Conf@ 1.5
Conf@ 2.0
L=3 Biased (Vanilla TF)
0.225
0.315
0,420
0.560
L=3 DDbias (Ours)
0.100
0.135
0.175
0.235
L=5 Biased (Vanilla TF)
0.275
0.380
0.500
0.660
L=5 DDbias (Ours)
0.115
0.155
0,200
0.270
1=7 Biased (Vanilla TF)
0.315
0.460
0.615
0.800
1=7 DDbias (Ours)
0.130
0.175
0.230
0.305
Table 19: Softmax, Prediction error of deeper, non-linear Transformers (ReLU, L=3 /
LayerNorm, Non-linear MI.ss, Softmax, Ratio=1.61e-3) under different confounding strengths r. Columns denote confounding strengths, rows show ICI. sample sizes and different model depths T.. Lower is better.
ICL Samples I.
Conf@0.5
Conf@ 1.0
Conf@ 1.5
Conf@ 2.0
I==3 Biased (Vanilla TF)
0.225
0.315
0.420
0.560
L=3 DDbias (Ours)
0.100
0.135
0.175
0.235
L=5 Biased (Vanilla TF)
0.275
0.380
0.500
0.660
L=5 DDbias (Ours)
0.115
0.155
0,200
0.270
L=7 Biased (Vanilla TF)
0.315
0.460
0,615
0.800
L=7 DDbias (Ours)
0.130
0.175
0.230
0.305
We report performance of vanilla TF, our DDbias with clean ICL examples, and our DDbias
with weakly confounded examples in Table 21 23 and 22 with difference prompting sample ratio;
37

---

Published as a conference paper at ICLR 2026
Table 20: Softmax, Prediction error of deeper, non-linear Transformers (ELU, L=3 / 7, LayerNorm, Non-linear MLPs, Softmax, Ratio=1.61e-3 under different confounding strengths Columns denote confounding strengths, rows show ICL sample sizes and different model depths L. Lower is better.
ICL Samples / L
Conf@0.5
Conf@ 1.0
Conf@ 1.5
Conf@2 2.0
L=3 Biased (Vanilla TF)
0.215
0.305
0,400
0.525
L=3 DDbias (Ours)
0.095
0.128
0.167
0.225
L=5 Biased (Vanilla TF)
0.260
0.365
0,485
0.640
1=5 DDbias (Ours)
0.110
0.150
0.195
0.260
L=7 Biased (Vanilla TF)
0,300
0.435
0,580
0.760
L=7 DDbias (Ours)
0.125
0.170
0.220
0.292
Table 21: Prediction error under weak confounding (Weak Conf@0.10)
Method / Sample Ratio (x10-3)
1.6
2.4
3.2
6.4
Biased (Vanilla TF)
0.240
0.300
0.360
0.430
DDbias (pure clean examples)
0.090
0.085
0.082
0.080
DDbias (weakly confounded)
0.105
0.100
0.097
0.094
The above results across consistently show the robustness of our proposed DDbias even with weakly confounded data;
At the same time, results in Table 2ahle24jan inform the cost of using weakly confounded data, i.e., the increase ICL sample number.
Finally, added experiments in our synthetic data setup, by simulating mixed ICL data:
To show more generlized conclusions, we adopt the non-linear, deep TF model (L=5, LayerNorm, relu activation) as the base model;
We report performance of vanilla TF, our DDbias with clean ICL examples, and our DDbias with mixed ICL examples in Table 10 with difference ratio p;
Results in Table 10 consistently show the robustness of our proposed DDbias even with small p, i.e. confounded examples consist of small part of the whole ICL sample set;
Table 22: Prediction error under weak confounding (Weak Conf@0.15)
Method / Sample Ratio (x (x10-3)
1.6
2.4
3.2
6.4
Biased (Vanilla TF)
0.260
0.330
0.400
0.490
DDbias (pure clean examples)
0.100
0.095
0.092
0.090
DDbias (weakly confounded)
0.110
0.105
0.102
0.099
Table 23; Prediction error under weak confounding (Weak Conf@ Conf@0.30)
Method Sample Ratio (x10-
1.6
2.4
3.2
6.4
Biased (Vanilla TF)
0.270
0.350
0.440
0.560
DDbias (pure clean examples)
0.110
0.105
0.102
0.100
DDbias (weakly confounded)
0.115
0.110
0.107
0.104
G.9 COMPARISON WITH IV-BASED ICL DEBIASING METHOD
38

---

Published as a conference paper at ICLR 2026
Table 24: Effect on the Prompting Sample Ratio (Weak Conf@0.10)
ICL prediction error
0.090
0.092
0.095
0.100
Sample ratio for DDbias (oracle) (x10-3)
6.4
3.2
2.4
1.6
Sample ratio for DDbias (weak) (x10->)
8.6
7.2
6.3
3.5
Table 25: Prediction error under mixed ICL prompting.
Method / Sample Ratio (x10-3)
1.6
2.4
3.2
6.4
Biased (Vanilla TF)
0.335
0.370
0.388
0.420
DDbias (pure clean examples)
0.145
0.132
0.117
0.106
DDbias (mixed examples, p 0.1)
0.190
0.140
0.132
0.118
DDbias (mixed examples, p 0.3)
0.218
0.176
0.144
0.120
DDbias (mixed examples, 0.5)
0.290
0.260
0.220
0.235
To compare DDbias with an IV-based approach, we extend our original confounded DGP by in- troducing instruments Z. We first draw Z ~ (0, Im), a latent confounder U ~ [0, o and noise e()) ~ (0, o²). Features are generated by combining instrument relevance and con- founding: zj (1) (Tz()), + a, + nxy where I controls instrument strength and a, matches the heterogeneous confounding coefficients Outcomes follow y = _(0) wc + +, + (O, with w' ~ 11). ). By construction, Z L. (U,e) (instrument exogeneity) and + X via (relevance), yielding clean comparison: IV exploits large confounded observational data with valid instruments, whereas DDbias requires only small unconfounded batch.
Remark 5. We do not simulate non-linear IV data generation, existing ICL-based debiasing method Liang el al. (2024) relies on the two-stage linear debiasing frumeworks (Angrist el al.) 1996). Although we nole that non-linear, deep IV methods also exist (Hartford al. [017]. extending IV-based approaches towards non-linear DGP under the regime of ICI is out of our scope in this paper.
Comparison with clean data. To be first, we compare our DDbias with IV-based approach Liang al. (2024) in a perfect regime, i.e., both unconfounded ICL samples and valid IVs can be acquired. We report results in Tab. 26] Experimental trends inform that, in both perfect regimes, our method slightly performs betther than IV approaches.
Table 26: f = lincar, Prediction bias comparison (IV vs DDbias) across Transformer depths and context sizes.
Model / Method
n=50
n=30
n=20
n=10
L=3 / IV
0.102
0.142
0.182
0.228
L=3 DDbias
0.090
0.130
0.170
0.220
L=5 / IV
0.118
0.165
0.208
0.268
L=5 DDbias
0.115
0.150
0.200
0.260
L=7 / IV
0.145
0.188
0.235
0.298
L=7 DDbias
0.135
0.171
0.222
0.290
Comparison with partially confounded data. Then, we compare our DDbias with IV-based approach in an imperfect regime, i.e., partially confounded ICL. samples are provided but valid IVs are accessible. In this regime, adopt the mixed ICL sample case. As reported in Tab, 27 we observe that with partially confounded data, our proposed DDbias achieves worse prediction performance than IV-ea ICL approach, while with large ICL samples (n = 50, method achieves certain robustness when is small).
Comparison with Confounded IVs. Finally, compare DDbias with IV-based approach in another imperfect regime, i.e., clean ICL samples are provided but only confounded IVs are accessible. As reported in Tab. 28 observe that with weakly confounded IVs, increasing ICL.
39

---

Published as a conference paper at ICLR 2026
Table 27: Prediction Bias comparison (IV vs DDbias) under Partially Confounded ICL Samples and
Valid IV.
Model Method
n=50
n=30
n=20
n=10
L=3 / IV (Valid IV)
0.105
0.145
0.185
0.222
L=3 DDbias (p = 0.3)
0.118
0.176
0.213
0.268
L=5 / IV (Valid IV)
0.118
0.162
0.205
0.265
L=5 / DDbias (p 0.3)
0.150
0.185
0.247
0.306
L=7 IV (Valid IV)
0.145
0.188
0.235
0.300
7 DDbias (p =p=00 0.3)
0.157
0.218
0.245
0.342
samples result significantly prediction bias, which opposite the trend of our DDbias under partially confounded data. Moreover, such a phenomenon informs that the imperfect acquisition of IVs results in structural bias in estimation, which cannot be canceled by the increasing ICL samples.
Table 28:
Prediction Bias under Invalid and Unconfounded ICL. Samples, where we set the correlation between IV Z and to 0.1 in the IV-oriented DGP.
Model / Method
n=50
n=30
n=20
n=10
L=3 IV (Weak IV)
0.423
0.351
0.280
0.265
L=3 DDbias
0.090
0.130
0.170
0.220
L=5 IV (Weak IV)
0.500
0.420
0.340
0.291
L=5 DDbias
0.115
0.150
0.200
0.260
L=7 IV (Weak IV)
0.600
0.500
0.400
0.312
L=7 DDbias
0.135
0.171
0.222
0.290
40

---

Published as a conference paper at ICLR 2026
L2 Div (log)
Unconf
Div (log)
10a
Unconf
12 Div (log)
10a-
Unconf
Div (log)
10a
Unconf
Conf_0 0.5
Conf 1.0
Conf 1.5
Conf 2.0
10
10
10
10-1
3000
6000
9000
3000
6000
9000
3000
6000
9000
3000 60b0 9000
Iteration
Iteration
Iteration
Iteration
1.0
1.0-
1.0-
1.0-
Unconf
Unconf
Unconf
Unconf
Cos Div
Cos Div
Cos Div
Conf 0.5
Conf 1.0
Conf 1.5
Conf 2.0
Cos
0.0
0.0-
0.0-
0.0-
3000 6000
90b0
6
3000 6000 9000 Iteration
0
3000 6600 9000
3000 60bo 9000
Iteration
Iteration
Iteration
Figure 14: Comparison between estimation with biased weights pre-trained from confounded data
and weights pre-trained on unconfounded data, with X ~ (0, Σ) and TF@
L2 Div (log)
10a_
Unconf
Div (log)
10a-
Unconf Conf_14
Div (log)
10a-
Unconf Conf 1.5
Div (log)
10a4
Unconf
Conf 0.5
1.0
Conf _.0
10-1
10-
10-1
10-1
10-2
10-1_
10-2.
10-2
3000
6000
9000
3000
6000
9000
3000
6000
9000
3000
6000
9000
Iteration
Iteration
Iteration
Iteration
1.0
1.0
Unconf
Unconf
Unconf
Unconf
Cos Div
Cos Div
Cos Div
Conf_0.5..
Conf
Conf
0.0
0.0-
0.0-
0.0-
3000 6000
9000
3000 6000
9000
3000 6000
9000
3000 6000
9000
Iteration
Iteration
Iteration
Iteration
Figure 15: Comparison between estimation with biased weights pre-trained from confounded data and weights pre-trained on unconfounded data, with X N(0, 1) and TF@3.
G.10 NON-GAUSSIAN ANALYSIS
We show the properties of the constructed U_ weights, deviation of pre-trained weights, inference
bias of the ICL stage and the effectiveness our proposed DDbias method the regime of Gamma
L2 Div (log)
10a
Unconf
Div (log)
10a
Unconf
Div (log)
10a
Unconf
L2 Div (log)
10a-
Unconf Conf_22 2.0
Conf 0.5
Conf 1.0
Conf 1.5
10
10-1-
10-1
10-2.
10-1_
10-2
1
3000
6000 9000
3000
6000
9000
3000 6000
9000
3000
6000 9000
Iteration
Iteration
Iteration
Iteration
1.0-
Unconf Conf_0.5
Unconf
Unconf
Unconf
Cos Div
Cos Div
Cos Div
Cos Div
Conf 1.0
Conf
1.5
Conf 2.0
0.0-
MJJUI
-0.5-
-0.5
6
3000 5000 9000
3000
600D 9000 Iteration
3000 6000 9000 Iteration
3000 6000 9000 Iteration
Iteration
Figure 16: Comparison between estimation with biased weights pre-trained from confounded data and weights pre-trained unconfounded data, with X N(0 Σ) and TF@3.
41

---

Published as a conference paper at ICLR 2026
Conf@2.0
Confo11.5 1.5
Conf@11.
Conf@o05
Pred Bias (log)
Blased DDbias-Ours
Pred Bias (log)
Blased
Pred Blas (log)
Blased
Pred Blas (log)
Blased
DDbias-Ours
DDblas-Ours
100
DOblas-Ours
10-1
10
10
10-1
1.6
2.4
6.4
1.6
3/2 (xle -)
16
3/2
16
3.2
Ratio (xle
-3)
Ratio
Ratio
(xle-3)
Ratio
(xle-9)
Figure 17: Prediction comparison of proposed Double-Debiasing method with increasing prompting/pre-tramin training sample ratio with Tf@ I, where X ~ (O,,E).
Conf@2.0
Conf@115
Conf@110
Conf@o 5
Pred Bias (log)
10-1_
DDbias
Blased
Pred Blas (log)
10-1-
DDbias-Oul
Blased
Pred Blas (log)
10-1
DObias-Our
Blased
Pred (log)
10-1
Blased DDbias-Ours
10-2-
10-2-
10-1
10-3-
10-3
10-3
10-4_
10
10
1.6 2.4
3.2
16
3.2
3.2
Ratio (xle
Ratio (xle
Ratio
(x (x1))3)
Ratio (xle-3)
Figure 18: Prediction comparison of proposed Debble-Debiasing method with increasing prompting/pre-trainn sample ratio with Tf@ where NOO,
Cont@2.0
Conf@1
Conf@1.0
Conf@0.
10
Pred Bias (log)
Blased
Pred Blas (log)
Blased
Pred Blas (log)
107
Blased
Pred Bias (log)
107
Biased
10
Ours
10
10
10
103.
10a
10
10a
10-3
10-1_
10-1-
1.
1.6
6.4
3.2
24 3.2
Ratio (xle
Ratio (xle
Ratio (xle-3)
Ratio (xle-3)
Figure 19: Prediction comparison of proposed Double-Debiasing method with increasing
prompting/pr--traning sample ratio with Tf@3, where X NO0,E).
Conf@0.5
Conf@110
Conf@1.5
Conf@2.0
0.8
0.8
Div
0.6
0.6
0.6-
Biased
Biased
0.4
Blased
0.44
Biased
0.2-
0.2
0.2
1.6
2/4
3/2
6.4
1.6
2.4
3.2
6.4
1.6
2.4
3
6.4
1.6
2.4
3/2
6.4
Ratio
(x)
Ratio (xle--)
Ratio (xle-3)
Ratio (xle-1)
Confg005
Conf@1.0
Confi
Conf@22.0
1.00-
1.00-
Cos Div
0.95
Oubs
0.95
DDbias-Durs
0.95
DDbias-Ours
0.95
DDbi as Ours
Cos
0.90
0.90
0.90
0.85
0.85
0.85
1.6
1.6
3.2 Ratio (xle-3)
24
16
2
3/2
6.4
Ratio (xle-3)
Ratio (xle-3)
Ratio (xle-3)
Figure 20: Comparision of Cos Divergence and 1.2 Divergence of our proposed Double-Debiasing method with increasing prompting/pre. traieiiiiinn sample ratio with Tf@ 1, where X X~NN N E)
42

---

Published as a conference paper at ICLR 2026
Conf@e05
Confe910
Confor1.5
Conf@22. 0
0.5
0.4
0.4-
0.4
Div
0.3
DOblas
0.3-
DDblas- Ours
0.3
DDbias
DiBlas-Ours
12
0.2
Blased
0.2
Biased
0.2-
Biased
Biased
0.1-
0.1
0.1
0.0
0.0
0.0
0.04
1.6
2/4
3/2
6.4
1.6
2.4
3.2
6.4
1.6
2.4
3.2
6.4
1.6
2.4
3/2
6.4
Ratio (x (xle-a)
Ratio (xle-3)
Ratio (x1e-3)
Ratio (xle-a)
Conígg0.5
Conf@1.0
Confg115
Conígi2.0
0.99
0.995
Cos Div
0.99
Cos Div
0.99
DOBaa--Our
0.98
DOblas-Ours
DDblas-Ours
0.990
DDtlas-Ours
0.98
cs
0.97
0.98
0.985
0.97
0.96
0.97-
0.980
3/2 Ratio (xle-²)
6.4
1.6
2/4
3/2
6.4
Ratio (x lele-)
Ratio (xle-)
Ratio (xle-)
Figure 21: Comparision of Cos Divergence and 1.2 Divergence of our proposed Double-Debiasing method with increasing prompiing/rre-traning sample ratio with Tf@3, where N(0, E)
Conf@005
Confe810
Confi81.5
Corf@220
10a
10a
109.
10a
13 Div
10a1
DDbias Ours
10
Dublas Ours Biased
10-1
DDbias Ours Biased
L2 Div
10
DDbias Durs
10-
Biased
10-2
Blased
10
10
10-1
10-3
1.6
2.4
3.2
6.4
1.6
2.4
3.2
6.4
1.6
2.4
3
6.4
1.6
2.4
3.2
6.4
Ratio (xle-3)
Ratio (xle-3)
Ratio (xle-3)
Ratio (xle-2)
Confe10.5
Conf@:10
0.75
0.75
0.8
0.8
Div
DDbias Durs
Cos Div
Doblas Ours
0.6
DDblas Our
0.6
DDblas Durs
0.4
0.25
0.25
0.2-
0.2-
0.0
0.0
2.4
16
2/4
3/2
6.4
Ratio (xle-3)
Ratio (xle-3)
Ratio (xle -3)
Ratio (xle-3)
Figure 22: Comparision of Cos Divergence and 1.2 Divergence of our proposed Double-Debiasing method with increasing prompting/pre-trainigg sample ratio with Tf@3, where ~ N(0 T).
43

---

Published as a conference paper at ICLR 2026
0.9
-102
Pred L2/222 Div
Cos Div
0.6
-10
Cos Div (L)
0.3
L2 Div (R)
-10
Pred Div (R)
0.0
-10
3000
6000
9000
lteration
Figure 23: Checking U_ weights on Non-Gaussian Data, with X Gamma(0.1, 10) and TF@ 1.
L2-Div (log)
L2-Div (log)
(6o1) 12--1A
Sane (log)
10a.
Wry-
100.
10a
100-
U-Weights
U-Weights
U-Weights
U-Weights
Conf_0.5
Conf 1.0
Conf 1.5
Conf 2.0
3000
6000
9000
3000
6000
9000
3000
6000 9000
3000
6000 9000
Iteration
Iteration
lteration
Iteration
1.0-
1.0-
1.0-
Cos Div
Cos Div
Cos Div
Cos Div
0.5-
0.5
0.5
U-Weights
U-Weights
U-Weights
U-Weights
Conf 0.5
Conf
0.0-
Conf
3000
9000
3000 6000 9000 Iteration
3000 6000
9000
3000 6000 Iteration
9000
Iteration
Iteration
Figure 24: Comparison between estimation with biased weights pre-trained from confounded data and U weights construct in the main paper, with X Gamma(0.11 10) and TF@1.
distribution, where X is sampled from (0.1, 10), together with the noise in Fig 23 and
24
25
L2 Div (log)
10a_
Unconf
12 Div (log)
10a_
Unconf
DDlvvi (log)
109_
Unconf
L2 Div (log)
10c
Unconf
Conf 0.5
Conf 1.0
Conf 1.5
Conf_22 2.0
12 DIV
3000
6000
9000
3000
6000 9000
2000 6obo 9000
3000
6000 900o
Iteration
Iteration
Iteration
Iteration
1.0
1.0
1.0
Unconf
Unconf
Unconf
Unconf
Cos Div
Div
Cos Div
Cos Div
Conf_0.5
0.5
Conf 1.0
0.5
Conf. 1.5
D.5-
Conf
0.0-
0.0
0.0
0.0
0
3000
6000
9000
3000
6000 9000
3000
6000
9000
300D 6000
9o00
Iteration
Iteration
Iteration
Iteration
Figure 25: Comparison between estimation with biased weights pre-trained from confounded data and weights pre-trained unconfounded data, with X Gomma and TF@3.
44

---

Published as a conference paper at ICLR 2026
Coní. U.S
Cont 1.0
Para Dif 24h DI
Cont 1.0
Cont 0.5
Canif 0.5
Cont 1,0
Cant 0.5
Cont 1.0
Conr 1.0
Carf 0.5
wee-
Cont.
Co
Cont.1.5
Cont
Pera Dif 4th
Cont.1.5
5000
Cont_1.5
sre
1099
206-
2009
5300
4CE3 6003
2003 4093 <E9O Iteration
ESSO
2006 A0DE 2048 0003
ACES 6093
2000 4006 2006 2006
Iteration
Iteration
Figure 26: Deviation of Pre-trained Weights along each feature dimension for the 1-th layer TF@33
Cont.
cont.0.5
Cont.o.5 Com_1.0
Cont.o.5
Cont.0.5
Diff 1-th
Co
Coet
Canf Cont1.0
Cant 3.5
Co
Cvef_1i 1
Guel
para Dif dl
20h 23n r00s vare Beeation
205h zPn FPSS Beration
28t
2991 v2nn
206
2P53 APSS
2001 25mh 1003 wee
Iberation
Iteaatio
Figure 27: Deviation of Pre-trained Weights along each feature dimension for the 2-th layer TF@3.
012
C.5
24h Dien
0.0
Eaei_o. 0.5
Conf 0.5 Curf 3.0
Coef 1.0
Cari
Coo
Cunl 1.0
Car
1.5
333
Cur Diffe
204
0.4
0.02
0.00
0.00
2950
FESS
1990
2350
#0D FESE
2093
2009
2890
2050 REES
1993
3950
495 F099
1990
Iteration
Figure 28: Deviation of Pre-trained Weights along each feature dimension for the 3-th layer of TF@3.
Conf@0.5
Confg1.0
Conf@115
Conf@22
2.0
1.00-
2.0
1.5-
Div
0.75
DDbiassOurs
DDbias-Ours
DDbias-Ours
1.5-
DDbias-Ours
1.0-
Blased
Blased
1.0-
1.0-
0.25
0.5-
0.00
0.0
1.6
6.4
1.6
2.4
1.6
2.4
3.2
6,4
1.6
2.4
3/2
6.4
Ratio (x le 3)
Ratio (xle
Ratio (xle 3)
Ratio (xle 3)
Configo.5
Conf@31.
Conf@e.5
Conf@2.0
1.0
1.0
0.9-
0.8
0.84
Cos Div
0.6
Dobias-Qurs
0.6
DDbias-Ours Blased
DDbies-Ours Blased
0.7
Biased
Gos
8
8
Blased
0.6-
0.4-
0.2
0.0
3/2 Ratio (x le
2/4
1.s
2'4
1/2 3)
6'4
1.6
2/4
3'2
6/4
1.s
3/2 Ratio (xle
2/4
2)
Ratio (xle
Ratio (xle
3)
3)
Figure 29:
Comparision of Cos Divergence and L2 Divergence of our proposed Double- Debiasing method with increasing prompting/pre-traiing sample ratio with TT@ 1, where X (011, 10)
45


Reviews:
[{"review_id": "Zxpw6Fwt6m", "reviewer": "Reviewer_gV7e", "summary": "This paper introduces Endogenous In-Context Learning, a new theoretical framework revealing that pre-trained Transformers can produce biased in-context predictions when trained on data containing hidden confounders. While prior ICL analyses assumed that there is no confounding, the authors show that endogeneity induces systematic prediction bias proportional to the confounding strength. They provide formal proofs that Transformer parameters learned on confounded data deviate from unbiased reference weights and this deviation propagates into biased ICL predictions. To address this, they propose Double-Debiasing (DDbias), a simple gradient-free prompting method that corrects such bias using only a few unconfounded examples by prompting the model twice. Several simulations on linear regression settings verify the theoretical results and demonstrate that DDbias effectively mitigates bias without retraining.", "questions": "* Comparison with existing deconfounding methods. If it is possible to identify a small set of debiased dataset, does the proposed method outperform other baseline debiasing methods? I am wondering if the authors can provide experimental results for performance comparison.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* This paper shows a novel theoretical problem formulation. While there is a similar work, this paper introduces the Endogenous In-context learning setting, addressing hidden confounders in ICL, which is previously under-explored source of bias from other existing theoretical works regarding ICL. \n* This paper provides well-grounded theoretical results providing clear formal analyses which characterizes how hidden confounders affect both the pre-training and inference results of Transformers. This shows that prediction bias scales proportionally to confounding strength. \n* Simulation results comprehensively verify the theoretical claims, demonstrating consistent trends between confounding strength, parameter deviation, and prediction bias reduction after applying their method.", "weaknesses": "* The theoretical analysis is limited to linear self-attention Transformers and linear regression tasks. It remains unclear whether the same bias characterization---particularly the proportionality to the confounding strength---extends to deeper, nonlinear, or multimodal Transformer architectures used in real-world ICL. Even if this might be very hard to be theoretically shown, I believe that providing experimental analyses about this scalability to larger models would make this paper much stronger.\n\n* The proposed method requires access to a small set of unconfounded examples. In realistic settings, how can such unbiased samples be obtained or verified? Are there practical diagnostics or robustness analyses for partially confounded data? And can we utilize such approaches to apply the proposed method?\n\n* Section 2.3 claims that a previous study (Liang et al) for confounding issue in ICL did not consider the gap between the traditional regression problem and ICL. However Theorem 2 later concludes that the ICL prediction bias is proportional to $r$ --- exactly the same form as in classical OLS theory. What specific term or mechanism in ICL makes this result fundamentally new rather than a re-derivation in a different setting? And are there concrete theoretical differences in constants, variance scaling or convergence behavior between EICL and standard OLS?\n\n* All experiments are synthetic and rely on controllable confounding  strengths r. It would be valuable to evaluate the framework on real-pre-trained models or naturally confounded datasets (e.g., sentiment review) to demonstrate practical relevance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ITT1RymsKY", "reviewer": "Reviewer_rnrM", "summary": "This paper addresses the common neglect of hidden confounders in current In-Context Learning (ICL) theory during both pre-training and inference, and proposes a new framework called Endogenous ICL (EICL). The authors theoretically prove that Transformers trained with confounded data produce prediction bias proportional to the confounding strength. They construct parameter settings (U_weights) that can yield unbiased predictions even when test prompts are confounded, and introduce a Double-Debiasing (DDbias) method that removes bias without gradient updates, requiring only a small number of unconfounded samples. Extensive synthetic experiments validate the theoretical results and demonstrate the robustness of the proposed method across different model depths, confounding strengths, and data distributions (including non-Gaussian cases), providing a new theoretical and methodological foundation for achieving unbiased predictions in ICL under realistic scenarios involving confounders.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. First to systematically investigate endogenous bias in ICL from hidden confounders, introducing the EICL framework with clear practical motivation.\n2. Strong theoretical contributions proving bias is proportional to confounding strength, alongside a lightweight, gradient-free debiasing method (DDbias) that uses minimal unbiased data.\n3. Extensive synthetic experiments across model depths, confounding levels, and distributions (including non-Gaussian) that convincingly support the theoretical claims.", "weaknesses": "1. The scope is restricted to linear attention Transformers and linear regression setups, with no extension to nonlinear architectures or tasks.\n2. Experimental validation relies entirely on simulated data, lacking tests using real-world datasets with genuine hidden confounders.\n3. The proposed DDbias method depends on the availability of a small set of completely unconfounded samples, which may be unrealistic in many real scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KZBIZ723eN", "reviewer": "Reviewer_P15F", "summary": "This paper tackles an important and previously overlooked aspect of In-Context Learning (ICL) in Transformers (TFs): the impact of endogeneity arising from hidden confounders (HCs) present during pre-training. The authors introduce the \"Endogenous ICL (EICL)\" problem setup and provide a theoretical analysis characterizing how confounding bias propagates from pre-training data to the final ICL predictions, specifically within the context of linear TFs and linear regression. They find that the resulting bias is proportional to the confounding strength. Building on this analysis, they propose \"Double-Debiasing (DDbias),\" a novel, gradient-free prompting method that utilizes a small set of unconfounded examples to correct the predictions of a biased, pre-trained TF. The theoretical claims and the effectiveness of DDbias are supported by experiments on synthetic linear regression tasks.\n\nThe paper makes a valuable contribution by formally introducing and analyzing endogeneity within the theoretical ICL literature, moving beyond the common assumption of causal sufficiency. The theoretical characterization of bias is insightful, and the proposed DDbias method offers a potentially practical, low-cost approach to mitigation.\n\nHowever, the study's reliance on strong simplifying assumptions (linear TFs, linear data generation) limits the immediate generalizability of the theoretical findings to complex, real-world TFs and tasks. Furthermore, the practical feasibility of the proposed debiasing method hinges on the availability and identification of truly unconfounded examples, which is often a significant challenge. While the empirical results validate the theory in the studied setting, broader empirical validation on more realistic tasks and datasets would strengthen the paper's impact.", "questions": "1.  Could you elaborate on the potential challenges or theoretical hurdles in extending the bias characterization and the DDbias methodology to non-linear TFs (e.g., with softmax attention, MLPs) and non-linear data generation processes?\n2.  Regarding the unconfounded data requirement for DDbias: How many unconfounded samples ($n_u$) were typically needed in your experiments to achieve effective debiasing (Figure 4 suggests ratios relative to pre-training data size)? How robust is DDbias if the provided \"unconfounded\" samples ($\\mathcal{D}^u$) still contain some small residual confounding?\n3.  Given the focus on characterizing bias, have you considered applying DDbias to scenarios beyond linear regression, perhaps using semi-synthetic NLP or vision tasks where confounders can be injected, to explore its empirical effectiveness more broadly?\n4.  Could you compare the practical data requirements for DDbias (small $n_u$ unconfounded samples) versus an IV-based approach (potentially larger $n$ confounded samples but including valid IVs)? In what scenarios might one approach be more feasible than the other?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  **Novel and Important Problem Formulation:** The formalization of Endogenous ICL (EICL) addresses a critical gap in existing ICL theory. Recognizing and analyzing the impact of hidden confounders, which are common in real-world data, is crucial for a more realistic understanding of ICL mechanisms.\n2.  **Pioneering Theoretical Analysis:** The paper provides the first (to the authors' knowledge) theoretical characterization of how endogeneity affects *both* the pre-training dynamics of TFs and the subsequent ICL inference bias. Linking the prediction bias to confounding strength ($\\mathbf{r}_j$) provides concrete theoretical insights.\n3.  **Simple and Practical Debiasing Method:** DDbias is an elegant, gradient-free prompting strategy that avoids costly fine-tuning or complex architectural changes. Its reliance on only a few unconfounded samples makes it potentially much more practical than methods requiring large-scale unbiased data collection.\n4.  **Clear Theoretical Framework:** The theoretical sketch (Figure 2) effectively outlines the paper's analytical steps: establishing grounding \"unbiased\" parameters (U-weights), showing pre-training deviation from these due to HCs, and linking this deviation to ICL prediction bias.\n5.  **Targeted Empirical Validation:** The experiments are well-designed to specifically verify the core theoretical claims regarding bias proportionality (Theorems 1 & 2) and the effectiveness of DDbias (Theorem 3 & Proposition 1) within the linear regression setting.", "weaknesses": "1.  **Strong Linearity Assumptions:** The entire theoretical analysis is predicated on linear TFs (specifically, linear self-attention without softmax) and linear data generation ($y = w_*^\\top x + \\epsilon$). While this is a common starting point for ICL theory, the extent to which these findings generalize to deep, non-linear TFs (like GPT models where ICL is prominent) and non-linear tasks remains an open question. The reliance on simplified TF parameters ($\\bar{S}, \\bar{T}_{:,j}$) further distances the theory from practical architectures.\n2.  **Requirement of Unconfounded Data for DDbias:** The proposed solution, DDbias, crucially depends on having access to a set of unconfounded examples ($\\mathcal{D}^u$). The paper suggests \"extremely few\" are needed, but doesn't quantify this rigorously or discuss the practical challenges of *obtaining* or *verifying* such unconfounded data in real-world scenarios where HCs are pervasive precisely because they are *hidden*. How sensitive is DDbias to imperfections or residual confounding in $\\mathcal{D}^u$?\n3.  **Limited Scope of Experiments:** The experiments exclusively use synthetic data generated from a linear model. While this allows for controlled verification of the theory, it doesn't demonstrate the method's applicability to the complex, high-dimensional, and often non-linear domains where large TFs and ICL are typically applied (e.g., natural language processing, vision). Validation on semi-synthetic or real-world benchmarks exhibiting known confounding (if available) would significantly bolster the claims.\n4.  **Comparison with Alternatives:** The paper contrasts DDbias with IV-based approaches mentioned in a concurrent work by highlighting that DDbias doesn't need IVs. However, it requires unconfounded data, which IV methods aim to circumvent. A more detailed discussion comparing the assumptions, requirements, and potential domains of applicability for DDbias versus IV regression or other potential debiasing strategies (e.g., causal representation learning adapted to ICL) would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hnWkI3P08N", "reviewer": "Reviewer_Uoeb", "summary": "This paper introduces a novel problem setting called Endogenous In-Context Learning (EICL), where hidden confounders (HCs) violate the causal sufficiency assumption commonly made in prior ICL studies, leading to dependencies between input features and noise terms. The authors provide the first theoretical analysis of how HCs affect transformer pre-training dynamics and subsequent ICL predictions, showing that pre-trained transformers exhibit prediction bias proportional to the confounding strength. To mitigate this bias without gradient updates or model fine-tuning, they propose Double-Debiasing (DDbias), a gradient-free method that prompts the biased transformer twice using a small number of unconfounded examples to yield unbiased predictions. Experiments on synthetic linear regression tasks with varying transformer architectures and confounding strengths validate the theoretical claims and the efficacy of DDbias.", "questions": "Please address the questions mentioned in the weaknesses. Especially, how can practitioners identify and collect \"unconfounded\" examples in practice without knowing the true data-generating process? What happens if these examples are only partially unconfounded?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Addresses an important and underexplored gap in ICL theory by incorporating hidden confounders, which are prevalent in real-world data (e.g., as illustrated in the Many-aspect Online Review example in Appendix B). This makes the work highly relevant for aligning ICL understandings with practical data structures.\n2. Empirical validation is thorough within its scope, using metrics like L2 divergence, cosine similarity, and prediction bias across confounded datasets, with results aligning well with theorems (Figs. 3-5). The method's efficiency (requiring only a few unconfounded samples) is demonstrated effectively.", "weaknesses": "1. The analysis is restricted to linear transformers and linear regression tasks under Gaussian assumptions, limiting generalizability to nonlinear architectures or more complex real-world ICL scenarios (e.g., NLP or vision tasks). While this simplifies proofs, it may not capture the full dynamics of modern transformers.\n2. Relies on access to a small set of perfectly unconfounded examples for DDbias, which could be challenging or costly to obtain/verify in practice when confounders are unknown. The paper lacks analysis of robustness to partially confounded \"unbiased\" data.\n3. Experiments are purely synthetic, with no evaluation on real datasets exhibiting natural hidden confounders. This reduces persuasiveness regarding practical impact, as synthetic setups may not reflect the nuances of actual data generation processes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)