Demystifying Spectral Feature Learning for Instrumental Variable Regression
Authors: Dimitri Meunier
Antoine Moulin
Jakub Wornbard
Vladimir R Kostic
Arthur Gretton
Venue: NeurIPS 2025 poster ()
Keywords: instrumental variable regression
NPIV
nonparametric statistics
feature learning
causal inference
operator learning
Primary area: theory

Abstract: We address the problem of causal effect estimation in the presence of hidden confounders, using nonparametric instrumental variable (IV) regression.
A leading strategy employs \emph{spectral features} - that is, learned features spanning the top eigensubspaces of the operator linking treatments to instruments.
We derive a generalization error bound for a two-stage least squares estimator based on  spectral features, and gain insights into the method's performance and failure modes. We show that  performance depends on two key factors, leading to a clear taxonomy of outcomes. In a \emph{good} scenario, the approach is optimal. This occurs with strong \emph{spectral alignment}, meaning the structural function is well-represented by the top eigenfunctions of the conditional operator, coupled with this operator's slow eigenvalue decay, indicating a strong instrument. Performance degrades in a \emph{bad} scenario:  spectral alignment remains strong, but rapid eigenvalue decay (indicating a weaker instrument) demands significantly more samples for effective feature learning. Finally, in the \emph{ugly} scenario, weak spectral alignment causes the method to fail, regardless of the eigenvalues' characteristics. Our synthetic experiments empirically validate this taxonomy. We further introduce a practical procedure to estimate these spectral properties from data, allowing practitioners to diagnose which regime a given problem falls into. We apply this method to the dSprites dataset, demonstrating its utility.

Demystifying Spectral Feature Learning for Instrumental Variable Regression
Dimitri Meunier' Gatsby Unit, UCL
Antoine Moulin Universitat Pompeu Fabra
Jakub Wornbard Gatsby Unit, UCL
Vladimir R. Kostic Istituto Italiano di Tecnologia & University of Novi Sad
Arthur Gretton Gatsby Unit, UCL
Abstract
We address the problem of causal effect estimation in the presence hidden confounders, using nonparametric instrumental variable (IV) regression. A leading strategy employs spectral features that is, learned features spanning the top eigensubspaces of the operator linking treatments instruments. We derive generalization error bound for two-stage least squares estimator based on spectral features, and gain insights into the method's performance and failure modes. We show that performance depends on two key factors, leading to a clear taxonomy of outcomes, In a good scenario, the approach optimal. This occurs with strong spectral alignment, meaning the structural function is well-represented by the top eigenfunctions of the conditional operator, coupled with this operator's 's slow eigenvalue decay, indicating strong instrument. Performance degrades in bad scenario: spectral alignment remains strong, but rapid eigenvalue decay (indicating a weaker instrument) demands significantly more samples for effective feature learning. Finally, in the ugly scenario, weak spectral alignment causes the method to fail, regardless of the eigenvalues' characteristics. Our synthetic experiments empirically validate this taxonomy. We further introduce a practical procedure estimate these spectral properties from data, allowing practitioners to diagnose which regime a given problem falls into. We apply this method to the dSprites dataset, demonstrating its utility.
Introduction
We study the nonparametric instrumental variable (NPIV) model
I
Y ho()) UU BIU 2 =0 0, V X,
(I)
where X is the treatment, Y the outcome, Z the instrument, U an unobserved confounder, and ho E L2(X) is the structural function to be learned, where we denoted L2(X) the Le space corresponding to the distribution Px. We assume common confounder of Z and Y. To illustrate this setting, consider a simplified example inspired by (00] Suppose we want to determine how ticket price (X) affects the number of flight tickets sold (Y). A potential confounder (U) could be major event (e.g., a large conference or festival) occurring at the destination. This event would likely increase demand for flights, and airline pricing algorithms would result in more expensive seats being sold; the confounder influences both X and Y. A naive regression of sales on price might then misleadingly suggest that higher prices lead higher sales. This occurs because the analysis fails account for the event, which independently drives up both demand (sales) and prices. To obtain a less biased estimate the price effect, we could use an instrument (Z), such as the cost of jet fuel. The
direct correspondence to dimitri.meuniert @ucl.ac.uk
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

cost of fuel is plausible instrument because: (i) it likely affects ticket prices (X) as airlines adjust fares based on operational costs, and (ii) it unlikely to be directly correlated with whether major event (U) is happening at the destination, thus satisfying E. U 7 O. By using fuel cost as an instrument, we can isolate the variation in ticket prices that is independent of the unobserved demand shock caused by the event, allowing for a more accurate estimation of the structural function ho-
Central to the NPIV model is the conditional expectation operator T (x) + Lz(Z), defined by Th(Z)= = E[h(X) Z]. Estimating ho amounts to inverting this operator from the following integral equation
The = ro: where (2) = F|Y Z]-
In practice, the operator T is unknown and must be estimated from data. Moreover, even if T were known, its inverse is typically unbounded, making recovery of hg ill-posed inverse problem that requires regularization.
A classical approach to solving NPIV is nonlinear two-stage least squares (2SLS), which projects X and Z into suitable feature spaces and performs regression in these representations. Let y and & denote feature maps for X and Z, respectively. In Stage one estimates the conditional expectation E (X) Z] by regressing (xX) on vb( (2), yielding an approximation of the form (Xp( Z] (Z), for some linear map A. This step isolates the component of (x() that is predictable from 2, filtering out variation due to the unobserved confounder. In Stage 2, the outcome Y is regressed on the predicted features (Xp(()) 2]. yielding an estimator of the structural function ho. which is modeled as a linear combination ho (X) = 3" (xx), for some coefficient vector B. Given that the feature maps ip and can be fixed (e.g.. polynomials, splines, or radial basis functions from kernel) or learned from data (e.g., using neural networks), a natural question arises: What specific information must these features encode enable efficient estimation?
The algorithm recently proposed by [36,, which learns features by minimizing spectral contrastive loss, suggests that the answer lies in the spectral structure T. In this paper. we rigorously investigate this perspective. We show that the performance of this approach hinges on two key factors: the spectral alignment of the structural function hg with the operator T, and the rate of its singular value decay. Namely, we identify three distinct regimes governing the effectiveness of spectral methods for causal inference
1. The good regime, where the structural function aligns strongly with the leading eigenspaces of the conditional expectation operator T, and the singular values of T decay slowly;
2. The bad regime, where spectral alignment holds, but rapid singular value decay leads to instability and degraded estimation quality;
3. The ugly regime, where the structural function is misaligned with the top eigenspaces of T, rendering spectral methods ineffective regardless of decay rate.
Overview of contributions. We investigate the effectiveness of spectral features in NPIV estimation, grounding their empirical success in principled theoretical analysis. Specifically:
By leveraging an upper bound on the generalization error for the 2SLS estimator and specializing to spectral features, we characterize the conditions under which each regime arises.
We clarify the theoretical foundation of the state-of-the-art causal estimation algorithm of (36 by showing that it implements a 2SI.S estimator with learned features that explicitly target the leading eigenspaces of the conditional expectation operator T. Although (36) suggested heuristic link to the Singular Value Decomposition (SVD) of T, we make this connection explicit by proving that their contrastive objective corresponds to a Hilbert-Schmidt approximation of T.
We empirically validate the "good-bad-ugly" taxonomy through synthetic experiments designed to isolate the effects of spectral alignment and singular value decay on generalization error. We further introduce practical, data-driven procedure to estimate these key spectral properties (spectral alignment and singular value decay), allowing practitioners to diagnose their problem's 's regime. We demonstrate this procedure on the dSprites dataset (3)
'The "good bad bad- ugly" terminology is borrowed from 224, who used it to describe performance regimes in Koopman operator learning. We adopt the same naming convention in different context: spectral feature learning for causal inference. The phrase itself, of course, originates from the classic (27).

---

Related work. Instrumental variable regression is often implemented as two-stage procedure, where the key design choice is the feature representation. Fixed features include polynomials, splines, while learned features are either derived from the conditional
or kernel methods [3
6
7 31 35
expectation operator T 36 or jointly optimized with the 2SI.S objective [23
33
40 Some
approaches replace first-stage regression with conditional density estimation 11 18
20
28
34
An alternative line of work reformulates IV regression as saddle-point optimization problem to These methods frame estimation as a min-max game between a candidate solution and dual witness function enforcing the IV moment condition. This approach sidesteps explicit conditional expectation estimation but relies on selecting tractable, fixed function classes to ensure the well well-posedness of the optimization problem.
avoid the bias introduced by nested conditional expectation estimation [2] 13
29
38
43
Features can be learned in various ways. One line of work learns them jointly with the regression objective, as in [6 23 33 40), who propose end-to-end frameworks that simultaneously optimize over the feature representations and second-stage parameters. In contrast, [36] 38 propose to learn features that reflect the eigenstructure of T, independently of the outcome Y, using spectral contrastive loss. This may appear suboptimal, since incorporating Y could, in principle, yield more informative representations, however it avoids the optimization difficulties inherent to nonconvex, joint objectives involving all three variables (X; Y, Z). This decoupled spectral approach was shown in certain experimental settings to outperform end-to-end alternatives, demonstrated in [36] Our work places this empirical finding in context, since it likely arises due to the experiments in that work satisfying the "good" regime in the present work. We provide further diseussion in Appendix D
Structure of the paper. Section 2 Lintroduces the notation and preliminaries. Section reviews the Sieve 2SLS estimator and presents generalization bounds highlighting the role of spectral features. Section [ shows how to learn such features via a contrastive loss and connects this to the method of [36.. Section [5] presents practical procedure estimate the spectral properties from data, enabling an empirical diagnosis of our taxonomy. Section 6 provides synthetic experiments, including an application to the dSprites dataset, that validate our thcoretical taxonomy. Proofs are deferred the Appendix.
2 Preliminaries
2.1 Background
Probability Spaces and Function Spaces. Y is defined on R, while X and take values in measurable spaces X' and Z, respectively, endowed with their a-fillds Fx and Fz. We let (Ω, IP) be the underlying probability space with expectation operator E. For R, E (X, Z, (X, 2)). let denote the push-forward of IP under R. and let πn @ a denote the product measure on the corresponding product -fiilld defined by (TR & T X B) = π TR( (A(A)Ts(B)) for measurable sets A, B. For R E {X, } and corresponding domain R E {X, Z). we abbreviate L2(Rt La (R, TR) as the space measurable functions f R R such that f [f d < oc, defined up to क g-almost everywhere equivalence.
Operators on Hilbert Spaces. Let H be Hilbert space. For bounded linear operator acting on H, we denote its operator norm by ||A| its Hilbert Schhiid norm by ||/4]] its Moore-Penrose inverse by A', and its adjoint by A For finte-dimeensionl operators, the Hilbert-Schmide norm coincides with the Frobenius norm. We denote the range and null space of A by R(A) and ((A), respectively. Given a closed subspace M S H, we write M 1 for its orthogonal complement, M for its closure, and II for the orthogonal projection onto M. The orthogonal projection onto M is denoted by (IIM) + IH 1IM = 1I -- For functions f. h e (2(X). E L2(Z), the rank-one operator g ② f is defined (g @ ()) (h, I) (x)9. This generalizes the standard outer product. For vectors Rdd, we write ||æ]|le, for the Euclidean norm.
Given n. m 2 0, we consider two indepen- dent datasets: an unlabeled dataset, Dm = {(,,, ". used to learn features for X and Z, and labeled dataset, D,, { (Z,,, Li, y:) 1n Ji) used to estimate the structural function. For R E {X, (X, (X, (X,Y), (Z,Y), Y),Y), Z, Y)} and measurable function f, we define the empirical expectation over Dm Em [f()) =
Data Splitting and Empirical Expectations.
m Di f(Fx), and similarly for E,, on Dn..
3

---

Feature Maps and Projection Operators. Let d 2 1, and let ④ X' - R be feature map with components 9; E L2(X) for i = I....,d. d. We define Hp.d = span (P....., (Pa). and let Il be the 1/2 ortorongganal projection onto this space. Analogous definitions apply for feature maps vb Rdd yielding Hadd and p..d
2.2 On the Conditional Expectation Operator: NPIV as an Ill-posed Inverse Problem
We begin by defining the conditional expectation operator. Let
L2 + (2),), - ([)(()))))))))))))) 2].
As highlighted by Eq. (2). this operator plays central role in the NPIV model. It encodes the conditional distribution of X given and provides convenient lens through which analyze how efficiently one can estimate ho. The NPIV equation admits solution if and only if ro E R(TT, and identifiable if and only 7 is injective. We therefore adopt the following assumption.
Assumption 1. T is injective and there exists a solution to the NPIV problem, i.e., (fro * 0.
The existence part is essential: without there no structural function hy satisfying the model in Eq. When T is not injective, we lose identifiability (i.e., unicity of the solution). In such scenarios, it possible to establish consistency to a minimum- norm solution of the inverse problem Eq. (2). namely h. = Ttro 15
We assume injectivity henceforth to simplify the discussion.
Eq. 2 is an inverse problem in which both the operator T and the right-hand side ro are unknown. This inverse problem is typically ill-posed, as T is generally not continuous were known, an approximation
f Hence, even if T rо would not ensure that T- is close ho T T--rp. rp. In such settings, regularization is required obtain stable approximate inverses. Two common regularization approaches are Tikhonov regularization, where T-1 xs (T*T+ + -1)--1",, 0, and spectral cut-off, which truncates the Singular Value Decomposition (SVD) 14]
When T is compact (and therefore admits countable sequence singular values), the degree of ill-posedness of the inverse problem is characterized by the decay rate of these singular values. Since the singular values of T-1 are the reciprocals of those of T, faster decay implics worse ill posedesesssss. in addition to the ill-posedness of
The NPIV problem can be viewed as particularly challenging, the operator T, one must also estimate it from data. We now introduce a sufficient condition for compactness of T.
Assumption 2. The joint distribution x is dominated by the product measure a & azz and its Rudon-Nikodym derivative, or "density", = dxx
dzxz belongs to L2(X' 2, a 8 Tz).
Assumption 2 is standard in the NPIV literature (Assumption A. in share variables or are deterministically related. For example, if Z (W, V) and X = V, then the joint distribution is not dominated by the product measure.
may fail when X and Z
Proposition 1. Under Assumption 2 T is Hilbert-Schmiad operator and thus compact.
This result is well known in the inverse problems literature; we provide proof in Appendix
Compact Decomposition and Eigensubspaces. As T is conditional expectation operator, it is non-expansive, i.e., || 1. Under Assumption 2
T is compact, and therefore admits SVD
T Σ 02u4 & U. 121
where {us} and {u:} are orthonormal basis of La (Z( and (x(X) respectively (left and right singular functions, respectively), and are the nonnegative singular values in nonincreasing order. The leading singular triplet is known explicitly: σ1 = 1, u1 1z and 01 1x corresponding to the constant functions. Under the assumption that T is injective (Assumption T a; > 0 for all i > 1. Finally, since T is Hilbert-Schmide Σiz1 of co. For any d define the leading eigensubspaces
U4 span uu,,..,, u.), 1lz,a orthogonal projection onto Va,
V4 = span (11..................... ......, va), Ix orthogonal projection onto Va-
"This corresponds to the unique element in pre-image ro under minimizes the L2-norm cquivalently characterized as the unique element the set 7 7 ((ro)) (T)) 2.
In particular, 7-1 discontinuous if is compact and not finite rank.

---

In addition, define Ta = definition of U4a Va and Tas we assume that de > dd+1- We refer Appendix A for more details on spectral theory.
Ei i= Ju @ Us the rank-d truncation of T. To avoid ambiguity in the
3 Solving NPIV with 2SLS
To understand how and when spectral features should be used, we first recall results from the Sieve 2SLS literature. This gives us clear picture of what terms are to be controlled in order to achieve good generalization, which we will use to demonstrate the effectiveness of spectral features and reveal our good-bad-ugly taxonomy.
3.1
Sieve 2SLS estimator
We start with the following characterization of the structural function:
ho = arg min E [YY - (Z))²
xx)
where the minimizer, hog is unique under Assumption I A popular strategy estimate ho is to consider a hypothesis space Hxx L2 (X) and procedure to estimate the action of T: Th Th for all h E Hx, in order to obtain
à E arg min E., (V-Thh(2) bEHx
possibly subject to regularization. In the algorithm 2SLS, we consider features on both X' and 2, and both steps (estimating h - Th and solving Eq. 4 are carried out via linear regression in feature space. The two sets of features may be infinie-dimmnnsioaal-eee using Reproducing Kernel Hilbert Spaces (RKHS) finte-dimensionall-ee When the feature dimension grows with sample size, obtain a sieve. Since our focus is on neural nework-bbseed features, we present the extend to
using splines, wavelets, or neural network features.
version, noting that the ideas naturally We refer to Appendix D
cases with proper regularization 55) a description of other 2SLS methods.
Consider feature map & X Rd such that H.x = fs + 0' (z). 0 E Rd) is included in Le (X). Then for any = ('x(()) E Hx. Th (2) = (x) 2/. Therefore, estimating (Z) (X) 2] allows us to estimate Th for any E Hx. Let vb feature map with H. = {2 + a a'v(z), Rd) L2(Z). We estimate F, with vector-valued linear regression in feature space by approximating F, with Ad)(), where A Rddd solving
+ Rdd be another
À =
E arg min É. [ll+((X)= (v) Aegaxa
The pscudo-inverse yields the minimum-norn solution in the case of non- con-uniqueness. Regularized versions, such as Trkulaov-regularizzed least squares (also known as ridge regression), can be obtained by penalizing with 7 |A|l for > 0. Computing Å is referred to as the first stage of the 2SLS algorithm. For the second stage, for any h 0 (-)( Hxx define Th(Z) = 0 F(Z) with F(Z) Áw(Z). Plugging this into Eq. 44) yields h h (T)(-)) with
=
E arg min [(Y 0 F()))
(5)
AER
Similarly the first stage, we also consider regularized version of the second stage, where penalize with 2|1013, for À > O. Je is the sieve NPIV estimator 3
6
2SLS versus saddle-point estimator. Based on features &p and v. 43 36 in saddle-poin form: hess (11 with
proposed an estimator
d = arg min max E, [a (V) - 0 px))) 0.5 (aT (2)) A|
BER4 aeRa
Albeit looking different, solving this min-max problem in closed-form leads to O = 0 (when we introduce regularization in Eq. (5) 1), and thus hai h. The saddle-poing form could be useful for
deriving convergence rates, see 2 for example, as it allows us use the theory of saddle-pon point problems. However, it is not necessary for the implementation of the algorithm.
5

---

3.2 Convergence rates for Sieve 2SLS
In order to present convergence rates for Sieve 2SLS, we introduce the sieve measure of ill -o--posedness.
Definition 1. Given d 2 and Hp,d = sp.n .pd. the sieve measure of ill ill-posedness is defined as
T
inf hEM.p. 1g [Thllea(z) beH [Alle,=1 11Thille2(2)
sup
This quantity can be interpreted as the operator norm of T restricted T T (HHg,a); that is, Tipd SuPgET(Hy..l) hz0 || (x)] the restricted norm T remains finite. It captures the degree of ill-posedness specific to the chosen hypothesis space H.p.d. As d increases and the hypothesis class becomes richer, Ti,d increases. Equipped with this quantity, now introduce stability conditions that will allow us to establish generalization bounds for Sieve 2SLS.
While the full operator T-1 is typically unbounded,
Assumption 3. (i) SUPNEH [|bll, [[ p Th (Z(Z)) = (Tol)): (ii) there is a constant C'> O such that T (Tp,aa 22)) SC - x [[ ho L2(X))
Assumption ()) is a condition on the approximation properties of the features used for the instrument space. It says the image of Hapdd through T lies almost entirely inside H...d i.e,, the projection onto the orthogonal complement vanishes faster than the inverse ill-posedness. Assumption (ii) is a stability condition sometimes referred as link condition that standard in the NPIV literature (see Assumption 6 in (3), Assumption 5.2(ii) in 77) and Assumption 3 in (23). It says that TI' becomes increasingly contractive on the component ho lying outside the sieve space as the sieve dimension increases. The next assumption ensures the conditional variance of the noise is uniformly bounded.
Assumption 4. There exists ō > 0 such that IE [U' z] S 6² < oc almost everywhere.
The following result is an upper bound on the generalization error from 6
Theorem 1 (Theorem B.1. K). Let h be the 2SLS estimator from Eq. 5 using features up X - B
and vb 2 - R.d. Suppose Assumptions and 4 hold and Tie v. (log d)/n o(1), where
Sp.ii = max(sup, [Xx](())(()]]] 1/2 (x(x))ll,, sup= (Z) 112 (1))l Then:
j ho 12(X)
= Op ( (I((Tp.a) + Ta.d Te.d
The first term captures the approximation error, while the second term corresponds to the estimation error arising from finite samples. When standard bases such as cosine, spline, or wavelet functions used, the approximation error well-understood. If the sieve ill-posedness Tp.d grows polynomial or exponential rate in d, one can choose = d(n) to balance the two terms optimally 3. Under such growth conditions and smoothness assumptions on ho (e.g., ho belongs to Sobolev ball), the resulting convergence rate is minimax optimal 8
3.3
Convergence rates for Sieve 2SLS with spectral features
drawback of using non-adaptive features (such as cosine or spline bases) that verifying Assump- tion 3Jr characterizing the growth of Fiud is generally difficult. In contrast, we now consider features constructed from the SVD of T, for which both conditions can be exactly characterized. Moreover, while estimators based universal bases achieve minimax optimality under smoothness as- sumptions on T and lg. they are often outperformed in practice by estimators using data-dependent features. This highlights that minimax rates, though important, do not capture the full picture of empirical performance. Recall from Eq. B the singular value decomposition of T,, and the definition of the top-d right eigenspace Va. The following result shows that this subspace achieves the minimal possible measure of ill posedness among all dddimensional subspaces of L2(X).
Proposition 2 (Lemma (3). Let Assumption 2 hold. For any 2 1, the smallest possible value of Tp.d is Te.d oj achieved when Hip.d Va.
Definition 2 (Spectral features). We say that the features vp and v are spectral features if
H.pdd spaufei..... val Va and Ho span/uu..... ..., ua) = Usg

---

By Proposition 2 spectral features minimize the sieve measure of ill poseddesss, and therefore the estimation error term. This leads the following corollary.
Corollary (Sieve 2SLS with spectral features). Let be the 2SLS estimator from Eq. ( using
spectral features. Let Assumptions 2 and hold and o "p....v (log d)/n o(1). Then:
= Op (I(TT (I(Txx,)) (oll + V non
No
(x)
We postpone the proof Appendix C.2
The good, the bad, and the ugly. The bound in Corollary [[ decomposes the generalization error into two terms: spectral alignment, encoded in the approximation enror [1 aa) koll. which measures how well 4gg lies in the top-d singular space; and ill-posedness, captured by the inverse singular value oT² In the good regime, ho lies mostly in Ug and σ₁ decays slowly, so both terms are small. In the bad regime, alignment is favorable but fast spectral decay inflates the estimation error. In the ugly regime, hg is misaligned with the top eigenspaces, so the approximation error dominates and estimation fails regardless of under a source condition and a singular value decay condition.
In Appendix B we show that the bound in Corollary 1 is tight
Sieve 2SLS with learned spectral features
In the previous section, we showed that spectral features minimize the sieve measure of ill- pl-pedddddsss and lead fast generalization rates when the structural function ho aligns well with the top eigenspaces of the conditional expectation operator TT. This motivates learning such features directly from data, especially in the case where the conditional distribution of X given the instrument is strong) and the alignment with the spectrum of T is favorable. We now demonstrate that the contrastive learning approach of [36] can equivalently seen as explicitly targeting the leading eigenspaces of TT Consider the following objective:
informative (i.e..
d
Calse, v = ||Ta(s9, 15) - -
Ta(a, Σ
4/ M100
Si
(6)
where 6i E L2(X), vi E L2(Z),
d....,d. As any rank-d operator can be decom- posed Ta(s,, 1²), we have the following version of the Eckart-Young-Mins Theorem for the Sllbert-Schmide norm:
Theorem 2 (Eckart Sockart-Young-.. Mirikyy Theorem). Let Assumption 2 hold and let d 2 1 be such that ad > Jd+1. For all feature maps : Rdd wb + Rd, we have
La(p, w) T Tall
of
i>d
where equality holds if and only if Ta(sp, v)) Ta in which case Hyped Va. Hypdd = Ua. Ta is the
rank-d truncation T introduced Section
While the Eckart- Young- Mirsky objective in Eq. (6) offers clean objective learn spectral features, appears impractical due to the unknown nature of the operator T. However, this objective admits an equivalent formulation spectral contrastive learning loss 377
Ca v = Ex Ez (((()T - 2Ex,z (0 (x) (Z)) + 1
where the last term is independent of (yp, v) We provide proof of this equivalence Appendix C.3 The population loss can be estimated from samples Dm = { mm as
2
La((), m(m
(((at)
(
1) (gfo
The spectral contrastive loss is the basis of the recent algorithm proposed in 36) where and ab are
parametrized by neural networks and optimized via stochastic gradient descent. While (36) motivate this objective heuristically. Eq. (6) shows that it directly targets the leading singular structure of T.
The use objectives similar to Eq. (6) for operator learning or Eq. 7 for representation learning has rich history that detail in Appendix D
7

---

Comparison to [36]. To justify the use of the contrastive loss in Eq. 8). 36 assume that the density p(a, 2) factorizes as
f = (()) x 2 - R4
(9)
We argue that this assumption is overly strong: it holds if and only if T has rank most d, in which case the spectral features perfectly capture the structure of T. This result is formalized in Proposition [] Appendix C.3 We should instead interpret Eq. (9) an approximation rather than literal assumption. In practice, the learned features so, 1/ provide a rank-d approximation of T, and the quality of this approximation governs the performance of the resulting Sieve 2SLS estimator.
Let Gē, & be d-dimensional features trained with loss Eq. with D, and define Ta = > Dr 2 Po. We make the following assumption:
Assumption 5. { } and {fI...., Pa) form linearty independent families.
This assumption is only made to simplify the discussion. (u1...., .....,,] or {&I..... Od) is not linearly independent, one can replace d in the following by the linear dimension of the family, Consider ad > 0d+1 so that the unique minimizer of the population loss given by Ta- We quantify how the distance between Ta and Ta affects the approximation error and the sieve measure of ill- posedness Tæ.d. We quantify how the distance between Ta and Ta affects the generalization error of the 2SLS estimator.
and hold and let éa = || Tall be such that éd
Theorem 3. Let Assumptions
(1 - 1/V2)oa-
i) od Tp.d S - 22a) (Ea)
ii) Let h be the 2SLS estimator from Eq. 5 using features & Rd and & + Rd..
Suppose (aa 26a) Sp (log d)/n = o(1) and Éd oa(a)), then
= =Op [I(Ixx))hollaa h + V2Ea (20u|hoollaa(x) ad
1 1 ho 12(x)
ad 28
The result quantifies how close the learned features must be to Ta in order to preserve the desirable properties of spectral 2SLS. Theorem 3 shows that as long as the learned operator Fa is a good approximation of the true rank-d truncation Ta the learned features inherit favorable measure of ill- ill-posedneess Theorem Bii) further ensures that if contrastive learning yields a near-optimal rank-d approximation of T. the downstream Sieve 2SLS estimator retains strong statistical guarantees. Importantly, the spectral features generalization error Ea encapsulates the statistical and optimization errors in learning spectral features. It depends on the number of samples used to train the contrastive loss, the expressivity and architecture of the neural networks parametrizing &p and v/, and the complexity of the leading eigensubspaces of 'T. As La(y, v converges to Ly = IIT Tall? & converges to 0 by Theorem 2 Precisely controlling Ea lies beyond the scope of this paper and remains an important question for future work.
We postpone the proof to Appendix
C.3
Estimating alignment and spectral decay from data
Having established the good-bad-ugly taxonomy based spectral properties and its interaction with the structural function hog we now discuss how to estimate these spectral properties from data. Namely, present methodology to estimate the spectral decay of the operator T as well spectral alignment with the structural function bn- Recall that T is compact and admits an SVD of the form T" Therefore, for all i 2 we have
Eq. 33), with the key relationship that 0.2% u4.
1 - (wi, Lo) 0i
1 - E u, (Z)).
(vi, Lo
- (T" u,, ho) (o) ai
(10)
where we used Thu ru. the definition of To and the tower property of the conditional expectation. This relationship shows that the alignment coefficients (v,, ho,, Lx(X) can be estimated from data we have estimates of the singular functions u, and singular values a,, In Section we provide practical estimator for Eq. 11pp using learned features and derive its estimation guarantees, which depend on the operator error Éd.
8

---

We apply this procedure in Section to diagnose the spectral properties of the dSprites dataset 30 popular benchmark in the nonparametric IV and proxy literature
36
0.7
0.5
0.5
04
0.3
02
0.1
0.0
0.7
0.5
0.5
0.3
0.2
0.1
0.0
622
021
020
0.2
217
038
E24
022
032
0.0
0.2
0.20
1:3
828
024
034
013
0.LL
0110
2.03
Egt Be-3loy singular value decay ratz
03T
016
0.5
314
033
E22
030
049
610
000
0.77
0.1G
0.5
824
832
033
0.00
0.99
0/0
0.37
3.22
312
233
630
046
610
0.20
812
0:09
017
0.5
8
000
9.8
0.0
0.9
1.0
0.59
000 0.0 0.00
332
1/0) Pg decay rate
Figure 1: Dependence of the MSE of instrumental variable regression on the decay rates of the spectrum and coefficients of the structural function. IV fitting and the MSE evaluations were repeated 500 times per parameter set, rendering the standard error of these estimates negligible. Left: Oracle spectral features; Right: Learned features.
Experiments
6.1 Synthetic datasets
We evaluate the main theoretical insight of the paper: Sieve 2SLS with spectral features performs best when by aligns with the top eigenspaces of T and its singular values decay slowly. Performance degrades with faster decay and weaker alignment. We design a synthetic NPIV setting where we control the conditional expectation operator T and its spectral decay and well the alignment of ho with its eigenspaces. To simulate such a setting, we rely on the following procedure for generating samples from the NPIV model:
Proposition 3. Let x be probability distribution X x Z with marginals xx and zz. Let T L2(X x + L2(2 πz) be the conditional expectation operator associated with x and let ho E (2 xx). Sample (X, 7) z. V ~~ ~N o²) and set U Tha 7) (X) + V and
Y = ho (X) + U. Then (U, 7, X,Y) Y) is a sample from the NPIV model of Eq. 0
Let d 11, X mm)
[0. 2m x² =011, f(x) = (sin(x), sin(x)),.. sin((d 1)x
g(=) = (sin(2),
...., sin((d 1)2)). and let P. Q be two orthogonal (d 1) x (d- 1)
matrices. Define v X' - Rd-1 and u 2 - Rd-1 as v(z) = Pf(x) and u(2) = Qg(z). We set T 1z 0 1x Σ d-1 0,44 m. (X, Z) is jointly sampled with rejection sampling. Z, X uniform on [0, 2Tr and X given Z admits T as conditional expectation operator. We then sample Y following Proposition 3 The multiplication of the trigonometric functions by the orthogonal matrices P. Q ensures that the consecutive singular functions of T are comparably complex. By keeping the difficulty constant per singular function we can isolate the influence of the singular value decay on the performance of the final estimator. We set the singular values {=: to decay linearly, from some
initial value σ1 to a dd-1 = C1 Co E [0. 1]. To control spectral alignment, we set hu = > C;22; subject to ||a]le, = 1, and allow {a,] 1d-1 to decay linearly from aj to Qd-1 = C. 01,
[0. 1]. Each value of cav E [o, 1] controls the alignment of ho with the top singular functions of T: larger values correspond to weaker alignment. Similarly, co [0, 1] controls the ill ill-posedness of the problem: smaller values indicate faster decay of singular values and hence more severe ill posddosessss.
In Figure [ we report the mean squared error (MSE) of two 2SLS estimators across range of spectral decay rates (c,) and structural function decay rates (c,,). The left panel uses 2 with fixed features f and g that form the exact spectral features. This represents an oracle setting. The MSE decreases from top to bottom as the problem becomes better conditioned (i.e., as e, increases). Additionally, the MSE decreases from left to right, smaller values of ca. correspond to stronger

---

Mar MyDAA
Good/had
Ugly
Very ugly
Comparison of the spectral methods' performance depending on the alignment of ho with Figure 2:
the singular functions of T. Blue: True ho: Orange: 2SL. estimate.
spectral alignment: more of hg is concentrated on the top singular functions of T. These trends fully consistent with the bound established in Corollary [ In the right panel, we lear 50 features from data by minimizing the empirical contrastive loss Eq. (8) using neural network. We then build a 2SLS estimator on top of the learned features. The same qualitative trends are observed: MSE decreases with slower singular value decay and better alignment. Moreover, when Co is small (top rows), learning the entire eigenspaces becomes difficult. In this regime, performance is most sensitive to Ca. In particular, if ho is concentrated on the top singular functions (small c.). good recovery is still possible. In contrast, when Co is large (bottom rows), the singular functions are equally important and more easily learned, rendering the alignment of hi less critical. The mismatch between the oracle and learned settings is because we do not exactly recover the singular feature spaces; as predicted by Theorem It is also worth noting that when he concentrated on high singular values, it may sometimes be beneficial for the singular values to decay quickly, by making it easier for the leature-learning model to learn the top singular functions well. This may explain the slight top-bottom increase in losses on the right end of the right panel.
The previous experiment assumes hg lies the span of the singular functions T. Figure illustrates three regimes. The left panel shows a well-aligned case ("good/bad"), where ho is entirely supported on the singular functions of 7, allowing for exact recovery as long as enough data is available. The middle panel represents a "partial alignment" regime, where ho has some overlap with the singular functions; in this case, recovery is imperfect, but the estimator can still capture some signal. Finally, the right panel illustrates the "misaligned" or "very ugly" regime, where ho is orthogonal to the singular functions. The instrument provides no information about the signal, and all spectral methods fail. We refer to Appendix E for further discussion of these scenarios and the experimental setup.
6.2 Dsprites dataset
The dSprites dataset, as employed in the evaluation of IV mod- els, consists of 64 x 64 noisy images of hearts, with varying x, positions, orientations and sizes. The full setting is as fol- lows; X E 1464x64 is the raw image, Z R3 consists of the x position, orientation and scale of the heart in the image, and the output is constructed a
12
Y = hop (X) + 32 (y position - 32) + UU U ~~~:~~0 ~ (O,
where
Figure 3: Example image from the dSprites dataset.
(x) (1 X112 3000)/500, A, 132 j1/32.
It was observed by Sun et al. (36) that spectral methods significantly outperform alternatives in this setting. We argue that is because the structural function employed in this benchmark lies in the "good" regime. Utilising the alignment estimation methods outlined in Section 5 we are able to provide evidence for this claim. We observe that the structural function is spanned by the leading singular functions of the conditional expectation operator associated with this model. The details of this experiment can be found in Appendix F
'The original formulation of the dSprites model for IV used Ar, ~ 21 1) 40 However, it is noted in 4I (arxiv version 18/06/2024, Appendix E.3) that it leads to identifiability issues and that the choice A.
132 jl/32 preferred.
10

---

Acknowledgements
D.M., J.W., and A.G. are supported by the Gatsby Charitable Foundation. V.R.K. is supported from NextGenerationEU and MUR PNRR project PE0000013 CUP JS3C22003010006 "Future Artificial Intelligence Research (FAIR)). A.M. has received funding from the European Research Council (ERC), under the European Union's 's Horizon 2020 research and innovation programme (Grant agreement No. 950180), and travel support from ELSA (Project ID: 101070617).
References
[1] Chunrong Ai and Xiaohong Chen. Efficient estimation of models with conditional moment restrictions containing unknown functions. Econometrica, (1(6):1795-1844 2003.
[2] Andrew Bennett, Nathan Kallus, Xiaojie Mao, Whitney Newey, Vasilis Syrgkanis, and Masatoshi Uehara. Minimax instrumental variable regression and 1_2 convergence guarantees without identification or closedness. In The Thirty Sixth Annual Conference on Learning Theory, pages
PMLR, 2023.
[3] Richard Blundell, Xiaohong Chen, and Dennis Kristensen. nomparonparrmettic iv estimation of shape-invariant engel curves. Econometrica, 75(6):613-1669, 2007.
[4] Vladimir Igorevich Bogachev and Maria Aparecida Soares Ruas. Measure theory, Vol. II. Springer, 2007.
[5] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a "siamese" time delay neural network. Advances in Neural Information Processing Systems, 6, 1993.
[6] Xiaohong Chen and Timothy M Christensen. Optimal sup-norm rates and uniform inference on nonlinear functionals of nonparametric iv regression. Quantitative Economics, (11:399-84 2018.
[7] Xiaohong Chen and Demian Pouzo. Estimation of nonparametric conditional moment models with possibly nonsmooth generalized residuals. Econometrica, 80(1):27--321, 2012.
[8] Xiaohong Chen and Markus Reiss. On rate optimality for ill-posed inverse problems in econometrics. Econometric Theory, 27(3):497-52,, 2011.
[9] Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, et al. Spectral methods for data science: A statistical perspective. Foundations and Trends@ in Machine Learning, 14(5):566 806, 2021.
[10] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively. with application to face verification. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'0S) volume pages 539-546. IEEE, 2005.
[11] Serge Darolles, Yanqin Fan, Jean-Pierre Florens, and Eric Renault. Nonparametric instrumental regression. Econometrica, 15): 2011.
[12] Zhijie Deng, Jiaxin Shi, and Jun Zhu. Neuralef:
Deconstructing kernels by deep neural networks. In International Conference on Machine Learning, pages 4976-4992. PMLR, 2022.
[13] Nishanth Dikkala, Greg Lewis, Lester Mackey, and Vasilis Syrgkamis. Minimax estimation of conditional moment models. Advances in Neural Information Processing Systems, 33, 2020.
[14] Heinz Werner Engl, Martin Hanke, and A. Neubauer. Regularization of Inverse Problems. Kluwer, 2000.
[15] Jean-Pierre Florens, Jan Johannes, and Sébastien Van Bellegem. Identification and estimation by penalization in nonparametric instrumental regression. Econometric Theory, 2773)447--46,, 2011.
[16] Yuri R. Fonseca, Caio F. L. Peixoto, and Yuri F. Saporito. Nonparametric instrumental vari- able regression through stochastic approximate gradients. Advances in Neural Information Processing Systems, 37, 2024.
11

---

[17] M.J. Greenacre. Theory and Applications of Correspondence Analysis. Academic Press, 1984. ISBN 9780122990502. httpsstsss//bbok google. wtt.kk/..oooo..e.oooooeee....... httpa///sssoooossedddaaaaaaaaaaa
[18] Peter Hall and Joel L Horowitz. Nonparametric methods for inference the presence of instrumental variables. Annals of Statistics, 33(6):2904-299)) 2005.
[19] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34, 2021.
[20] Jason Hartford, Greg Lewis, Kevin Leyton-Brown, and Matt Taddy. Deep iv: A flexible approach for counterfactual prediction. In International Conference on Machine Learning, pages 1414-1423, PMLR, 2017.
[21] Hsiang Hsu, Salman Salamatian, and Flavio P Calmon. Correspondence analysis using neural networks. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2671-2680. PMLR, 2019.
[22] Hsiang Hsu, Salman Salamatian, and Flavio P Calmon. Generalizing correspondence analysis for applications in machine learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):934799366 2021.
[23] Juno Kim, Dimitri Meunier, Arthur Gretton, Taiji Suzuki, and Zhu Li. Optimality and adaptivity of deep neural features for instrumental variable regression. xrviv preprint Tt::2501.04889 2025.
[24] Vladimir Kostic, Karim Lounici, Pietro Novelli, and Massimiliano Pontil. Sharp spectral rates for koopman operator learning. Advances in Neural Information Processing Systems, 36, 2023.
[25] Vladimir Kostie, Grégoire Pacreau, Giacomo Turri, Pietro Novelli, Karim Lounici, and Massim- iliano Pontil. Neural conditional probability for uncertainty quantification. Advances in Neural Information Processing Systems. 37, 2024.
[26] Henry Oliver Lancaster. The structure of bivariate distributions. The Annals of Mathematical Statistics, 29(3)7719-736, 1958.
[27] Sergio Leone. The good, the bad and the ugly. United Artists, 1966.
[28] Zihao Li, Hui Lan, Vasilis Syrgkanis, Mengdi Wang, and Masatoshi Uchara. Regularized deepiv with model selection. arXiv preprint ar X4Xx:2243.002266 2024.
[29] Luofeng Liao, You-Lin Chen, Zhuoran Yang. Bo Dai, Mladen Kolar, and Zhaoran Wang. Provably efficient neural estimation of structural equation models: An adversarial approach. Advances in Neural Information Processing Systems, 33, 2020.
[30] Loic Matthey Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentangle- ment testing sprites dataset.
2017.
[31] Dimitri Meunier, Zhu Li, Tim Christensen, and Arthur Gretton. Nonparametric instrumental regression via kernel methods is minimax optimal. arXiv preprint arXiv:2411.19653. 2024.
[32] Whitney K Newey and James L Powell. Instrumental variable estimation of nonparametric models, Econometrica, 71(5):1565 1578, 2003.
[33] Ieva Petrulionyte, Julien Mairal, and Michael Arbel. Functional bilevel optimization for machine learning. Advances in Neural Information Processing Systems, 37, 2024.
[34] Daqian Shao, Ashkan Soleymani, Francesco Quinzan, and Marta Kwiatkowska. Learning decision policies with instrumental variables through double machine learning. arXiv preprint arXiv:2405.08498. 2024.
[35] Rahul Singh, Maneesh Sahani, and Arthur Gretton. Kernel instrumental variable regression. Advances in Neural Information Processing Systems, 32, 2019.
12

---

[36] Haotian Sun, Antoine Moulin, Tongzheng Ren, Arthur Gretton, and Bo Dai. Spectral repre- sentation for causal estimation with hidden confounders. arXiv preprint ar arXiv:2407.10445 2024.
[37] Yao-Hung Hubert Tsai, Han Zhao, Makoto Yamada, Louis- Morency, and Russ R Salakhutdinov. Neural methods for point-wise dependency estimation. Advances in Neural Information Processing Systems, 33, 2020.
[38] Ziyu Wang, Yucen Luo, Yueru Li, Jun Zhu, and Bernhard Schölkopf. Spectral representation learning for conditional moment models. arXiv preprint arXiv:2210.16522 2022.
[39] Ziyu Wang, Yuhao Zhou, and Jun Zhu. Fast instrument learning with faster rates. Advances in Neural Information Processing Systems, 35:16596-16611, 2022.
[40] Liyuan Xu, Yutian Chen, Siddarth Srinivasan, Nando Freitas, Arnaud Doucet, and Arthur Gretton. Learning deep features in instrumental variable regression. In International Confer- ence on Learning Representations, 2020.
[41] Liyuan Xu, Heishiro Kanagawa, and Arthur Gretton. Deep proxy causal learning and its application to confounded bandit policy evaluation. arXiv preprint arXiv:2106.03900 2021.
[42] Xiangxiang Xu, Shao-Lun Huang, Lizhong Zheng, and Gregory W Wornell. An information theoretic interpretation to deep neural networks. Entropy, 24(1):135, 2022.
[43] Rui Zhang, Masaaki Imaizumi, Bernhard Schölkopf, and Krikamol Muandet. Instrumental variable regression via kernel maximum moment loss. Journal of Causal Inference, 11(1): 20220073, 2023.
[44] Liu Ziyin, Tilman Hartwig, and Masahito Ueda. Neural networks fail to learn periodic functions and how to fix it. Advances in Neural Information Processing Systems, 33:1583-1594, 2020.
13

---

Demystifying Spectral Feature Learning for Instrumental Variable Regression
Supplementary Material:
A Background on Spectral Theory
For completeness, we briefly recall key notions from spectral theory used throughout the paper. Our goal is to provide intuition and the minimal mathematical tools required interpret concepts such as compactness, singular value decomposition (SVD) of operators, and eigensubspaces.
Let H1 and H2 separable Hilbert spaces with inner products (,,) and norms || || v 1,2.
Hilbert spaces and bounded operators. linear map A HI + H2 is said be bounded if there exists C > 0 such that || I Cl for all h E H1. The smallest such constant is the operator norm, denoted |Allop'
Adjoint and self-adjoint operators. Every bounded operator A H1 - H2 admits an adjoint A" H2 H1 defined by (Ah, n2 (h, A*g) uh. If Hi H2 and A", the operator is self-adjoint.
Compact operators. A linear operator is called compact if maps bounded sets in H1 to rela- tively compact sets in H2, that is, the image of any bounded sequence has a convergent subsequence in H2. All compact operators are bounded, but not all bounded operators are compact. Any compact operator is limit (in operator norm) of finite-rank operators, that the class of compact operators can be defined alternatively as the closure of the set of finite-rank operators the operator norm topology. The conditional expectation operator is canonical example of compact operator (see Proposition below).
Hilbert-Schmidt operators and norm. Let A: H1 - H2 be a bounded linear operator. Choose any orthonormal basis {ejb21 of H.. The operator A is called Hilbert-Schmidt if
|
Σ
Ae <0.
j21
The value of the sum above is independent of the chosen basis of H₁. Hence the quantity (A, B) Σyz1 Bey)H, defines an inner product on the space of Hilbert-Schmide operators, and || the induced norm.
Singular Value Decomposition (SVD). When A: H1 - H2 is compact (not necessarily self- adjoint), it admits singular value decomposition:
A =
T U
where 01 2 02 2..2 0 are the singular values A such that o; + 0, and {er} C H1, {u,} C Hz form orthonormal bases satisfying Au; det, and A A'u; d,U; This generalizes the matrix SVD to
spaces. Important properties we use below:
The above is finite if and only if A has finite rank.
|| A llap = d1-
If A is Hilbert-Schmiit then A is compact such that ||||| = oo, hence ||A|..p S || Thus Hilbert-Schmidt operators are bounded, but the converse need not hold.
B Learning Rate with Spectral Features
In this section, we formalize the learning rate that can be obtained from Corollary 1 To do so, employ two standard assumptions from the nonparametric IV literature X source condition and singular value decay assumption the operator T.
14

---

Assumption 6 (Singular Value Decay). Let p 2 1. There are constants < S C2 < oo such that for all L. c o, S Cat
This assumption is standard in the nonparametric IV literature B and characterizes the difficulty of the inverse problem defined by the operator T. A larger p indicates faster decay and thus greater ill-posedness. Alternative ill-posedness characterizations can be considered, such exponential decay [8].
Assumption 7 (Source Condition). Letr > O. There is a constant B < oc such that
(ho.4;) Lx < B.
h = [1(T)T) -/2 [(x())
0
This condition quantifies how "smooth" ho relative to the spectral decomposition of T, or equiva- lently, how quickly the alignment coefficients (hg, u4) (X) decay relative the singular values. A larger implies ho is "smoother" or better aligned with the less ill-posed directions of T. Note that
under the "bad" scenario where the eigen-decay fast (p is large), stronger alignment is necessary for [| to be finite for large value of r. While in the "good" scenario with p 1, 1//b < +o0 more likely to hold with a large value of r.
Under Assumption 7 we have
Kho, u() o?r
(IIx,a) ho (x)
Σ
(ho, u.) S o
oz 1holl
a>d
Combined with Assumption we obtain from Corollary that:
holl Or (Iholldd
dL+2z
n
Balancing both terms in d as function of n,, obtain
l1 holl hollo =0
Op nn 10+1)+t-
with
d(n) = n 3.I-
This bound tight as a matching lower bound is obtained in Theorem 1-(i) 8 From this tight rate, we see that both the ill-posedness (p) and the relative smoothness (r) fundamentally capture the effectiveness of 2SLS with spectral features, and their impacts are inherently intertwined.
A low relative smoothness (small r) means in has significant components aligned with directions corresponding to very small singular values, even after accounting for the singular value decay. This also directly slows the rate, effectively rendering the problem very hard even impossible (r + 0+ makes the rate vacuous, representing our "ugly" scenario).
A fast decay (large p) directly slows down the rate regardless of It means singular values drop quickly, making fundamentally harder to resolve higher-frequent When p is large
components of hg- either in the bad or in the ugly scenario. A large p also slows down the rate indirectly by making it harder to achieve a high relative smoothness r.
When working with learned features. We see from Theorem 3 that when learn the spectral features using contrastive learning, we incur an additional penalty related to éa/0a in the upper bound. This eтror shows that a faster eigendecay makes the 2SLS estimator more sensitive to errors in the learned features. Alignment of hn does not enter this factor, as learning the eigensubspaces of is unrelated to loo Intuitively, features corresponding to small singular values can empirically difficult to learn, even with very large number of (Z, X) samples.
C Proofs
In this section we recall and prove the results of the main section.
15

---

C.1 Proofs of Section 2
We refer the reader to
background on measure-theoretic notions used throughout this section. We assume that X' and 2 are standard Borel spaces, i.e., Borel subsets of complete separable metric spaces. This ensures the existence of regular conditional distributions and supports the disintegration of the joint law x. Working directly with measures allows to treat both continuous settings, where distributions admit densities with respect to the Lebesgue measure, and discrete or more general settings, where such densities may not exist.
To prove properties the conditional expectation operator T, is helpful to express terms of a Markov kernel. We define Markov kernel p Z x Fx [0, 1] such that for any measurable set A E Fx.
((2 Pix A Z = =],
meaning p(,, is a regular conditional distribution X given = 2. Then for all E L2(X),
Th(z) - - Mx)) Tz-a.
(11)
By the disintegration theorem. for all measurable sets A X and B Z, the joint distribution admits the disintegration
B) (A,ple, dr dzz(2),
xx
and therefore, the joint distribution πx, can be decomposed as
dxx,z(zx= p(z, dx) dzz(=),
(12)
where p(z, dx) is a Markov kernel as defined above. This decomposition holds in general, even when Xx,2 is not absolutely continuous with respect to т @ π2.
Under Assumption 2 the joint distribution Tx,z is absolutely continuous with respect to the product measure π @ π2, and thus admits density
dxx.z
p(x, 2) = d(πx @ T2)
(xr, 2).
(13)
In order to prove Proposition we use the following lemma.
Lemma 1. Under Assumption for @ TT almost every (I, 2).
dp(z,-) (Ir).
p(z, 2) d
Proof. From Eq. [12), we have
d dxx,2(0, p(z, dx) 2
On the other hand, by Eq.
d 2) = p(z, 2) d (x) dr (2).
Comparing both expressions and using the uniqueness of the Radon-Nikodym derivative, we conclude that for π @ π az-almost every (x, 2).
p(z, dx) p(z, 2) x (x),
i.e., p(z.)) & Rx, and
dp(z, (x) = p(x,2))
dux
This concludes the proof.
By Lemma I under Assumption 2 for all h E L2(X) and mz-ac.
Th(=) / hxx) px) / h())p(a, =)dxx()))))))))))))))))))))))))) (hi,p,, (0),z Lx())-
(14)
We recall and prove Proposition I This is a classical result, see, e.g., (26]
16

---

Proposition 1. Under Assumption 2
T is Hilbert- Selbbert-Schmidt operator and thus compact.
Proof. Take (ex) any orthonormal basis (ONB) of L2(X). Then, using the definition of the norm:
I
Σ
(Ti
(definition of HS norm via ONB)
121
Σ
[Tes())]* drz(=)
(expand (Z) norm)
12
Σ (es.p(:2)) 2) L2(x) (z)
(by Eq.
11
(c..p(: 2) =))) (zz
(Fubini's theorem)
21
/ [|p(s 2) 2)x)2x(x) (z)
(Parseval's identity)
[ p(x, 2)² dx (x) (z)
(expand L2 norm)
Xx2
< +00
(by Assumption since E L2(TT @ T2)).
Note that the proof shows addition that |/ = 1Illl
C.2 Proofs Section 3
Proposition 4. Let Hppdd = span(pp.nnn...... .aa. and let dim(H.p,a). Then:
Ta,d _07 ii) H4p, V. span(er..... v. then Te,d S o. iii) For any 2 1, the minimal value Tipd 0g is achieved when H.p.d Va-
Proposition is due to Lemma of K and Proposition] correspoods to part (iii). We provide full proof for completeness.
Proof. We first prove (i). Since dim(Hpp.a) w, and the subspace Vw-1 has dimension 1, there
exists Hypd O V4-1 with [[lllex(x) 1. Then,
T
inf heH
||Th|l2,(2)
[| (x)==1
|| (2)))))
[T
sup hEVI-1 [1)]
sup (T'Th,h) o hev
||
where the last line follows from the min-max theorem for compact self-adjoint operators.
We now prove (ii). If l Hyped C and [ then
Σ (h, v.)ur
17

---

so by orthonormality of (u:),
lTTiliin2)) =
Σ oe us)ui
Σ oi (h, vi 2 2 o Σ (h, 0i)2 o.
(2(
Taking the infimum over unit-norm Hapd gives T 2 0², i.e. Te.d S o.
Finally, (iii) follows by taking Hipdd Va, so that w d i, and both bounds in (i) and (ii) match.
We recall and prove Corollary
Corollary (Sieve 2SLS with spectral features). Let be the 2SLS estimator from Eq. 5 using
opattar Let Assumptions and 4 hold and o Sp.s.d v (log d)/n o(1). Then:
A ho
h
=0p (I(Ilxa))oolllzx)) + (2(X)
no?
Proof. Let verify that Assumption holds under Definition iee, when Hord Va and Hord Ua.
(i) Since the image of Va under T lies in Ua, have
(Ih.a)iTh=a = 0 for all h E H.p.d-
Hence,
sup heH
[[(U).))
(T
|Alla =1
and condition (i) is satisfied.
(ii) Let ho L2(X), and consider its projection onto the orthogonal complement of H Heed Since T
compact with singular value decomposition Σi= Jru; @ o under Assumption 2 we have
(T(Tp.) (220(2)
of (ho, 02)
o
(ho: v2)² 0 (Tp.))) ho (x)
i>d
+
This implies, using that Tp,d = o by Proposition 2
(IIp.a))holllttt I (II holhllllll))))) = ( holll>(())))))))
Assumption (ii) holds with constant C' 1.
Since both parts of Assumption hold, and Tp.d = oa by Proposition 2 which yields the desired result.
we may apply Theorem
C.3 Proofs of Section 4
The following result shows that Eq. 9 holds and only if rank
S d. Proposition 5. Let ip: X Rd 3 Rd with components ip: E L2(X), db L2(Z). Then
= ( if and only if T 1 x p..
Proof. * Assume p(u, =) (bx)) Then, by Eq. for all L2(X).
Th(2) / h(x)p(2, dr) / 2))
(by Lemma
/ (x)x(x) (=) o(>)" M()))(()dxxx())
d
Σ (h(z))h, =
Σ
v/ 2 ps (n))E),
18

---

so T = - 1= Vi 2 9r.
c Now assume T = -Z the pz. pz. Let f E 12(xxx @ Ø2). We want to show:
2())) &T))(2;)) / (x,, [l2,2))))) @ )z)
Starting with the right-hand side:
I 2(zx2) dix (Tx)) ((a=)))laal(())dxxxxooooo
Σ / f(x, (1)22))((())())))
Σ / v1(()
dTx(()) dxz(2)
Σ ()())(((s) dTz(2)
=
z [T (T (z) dTz(22 (since T Σ t/ (.)
/ (J fx, 21) dTz((2 (by Eq.
f(2,2)p(22 dxx (x) (by Lemma xxz
Since both integrals agree for all f E L2(xx & πz), and p(r, 2) is the Radon-Nikodym derivative of πxz with respect xx æz, it follows by uniqueness that
p(z, 2) = ( TT)))(()))))))))))))))) for (Tx 6 malmmo every (x, z).
In order to prove Theorem we prove intermediate results. Proposition Assumption holds and only if rank(fa) d.
Proof. Assume that Assumption 5 holds, iee, (G1...., pa) and {ui...., Aat are linearly indepen- dent. We show that rank d.
Since Tah EE (h, (r, we can write Fr = A t, where:
④ (X) Rd. defined by h + ((h, (1),,.)).... (h, (a)).
A: Rdd > L (Z), defined by a Σ 0,44..
Under Assumption 5 the family } is linearly independent, so + is surjective. Similarly, the family { } is linearly independent, A is injective. Therefore, the composition A o & has image of dimension
rank(Ta dim dim ))) dim (R(A)) d,
where the second equality follows from the surjectivity of db, and the third from the injectivity A.
Conversely, suppose rank(Ta) = d. Then Ta has image of dimension d, which implies that the v must be linearly independent. The same reasoning applied T shows that {st,.... Ga3 is linearly independent.
Under Assumption 5 we can therefore write Ta in the following SVD form:
f.
Σ
ā x 6,
(15)
19

---

where 81 2 2 ... 0d > 0 are the singular values of T and ûz and 0; are the left and right
singular functions of T.
Proposition 7. Under Assumption
5 Hig.d = span (01, ..... da} and H.sd" = span (.I..... ud).
Proof. We have by definition of the SVD that
N(Ta) span ( 6a) To show that Hppdd span {0r, .......; 0d). . suffices to prove that
(TTa) span (fi,..., pat
Let Lx(X). Then:
h he (Taa)
= Tah 0 Σ (h, v.)
d
Σ (h, Gi) =
(h, p.) = vi [d]
where the last equivalence follows from the linear independence of {
Therefore, N((aa) = (p .Ga]+ - which implies
Hapd _TTa span {01 da}
The proof for H.g.a span fūi..... ú.) analogous, using the adjoint Ti.
We now recall and prove Theorem
4 and 5 hold and let Ea = | Ta be such that éa <
Theorem 3. Let Assumptions
(1 - 1/V2)aa-
i) oā S Ta.a S
- (aa-28d) -1;
ii) Let h be the 2SLS estimator from Eq. 5 using features & X - Rd and J 2 + Rdd.
Suppose (os 2Ea) -1 V (log d)/n o(1) and E. oa(o))) then
=0p (I(TIx)) ( holl +
V2Ea [2el|hollLax()
- hm 12(x)
ad
ad 28d
Proof. We first show that oā < Tp.d S (a 2Ea) First note that if h E Hyp.d then by Proposition h E Va. Using the SVD of Ta from Eq. 15). and writing h nd (h, 0:)0 we
obtain:
d Fahe Σ (h, so [Tah|li
Σ
of (h, 02) 02111122(()
Next, observe that for all ha E L2(X).
||Th|laa(2) 2 Tah Lz(2))
By the reverse triangle inequality, the bound | |Ta Tal values, we have, for all E (x)). with ||h [|/]le((x) = 1:
Eed and Weyl's inequality for singular
T (z)) 2 [T//|l
2 | Lx(2) - f/a Tall ||Alle(())
> (Ga- &a)l| (x())
δ Ed
2 σa 28
20

---

This shows that
2 0d
29d
Tp. S (od - 28a)
which holds under the assumption Ed < (1 1/V2)ar < 0a/2. On the other hand, since dim(Hppa) = d by Assumption 5
Proposition (i) implies
dd
T o
Next, we show how to bound the sieve approximation emror, write:
( (x())
[(Llp,a) ((1xx (IIx,a) (x)) + [[ITtaa__lxx a (xg)) ((Ix,a)_hoolz +
(lx,aho))l
( [[([[, Hx,,allpp [|oollz,(X)-
By Wedin's sin-O theorem (Theorem 2.9 and Eq. (2.26a) in 9 and under the assumption éa < (1 - 1/v2)aa, we have:
V2Ea
Fd
Therefore,
2 (
(I (x) S a +
To conclude now check Assumption
Let introduce
[1
sup KeHap Reg
2 Tq,d.
3.p.d
||I T (all).(()
The unique use of Assumption 3
i) the proof of Theorem B.1 6 prove that s-1
O(T.,))
as d - t00, which now directly prove. First, note that for any L2(X), U.2,d
I (Tl + Ixx Nx,a) hill (
< [Tahlles(2) + ||T ax l1All
[2aal1bllaa(x)
$ [Tahllls +
0d
where we used | < the definition of Ta and Wedin's sin-O theorem. On the other hand,
[T T
TT a
TT
p.ah
| (TTTTTT.... IIx,a) h
L2(2)
Lz(2)
L2(2)
V28al [b] Lx(X)
-I(IIp.d II TI
[ a pII..pppaih s.hh Lz(
L2(Z)
0d
2
[|Tab (z(2)
0d
We therefore obtain that,
S sup
S Tg.d X
3.p.d
NeHo,a, hy 11 sh lzz(2) (
Iv aa
1
33/28
0d
where the inequality is valid for large enough as long as 3 3/ 2 Ta.d I (oa 26a)-1 a sufficient condition
o(1). Re-using that
3
28a = o(1),
which is satisfied if & o(o3). We therefore conclude that if E = o(a3) then 3.p.d O(T..))
Finally, check Assumption ii). First note that by definition of 11ped and Ta. we have
21

---

Ta(Ilp.)) = O, therefore,
Ta.a (I1 holl
Tp,d (T -
(T) (1 (.aa)) ho
S Tppd (II'T -Ta) p.a) holl + (Ta (T)
S (a E.) p(( holl
v 0a+1 + Edd
[( .). koll,
2dd
We conclude using that éd o implies Ea = o(aa) which implies (aa+1 + Ed)(Os - 2Ed) ((1).
We now prove the equivalence between Eq. 6 and Eq. 7 While the proof strategy follows that of Kostic al. [25] Theorem 1]. we include our own version for completeness, as the parametrization of the learned operator differs. Specifically, (25] directly approximate the truncated operator Ta using singular value decomposition of the form 1z @ 1x + Ei= o @ P.. where the singular values o; are learned explicitly via a separate network. In contrast, our approach learns only the feature maps (fit 4/:) and defines the approximation Ta( } Σe Vs @ Gr. As such, we provide short self-contained derivation adapted our setting for clarity. Recall that the Eckart- formulation of the objective is defined as
La(e, vb) [TTa vb) TIis (Tta,, 2(Ta(),,), v), T) us + (Tale,
Proposition 8. It holds that,
[Ta((e, ) ()) = ExEz (((()) -2Ex,2 (p(X)'
Proof. Let us introduce
d Φ : - L2(X), c Σ
d
CpPp:
v: L2(Z). c
cr..
1=1
such that Te((, VØ* vo = Elv(Z)) and --0 Elp(X) On one hand, we have:
[[Ta( 1)
|V 12
Tr "TV"TT
TT (T
Ex (Tr (I((((2)0(22))(()xx(()))
ExEz (()(2)((X))}]
On the other hand, we have:
(Talp,, v). Th = Tr (V
= Tr (E [M 2)
= =Exz [Ir (x(Z)x(x)))
-Exz +())))(()))
which conclude the proof.
As noted in (25), the above result does not require T to be Hilbert-Schmidt or even compact. Indeed as Ta(ip, (b) is finite rank operator, (Taly, +), T) as always well defined.
22

---

D Extended Related Work
We now discuss the various ideas that have been proposed to solve NPIV problems. Extensive benchmarking for these methods has already been conducted in prior work; refer, for instance, to
336
As mentioned in Section T a central challenge in NPIV estimation is that it constitutes an ill-posed inverse problem. The literature has evolved along several methodological lines to address it. Two broad classes of estimation strategies have become particularly prominent:
Two-Stage Least-Squares (2SLS), This classical approach and its various generalizations involves a sequential estimation procedure. Typically, the first stage estimates the conditional expectation of the endogenous variable (or its features) given the instrument, and the second stage uses these predictions to estimate the structural function.
Saddle-point optimization. These methods, often rooted in generalized method of moments (GMM) framework or duality principles, reformulate the NPIV estimation as finding an equilib- rium in min-max game. This often involves optimizing objective function over hypothesis space for the structural function and a test function space for moment conditions.
These two methodologies developed with distinct focuses. The 2SLS methods, with roots in classical econometrics [2] M. have progressively incorporated more sophisticated nonparametric techniques. the other hand, have gained traction with the rise of machine learning,
Saddle-poin point methods, offering powerful tools for handling complex optimization problems arising from conditional moment restrictions [3] 29 2
2SLS approaches. Early and influential nonparametric extensions of 2SLS employed sieve or series estimators. These methods approximate the unknown functions using finite linear combination of basis functions, such as polynomials, splines, or Fourier series. The number of basis functions, or the "sieve dimension", is allowed to increase with the sample size, enabling consistent estimation of the nonparametric functions.
A seminal contribution, by 32 provided identification results and a consistent nonparametric estimator for conditional moment restrictions. Their proposed NPIV estimator an analog of 2SLS, where the first stage involves nonparametric estimation of conditional means of basis functions of X given Z, and the second stage uses a series approximation for the structural function based on these first-stage predictions. Regularization is achieved by controlling the number of terms in the series approximation.
Building similar principles, [8 proposed nonparametric methods based on both Kerneertcchnique and orthogonal series for estimating regression functions with instrumental variables. For their orthogonal series estimator, they transformed the marginal distributions of X and Z to be uniform and used Fourier expansions. The estimated Fourier coefficients of the structural function were obtained via a regularized regression involving estimated coefficients from the first stage. They were among the first to derive optimal convergence rates for this class of problems, explicitly linking these rates to the "difficulty" of the ill-posed inverse problem, which is characterized by the eigenvalues of the underlying integral operator.
11 proposed estimation procedure based on Tikhonov regularization. This involves regularizing the inverse of the integral operator T (or its empirical counterpart) to stabilize the solution. Specifi- cally, denoting / and în empirical estimates (in their case, computed with kernel density estimators),
the Tikhonov regularized solution is of the form ha = - aa) t*fo. where > 0. They presented asymptotic properties of their estimator, including consistency and convergence rates, which depend on the smoothness of the structural function and the degree of ill-posedness of T.
35 introduced Kernel Instrumental Variable Regression (KIV), a direct nonparametric generalization of 2SLS. KIV models the relationships between the different variables as nonlinear functions in reproducing kernel Hilbert spaces (RKHSs). In stage 1, it learns a conditional mean embedding m{=)= E [x( Z 2] where (xX) represents features of X mapped into RKHS Hx. This learning is framed as a vector-valued kernel ridge regression, effectively estimating the conditional
""Keneel" meant here in the sense of density estimation, and not reproducing kernl:
18 eq. (2.4)).
23

---

expectation operator E Hxx - Hz. In stage 2, a scalar-valued kernel ridge regression of the outcome Y on the estimated means (z(Z) is performed to estimate the structural function. The authors prove the consistency of KIV in the projected norm under mild conditions and derive conditions under which KIV achieves minimax optimal convergence rates. The analysis was later improved by 31. who established convergence in L22 norm rather than the projected norm.
Another related approach, "fast IV" (FIV), was proposed in 39 FIV studies nonlinear IV with high-dimensional instruments and proposes two-stage pipeline that keeps the outcome model in a fixed RKHS/GP space but learns the first-stage kernel from a black-box adaptive regressor (e.g., neural network) distilled into a compact kernel basis for estimating the conditional expectation operator. The resulting estimator plugs this learned kernel into standard kernelized IV schemes, achieving rates that adapt to the dimensionality of informative instrument features. Their work focuses on learning a kernel basis for the instrument. In contrast, our method learns paired spectral features for both the instrument and the treatment to approximate the top singular subspaces of the conditional expectation operator.
Saddle-point approaches. These arise from reformulating the conditional moment restrictions that define the NPIV problem (F D. frequently conceptualized as a zero-sum game. One player selects function h from hypothesis space H. to minimize the objective. The other player, the "adversary" or "witness"" selects a test function g from a test function space 9 to maximize the objective. The adversary's 's role is select the function that maximizes the violation of the moment condition by the current choice of h. A notable advantage of this formulation is that it can allow bypassing the direct estimation of conditional expectations. Saddle-point objectives in NPIV regression can be derived in different ways.
One can frame the NPIV problem finding solution bn that minimizes a certain criterion (e.g., a norm) subject to satisfying the moment condition The = ro. The Lagrangian of this constrained problem then leads to a minimax objective. [2] target the least-norm solution
ho arg min AETa(X) 2
1
|I (x)) subject to Th ro-
The corresponding Lagrangian is given by Lff = [ + (ro - Th, L2(2) for g E Lz).). Their method achieves strong L2 error rates under source condition and realizability assumptions. Notably, their approach does not require the often- atten-problematc closedness condition or uniqueness of the IV solution. [29 consider similar approach, focusing the setting where the function classes are formed by neural networks, but use different techniques to analyze their method (online learning and neural tangent kernel theory).
Another approach to deriving a saddle-point problem involves using Fenchel duality to transform a squared error loss involving conditional expectations. This is the approach taken by (36), which we study in this paper.
A third method is directly use the unconditional moment formulation E. E{( -(X)) 0. define their criterion as the maximum moment deviation,
13
ho arg inf sup E [(Y - h(X)) (( (h,g).
REH geG
and define the estimator h arg min, sup, E, [(Y h(X)) + (h) ARz(g), where R1, Ri are regularizers. Their key theoretical result is that the statistical estimation rate, in terms of projected mean squared error [ ho) |l, scales with the critical radii of the hypothesis space H. and the test function space G. This holds under some closedness assumption, namely that E [X) - h'(X) 2] E G for any h, h' e H. We note that the method introduced by 43 defines a risk functional in terms of the squared moment deviation and is thus related to GMM.
The role and interpretation of the "adversary" function g E G and its associated objective function vary subtly across different saddle-poins formulations, which in turn influences the types of assumptions required. This distinction is important: if is intended to approximate E [Y - h(X) 2), then the space G must be sufficiently rich to do so, which is reflected via closedness condition. y is primarily a Lagrange multiplier, its existence and properties are more directly tied to conditions like source condition. This difference helps explain why 2 can dispense with the closedness assumption while other methods may require it.
24

---

About the contrastive loss. The contrastive loss used to learn spectral features derives its name from the foundational concept of contrastive learning, which traces back the early 1990s. Notably, [S] introduced the Siamese network architecture for signature verification, which utilized form similar Eq.(8) to measure the distance hetween input pairs. Later, [TO] formalized the contrastive loss function for face verification tasks. This loss is, in turn, linked to a truncated conditional operator (see Eq. ()). The underlying principle has been frequently redeveloped under various names, such correspondence analysis [17], principal inertia components [21 [2] for finite alphabets, the contrastive kernel [19 121. and pointwise dependence [37] in self-supervised representation learning. also proposed and studied by 38 and used as local approximation to the log-loss of classification deep neural networks [42]. More recently, [25 linked this same objective to the SVD of the conditional expectation operator.
The objective in Eq.
E Experimental Details
In this section, we provide additional details for the experiments presented in the main text. Let (Z, X be generated as described Scction [6] with n 100000. Figure [ displays both the true data-generating density and the density corresponding to set of learned spectral features. Even for = 11, the resulting densities are complex. While it is feasible to conduct analogous experiments with higher values of d, observed no qualitative changes in the outcomes. However, training and hyperparameter tuning became increasingly challenging as d grew.
Fitted density
Orginal Density
0,0425
0.0375
0.0325
2-
0.0175
0.0125
0.0025
Figure 4: Comparison of density corresponding to set of learned spectral features (left) and the true data generating density (right).
E.1 Models Employed
The features were learned using two-hidden-layer neural networks. All models shared the same architecture, with layer widths [1, 50, 50]: the input is one-dimensional, and the final layer outputs 50 learned features. To encourage the models learn more oscillatory functions, the first layer used the activation + + sin? (z), as introduced by 44, followed by UELU-actietted hidden layers and final linear layer.
We note that the first singular eigenfunctions of the conditional expectation operator T are always the constant functions (see Section (). Therefore, we hard-code the constant feature 1 1x into the model and restrict the following learned features to be mean-zero.
In addition, we included regularization term penalize both feature collinearity and large feature norms. This regularizer is a sample- based approximation of the quantity defined in Equation (10) of 25
E (Ix)())((X'' - -111 + (1L(Z))(Z)* - 1|3 - + 2E [X){[ 21 [1)
25

---

The spectral features were trained on 100,000 samples of (Z, (),X), while the 2SLS estimator built from the learned features used a separate dataset of 10,000 samples of (Z,X, Y).
E.2 Expanded example of the ugly scenario
As noted in the main text, the difficulty of recovering ho increases when it is not well supported on singular functions of the conditional expectation operator T. If the projection ho onto the span of the learned singular functions is sufficiently large, the missing components may not significantly harm the quality of the estimate. We illustrate this property with controlled experiment designed to vary the amount of support ho has in the singular space of T.
We fix 1 orthonormal features u, for Z and iy for X, and define the operator:
1 8 1x
Σ oju Us
mirroring the setup from Section We vary the spectrum by setting disk = c and 0k+1:d-1 = for k E {1...., 1}. We then define the target function:
ho
Σ
Vd
which is uniformly spread across the feature directions. l increases, the projection of hg onto the singular space of T increases in discrete steps from 0 to 1. This results in a corresponding qualitative improvement in the accuracy of the 2SLS estimator for hg- Figure SegureBillusee illustrates this behavior: for small values of k where ho is largely orthogonal to the singular functions of T, the estimate fails to recover hg. As grows, and more of ho 's energy lies in the span of these singular functions, the 2SLS estimate increasingly aligns with the true function.
F Estimating spectral alignment
In this section, we present the details of the methodology introduced in Section 5 We recall the central formula provided in Eq. (TO).
FE -BY uz (Z)).
(w,, ho) Lx())
(16)
a;
To evaluate spectral alignment, we shall approximate the RHS of the above. Suppose we have learned an operator Ta = Σi= vs @ Φ = vr which approximates the true conditional mean operator T. Here W: Rd - (Z) and Φ: Rd + (x) are the maps sending & - Σd ayyi and a + Σi= a; One can compute the SVD of Fa two steps. We start with the derivations assuming access to population quantities. Let Cz. and Cx be the d x d population covariance matrices of the learned Z, and features, respectively. For any 2 and x, we then introduce the whitened feature vectors (=)=) (C>1/2((>))) and (x(x) (x))) The corresponding operators, V VC 1/2 and &' = xCx 1/2 are isometries. We can then write
Ta
C C//2 a d x matrix, let its SVD be //2 OE where Eggy is positive-definite diagonal matrix with diagonal entries a. Now we can write
Fa
Finally letting Ug.ab. = O'Oe;, and Va. Φ Pe; with the corresponding operators Us,e Rd (Z), and Vo I + (XX),, we get the SVD
Fa Us Vo. V V....
In practice, given (Z, X) dataset, possibly the same one as was used train Ta one can perform sample-based counterparts to the above procedures to get approximate SVD.
26

---

k-1
K-2
ho mymo
10-
ho
-0.5
-1.0
1.3
0.0
-0.5
-1.5
k-B
k-10
3.5
0.5
0.0
1.0
1.5
X2
X2
Figure 5: Qualitative improvement the 2SLS estimate of he k increases. When 1, hy is orthogonal to the singular functions of T; when k = d 1, it is fully contained in their span.
Rationale for the alignment approximation method. We are interested in evaluating the length of the projection of ho onto the leading learned singular functions of Ta. that is of (Coww, ho). Following Eq. 100, we have
(Vo.wii ho) = a.d (Tá uo.ri ho) = σ l Tahio)
d (uo,ord Tho) + Iu. (Ta - hT)o).
The first term on the RHS above can be equivalently written as -. (up.p.. Tho) To... Blua.. (Z)Y), for which it is easy to compute sample-based approximation. This ap- proximation should be close the true value of (Vags. ho) provided that the second RHS term is small. Letting Ta be the rank-d SVD truncation of T we further telescope the second term on the RHS to get
Toy. (Muww (Ta Ta) + (Mo. (Ta -T)ho))))
(17)
Since Ta should converge to Ta for a sufficiently flexible feature-learning model class, is sensible to bound the norm of the first term above with _ o. l Tall [holl For any < d, this upper bound should converge to 0 with the size of the feature-learnigg sample size. To hopper-bound the second term of Eq. (17), note that by Wedin Sin-O Theorem, we have
TTa
Fal
II
27

---

Thus
Ta -Tll
(usaoa (Ta- 1
II no (Ta-T)hal! <
[ana (Ta Thholl +
(Ta-T)hul)
oa
=0
By construction, Span(una)) is orthogonal to the image of Ta T. Hence the first term on the RHS vanishes, leaving us with
(m ho) o.bba F[Yus...l S 0.0i Ta Tall|boll + oa b, ,, I Talll hholl
(18)
The essential takeaway from the bound above that as long as the feature-learning sample size can increase to ensure li Tall is small, approximating the alignment with v. uo,at, is consistent. Moreover, the bound becomes tighter when,
1. dd is big (close 1), which corresponds slow spectral decay looking the top singular function.
2. hg lies in the top of the spectrum of T that (Ta To |I is small.
We can additionally compare how close is 0 Bl ud.ba the alignment with the true eigenfunc- tion of T: Seg,hg): so that denote a; (, ho) and à; 00,0, Y us (Z)).
evaluate if we in the "good" or "bad" scenario. For i s d, The previous bound shows that
a: ā:l < ErT,,
EmT, of oe Ta
Tal holl oo oo Fallll(Ta -Thool
Then
I ee holl
Σ
o D.v
E us. (Z)]]
Σ
2
a,)) Σ
(a. á.)(as + ai)
i=1
i=1
i=1
d
S
Σ
a;
à;l la; + àzl
Next,
|a: + ā,l = =12a - (a: --)ll < 2/0,1 + la
àsl S 21/ool + Em,,
and
holl holl
Σ
Uns vo Yyu,
2
Σ Em (2] || Err.)
i=1
d h Em, (Em,) 2
i=1
Finally, to assess how well we get the alignment the eigenfunctions of observe that
T
Fal
I holl
[1 < (11 (Ilua.e.ad (Ilise
holl
/holl.
aa
Practical considerations. practice, we want to perform sample-based approximations of the procedure described above, and while the top singular functions of T and the corresponding singular values can be reliably estimated, learning the bottom the spectrum is more unreliable. Given that we divide E| ug... [Z)] by Tio... for singular values close to O, small error in estimating them is inflated by the inversion. Hence, we resort to a heuristic that allows us to decide which features and singular functions are learned reliably. Let T. = Ü. E.. Do.v be finite-sample approximation of the SVD of πे. That is, perform the SVD computation procedures of the preceding paragraph but relying on sample feature covariance matrices Öx and Čz. After fitting the SVD once, we recompute the covariance of fe.... and (.. on resampled (Z(M) x()) data with b 1,... and extract
its diagonal terms, which we refer to á (b)
1. Letting the original singular value estimates be
28

---

0o.., = (0) 0.0. we can use the sets (6 6.0 2 to evaluate the reliability of given singular value
estimate. If the variance of the estimates in given collection of a (b) 9. is large relative their mean, we treat the value unreliable and discard all the singular values 0...2k from further analysis.
For remaining singular values, 0,..., ".. ,.o.:,."..,,,vv. which we treat as "reliably learned" we utilise the quantiles of bootstrap samples construct pseudo-confidence-sets & d.v.. for de.eh. We use these as proxies for upper and lower bounds on o.... These provide us with matching pseudo-bounds on our estimates of (Vo.o.i ho). That is, the "confidence set" for (0i, ho? is the interval
(amax
ug (2)), (ow)))) - -B ".. EE RE(Yú,,e..o
together with the central estimate (o - YY
(.
F1 Spectral alignment in synthetic examples
Note that whenever the structural function is available, as is the case in our one-dimensional synthetic example and in the dSprites experiment, one can evaluate the correctness of the approximation proposed above by comparing it to (ho, vo.p,)). ) In Figure we observe that the method is reliable the decay of singular values is sufficiently slow for feature-Tearning pick up the singular function pairs reliably. For Co = 0.8 the approximations are very good while for Ca = 0.2 they become entirely unreliable. A more thorough theoretical analysis of these estimates is a topic warranting further investigation but these observations are generally line with Eq. (18). In particular, small values of d inflate the approximation error.
F.2 dSprites is in the good regime
We fitted the spectral -eattre models for dSprites using the same architectures as proposed by [36]. Analogously the fully synthetic experiment, measure where in it, ho is supported. As seen in Figure 7 the top of the learned spectrum is very flat, and hence, in line with what observed in the previous example, the alignment estimates are reliable. Our experiments confirm that bg is usually spanned by the leading 32 spectral features and 85% of its squared norm spanned by the leading 10 functions. Hence, dSprites is indeed example of the good case.
able to evaluate the spectrum of T and directly
29

---

inve Ae true (right)
944
Youre Ael (stimate(righhs)
apasl " true (left)
Commllezin Altannnul 2
Indiedual Pesture Algunnut
Buul (lcf))
x
Indivitoal
Singsar Value Inderx
Singnlar Value Indect
Co 0.2, Ca. 0.2
(b) 0.2, Ca. 5.0
aediit Aliqunzzii
Commlative Allemmon
C
Singraa Value Indeex
Singular Vaiue (d) 0.6, 5.0
(c) Ca 0.6, Ccr 0.2
Indrediaal Festure Aligmment
14
614
Ally
(e) Cr 0.8, Ciy 0.2
( c, = 0.8, c.. = 5.0
Figure 6: Evaluation of the reliability of spectral alignment estimation depending on the real spectral alignment measured in terms of and the rate of singular value decay
30

---

Maunt I Feliiatte
Slogular Value Betimote Distetiotion
1a Au true
Singslar Value Inder
(a) Distributions resampled values (b) evaluated for a single Ta model.
(b) Estimates and true values of the projection length of yu onto the leading features.
Figure 7: Evaluation of dSprites spectral alignment. Left: the confidence intervals correspond to bootstrapped refits of the feature covariance for individual model. Right: the confidence intervals are obtained from independently trained models with identical hyperparameters but different random seeds.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes]
Justification: We specifically point out the settings and detailed contributions of our work in both abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: describing our assumptions as well as discussing our theoretical results, we specifically list our limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We specifically list all assumptions in the main text and refer to each as- sumption individually for every theoretical result we obtain. All proofs are reported the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
32

---

All the theorems, formulas, and proofs in the paper should be numbered and cross
referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We provide complete description of the data generation process, experimental setup, and hyperparameters used in all simulations, ensuring full reproducibility of our results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken
to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide our code in a zip file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
33

---

reproducible, they While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
httpa///guidee//eeuuubiiiiseee / //nips.cc/ wwtppblic/guiiiiaacccc des/ CoodeSubiisss cettuubmssiioo stipoiiissioooooccii.. onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide a complete description of the data generation process, experimental setup, and hyperparameters used in all simulations, ensuring full reproducibility of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes]
Justification: Our experimental results are based on multiple realizations, standard deviation are reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2- 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
34

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our experiments are lightweight and can be reproduced on a standard laptop without specialized hardware.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttga////eurrsss..////blb///bba http://tthicccccdeeeeeeeeeeee Answer:
[Yes]
Justification: We have reviewed the Code of Ethics this paper conforms with the Code. Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction)
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes]
Justification: Our paper mainly focuses on investigating the learning efficiency a two stage least squares estimator with spectral features. We therefore believe our paper does not have any negative societal impacts. We explain how our theory can improve our understanding of various causal learning algorithms in the introduction which could be potential positive societal impacts. Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal
impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
35

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: The paper does not use existing assets. Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. wwt.coo//aaaaa com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots. if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:
36

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should be described.
37


Reviews:
[{"review_id": "uQAQc5piAK", "reviewer": "Reviewer_9cMm", "summary": "- This paper studied the problem if using instrument variable for causal effect estimation when there is an existence of unobserved confounders. A potential unobserved confounder may affect both the treatment and effect, leading to incorrect causal effect estimation.\n- In particular, the paper focused on the nonlinear two-stage least squares method for using instrument variables. Under the nonlinear 2SLS framework, regressions are done between the projected feature space of the treatment and instrument variable. Then the outcome is regressed   further against the predicted features. \n- This paper investigated in the conditions where the nonlinear 2SLS approach perform well or badly, and identify three different regimes where the effectiveness of the framework is affected by the alignment between the structural function and the operator, and the rate of the singular value decay. Numerical simulations support the theoretical analysis.", "questions": "- how does the misalignment between h0 and the top eigenspaces of T compare to a fast eigenvalue decay? Which source is more significant in terms of damaging the effectiveness of the 2SLS?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n- Overall the paper is well-organized and presented clearly. This work considered an interesting angle of nonlinear 2SLS: what are the good or bad features to choose - which significantly affects the effectiveness of the regressions.  The framework was illustrated with practical examples, and each step of the analysis is sketched out in sequel.\n- The result decomposes clearly the sources of errors in feature representation and approximations, demonstrate that the 2SLS with spectral features performs the best when h0 aligns with the top eigenspaces of T and its singular values decay slowly. This insight is useful for practitioners to perform the best practices when using 2SLS.\n\nWeakness:\n- although the paper studied nonlinear 2SLS and it's good / bad scenarios, the theoretical innovations are relatively weak. It is natural to see that due to the assumptions and design of the 2SLS framework, the best scenario occurs when h0 aligns with the top eigenspaces of T and its singular values decay slowly. This finding is more on the incremental side and lacks more innovations.\n- the numerical experiments does not use any real-world datasets but only simulated data. The experimental results can be strengthened with more complex datasets and comparisons to other existing approaches.", "was_revised": true, "final_justification": "I had concerns on the theoretical contribution and lack of more complex real-world data in the experimental secion. The authors response partially addressed these concerns, however my concern on the technical innovations remains. I will update my score to positive.", "rebuttal": "We thank the reviewer for the time spent in reviewing our manuscript, for the encouraging evaluation of our work, and for the helpful feedback. We address the questions and concerns of the reviewer below, and have proposed updates to the manuscript to address the latter.\n    \nWe would be grateful if the reviewer might consider increasing their score in case they are satisfied with the proposed changes.\n\n- **W1. Regarding the theoretical contribution**. While the high-level intuition that spectral alignment and singular value decay matter seems natural in retrospect, this understanding remained at the level of folklore, lacking a rigorous treatment. Our primary contribution is to systematize this intuition, characterizing the different performance regimes and, crucially, the failure modes of spectral feature learning for NPIV. By establishing a concrete \"good-bad-ugly\" taxonomy, along with a method to empirically evaluate which case a given problem falls into (see our answer to Weakness 1 of **Reviewer m9NX**), we offer valuable insights to both theorists and practitioners.\n\n- **W2. Regarding the use of real-world data**. We used synthetic data to cleanly isolate the effects of spectral alignment and singular value decay, as these ground-truth properties are unknown in real-world settings. We agree that validating our taxonomy on more complex datasets is an important step. As detailed in our response to Weakness 1 of **Reviewer m9NX**, we have a procedure to estimate spectral properties from data and have applied it to the dSprites dataset, confirming it falls into the \"good\" regime. We will add these results to the final manuscript.\n    \n- **Q1. Comparison between misalignment and fast decay**. We split our answer in two parts: first, assuming we work with the true singular functions of $\\mathcal{T}$ as features; second, when we work with learned features obtained from our constrastive learning procedure.\n    \n    **Assuming one has access to the true normalised features** (i.e. the singular functions) $u_i,v_i$ of the conditional mean operator $\\mathcal{T} =\\sum_{i \\geq 1 }\\sigma_i v_i \\otimes u_i$. The overall generalization error rate for $h_0$ depends on two key factors:\n        \n    - *Operator ill-posedness*: The rate of singular value decay of $\\mathcal{T}$, typically $\\sigma_i \\asymp i^{-p}$ for $p \\geq 1$. A larger $p$ indicates faster decay and thus greater ill-posedness. \n    \n    \n    - *Solution smoothness/alignment*: The source condition\n    $$\n    \\left\\lVert h_0 \\right\\rVert^2_{r} := \\left\\lVert (\\mathcal{T}^*\\mathcal{T})^{-r/2}h_0 \\right\\rVert^2 = \\sum_{i\\geq 1} \\frac{\\langle h_0, u_i \\rangle_{L_2(X)}^2}{\\sigma_i^{2r}} < +\\infty\n    $$\n    This condition, for $r \\geq 0$, quantifies how \"smooth\" $h_0$ is relative to the spectral decomposition of $\\mathcal{T}$, or equivalently, how quickly the alignment coefficients $\\langle h_0, u_i \\rangle_{L_2(X)}$ decay relative to the singular values. A larger $r$ implies $h_0$ is \"smoother\" or better aligned with the less ill-posed directions of $\\mathcal{T}$. Note that under the \"bad\" scenario where the eigendecay is fast ($p$ is large), a stronger alignment is necessary for $\\left\\lVert h_0 \\right\\rVert^2_{r}$ to be finite for a large value of $r$. While in the \"good\" scenario with $p \\sim 1$, $\\left\\lVert h_0 \\right\\rVert^2_{r} < +\\infty$ is more likely to hold with a large value of $r$.\n    \n    Under the source condition, the approximation error when using as features the top-$d$ singular functions of $\\mathcal{T}$ is:\n    $$\n    \\sum_{i>d} \\langle h_0, u_i \\rangle_{L_2(X)}^2 \\leq \\sigma_d^{2r} \\sum_{i>d} \\frac{\\langle h_0, u_i \\rangle_{L_2(X)}^2}{\\sigma_i^{2r}} \\leq \\sigma_d^{2r}\\left\\lVert h_0\\right\\rVert^2_{r}.\n    $$\n    Combined with the eigenvalue decay (involving $p$), we obtain from Corollary 1 in our submission that:\n    $$\n    \\left\\lVert \\hat{h}-h_0\\right\\rVert_{L^2}=O_p\\left(\\left\\lVert h_0 \\right\\rVert_{r}\\sigma_d^{r}+\\sqrt{\\frac{d}{n\\sigma_d^2}}\\right) = O_p\\left(\\frac{\\left\\lVert h_0 \\right\\rVert_{r}}{d^{r\\times p}}+d^{1/2 +p}\\sqrt{\\frac{1}{n}}\\right).\n    $$\n    Balancing both terms in $d$ as a function of $n$, we obtain \n    $$\n    \\left\\lVert\\hat{h}-h_0 \\right\\rVert_{L^2}=O_p\\left(n^{-\\frac{r}{2(r+1) + p^{-1}}}\\right).\n    $$\n    with $d(n) = n^{\\frac{1}{1 + 2p(r+1)}}$. This bound is tight as a matching lower bound is obtained in [1, Theorem 1-(i)]. \n    \n    From this tight rate, we see that both the ill-posedness ($p$) and the relative smoothness ($r$) fundamentally damage the effectiveness of 2SLS, and their impacts are inherently intertwined.\n    \n    - A fast decay (large $p$) directly slows down the rate regardless of $r$. It means singular values drop quickly, making it fundamentally harder to resolve higher-frequency components of $h_0$.\n    - A low relative smoothness (small $r$) means $h_0$ has significant components aligned with directions corresponding to very small singular values, even after accounting for the singular value decay. This also directly slows the rate, effectively rendering the problem very hard or even impossible ($r \\to 0_+$, makes the rate vacuous, representing our \"ugly\" scenario).\n    \n    **When working with learned features via contrastive learning.** We see from our Proposition 3 that when we learn the spectral features using contrastive learning, we incur an additional penalty $\\varepsilon/\\sigma_d$ in the upper bound, where $\\varepsilon$ is the optimality gap on the contrastive learning loss. The alignment of $h_0$ does not enter this factor, as learning the eigensubspaces of $\\mathcal{T}$ is unrelated to $h_0$. This term shows that a faster eigendecay makes the weight of the error incurred by contrastive learning grow faster. Intuitively, features corresponding to small singular values can be empirically difficult to learn, even with a very large number of $(Z,X)$ samples.\n    \n    We will add this discussion as a remark in the main paper, with full details in the Appendix. We think that it is a great addition to the discussion in our manuscript.\n    \n[1] Xiaohong Chen, Markus Reiss. \"On rate optimality for ill-posed inverse problems in econometrics.\" Econometric Theory, Vol. 27, No. 3, INVERSE PROBLEMS IN ECONOMETRICS (June 2011), pp. 497-521."}, {"review_id": "hObQaMPcud", "reviewer": "Reviewer_m9NX", "summary": "The paper provides a theoretical analysis of the spectral feature learning for instrumental variable regression. Via a detailed analysis of the conditional expectation operator $\\mathcal{T}$ between the features and the instrument, the work complements previous empirical observations that the performance of two stage IV regression estimator depends on the spectral structure of the operator and its alignment with the ground-truth link function. The paper shows that spectral contrastive learning, introduced in previous work, approximates the SVD of $\\mathcal{T}$ and identifies three regimes which determine the effectiveness of such spectral-based methods. These regimes depend on the spectral alignment and singular value decay of the conditional expectation operator. Finally, the authors illustrate their theoretical results and the identified regimes with numerical simulations, where they explicitly control the decay and alignment parameters.", "questions": "1. How robust are the results to \"slight\" violations of Assumption 1? Partial identifiability frequently arises in applications where one uses instrumental variable methods, cf. , for instance, anchor regression [1]. What would the error decomposition roughly look like in such a scenario? \n2. In Proposition 3, the choice of the cutoff dimension $d$ seems to depend on the decay of singular values of $\\mathcal{T}$ which is a priori unknown. How would one choose $d$ empirically? \n\n[1] Rothenhäusler, Dominik, Nicolai Meinshausen, Peter Bühlmann, and Jonas Peters. \"Anchor regression: Heterogeneous data meet causality.\" Journal of the Royal Statistical Society Series B: Statistical Methodology 83, no. 2 (2021): 215-246.", "limitations": "The authors have properly addressed the limitations of the study.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper presents novel theoretical contributions: it shows that the current state-of-the art NPIV estimator, based on a contrastive learning objective, approximates the spectral structure of the conditional expectation operator. Based on this, the authors propose a useful and intuitive decomposition of its generalization error and propose regimes based on its behaviour. Importantly, the results seem to suggest that the empirical success of the state-of-the art 2SLS estimator does not translate into universal theoretical guarantees, but rather only illustrates the \"good\" regime, which is a valuable insight for theorists and practitioners in the field.\n2. The paper is very well-written, self-contained and is very clear about its contributions and limitations. \n3. All assumptions and results are appropriately introduced and additionally discussed informally, providing intuition.\n\nWeaknesses:\n\n1. The main weakness of the paper seems to be the experimental evaluation. For instance, in lines 92-94, it is suggested that the empirical success of the decoupled spectral approach in [32] is due to the experiments in that work satisfying the \"good\" spectral scenario. However, this claim is not backed by evaluation of any \"proxies\" for alignment/decay on the datasets in [32] (dSprites and Demand Design) placing them in a \"good\" scenario. Additionally, the work would profit from real-data or semi-synthetic experiments illustrating the bad/ugly scenarios, since this would provide evidence that those scenarios do occur in realistic datasets. \n2. If I am not missing anything, the work does not seem to provide corresponding lower bounds for the generalization error. This would be useful, for instance, to prove optimality of spectral methods in the \"good\" regime. Since the upper bound in, e.g., Theorem 1, is not provably tight, the reader cannot conclude whether the three scenarios provide a fundamental limitation for Sieve 2SLS, information-theoretic limitations for any method or are artifacts of the upper bound. Thus, a tight analysis would greatly improve the interpretability of the scenarios. \n3. In Proposition 3, the $\\epsilon$ is defined implicitly, and the decay rate of the resulting $\\frac{\\epsilon}{\\sigma_d}$ is unclear. This slightly weakens the interpretability of the approximation error upper bound.", "was_revised": true, "final_justification": "The authors' rebuttal has clarified several aspects of the paper and improved my understanding of its contributions. I believe that the added discussion of the empirical evaluation of spectral alignment improves the scope of the paper. Thus, my concerns have been largely addressed, and I would recommend the paper for acceptance.", "rebuttal": "We thank the reviewer for their positive assessment of our contribution and valuable comments! We have addressed all points of concern and have proposed edits to the manuscript. We elaborate on the details below.\n\nWe would be grateful if the reviewer might consider increasing their score in the event that they are satisfied with the proposed changes.\n    \n- **Q1. Robustness to violations of Assumption 1.** As noted, estimating the structural function ($h_0$) in the NPIV model is equivalent to solving the ill-posed inverse problem $\\mathcal{T}(h) = \\mathbb{E} [Y∣Z]$ in $h$, where identifiability of $h_0$ hinges on the injectivity of the operator $\\mathcal{T}$ (Darolles et al [9]). When $\\mathcal{T}$ is not strictly injective, we face partial identifiability. In such scenarios, while unique identification of $h_0$ is lost, it is possible to establish consistency to a minimum-norm solution of the inverse problem. This approach is common in partially identified instrumental variable models (Bennett et al [2], Meunier et al [27]]).\n    \n    Regarding the second part of Assumption 1 (existence of a solution, $\\mathcal{T}^{−1}(\\{\\mathbb{E}[Y∣Z]\\}) \\ne \\emptyset$): This is a fundamental model specification assumption. A violation here implies that our NPIV model structure is misspecified for the data, meaning no true $h_0$ exists that satisfies the moment condition of the model. This is a much more severe form of \"violation\" than a lack of injectivity, as it implies the problem itself has no solution consistent with the model, and therefore cannot be \"slightly\" violated in a meaningful way for consistent estimation. Any prior work on NPIV relies on this assumption for the existence of a suitable $h_0$.\n    \n    Finally, we thank the reviewer for pointing us to anchor regression. It is crucial to clarify that anchor regression addresses a different challenge than violations of our Assumption 1. While our assumption pertains to the existence and injectivity of the operator $\\mathcal{T}$ for the identifiability and consistent estimation of the structural function $h_0$ in the NPIV model, anchor regression is designed for scenarios where traditional instrumental variable assumptions, specifically the exclusion restriction, are violated (i.e., the instrument directly causes the outcome). This violation does lead to a loss of identifiability for the causal effect in classical IV setups. However, anchor regression's primary objective is distributionally robust prediction, providing a stable predictive model even when an 'instrument' is technically invalid, rather than focusing on the consistent identification of the underlying structural function as our NPIV framework does.\n\n    For conciseness, we did not elaborate on this in the main text. However, we agree this is an important nuance and will add a remark after Assumption 1 to clarify this point and its implications for partial identifiability.\n\n- **Q2. Empirical choice of cutoff dimension (d).** We agree that the optimal choice of the latent dimension $d$ (cutoff dimension) inherently depends on the decay of the operator's singular values, which is *a priori* unknown. This is a common challenge in ill-posed inverse problems and spectral methods, analogous to selecting the effective rank in noisy matrix completion. Theoretically, this can be addressed using concepts like the Baik–Ben Arous–Péché (BBP) transition [R1], which provide a formal prediction for the \"edge\" of the spectrum that arises purely from statistical noise. Any singular value found in the data that is significantly larger than this theoretical edge would be considered part of the signal. A simple and practical way to find this same signal-noise boundary is to perform a stability check: Repeatedly estimate the singular values on different bootstrap samples of the data. The estimates of the true \"signal\" singular values will be stable (low variance), while the \"noise\" values will be unstable (high variance). Then, choose $d$ at the \"elbow\" where this transition occurs. We refer to our answer to W1 below for further discussion.    \n    \n- **W1. Experimental evaluation, spectral scenarios.** Empirically evaluating proxies for spectral alignment is indeed essential for assessing whether practical examples fall into the \"good\" spectral scenario.\n    \n    **Quantifying spectral alignment in dSprites**\n    \n    After submission, inspired by the numerical methods for linear inverse problems [R2], we developped a strategy to quantify the spectral alignment, $a_i = \\langle h_0 ,u_i\\rangle_{L^2(X)}$ where $u_i$ are the left eigenfunctions of $\\mathcal{T}$: $\\mathcal{T} = \\sum_{i \\geq 1} \\sigma_i v_i \\otimes u_i$. Coeficient $a_i$ directly measures the contribution of the $i$-th singular function to $h_0$ in the $L_2(X)$-norm, that is $\\lVert h_0 \\rVert_{L_2(X)} = \\sum_i a_i^2$.  Since $h_0$ is unknown, we exploit the identity:\n    $$\n    a_i = \\frac{1}{\\sigma_i} \\langle h_0, \\mathcal{T}^*v_i \\rangle_{L_2(X)} = \\frac{1}{\\sigma_i} \\langle \\mathcal{T} h_0, v_i \\rangle_{L_2(Z)} = \\frac{1}{\\sigma_i} \\mathbb{E}[(\\mathcal{T}h_0)(Z)v_i(Z)] =  \\frac{1}{\\sigma_i} \\mathbb{E}[\\mathbb{E}[Y \\mid Z]v_i(Z)] =  \\frac{1}{\\sigma_i} \\mathbb{E}[Yv_i(Z)]\n    $$\n    where we used the singular function relationship $\\sigma_i u_i = \\mathcal{T}^*v_i$ and the fact that $h_0$ satisfies $\\mathcal{T}h_0=\\mathbb{E}[Y \\mid Z]$.\n    \n    This identity shows we can estimate the coefficient $a_i$ by estimating the singular value $\\sigma_i$ and the left singular function $v_i$ from $(Z, X)$ samples, and then estimate $a_i$ using $(Y, Z)$ samples. More specifically, the procedure is as follows.\n    1. Learn spectral features using the spectral contrastive method from our paper.\n    2. Estimate SVD components: Perform a canonical correlation analysis (CCA) on these features to get estimates of the leading singular values $\\hat \\sigma_i$ and singular function $\\hat v_i$.\n    3. Quantify stability: Perform subsampling and get a measure of uncertainty for our estimates (e.g., define confidence sets using quantiles). One would typically expect the estimates for the leading components to be stable, while uncertainty grows for components corresponding to smaller singular values.\n    4. Estimate alignment: using these stable estimates, calculate the proportion of the $L_2$ norm of $h_0$ captured by the leading features.\n    \n    Applying this procedure to the dSprites dataset reveals:\n    - The singular value spectrum is initially flat and then decays sharply, indicating a spectral structure that exhibits the BBP-type transition discussed in our response to Q2.\n    - Our alignment estimates show that approximately 85% of the squared norm of $h_0$ is explained by the top 10 features, and about 95% by the top 20.\n    \n    Together, these findings provide evidence that the dSprites experiment falls into the \"good\" regime. This methodology provides a practical tool for assessing spectral properties in real-world settings. We will add a full description of this procedure and the new results to the appendix.\n    \n    **Demand Design dataset**\n\n    The Demand Design dataset includes observed confounding variables. This leads to the conditional mean operator being non-compact. Consequently, direct notions of spectral alignment and decay for $\\mathcal{T}$ become significantly more involved and are outside the scope of our current work, though an interesting and important problem for future research.\n    \n    **Illustrating bad and ugly scenarios**\n    \n    Illustrating the \"bad\" and \"ugly\" scenarios on real datasets is an important direction. Our primary goal in this work was to first establish a clear theoretical framework for these failure modes and validate their existence in a controlled setting. The empirical tools we have now developped (as discussed for dSprites) make it feasible to search for these scenarios in practice.\n    \n    While we have not yet run specific experiments, we can describe plausible bad/ugly real-world situations:\n    - \"Bad\" scenario (weak instrument): this is a well-known challenge in applied econometrics, the bad regime can arise when an instrument has only a weak effect on the treatment. For instance, using small, local policy variations as instruments for nationwide economic behavior often leads to this issue.\n    - \"Ugly\" scenario: this failure mode can occur if the instrument influences the treatment through a mechanism that is irrelevant to the structural function. For example, if an instrument (like an insurance plan) predicts treatment (a drug), but the health outcome depends on a factor (like a gene) that is uncorrelated with the instrument, then the estimation of the structural function would fail because the instrument provides the \"wrong\" kind of variation in the treatment.\n\n- **W2. Lower bound.** We agree with the point made on the lower bound. As explained in our answer to Q1 of **Reviewer 9cMm**, we can now show that our upper bound in Corollary 1 is tight, addressing the optimality of spectral methods and the fundamental nature of our proposed scenarios. This will be added as a remark, with full details in the appendix.\n            \n- **W3. Control on $\\varepsilon$.** We agree that the absence of precise characterization of $\\varepsilon$ in Proposition 3 weakens interpretability. However, representation learning is challenging enough on its own, and uncovering tight theoretical bounds for $\\varepsilon$ would require a dedicated and substantial investigation (e.g., a full paper on spectral contrastive learning generalization), which lies beyond the scope of our current work. We will add a remark near Proposition 3 to explicitly highlight this as an important and open problem for future research.\n    \n[R1] Baik et al. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. Annals of Probability, 2005\n\n[R2] Hansen et al. Deblurring Images: Matrices, Spectra, and Filtering, SIAM, 2006"}, {"review_id": "qRtzN5sF7V", "reviewer": "Reviewer_baed", "summary": "This paper studies the problem of causal effect estimation under hidden confounding using nonparametric instrumental variable regression. The authors propose a two-stage least squares estimator based on spectral features derived from the conditional expectation operator. A key contribution is the theoretical analysis that establishes a generalization error bound and highlights two critical factors affecting performance: the spectral alignment between the structural function and the operator’s eigenfunctions, and the decay rate of the operator’s singular values, which reflects instrument strength. Based on these insights, the paper introduces a good–bad–ugly taxonomy that characterizes when the method performs well or fails. Synthetic experiments are carefully designed to validate this taxonomy.", "questions": "Add more comparison experiments: One notable limitation of the experimental section is the absence of comparisons with established baseline methods such as Kernel IV or DeepIV. These kernel-based and deep learning-based IV models are commonly used in causal inference under endogeneity and provide strong empirical baselines. Adding such comparisons would make the evaluation more convincing by showing where spectral feature learning performs better or worse in more realistic and challenging scenarios. Without these baselines, it is difficult to tell whether the proposed method brings practical improvements over existing approaches, especially when the spectral alignment or properties of the operator cannot be directly observed or controlled.", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "S1: The paper provides a rigorous theoretical analysis of spectral 2SLS for NPIV regression, including provable generalization bounds.\n\nS2: The experiments are carefully designed, I like the setting for good–bad–ugly taxonomy.\n\nS3: The integration of spectral learning theory with causal IV estimation is novel\n\nW1: While the theoretical results are solid, no real-world datasets are used to validate the findings. \n\nW2: The paper occasionally assumes familiarity with advanced spectral operator theory, which may be difficult for broader NeurIPS audiences (from my point of view).\n\nW3: The  practical applicability is somewhat limited by the fact that real data rarely comes with ground truth for spectral alignment or singular value decay, making it hard to operationalize the taxonomy without strong assumptions.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful reading and positive feedback on our paper. We have addressed the reviewer concerns and questions below, and we believe our proposed revisions significantly strengthen the manuscript.\n    \nWe would be grateful if the reviewer might consider increasing their score in case they are satisfied with the proposed changes.\n\n- **W1. Lack of real-world datasets.** We agree with the reviewer that incorporating more complex datasets would strengthen our experimental validation. To address this, we have added empirical results on the dSprites dataset. This is a well-established, high-dimensional dataset for NPIV, serving as a crucial example of a complex scenario where our method demonstrates strong applicability and scalability. We refer to our answer to Weakness 1 of **Reviewer m9NX** for details.\n    \n- **W2. Assumption of familiarity with advanced spectral operator theory.** We agree that parts of the paper might assume a high degree of familiarity with advanced spectral operator theory, potentially limiting accessibility for a broader NeurIPS audience. To address this, we will add a dedicated section in the appendix to provide a concise and accessible introduction to the necessary concepts from spectral operator theory. This section will clarify key definitions and notations essential for understanding our theoretical framework, aiming to make the paper more self-contained and interpretable without burdening the main text.\n    \n- **W3. Practical applicability and unobservable spectral properties.** We agree that the unobservability of these properties in practice is a significant challenge. Crucially, since submission, we have developed a strategy to empirically quantify spectral alignment and eigendecay and have tested it on the dSprites dataset. We also show how to empirically identify the effective cutoff dimension ($d$). This provides a means to gauge the decay of singular values and thus the effective ill-posedness, helping to operationalize aspects of the \"good,\" \"bad,\" and \"ugly\" scenarios in practice. We refer to our answer to Question 2 & Weakness 1 of **Reviewer m9NX** for details.\n\n    We will incorporate details of these empirical strategies into the paper, demonstrating how practitioners can gain insight into the spectral properties of their data, even without ground truth.\n    \n- **Question. Absence of comparisons with established baseline methods.** It is important to clarify that our paper primarily focuses on the theoretical characterization of spectral features and their performance regimes (good, bad, ugly) for NPIV. The spectral features framework, which our paper analyzes theoretically, has already been empirically benchmarked extensively against a range of other state-of-the-art methods by [1] (including KIV, DFIV, DeepGMM and Dual Embedding). That study provides comprehensive empirical evidence of its practical performance against established benchmarks. Therefore, a full-scale empirical comparison was not the main objective of the present theoretical contribution. We will add a sentence to the experimental section of the main paper to explicitly direct readers to this relevant work and its comprehensive benchmarks.\n\n[1] Haotian Sun, Antoine Moulin, Tongzheng Ren, Arthur Gretton, Bo Dai \"Spectral Representation for Causal Estimation with Hidden Confounders\", AISTATS, 2025."}, {"review_id": "7TW8chfP2o", "reviewer": "Reviewer_QU6R", "summary": "This paper learns spectral features for NPIV regression to estimate causal effects in the presence of hidden confounders. It derives a generalization error bound for a 2SLS estimator utilizing these features, identifying three performance regimes: good, bad, and ugly.", "questions": "1. Provide a dedicated subsection (e.g., in the related work or methodology) that offers a detailed comparative analysis with \"Kernel Instrumental Variable Regression\" (Singh et al., 2019), \"Dual Instrumental Variable Regression\" (Bennett et al., 2023), and \"Fast Instrument Learning with Faster Rates\" (Kim et al., 2022). While some of these works are cited, the current discussion is too brief and appears to misrepresent aspects of their original formulations.\n    \n2. Could the authors expand the experimental validation to include comparisons with a broader range of advanced instrumental variable methods (e.g., Kernel IV, Dual IV, AGMM, DeepGMM, Fast IV, DFIV, DeepIV) and, ideally, incorporate experiments on at least one high-dimensional, complex real-world dataset?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper studies nonparametric instrumental variable regression, a critical problem in causal effect estimation where hidden confounders exist.\n    \n2. The paper derives a generalization error bound for a two-stage least squares estimator based on spectral features.\n    \n3. While spectral features have been heuristically linked to NPIV in prior work , this paper provides a novel and rigorous theoretical characterization of their performance through the generalization error bound and the \"good-bad-ugly\" taxonomy.\n\n**Weakness:**\n\n1. The paper acknowledges that instrumental variable regression is often implemented as a two-stage procedure and mentions fixed and learned features. However, while it references other works that formulate IV regression as saddle-point optimization , it could provide a more detailed comparative analysis of its proposed method with specific, related prior works such as \"Kernel Instrumental Variable Regression\" (Singh et al., 2019 ), \"Dual Instrumental Variable Regression\" (Bennett et al., 2023 ), and \"Fast Instrument Learning with Faster Rates\" (Kim et al., 2022 ). Explicitly highlighting the similarities, key differences, and unique advantages or limitations compared to these methods would strengthen the paper's positioning.\n    \n2. The experimental section is simple, primarily relying on synthetic data where feature spaces are modeled using trigonometric functions. Its generalizability to highly complex, real-world, high-dimensional data (e.g., images, text) where neural networks are typically employed for feature learning is not fully explored or demonstrated.\n    \n3. The experimental section would be significantly strengthened by including comparisons with state-of-the-art methods such as Kernel IV, Dual IV, AGMM, DeepGMM, Fast IV, DFIV, and DeepIV.\n    \n4. The synthetic experiments use trigonometric functions for feature spaces. While useful for isolating effects, the generalizability of these findings to highly complex, high-dimensional real-world data where neural networks are typically employed might warrant further discussion or empirical demonstration.\n    \n5. The main body of the paper lacks a dedicated \"Conclusion\" section. Additionally, while the authors state in their checklist that they discuss the limitations of their work, this discussion is not apparent in the corresponding sections of the paper.", "was_revised": true, "final_justification": "Authors agreed to expand the theoretical comparison and add dSprites experiments, partially addressing my concerns. However, the lack of a direct head-to-head comparison with Kernel IV, Dual IV, AGMM, DeepGMM, Fast IV, DFIV, and DeepIV persists. I therefore raise my score but not to be 5/6.", "rebuttal": "We thank the reviewer for their careful reading and insightful comments on our paper. We have addressed the reviewer concerns and questions below, and we believe our proposed revisions strengthen the manuscript.\n    \nWe would be grateful if the reviewer might consider increasing their score in case they are satisfied with the proposed changes.\n    \n- **Weakness 1 & Question 1. Detailed comparative analysis in literature review**. We agree that given the large amount of NPIV papers, a more explicit discussion of the similarities, key differences, and unique theoretical advantages or limitations of our approach compared to these relevant methods would strengthen the paper's positioning within the literature. We will revise the related work in the camera-ready version to include this expanded theoretical comparative analysis and the missing citation to \"Dual Instrumental Variable Regression\" and \"Fast Instrument Learning with Faster Rates\".\n    \n- **Weakness 3 & Question 2. Experimental comparisons with state-of-the-art methods**. It is important to clarify that our paper primarily focuses on the theoretical characterization of spectral features and their performance regimes (good, bad, ugly) for NPIV. The spectral features framework, which our paper analyzes theoretically, has already been empirically benchmarked extensively against a range of other state-of-the-art methods by [1] (including KIV, DFIV, DeepGMM and Dual Embedding). That study provides comprehensive empirical evidence of its practical performance against established benchmarks. Therefore, a full-scale empirical comparison was not the main objective of the present theoretical contribution. We will add a sentence to the experimental section of the main paper to explicitly direct readers to this relevant  work and its comprehensive benchmarks.\n    \n- **Weaknesses 2 & 4. Experimental generalizability to complex, high-dimensional data.** We agree that our experimental section could be expanded to better illustrate generalizability from synthetic data to complex, high-dimensional, real-world settings where neural networks are typically employed.\n\n    To address this, our paper now includes empirical results on the dSprites dataset. This is a well-established, high-dimensional dataset for NPIV, serving as a crucial example of a complex scenario where our method (which uses neural networks for feature learning) demonstrates strong applicability and scalability. We refer to our answer to W1 of **Reviewer m9NX** for further details. Furthermore, the spectral contrastive learning approach has previously achieved state-of-the-art results on challenging Demand Design datasets [1], further proving its efficacy on complex real-world data.\n\n    While our synthetic experiments use trigonometric functions to clearly isolate theoretical effects, we believe our new results on dSprites and the performance on Demand Design in prior work provide strong evidence for the generalizability of our findings to more complex settings.\n    \n- **Weakness 5. Missing conclusion section.** We will use the extra page given for the camera ready version to add a concluding discussion and perspective on future works. We acknowledge that key limitations of our submitted work included the absence of a strategy to empirically quantify spectral alignment and decay for identifying the \"good, bad, or ugly\" scenarios, and the lack of a precise theoretical characterization of the error term $\\varepsilon$ in Proposition 3. We have addressed the former through a newly developed empirical strategy for spectral alignment, as elaborated in our response to W1 for **Reviewer m9NX**. We also recognize the latter as an important and open problem for future research, which we briefly discussed on page 8 of our manuscript and further elaborate on in our response to W3 for **Reviewer m9NX**. We will explicitly discuss these limitations in the conclusion.\n\n[1] Haotian Sun, Antoine Moulin, Tongzheng Ren, Arthur Gretton, Bo Dai \"Spectral Representation for Causal Estimation with Hidden Confounders\", AISTATS, 2025."}]

Author rebuttal:

Meta-review: This work analyzes nonparametric IV regression with spectral features, deriving generalization bounds and showing that performance depends on spectral alignment and eigenvalue decay, leading to a “good–bad–ugly” taxonomy that is theoretically characterized and empirically validated.

All reviewers viewed this paper positively and recommend acceptance. In my own reading this work makes a good contribution worthy of publication at NeurIPS.
Decision: Accept (poster)