Title: Demystifying Spectral Feature Learning for Instrumental Variable Regression
Venue: NeurIPS 2025
Accepted: yes
Keywords: instrumental variable regression

NPIV

nonparametric statistics

feature learning

causal inference

operator learning
Abstract: We address the problem of causal effect estimation in the presence of hidden confounders, using nonparametric instrumental variable (IV) regression.
A leading strategy employs \emph{spectral features} - that is, learned features spanning the top eigensubspaces of the operator linking treatments to instruments.
We derive a generalization error bound for a two-stage least squares estimator based on  spectral features, and gain insights into the method's performance and failure modes. We show that  performance depends on two key factors, leading to a clear taxonomy of outcomes. In a \emph{good} scenario, the approach is optimal. This occurs with strong \emph{spectral alignment}, meaning the structural function is well-represented by the top eigenfunctions of the conditional operator, coupled with this operator's slow eigenvalue decay, indicating a strong instrument. Performance degrades in a \emph{bad} scenario:  spectral alignment remains strong, but rapid eigenvalue decay (indicating a weaker instrument) demands significantly more samples for effective feature learning. Finally, in the \emph{ugly} scenario, weak spectral alignment causes the method to fail, regardless of the eigenvalues' characteristics. Our synthetic experiments empirically validate this taxonomy. We further introduce a practical procedure to estimate these spectral properties from data, allowing practitioners to diagnose which regime a given problem falls into. We apply this method to the dSprites dataset, demonstrating its utility.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper learns spectral features for NPIV regression to estimate causal effects in the presence of hidden confounders. It derives a generalization error bound for a 2SLS estimator utilizing these features, identifying three performance regimes: good, bad, and ugly.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper studies nonparametric instrumental variable regression, a critical problem in causal effect estimation where hidden confounders exist.\n    \n2. The paper derives a generalization error bound for a two-stage least squares estimator based on spectral features.\n    \n3. While spectral features have been heuristically linked to NPIV in prior work , this paper provides a novel and rigorous theoretical characterization of their performance through the generalization error bound and the \"good-bad-ugly\" taxonomy.\n\n**Weakness:**\n\n1. The paper acknowledges that instrumental variable regression is often implemented as a two-stage procedure and mentions fixed and learned features. However, while it references other works that formulate IV regression as saddle-point optimization , it could provide a more detailed comparative analysis of its proposed method with specific, related prior works such as \"Kernel Instrumental Variable Regression\" (Singh et al., 2019 ), \"Dual Instrumental Variable Regression\" (Bennett et al., 2023 ), and \"Fast Instrument Learning with Faster Rates\" (Kim et al., 2022 ). Explicitly highlighting the similarities, key differences, and unique advantages or limitations compared to these methods would strengthen the paper's positioning.\n    \n2. The experimental section is simple, primarily relying on synthetic data where feature spaces are modeled using trigonometric functions. Its generalizability to highly complex, real-world, high-dimensional data (e.g., images, text) where neural networks are typically employed for feature learning is not fully explored or demonstrated.\n    \n3. The experimental section would be significantly strengthened by including comparisons with state-of-the-art methods such as Kernel IV, Dual IV, AGMM, DeepGMM, Fast IV, DFIV, and DeepIV.\n    \n4. The synthetic experiments use trigonometric functions for feature spaces. While useful for isolating effects, the generalizability of these findings to highly complex, high-dimensional real-world data where neural networks are typically employed might warrant further discussion or empirical demonstration.\n    \n5. The main body of the paper lacks a dedicated \"Conclusion\" section. Additionally, while the authors state in their checklist that they discuss the limitations of their work, this discussion is not apparent in the corresponding sections of the paper.\n\nQUESTIONS:\n1. Provide a dedicated subsection (e.g., in the related work or methodology) that offers a detailed comparative analysis with \"Kernel Instrumental Variable Regression\" (Singh et al., 2019), \"Dual Instrumental Variable Regression\" (Bennett et al., 2023), and \"Fast Instrument Learning with Faster Rates\" (Kim et al., 2022). While some of these works are cited, the current discussion is too brief and appears to misrepresent aspects of their original formulations.\n    \n2. Could the authors expand the experimental validation to include comparisons with a broader range of advanced instrumental variable methods (e.g., Kernel IV, Dual IV, AGMM, DeepGMM, Fast IV, DFIV, DeepIV) and, ideally, incorporate experiments on at least one high-dimensional, complex real-world dataset?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of causal effect estimation under hidden confounding using nonparametric instrumental variable regression. The authors propose a two-stage least squares estimator based on spectral features derived from the conditional expectation operator. A key contribution is the theoretical analysis that establishes a generalization error bound and highlights two critical factors affecting performance: the spectral alignment between the structural function and the operator’s eigenfunctions, and the decay rate of the operator’s singular values, which reflects instrument strength. Based on these insights, the paper introduces a good–bad–ugly taxonomy that characterizes when the method performs well or fails. Synthetic experiments are carefully designed to validate this taxonomy.\n\nSTRENGTHS AND WEAKNESSES:\nS1: The paper provides a rigorous theoretical analysis of spectral 2SLS for NPIV regression, including provable generalization bounds.\n\nS2: The experiments are carefully designed, I like the setting for good–bad–ugly taxonomy.\n\nS3: The integration of spectral learning theory with causal IV estimation is novel\n\nW1: While the theoretical results are solid, no real-world datasets are used to validate the findings. \n\nW2: The paper occasionally assumes familiarity with advanced spectral operator theory, which may be difficult for broader NeurIPS audiences (from my point of view).\n\nW3: The  practical applicability is somewhat limited by the fact that real data rarely comes with ground truth for spectral alignment or singular value decay, making it hard to operationalize the taxonomy without strong assumptions.\n\nQUESTIONS:\nAdd more comparison experiments: One notable limitation of the experimental section is the absence of comparisons with established baseline methods such as Kernel IV or DeepIV. These kernel-based and deep learning-based IV models are commonly used in causal inference under endogeneity and provide strong empirical baselines. Adding such comparisons would make the evaluation more convincing by showing where spectral feature learning performs better or worse in more realistic and challenging scenarios. Without these baselines, it is difficult to tell whether the proposed method brings practical improvements over existing approaches, especially when the spectral alignment or properties of the operator cannot be directly observed or controlled.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a theoretical analysis of the spectral feature learning for instrumental variable regression. Via a detailed analysis of the conditional expectation operator $\\mathcal{T}$ between the features and the instrument, the work complements previous empirical observations that the performance of two stage IV regression estimator depends on the spectral structure of the operator and its alignment with the ground-truth link function. The paper shows that spectral contrastive learning, introduced in previous work, approximates the SVD of $\\mathcal{T}$ and identifies three regimes which determine the effectiveness of such spectral-based methods. These regimes depend on the spectral alignment and singular value decay of the conditional expectation operator. Finally, the authors illustrate their theoretical results and the identified regimes with numerical simulations, where they explicitly control the decay and alignment parameters.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper presents novel theoretical contributions: it shows that the current state-of-the art NPIV estimator, based on a contrastive learning objective, approximates the spectral structure of the conditional expectation operator. Based on this, the authors propose a useful and intuitive decomposition of its generalization error and propose regimes based on its behaviour. Importantly, the results seem to suggest that the empirical success of the state-of-the art 2SLS estimator does not translate into universal theoretical guarantees, but rather only illustrates the \"good\" regime, which is a valuable insight for theorists and practitioners in the field.\n2. The paper is very well-written, self-contained and is very clear about its contributions and limitations. \n3. All assumptions and results are appropriately introduced and additionally discussed informally, providing intuition.\n\nWeaknesses:\n\n1. The main weakness of the paper seems to be the experimental evaluation. For instance, in lines 92-94, it is suggested that the empirical success of the decoupled spectral approach in [32] is due to the experiments in that work satisfying the \"good\" spectral scenario. However, this claim is not backed by evaluation of any \"proxies\" for alignment/decay on the datasets in [32] (dSprites and Demand Design) placing them in a \"good\" scenario. Additionally, the work would profit from real-data or semi-synthetic experiments illustrating the bad/ugly scenarios, since this would provide evidence that those scenarios do occur in realistic datasets. \n2. If I am not missing anything, the work does not seem to provide corresponding lower bounds for the generalization error. This would be useful, for instance, to prove optimality of spectral methods in the \"good\" regime. Since the upper bound in, e.g., Theorem 1, is not provably tight, the reader cannot conclude whether the three scenarios provide a fundamental limitation for Sieve 2SLS, information-theoretic limitations for any method or are artifacts of the upper bound. Thus, a tight analysis would greatly improve the interpretability of the scenarios. \n3. In Proposition 3, the $\\epsilon$ is defined implicitly, and the decay rate of the resulting $\\frac{\\epsilon}{\\sigma_d}$ is unclear. This slightly weakens the interpretability of the approximation error upper bound.\n\nQUESTIONS:\n1. How robust are the results to \"slight\" violations of Assumption 1? Partial identifiability frequently arises in applications where one uses instrumental variable methods, cf. , for instance, anchor regression [1]. What would the error decomposition roughly look like in such a scenario? \n2. In Proposition 3, the choice of the cutoff dimension $d$ seems to depend on the decay of singular values of $\\mathcal{T}$ which is a priori unknown. How would one choose $d$ empirically? \n\n[1] Rothenhäusler, Dominik, Nicolai Meinshausen, Peter Bühlmann, and Jonas Peters. \"Anchor regression: Heterogeneous data meet causality.\" Journal of the Royal Statistical Society Series B: Statistical Methodology 83, no. 2 (2021): 215-246.\n\nLIMITATIONS:\nThe authors have properly addressed the limitations of the study.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This paper studied the problem if using instrument variable for causal effect estimation when there is an existence of unobserved confounders. A potential unobserved confounder may affect both the treatment and effect, leading to incorrect causal effect estimation.\n- In particular, the paper focused on the nonlinear two-stage least squares method for using instrument variables. Under the nonlinear 2SLS framework, regressions are done between the projected feature space of the treatment and instrument variable. Then the outcome is regressed   further against the predicted features. \n- This paper investigated in the conditions where the nonlinear 2SLS approach perform well or badly, and identify three different regimes where the effectiveness of the framework is affected by the alignment between the structural function and the operator, and the rate of the singular value decay. Numerical simulations support the theoretical analysis.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- Overall the paper is well-organized and presented clearly. This work considered an interesting angle of nonlinear 2SLS: what are the good or bad features to choose - which significantly affects the effectiveness of the regressions.  The framework was illustrated with practical examples, and each step of the analysis is sketched out in sequel.\n- The result decomposes clearly the sources of errors in feature representation and approximations, demonstrate that the 2SLS with spectral features performs the best when h0 aligns with the top eigenspaces of T and its singular values decay slowly. This insight is useful for practitioners to perform the best practices when using 2SLS.\n\nWeakness:\n- although the paper studied nonlinear 2SLS and it's good / bad scenarios, the theoretical innovations are relatively weak. It is natural to see that due to the assumptions and design of the 2SLS framework, the best scenario occurs when h0 aligns with the top eigenspaces of T and its singular values decay slowly. This finding is more on the incremental side and lacks more innovations.\n- the numerical experiments does not use any real-world datasets but only simulated data. The experimental results can be strengthened with more complex datasets and comparisons to other existing approaches.\n\nQUESTIONS:\n- how does the misalignment between h0 and the top eigenspaces of T compare to a fast eigenvalue decay? Which source is more significant in terms of damaging the effectiveness of the 2SLS?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}