Title: Optimal Nuisance Function Tuning for Estimating a Doubly Robust Functional under Proportional Asymptotics
Venue: NeurIPS 2025
Accepted: yes
Keywords: Debiased Estimation

Sample Splitting

Nuisance Parameter Selection

Proportional Asymptotics

Semiparametric Inference
Abstract: In this paper, we explore the asymptotically optimal tuning parameter choice in ridge regression for estimating nuisance functions of a statistical functional that has recently gained prominence in conditional independence testing and causal inference. 
  Given a sample of size $n$, we study estimators of the Expected Conditional Covariance (ECC) between variables $Y$ and $A$ given a high-dimensional covariate $X \in \mathbb{R}^p$. Under linear regression models for $Y$ and $A$ on $X$ and the proportional asymptotic regime $p/n \to c \in (0, \infty)$, we evaluate three existing ECC estimators and two sample splitting strategies for estimating the required nuisance functions.
  Since no consistent estimator of the nuisance functions exists in the proportional asymptotic regime without imposing further structure on the problem, we first derive debiased versions of the ECC estimators that utilize the ridge regression nuisance function estimators. 
  We show that our bias correction strategy yields $\sqrt{n}$-consistent estimators of the ECC across different sample splitting strategies and estimator choices. We then derive the asymptotic variances of these debiased estimators to illustrate the nuanced interplay between the sample splitting strategy, estimator choice, and tuning parameters of the nuisance function estimators for optimally estimating the ECC.
  Our analysis reveals that prediction-optimal tuning parameters (i.e., those that optimally estimate the nuisance functions) may not lead to the lowest asymptotic variance of the ECC estimator --  thereby demonstrating the need to be careful in selecting tuning parameters based on the final goal of inference.
  Finally, we verify our theoretical results through extensive numerical experiments.


Peer reviews:
comments: SUMMARY:
The authors study parameter tuning for estimation of the ECC in linear models under the asymptotic regime $\frac{p}{n} \rightarrow c$ as $n \rightarrow \infty$ (including $p > n$). In this setting, the linear model weights are nuisance parameters for which naive plug-in, Newey--Robins or doubly robust estimators have worse-than-$\frac{1}{\sqrt{n}}$ asymptotic bias. The authors propose debiased versions of these estimators. The authors derive asymptotic variances of their estimators---including estimators utilizing with additional subsampling---as a function of the ridge-regression regularization parameters for the linear weights.

STRENGTHS AND WEAKNESSES:
The paper was a pleasure to read, with contributions well motivated and technical results clearly spelled out. The authors compare a wide range of approaches and make clear which results are novel and/or surprising. Applications to structure discovery and/or covariate selection are clear.

QUESTIONS:
Overall, I am very happy with this paper. No questions/suggestions significant enough to influence my score come to mind. I am keen to see other reviewers' comments and will follow up with any significant questions during the next stage.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- DML relies on estimating nuisance functions (eg. Propensity score, outcome regression) accurately. However, in high-dimensional settings, ridge, Lasso, and many other estimators become inconsistent. Besides, in the propotional regime, the invalidity of orthogonalization may be invalid, thus the product of two inconsistent nuisance estimators can introduce substantial bias. Furthermore, cross-fitting of DML becomes problematic: estimates from different folds can be correlated;\n\n- This paper studies how to optimally tune nuisance function estimators in high-dimensional settings of the ECC. They derive bias-corrected versions of three ECC estimators (plug-in, Newey-Robins, and doubly robust) that achieve $/sqrt{n}$-consistency, each applied under two distinct sample-splitting strategies. They also provide exact asymptotic variance formulas for these estimators, showing that the tuning parameters that are optimal for prediction do not necessarily minimize the variance of the final ECC estimator.\n\n- Simulations validate the theoretical findings and provide practical insights into selecting tuning parameters for reliable inference in high-dimensional settings.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The paper is technically strong. It addresses a challenging of inference for doubly robust functionals when nuisance functions cannot be consistently estimated due to high-dimensionality.\n- The authors provide rigorous bias correction techniques for three estimators (integral-based, Newey-Robins, and doubly robust), and prove $\\sqrt{n}$-consistency and asymptotic normality.\n- The asymptotic variance analysis is detailed, revealing a nuanced dependence of estimator performance on tuning parameters and sample splitting schemes.\n- The simulation studies illustrating theoretical findings.\n- The insights regarding inference-optimal and prediction-optimal can be useful for the causal and robust statistical inference applications. \n- The paper builds on concepts like sample splitting and debiasing from classical DML literature, its methodological contributions—especially the derivation of explicit variance formulas under ridge regularization—are novel.\n\n**Weakness:**\n\n- This paper only analyzes ridge regression for estimating nuisance functions $E[A|X]$, and $E[Y|X]$ under the linearity assumptions. While this helps derive the closed-form asymptotic variance but narrows the applicability. \n- The results do not generalize to model nonlinearity or non-Gaussian errors. It is unclear whether similar variance or bias correction methods extend to more realistic scenarios. \n- The asymptotic results depend on the assumption of Gaussian and sub-Gaussian. It would be great if the author could add a discussion of the robustness of the results to deviations from Gaussianity. Though it can be a future work for the theoretical study, adding empirical verification of performance under violations of this assumption might be helpful to explain the robustness of the results.\n- Any minimax lower bound for estimating ECC under proportional asymptotic? \n- How sensitive are confidence intervals to variance estimation? \n- In theory, it’s elegant to minimize asymptotic variance w.r.t. $\\lambda_1$, $\\lambda_2$, but these variance expressions are complicated, require knowledge of unknown population parameters (eg. Inner product between nuisance coefficients). However, there’s no practical method is proposed for running parameter sections, may not be practically estimable in finite samples.\n\nQUESTIONS:\nSee my questions in the above “Weakness” part and below:\n\n* Section 3.2 and Section 4, no concrete method is provided for how practitioners can implement this in real-data setting when ground truth is unavailable. Can you add empirical study or a practical algorithm or heuristic for tuning under real constrains?\n* Can you add a simple simulation example under eg non-Gaussian error, nonlinearity in E[Y|X] and E[A|X]? This would strengthen the empirical case with robustness to non-Gaussian and misspecified settings.\n* In the two-split simulations, the authors note that the variance “blow up” around a certain $\\lambda$, but this is unexplained? Any insights?\n\nLIMITATIONS:\nSee the limitations also in the above “Weakness” and “Questions” part.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper studies the optimal tuning parameter choice in ridge regression to estimate the Expected Conditional Covariance in the setting of linear data-generating model and the proportional asymptotic regime (where $p>n$ is allowed). It studies three estimators and two data-splitting methods, and proves the $\sqrt{n}-$consistency and derives the asymptotic variances of the estimators.

STRENGTHS AND WEAKNESSES:
The paper clearly explained the novelties compared with existing work and proves novel results for proportional asymptotic regime. It reveals that prediction-optimal tuning parameters may not lead to the lowest asymptotic variance of the ECC estimator. 

The proposed sample-splitting approach appears to sacrifice the sample efficiency of DML. In addition, there is a considerable body of work on high-dimensional linear regression, where RMT is widely applied. The paper can benefit from discussing connections to this literature.

QUESTIONS:
The proofs rely heavily on results from Random Matrix Theory. To what extent can the techniques be extended beyond the restrictive assumptions of a linear model and the ridge estimator?

The paper provides analytic expressions for the asymptotic variances. Do these expressions offer any intuitive understanding of the influence of the parameters involved? Can they help explain the variance blowup observed in the experiment?

LIMITATIONS:
The paper discusses the limitation on Gaussian assumption. The limitations of the proving technique applied to linear model and ridge estimator merits consideration.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the expected conditional covariance (ECC) under proportional asymptotics. The regressions are assumed to be linear with independent Gaussian noise. For some results the covariates are assumed to be independent and sub-Gaussian. For the main results they are assume to be independent and Gaussian.\nThe first result proves that bias corrected ridge estimators are n^{1/2} consistent. There are six variations of the estimator.\nThe second results gives the asymptotic variances of the six estimators.\n\nSTRENGTHS AND WEAKNESSES:\nQuality\n\nThe main weakness of the paper is that the results do not line up to the title and seem incomplete. First, the paper is about ECC not general functional estimation. Second, the assumptions in (2.1) are already very strong. Third, asymptotic normality is not proved, instead normality of all variables is assumed with independent covariates. Fourth, the way to operationalize (3.11) is not described. The variance seems to contain ECC=rho and also two-norm limits that are unknown, so it does not seem to given a feasible estimate of lamba1 and lambda2. Fifth, there is no analysis of a data-driven choice of lambda1 and lambda2. Sixth, the debiasing functions called g are not given in the main text so it is unclear what they depend on and how to implement them.\n\nIf I understand correctly, optimality is never established, even though it appears in the title. For a given ridge estimator on Gaussian data, the best ridge regularization is described as the minimizer of some objective. But this ridge estimator is not optimal, and so terminology  like “sharp asymptotic analyses” seems misleading. \n\nClarity\n\nThe main strength is that the paper summarizes the literature and ideas of the inconsistent regime well. The ideas are clearly presented.\n\nSignificance\n\nI am concerned about how significant the result is because it ultimately does not derive the asymptotic variance without assuming all the data are Gaussian. The result also does not give practical guidance on choosing lambda1 and lambda2, since they seem to depend on unknown limits and maybe a preliminary estimate of ECC, which is not discussed.\n\nOriginality\n\nThe originality depends on how much the derivations of the debiasing g functions departs from the literature on the inconsistent regime. I have not studied the appendix closely. If it is using new techniques, then there are originality points. If it is using the same techniques, the less so.\n\nQUESTIONS:\nWhat is the step by step method that a researcher should use to tune the nuisance functions in the inconsistency regime?\n\nHow does this step by step method perform in coverage simulations?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}