Title: Ridge Boosting is Both Robust and Efficient
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multiaccuracy

Semiparametric efficiency

Covariate shift
Abstract: Estimators in statistics and machine learning must typically trade off between efficiency, having low variance for a fixed target, and distributional robustness, such as \textit{multiaccuracy}, or having low bias over a range of possible targets.
In this paper, we consider a simple estimator, \emph{ridge boosting}: starting with any initial predictor, perform a single boosting step with (kernel) ridge regression. 
Surprisingly, we show that ridge boosting simultaneously achieves both efficiency and distributional robustness: for target distribution shifts that lie within an RKHS unit ball, this estimator maintains low bias across all such shifts and has variance at the semiparametric efficiency bound for each target.
In addition to bridging otherwise distinct research areas, this result has immediate practical value. Since ridge boosting uses only data from the source distribution, researchers can train a single model to obtain both robust and efficient estimates for multiple target estimands at the same time, eliminating the need to fit separate semiparametric efficient estimators for each target.
We assess this approach through simulations and an application estimating the age profile of retirement income.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of estimating the mean outcome $\\mathbb{E}_Q[Y],$ in a target population $Q$, denoted given labeled data $(X_i, Y_i)$ from a source distribution $P$. The authors adopt the \\emph{target-independent learning} setup introduced by Kim et al. (2022). In particular, Kim et al. (2022) establish an interesting connection between multi-accurate predictors (a concept in fair machine learning) and this distribution shift problem, showing that multi-accurate predictors are ``universally adaptable'' meaning that they achieve low bias for any target distribution $Q$ for which the auditing class $\\mathcal{C}$ that well approximates $dQ(x) / dP(x)$. \n\nThis paper expands upon the connections between multi-accurate predictors and causal inference. The main result establishes a numerical equivalence: when the auditing class $\\mathcal{C}$ is a Hilbert ball, then the multiaccurate predictor can be implemented as a boosted ridge regression $\\hat{\\psi}(X) = \\hat{\\gamma}(X) + X^\\top \\hat{\\beta}$ and consequently it is numerically equivalent to the automatic debiased machine learning estimator (e.g., Chernozhukov et al. 2022). Consequently, the multi-accurate predictor is not only universally adaptable, but it is also asymptotically normal for any target distribution and semi-parametrically efficient. One practical upshot of this connection is that we can construct valid confidence intervals for $\\mathbb{E}_Q[Y]$ based on this multi-accurate predictor using standard techniques as illustrated in their Monte Carlo simulations.\n\nSTRENGTHS AND WEAKNESSES:\nI found the paper to be very well-written, effectively presenting its main result and describing how it builds upon the connections between fair machine learning and causal inference that were initiated in Kim et al. (2022). The paper's main result is ex-post obvious once it is shown, but I certainly had not noticed this nice connection before despite having familiarity with both multi-accuracy and the auto-DML framework. In this sense, the paper provides a nice bridge between fair machine learning and causal inference/semi-parametric theory, which have two active areas at the intersection of statistics and machine learning.\n\nI have one main concern / point of clarification for the authors' main result. The central assumption in the paper is that the density ratio $dQ/dP$ can be well approximated by a linear function of the covariates $X$ (the first condition in Theorem 2). However, as stated in the main text, this assumption appears slightly stronger than those typically made in the Auto-DML literature (e.g., Chernozhukov et al. 2022). In particular, the Auto-DML framework typically assumes that the density ratio lies in the span (or is well approximated) by a (possibly growing) class of nonlinear transformations of the covariates $\\frac{dQ}{dP}(x) \\approx \\sum_{j=1}^K \\delta_j \\phi_j(x), \\quad \\phi_j \\in \\mathcal{F}$, where $\\mathcal{F}$ is a dictionary of basis functions (e.g., polynomials, splines, neural networks). I was thinking through the simple example where $X \\sim N(0, 1)$ on $P$ and $X \\sim N(1, 1)$ on $Q$ in which case the density ratio is $dQ / dP(x) = \\exp(x - 0.5)$, which is clearly not linear in $x$.\n\nOf course, the feature space $X$ in this paper could be re-interpreted in a broader view -- for example, we could think of $X$ as already the non-linear bases typically used in the Auto-DML framework. Indeed, I looked in the Appendix and the authors already noted this, and so I would strongly encourage them to move this into the main text as a remark. Of course, this resolution also highlights a limitation of this connection --- for this construction, the researcher must pre-commit to the feature map ex-ante, rather than trying to learn a sufficiently expressive representation.\n\nQUESTIONS:\nPlease see my earlier discussion of strengths/weaknesses of the paper.\n\nLIMITATIONS:\nPlease see my earlier discussion of strengths/weaknesses of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of estimating $\\mathbb{E}_Q[Y]$ from labeled samples $(X_i, Y_i) \\sim P$ under covariate shift, where $\\mathbb{E}_P[Y|X] = \\mathbb{E}_Q[Y|X]$. The goal is to construct an estimator that generalizes well to the target distribution.\n\nThe main idea of this work is to apply multiaccuracy: a predictor $\\hat{f}$ is said to be $(\\mathcal{C}, \\alpha)$-multiaccurate if\n$$\n\\sup_{c \\in \\mathcal{C}} \\left| \\mathbb{E}_P\\left[ (Y - \\hat{f}(X)) \\cdot c(X) \\right] \\right| \\leq \\alpha.\n$$\n\nIf $dQ/dP \\in \\mathcal{C}$, then this guarantees that the expected bias  $\\mathbb{E}_Q[Y - \\hat{f}(X)]$ is at most $\\alpha$. \n\nThe estimator studied in the paper is a ridge-boosted predictor of the form\n$$\n\\hat{\\psi}(X) = \\hat{\\gamma}(X) + X^\\top \\hat{\\beta},\n$$\nwhere $\\hat{\\gamma}$ is any initial predictor trained on $(X_p, Y_p)$ and $\\hat{\\beta}$ is obtained by ridge regression on the residuals $Y - \\hat{\\gamma}(X)$ using source data.\n\n**Theorem 1** shows that this ridge-boosted predictor has lower multiaccuracy error than the initial predictor. That is, the worst-case residual correlation with functions in $\\mathcal{C}$ is reduced by the boosting step. In the case where the $P$ has a diagonal covariance matrix, they can show that multiaccuracy tends to 0 as the regularization, $\\lambda$ goes to 0. \n\nIn their main result, **Theorem 2**, they  show that the average prediction of the ridge-boosted estimator over target samples is asymptotically normal and achieves the semiparametric efficiency bound for estimating $\\mathbb{E}_Q[Y]$, provided that $\\hat{\\gamma}$ is consistent and $dQ/dP$ is well-approximated in $\\mathcal{C}$.\n\nFinally, they experimentally validate their results over synthetic data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: this work studies an interesting connection between multiaccuracy and OOD generalization. It also provides a relatively thorough analysis of the proposed algorithm (showing that the estimator achieves the lower bound on variance) which is a nice guarantee. \n\nWeaknesses: \n\n1. The setting where $\\mathcal{C}$ only comprises of linear functions is rather limited. It doesn't feel very plausible to me that dQ/dP would follow this form in almost any situation. If the authors don't agree here, I would find some examples very helpful. At the very least, arguing why this is an important theoretical step would be appreciated.\n\n2. Theorem 1 feels a bit weak. Merely showing that the multiaccuracy is 'smaller' than the previous one doesn't feel like much of a result on its own. This is only helpful in the context of multiaccuracy being a goal (from some kind of fairness perspective). However, for the OOD setting (where the goal is to just achieve a good loss over Q) I don't really see how this is relevant.  \n\n3. Theorem 2 feels like a relatively straightforward consequence of the Corollary from Chernozhukov et. al. 2023. This wouldn't be an issue if Theorem 2 were characterizing a more relevant or practical problem, but given that Theorem 2 is restricted to a very limited theoretical setting (weakness 1) i do feel like there is a higher onus for providing new technical innovatinos. \n\n4. The presentation of Theorem 2 is sloppy. what does $\\min_{c \\in C} ||\\frac{dQ}{dP} - c|| \\to_p 0$ mean? What are we taking the limit with respect to? None of those quantities vary with the sample size. Furthermore, insepcting the proofs, it seems that this work effectively just plugs this in all over the place to achieve other limits going to 0. What this means is that simply stating $\\frac{dQ}{dP} \\in \\mathcal{C}$ would have equivalently sufficed. As far as I can tell, $\\frac{dQ}{dP} \\in \\mathcal{C}$ is the only reasonable interpretation of this statement, but I'm happy to be corrected. \n\n5. The general presentation can be significantly improved. There are several passages where algebra is done to provide intuition, but I think that this can be mostly omitted -- the manipulations are relatively standard and mostly take up space. Instead, this space could be used to argue more effectively for the relevance of this setting and to provide more proof details. The supplementary material of this paper is quite short suggesting that some of the proofs could be included in the main body.\n\n6. Additionallly the notation feels cumbersome and often poorly defined. For example, $\\Psi(Q)$ is never defined and appears for the first time in Theorem 2. I'm assuming that $\\Psi(Q)$ simply means $\\mathbb{E}_Q[Y|X]$.\n\nQUESTIONS:\nPlease see the list of weaknesses I provided. In general, I think this paper could do with significant revision. It also needs to justify its setting (where $\\mathcal{C}$ is just a Hilbert ball more).\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper bridges fairness-inspired multiaccuracy estimation with semiparametric efficiency from causal inference. It shows that when the auditing class is a Hilbert space, a simple ridge-boosting step applied to the residuals of a base estimator yields a multiaccurate predictor that is numerically equivalent to an augmented balancing weight (ABW) estimator. As a result, this estimator inherits both universal adaptability (robustness under unknown covariate shifts) and semiparametric efficiency (minimum asymptotic variance), even without access to target distribution data. The authors provide theoretical justification, draw connections to Riesz regression and TMLE, and validate their claims through simulations demonstrating improved confidence interval coverage under distribution shift.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper establishes a link between multiaccuracy and semiparametric efficiency, uniting fairness and causal inference methodologies.\n\nThe estimator does not require access to target distribution data, making it highly applicable in privacy-sensitive or resource-constrained settings.\n\nThe method is simple, which implements a practical, one-step ridge-boosting procedure that is easy to use and analyze.\n\nAt last, the paper provides empirical evidence that the once-boosted estimator achieves valid confidence interval coverage under covariate shifts.\n\nWeakness:\nOnly one synthetic simulation setting is presented, with no real-world benchmark or application to support claims of practical effectiveness.\n\nKey efficiency results rely on assumptions such as consistent outcome regression and the density ratio being well-approximated within the Hilbert space. In practice, these assumptions may not hold, and there is no empirical test or sensitivity analysis of robustness to assumption violations.\n\nWhile the connection to existing debiasing methods is insightful, the novelty lies more in interpretation than in introducing a new estimator. The paper might be perceived as a reinterpretation of known techniques rather than a fundamentally new method.\n\nQUESTIONS:\nHave you evaluated the proposed estimator on real datasets with covariate shift? If not, do you anticipate any challenges in doing so?\n\nYour simulation assumes a linear-Gaussian data-generating process. How do you expect performance to change in more complex, nonlinear settings?\n\nLIMITATIONS:\nThis is a theoretical contribution that adds conceptual clarity and unifies ideas across fairness, robustness, and causal inference. Its practicality could be enhanced by empirical validation on real datasets and extending beyond linear function classes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper connects two subareas of machine learning, multiaccuracy and semiparametric efficiency. It focuses on the case where the class of tests C (for multiaccuracy) is a Hilbert space. With some additional niceness assumptions, the ridge boosting estimator is multi-accurate and semiparametrically efficient. Together, these properties imply that the estimator has small bias and asymptotically efficient variance on unseen distributions Q (i.e., not used for training), for Q induced by the set of tests C. \n\nThis paper is in a line of very recent papers (last couple years) connecting multicalibration and causal inference. For example, Kim et al. 2022 showed that multiaccurate predictors can achieve low bias across certain (unseen) target populations, even when the distribution shift is unknown. However, multiaccuracy alone only guarantees bounds on the bias, not also the variance. In this paper, the Hilbert ball assumption is used to help bound the variance. Theorem 1 bounds the multi-accuracy error, and Theorem 2 bounds the bias and variance. \n\nThe paper also includes an experiment with synthetic data, comparing the coverage of single ridge outcome regression and ridge regression with auditing. On the synthetic data, with the same tuning parameter lambda, the boosted ridge has better coverage per sample size.\n\nSTRENGTHS AND WEAKNESSES:\nQuality:\nThe claims are primarily theoretical and supported by proofs. The proofs appear to be complete and correct. The paper makes some claims about the practicality of multiaccurate learning algorithms (in general), but the experiment that appears in this work is only on synthetic data. \n\nClarity:\nThe paper is nicely organized, and the background sections set the stage well for the technical content of the paper. Additional context in the exposition of the scope of the work's results would be appreciated, as discussed by my review and other reviewers. \n\nSignificance:\nThis paper attempts to connect the multicalibration and semiparametrics literatures. In the special case when the multiaccuracy class is the Hilbert ball, ridge regression leads to an estimator that implicitly performs Riesz regression. It shows that the notions from multiaccuracy and semiparametric efficiency can coincide, and that there may be other possible connections between these two subareas. This could lead to future projects by researchers from either areas. \n\nOriginality: \nMany comments about significance above also apply for originality. The paper draws heavily from both multicalibration and semiparametrics, as it should for a paper attempting to combine ideas from both areas. Relevant citations are provided to distinguish this work from previous work.\n\nQUESTIONS:\n---With respect to the assumptions 1-5 in the paper:\n-Why is assumption 5 (supplemental pg 2) necessary?\n-On pg 7, main body, assumption 3 and 4 are claimed to be standard in the semiparametric efficiency literature. What is an example of a problem for which some of the assumptions fail, and this argument cannot be applied? Are there \"natural\" extensions (e.g., slightly different problem settings) that are not captured by this approach (and perhaps worthy of future investigation)?\n\n\n---What other auditing function classes (besides Hilbert balls) are plausible candidates for connecting multiaccuracy (or multicalibration definitions more generally) to the semiparametric efficiency literature?\n\n---Minor typos:\nLine 228: derive -> deriving\nLine 319: are results -> our results\n---How would this algorithm perform on a \"real-world\" data set? Which applications could plausibly satisfy (or be close to satisfying) the 5 assumptions necessary for Theorem 2? To which applications is the estimator in this paper particularly well-suited? (e.g., If you had to pick one \"real-world\" application, what would it be?)\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors analyze properties of \"multiaccurate estimators\" in the covariate shift setting (labeled source data, unlabeled target data, the distribution of features $X$ changes). With the goal of estimating the expectation of Y (label) in the target distribution, and for a defined \"auditing function class\" (in their case a Hilbert ball), the authors analyze a previously derived multiaccurate model using a ridge boosting estimator (an additive term that corrects a model fit to the labeled source data using, e.g., ERM). The authors show the ridge boosted multiaccurate estimator is semiparametrically efficient for any member of the Hilbert ball (and equivalently, any corresponding covariate shift-correcting density ratio). This means that despite needing to estimate the source outcome regression function (\\gamma), and the auditing functions, the boosted multiaccurate estimator of $E_Q[Y]$ is asymptotically $\\sqrt{N}$-consistent and achieves the optimal variance bound. In a simple linear regression simulation, the authors show that the boosted ridge regression estimator achieves correct confidence interval coverage as opposed to undercoverage by standard ridge regression.\n\nSTRENGTHS AND WEAKNESSES:\nFor context, I reviewed this as someone who has done a lot of work on model training and evaluation under distribution shifts, and who is familiar with the causal inference literature. Notably, I did not have prior familiarity with multiaccuracy (though I had heard of multicalibration).\n\n### Clarity:\n- I think the biggest weakness of the paper is that it is written in a way where it heavily depends on the reader being intimately familiar with multiaccuracy literature. This may be the simplest presentation of the material, since it demonstrates a property of the ridge boosting estimator from Kim et al. (2022), but this means that the paper does not stand well on its own without having read the original paper first.\n\n### Originality/Significance:\n- Because the paper is written to heavily rely on familiarity with Kim et al. (2022), the potential significance and impact of this paper is much lower than it might otherwise be. To grossly oversimplify, it seems the main result is to show that, for a particular class of auditing functions, the estimator from Kim et al. (2022) achieves semiparametric efficiency \"for free.\" I will let a more theoretically knowledgeable reviewer speak to the significance of this result, but speaking from the perspective of a reader interested in theory + applications of distribution shift who is not as well-steeped in this literature, the paper does not speak to why this result is surprising, or what this result makes possible that could not be done before.\n- I think the authors could better motivate the choice of $E_Q[Y]$ as their estimand, since $E_Q[Y]$ is not usually the quantity of interest in machine learning. That said, I can think of examples where it is useful. For example, estimating performance on new data (under covariate shift) in settings where labels can't be easily obtained in a timely manner (such as post-deployment monitoring for long term risk predictors). Or estimating the average treatment effect under a covariate shift. \n- The form of multiaccuracy in Definition 1, and its relation to a \"collection of density ratios\" (Ln 144) makes it look quite similar to the form of Distributionally Robust Optimization. For readers familiar with DRO but not with multiaccuracy, it would be useful for the authors to include a discussion of the similarities and differences.\n\n### Quality:\n- I don't have much to comment on re: quality. I leave the more technical details to other reviewers.\n- The simulated experiment confirms the asymptotic normality result of Theorem 2 in finite samples for a very simple linear regression setup.\n\nQUESTIONS:\nQ1: If, as the authors state, one lets the auditing function class be a \"collection of density ratios\", how does multiaccuracy differ from Distributionally robust optimization (DRO) (e.g., [1, 2]) (Equations 3 and 4 in [1] might be most relevant.)? In DRO we might be interested in minimizing the maximum error across distributions in a bounded density-ratio ball. If this worst-case error is below $\\alpha$, and the density ratio ball is the same as the auditing function class C, is the solution to this DRO problem a multiaccurate estimator? If so, discussion and comparison to DRO approaches seems interesting and relevant. If they are related, how does your efficiency result compare to the statistical properties of DRO estimators under covariate shift?\n\n\n[1] Duchi, J., Hashimoto, T., & Namkoong, H. (2023). Distributionally robust losses for latent covariate mixtures. Operations Research, 71(2), 649-664.\n\n[2] Duchi, J. C., & Namkoong, H. (2021). Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 49(3), 1378-1406.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-24"}