Ridge Boosting is Both Robust and Efficient
Authors: David Bruns-Smith
Zhongming Xie
Avi Feller
Venue: NeurIPS 2025 spotlight ()
Keywords: Multiaccuracy
Semiparametric efficiency
Covariate shift
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Estimators in statistics and machine learning must typically trade off between efficiency, having low variance for a fixed target, and distributional robustness, such as \textit{multiaccuracy}, or having low bias over a range of possible targets.
In this paper, we consider a simple estimator, \emph{ridge boosting}: starting with any initial predictor, perform a single boosting step with (kernel) ridge regression. 
Surprisingly, we show that ridge boosting simultaneously achieves both efficiency and distributional robustness: for target distribution shifts that lie within an RKHS unit ball, this estimator maintains low bias across all such shifts and has variance at the semiparametric efficiency bound for each target.
In addition to bridging otherwise distinct research areas, this result has immediate practical value. Since ridge boosting uses only data from the source distribution, researchers can train a single model to obtain both robust and efficient estimates for multiple target estimands at the same time, eliminating the need to fit separate semiparametric efficient estimators for each target.
We assess this approach through simulations and an application estimating the age profile of retirement income.

Ridge Boosting is Both Robust and Efficient
David Bruns-Smith (Stanford University)
Zhongming Xie (UC Berkeley)
Avi Feller (UC Berkeley)
Abstract
Estimators in statistics and machine learning must typically trade off between efficiency, having low variance for a fixed target, and distributional robustmess, such multiaccurrcyy, or having low bias over range of possible targets. In this paper, we consider a simple estimator, ridge boosting: starting with any initial predictor, perform single boosting step with (kernel) ridge regression. Surprisingly, we show that ridge boosting simultaneously achieves both efficiency and distributional robustness: for target distribution shifts that lie within an RKHS unit ball, this estimator maintains low bias across all such shifts and has variance the semiparametrie efficiency bound for each target. In addition bridging otherwise distinct research areas, this result has immediate practical value. Since ridge boosting uses only data from the source distribution, researchers can train a single model to obtain both robust and efficient estimates for multiple target estimands the same time, eliminating the need to fit separate semiparametric efficient estimators for each target. We assess this approach through simulations and an application estimating the age profile of retirement income.
1 Introduction
Estimators in statistics and machine learning must typically trade off between efficiency and robust- ness. Efficient estimators, largely developed in semiparametric statistics and econometrics, focus on having the smallest asymptotic variance (the "efficient variance") among unbiased estimates for a single target estimand. Importantly, such estimators provide valid asymptotically Normal confidence intervals- critical in many empirical applications-anne and these intervals have the smallest possible width. By contrast, robust estimators, the focus of an active literature in Distributionally Robust Optimization (DRO) and other subfields, instead aim have good performance for many, possibly unspecified targets. For example, Kim et al. [2022] show that class of "multi-accurate" estimators- based on boosting an initial predictor-constraiie unknown mean in large class of covariate shift problems. In general, we expect that controlling worst-case bias across many estimands would come at the cost of increased variance.
worst-case bias for predicting the
Surprisingly, we show that a simple version of boosting, once-boosting with ridge regression, is simultaneously robust over a large of possible distribution shifts, while also achieving the efficient variance and smallest possible confidence interval for each estimation target separately. Constructing this ridge boosting predictor is simple: we start with any initial predictor, and then perform step of boosting using ridge regression populations whose density ratio with respect the source population is well-approximated by the RKHS, the resulting estimator is approximately unbiased and achieves the semiparametric efficiency bound. This very general (but completely general) class of distribution shifts: it includes any shift whose density ratio can be expressed as linear in a fixed transformation of the covariates, even infinite dimensional transformations. For example, this includes distribution shifts whose density ratio can be approximated as linear in the last-layer embedding of pre-trained large language model, but not the more general class that would involve fine-tuning the neural network.
Reproducing Kernel Hilbert Space (RKHS). For all target
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We similarly establish this result for broad class of linear estimands, generalizing the results from Kim al. [2022] beyond the missing mean to more complex targets like average derivatives and impulse responses. In this more general setting, we replace the density ratio with the more flexible Riesz representer corresponding to the estimand of interest [Chernozhukov et al., 2021]. Our key technical insight is that kernel ridge regression implicitly estimates the Riesz representer needed for semiparametric estimation: the ridge boosting estimator we analyze is in fact numerically equivalent to the "Automatic Debiased Machine Learning" estimator of Chernozhuko et al. [2021] and inherits its optimality properties. As a result, we can train a single predictor using only source distribution data. Deploying this predictor to estimate any target estimand (whose Riesz representer is in the RKHS) will then have both low bias under distribution shift and asymptotically optimal confidence witthot ever explicitly computing target-specific bias correction terms.
Our results have immediate practical implications. In settings where practitioners must estimate many related quantities under different covariate shifts-such as estimating health outcomes across multiple hospitals, or computing age profiles of economic variables- our approach eliminates the need to fit separate semiparametric efficient estimators for each target. As we show in simulations, this approach also yields valid confidence intervals for scalar estimands, an important requirement for many applications.
Paper organization. The paper proceeds as follows. Section 2 formalizes the problem setup, defining our estimation targets and contrasting robustness and efficiency. Section 3 introduces ridge boosting and proves it is both multiaccurate and semiparametrically efficient. Section 4 demonstrates performance through simulations and an empirical application. Section 5 concludes with limitations and future directions.
1.1 Related literature
Multiaccuracy and Multicalibration: Multi-calibration, introduced by Hébert-Johnson et al. [201 8), is a refinement of group calibration that requires a predictor to be simultancously calibrated across a rich collection of (potentially overlapping) subpopulations. For a prediction task, calibration requires that among the individuals which receive prediction f(z) = v, the true expectation is v. Variants of the original definition have been studied by number of works [Kim et al., 2019, 2022, Deng et al., 2023, Jung et al., 2021, Gopalan et al., 20221). Multiaccuracy [Kim et al., 2019] is a weaker version of multi-call calti-calibratton:: it weakens multi-calibration by removing conditioning on the predicted values. Both concepts strengthen classical group fairness by ensuring fine-grained predictive accuracy without sacrificing overall performance. The multiaccuracy criterion is a special case of DRO [Hastings et al., 2024]. The link between multicalibration and boosting is discussed extensively in Globus-Harri al. [2023]. Long et al. [2025] consider boosting over an RKHS to achieve multiaccuracy, but for classification. It would be interesting to see whether or not we could extend our result to their setting.
Semiparametric efficiency and doubly robustness: Semiparametric efficiency theory provides a rigorous foundation for the efficient estimation of target parameters in models that incorporate both parametric and nonparametrie components [Bickel al., 1993, Newey, 1994]. In the context of causal inference, doubly robust estimators [Robins et al., 1994, Kennedy, 2024] form a central class of methods that can attain semiparametric efficiency under correct specification of both. One motivation for these estimators comes from orthogonal (or Neyman-orthogonal) estimating equations [Chernozhukov et al., 2018, 2021, Foster and Syrgkanis, 2023], which reduce sensitivity errors in nuisance function estimation. A complementary line of work [Zubizarreta, 2015, Ben-Michael al., 2021, Athey et al., 2018, Hirshberg and Wager, 2021, Bruns-Smith et al., 2025a] focuses on balancing weights, which aim to reweight samples so that covariate distributions are matched across treatment groups. When balancing weights are combined with outcome regression, the resulting augmented estimators inherit both double robustness and semiparametrie efficiency. In parallel, targeted maximum likelihood estimation (TMLE) [Van Der Laan and Rubin, 2006, Van der Laan et al., 201 1] shows that, by incorporating a targeting step based on the efficient influence function of the parameter of interest and grounded in likelihood theory. TMLE achieves semiparametric efficiency. Cho et al. [2024] consider the TMLE update in an RKHS, and find closely related universal adaptability property. future work, it may be possible to unify their results with our boosting and multicalibration setting.

---

Connection between multicalibration and causal inference: There are several recent papers discussing the connection between multicalibration and causal inference. Wu al. [2024] show the connection between invariant risk minimization and multicalibration in the context of concept shift. Ye and Li [2024] explores multicalibration and universal adaptability in survival analysis. Kern et al. [2024] shows that the multi-accurate conditional average treatment effect estimate is robust to unknown covariate shifts. Van Der Laan al. [2023] also calibrate a baseline model to achieve semiparametric efficiency, albeit without using multicalibration.
2 Problem setup: Robustness vs efficiency
2.1
Notation
Let E X' denote covariates and Y E y S an outcome of interest. We consider a source distribution P over (X, We assume that have ny independent and identically distributed observations from P. denoted by {(X,, Y)) Miid. P. We let X, R denote the matrix of observed covariates and Yp R"v the corresponding vector of outcomes. Define (0(z) = Er[YY] zr], the optimal mean-squared error predictor of Y given X in P.
2.2 Defining our estimation target
We consider the goal of estimating scalar summary of the optimal predictor "No [see Chernozhukov al., 2018]. Examples include estimating missing mean under covariate shift, estimating an average treatment effect, and estimating an average derivative. This setup generalizes the estimands considered Kim et al. [2022].
Definition (Target Estimand). For any function f: + y, define:
Ouarget (f) Eplm(S, X)):
where m some saul- and X such that Osrrer linear in f. Our target estimand is:
0g = (1orger(70)).
Assumption (Continuity). We assume that Osarget is a continuous linear functionall That is, there
exists constant C > O such that:
O()) S CEp((())))
for all f with [X)(
< Xc.
We now make this concrete with some examples.
Example 1 (Missing Mean Under Covariate Shift): We begin with example that will be familiar to machine learning practitioners. Let Q be another distribution on (X,Y). We assume that we observe samples of X drawn from Q, but that is unobserved. Our goal is to estimate the missing mean Eq [Y]. For example, if we collect health outcomes (Y) in New York City (P), our goal might be use that data infer average health outcomes in another city like Raleigh (Q).
The issue is that New York and Raleigh are very different cities. But under the covariate shift assumption that Ep Y X Eq (Y XX), the missing mean can be written
dQ dP (x))(())
E2
Y Ro (X) E,
(70))
exactly as in Definition In this example, Assumption holds if and only if Q is absolutely continuous with respect to P and
dQ dP (x)²
Ep
<C.
The analogous example in causal inference is estimating the counterfactual potential outcome for treated units when targeting the Average Treatment Effect on the Treated: P. are the control units, Q the treated units, and Y (0) replaces Y. See Johansson et al. [2022] for discussion.

---

Example 2 (Average Derivative); We now consider an example common in applied economics. Let X, denote the first component of X. Then define the average derivative as:
[x)
Ourgal (70) = Ep
0x1
This Orrgett is also a linear functional. For example, if Y is household spending, X₁ is household income, and the remainder of X contains other household characteristics, then (70) measures the average spending response to change in income, known as the "Marginal Propensity to Consume." "
A central object in what follows will be the Riesz representer corresponding to the estimand Ourgeri Definition 2 (Riesz representer). Every continuous linear functional 0 has corresponding Riesz representer, a unique function au (x)(x) (z) such that:
f(f) = (X]aa(()f(C)
for all f such that Er]f(X)?] < co. We will write Clarger (x) denote the Riesz representer of o zarger
Example (Density Ratio): When (f) = Eq (X)],, then the Riesz representer is the density ratio, Orurges (a) dz)) This has known analytic form: ('urge( e(x)/(1 (e(])) where e(x:) is the propensity score or domain classifier for Q vs. P.
Note that our setup focuses scalar summaries of the optimal predictor, and does not, for example, consider finding a predictor that achieves small mean squared error uniformly over a target distribution Q. Recent work in Kern al. [2024] suggests that we could extend our results hold uniformly over X. We leave such an extension to future work.
2.3 Plug-in estimation and regularization bias
Before turning to robustness and efficiency, we introduce natural starting place, the plug-in in estimator. This first fits (X)) by predicting Y from X using samples from population and then computes:
npp Σ np
1
uurget
In the special case of the missing mean (Example 1), we fit predictor under P. but apply it to covariates drawn from Q. That is, say that we observe ny iid samples of from Q. The plug-in
estimate is then:" Bungga == 1 ny (Xy))
The core difficulty with the plug-in estimator is regularization bias. When fitting f via machine learning, standard methods regularize the predictor to generalize better out-of-sample. Unfortunately, might regularize away parts of the sample-space that are important for Oargett Say there is a particular combination of X that is very common in Q, but relatively rare P. Then a cross-validated predictor trained under P might regularize away the predictions optimal for prediction under P, this would lead to meaningful bias for Eo(Y), which in turn could lead to very poor estimate of the target estimand. Furthermore, bias means that the estimate will not be asymptotically normal, meaning that standard confidence intervals will not be valid often an important desideratum in applied work.
those values of X to reduce variance. While
2.4 Robustness: Constraining worst-case bias across many unknown targets
large literature in robust optimization and algorithmic fairness focuses on constructing estimators that modify y above such that the plug-in estimate 6(4) has small bias for range of target quantities [Kim et al., 2022]. For example, say observe data from a single source hospital P. but want to estimate Eo( [Y] across many different target hospitals Q1, Q2, Q k. where collecting unlabeled data and estimating density ratios ag for each target site would be costly possibly infeasible due to privacy concerns. Can we still estimate a y from that is robust to many unknown distribution shifts? Kim al. [2022] show that the answer is yes- although as we discuss below, we might be concerned about this procedure inflating the variance.
Wee can technically write this as special case of ml(; X.) over a single population by combining the two populations and introducing an indicator variable for membership in Q.

---

Consider target estimand Burger satisfying Definition 1 and Assumption 3, but now assume that we do not have access (arget ahead of time. Our goal to use the observations in to construct predictor y such that the resulting plug-in estimator for Ouaget is approximately unbiased. In other words, we want to control the worst-case bias over a large set of possible estimands. the fair machine learning literature, this condition is known as multiaccuracy:
Definition 3 (Multiaccuracy)). Given "auditing" " function class C and source population P. predictor f is (C, a) -mmuliicccurute if for every functtion ((X) E C,
sup EE ( ( f(X)) c(X)] S a.
cEC
Kim et al. [2019] show that initial predictor Miiit can be modified to be multiaccurate by running simple boosting procedure, including a version of our main proposal of boosting with ridge regression.
We now generalize the result in Kim et al. [2022], which considered the specific case of estimating a missing mean under covariate shift, to the more general class defined in Definition where we replace the density ratio with the Riesz representer. Even if we do not know Ourget in advance, if we can construct a set O such that we believe Ourget E O, then we can still obtain an unbiased estimator of Orarget if we can construct multiaccurate predictor "mm--
Proposition 1. Let θ be some set of functionals such that Definition I and Assumption 3 hold. Let A be the corresponding set of Riesz representers:
A = {ax: 30 E θ s.t. ( = (X)((X)((X)) with B[((x)* < co).
Then ima is (A,a))mmu mammiiacccccceatee
(0 ((7)) S a, e
Remark (Distributtonally-Roo) Robuut Optimization). Let contain O(f) Equ (X)) many dis- tributions Qu. where A contains the corresponding density ratios, /dP.P In this case, Definition 3 a special case of the more general literature on Distributionally- Robust Optimization [Hastings et al., 2024]. For target estimand, Puarge(f) [X)), if dQ uarge// dPP E then Burged (ma)) is approximately unbiased for Purrye(fio)
However, we might be concerned about the cost of robustness in terms of additional variance. Since we enforce small bias over a potentially large class of target estimands 9 E O, we would therefore expect larger variance for our specific target estimand uurget
2.5 Efficiency: Unbiased estimate with the smallest variance for a single, known target
In many applications, know our target functional Ourger in advance, such as if we observe samples of X from the distribution Q at training time. In this case, one popular strategy to "bias correct" the initial estimate duager (f). a problem studied extensively in the semiparametric statistics literature [Chernozhukov al., 2024]. Importantly, the resulting estimator has the smallest possible variance among all unbiased estimators [Chernozhukov et al., 2018).
Following the general setup in Chernozhukov et al. [2021], we focus on bias correction using the Riesz representer Ourget. Under minimal conditions, if & is a consistent estimator of [ and à is a consistent estimator of Ortrgee (X), then the estimator,
Thp
Deniiient == Ounger(() (4)
à (()) -
nip
bias correction
has the following three properties, asymptotically: (1) is unbiased, i.e., [Peeeteienll O (70): (2) it is normally distributed; and (3) it has the smallest variance of all regular asymptotically-Ii linear estimators. This third property is called semiparametric efficiency, and the corresponding variance is called the semiparametric efficiency bound, denoted Vğ for 0(70). Formally, we have:
Vn(eeeeeen Prarge((0)) + (0, Vongge ). and V. +p Vougee
where V the sample variance,
2
( -((x)) Oefficien
7pp
See e.g. Chernozhukov et al. [2023] for a set of minimal conditions under which this result holds.
5

---

3 Ridge boosting simultaneously achieves robustness and efficiency
Thus far, we have explored two classes of estimators: robust estimators that have low bias over many estimands, and bias-correcter estimators that are unbiased and efficient for specific target estimand. In this section, we demonstrate that it is possible to construct an estimator that is both robust and efficient. Specifically, when the set of target Riesz representers AA is the norm ball in an RKHS, we can construct multiaccurate predictor that has small worst-case bias over the corresponding θ while simultaneously achieving the semiparametric efficient variance for every target 0 E O. We refer to resulting procedure as once-boosting with ridge regression or, more simply, ridge boosting.
While our most general theoretical results only hold when A is norm-ball in an RKHS, in the Appendix we sketch out a version of our result for boosting with Random Forests, Whether there exists a more general result is a exciting topic for future work.
3.1
Ridge boosting
In this section, we introduce the main estimator analyze, once-boosting with ridge regression.
Setup. An RKHS H is a set of functions h : X' + R defined by an inner product. In the most general case, for all X' there exists ((x) E H such that for any h E H. h(2) (h, d(x))- One special case is the finite dimensional Hilbert space where ((x) is some feature map from X' - R
3 6(x) B E R4;; our results, however also hold for RKHS's. For any H. we define the norm || = (h, h).
and H. = {h(x)=
For some RKHS H. we consider the following function class:
A= {h H. |l B).
for some B > 0. We set B (which will be without loss of generality), that A forms unit ball. Recall that in our robustness setup, AA corresponds the set of Riesz representers for all 0 E the covariate shift setting, A is the set of density ratios. Restricting our attention to Riesz representers that belong to such an very general, but not fully general. Such set can include highly non-linear functions of x, but only functions that can be written in terms of the fixed basis 6(x)) For example, A could be a set of functions that are linear the last-layer embedding a pre-trained large language model (LLM). But A could not include all functions achievable by fine-tuning that pre-trained LLM.
Estimator. We now introduce once- boossboo with ridge regression. For notational simplicity, we present the algorithm in the case where H is finiteddimensional RKHS with <(x) E RR - the arguments are identical in the infinite-dimensionnn with rows o(z;) for each observation Assume that we have an initial estimator of Ep[Y]X], (X),, which could have been fit with some arbitrary machine learning algorithm. We then perform a ridge boosting step on the residuals Yp Tamil (Xp))
case. We will write Φ, E R" for the matrix
min BER
{Ilyp Mumil (Xp)
+ p + {113113
Call the solution Braoa and define
Thoose (Ix) = ((x))
Broust- Then define:
Ama (x) 'inii (1))s((:))
3.2 Ridge boosting is multiaccurate
Next we will show that fm is indeed multiaccurate. other words, 0(fma) is an approxiamtely unbiased estimate of ((r)), for all 0 E ④ We first define the notion of the multiaccuracy error.
Definition 4 (Multi-ace nulti-accuuacy error). For a given auditing function class C and source population P. the multiaccuracy error of an estimator f(X) and its sample analog are defined as:
MfEe(f) = ssup [ [c(X) -Y-------- (f-f(X))]I. MAFc(f)
sup (c(Xp)) (Yp -((x)))) CEC
<Ec
Then we have the following guarantees:

---

Theorem 1. For Fi defined in (2), we have:
À + of
MAE S max IS1Sd À
MAE
where of are the eigenvalues of 4 Dy. Under standard regularity conditions, with probability -
(fma()ima) 50
[6n+ V
1/n Ppp
for on such that 8, -> as 71 + op.
We provide proof and additional discussion in the Appendix. The first result shows that step of ridge boosting is guaranteed to decrease the sample multiaccuracy error. The second result shows can generalize of sample. The rate of convergence of 6, depends the dimensionality and smoothness of H. When H is fintte-dimensional with dimension 5n S vd/n.
We emphasize that Theorem 1 is not fundamentally new result. The multiaccuracy literature already proves generalization bounds on the multiaccuracy error for boosting estimators using ridge regression; see Kim et al. [2019]. However, specifics (e.g. linear boosting instead of exponential weighting) and so we provide Theorem I for completeness. Our proof uses standard techniques from the analysis of kernel ridge regression.
Remark 2.
boosting procedure differs slightly in its
3.3 Ridge boosting is semiparametrrially efficient
We showed above that boosting with ridge regression produces a predictor that is multiaccurate with respect A. We now show that this multiaccurate estimator is also semiparamettrcally efficient for all E @ Specifically, we will show that ridge boosting implicitly estimates the Riesz representer and performs semiparametric bias correction. In fact, the resulting estimator ((Smm) is numerically equivalent to special case of Automatic Debiased Machine Learning using kernel Riesz regression Chernozhukov al. [2021], Singh [2024].
3.3.1 Ridge regression implicitly estimates Riesz representers
The key fact that will lead to our main result is that ridge regression implicitly estimates Riesz representers. To see this, notice that for any continuous linear functional the Riesz representer (x(X) is the unique solution to the following loss minimization problem
0g
(Epla(x))} -2((a). [0)
argmin
See Chemozhukov al. [2021] for more discussion. One way to approximate (x(X) is to minimize (3) over an RKHS H. The sample version this optimization problem is:
1 T -, - + Allmll
min Ty
n2p
with minimizer ńix and corresponding Riesz representer estimate á (a) = ((x) fj. See Singh [2024] for an analysis of this estimator.
Ridge regression when used to estimate 0 can always be rewritten weighting estimator with weights & (X,), as we show in the following proposition.
Proposition Let 0 be any continuous linear functional, and define the Riesz representer estimate a from (4). Let Zp be any function of X, and Yp- Consider the ridge regression in H that predicts Zp given Xpi
min 1 BERS
2 + A113112.
Call the solution Bridge and the corresponding predictor Gridge (x) = Φ(x) Bridge- Then:
0 (frriddge)
- (Xp) )) 7.p-
n2
7

---

This is well-know result; see Kallus [2020], and see Bruns-Smith et al. [2025a] for an extensive discussion of the implications for semiparametric estimation.
Remark 3. Ridge regression can be wrillen as a linear smoother. The result above says that when we compute (. the smoother weights estimate the Riesz representer. Random forests can also be written as linear smoother, and Lin and Han [2022] show that the weights converge to the Riesz representer similar sense. We use this connection show that the same robustnessleffcienny properties apply to boosting with Random Forests in Appendix B.
3.3.2 Main result: Semiparametric efficiency for all
We now apply Section 3.3.1 to our estimator 9mm to establish our main result: ((fma) does not just have small worst-case bias over all A θ, it is semiparametrically efficient for each individual 0 E θ.
For any 0 O we have the following:
((1ma) - 0(finit) + 0(fhhnn)
1
-f(ima)) + (xp))) Mimi (Xpp),
(5)
np
where the second equality follows from applying Proposition for Zp Y -inin
Note that (6) has exactly the form of fetiiiient from Section 2.5. In fact, this estimation strategy in which we fit an arbitrary machine learning estimator fimit and use a (x(X) that minimizes (3) for bias correction - is a well-studied estimator from the semiparametric statistics literature [Athey et al., 2018, Hirshberg and Wager, 2021, Chernozhukov et al., 2021, Bruns-Smine et al., 2025a]. The particular form of (X) used here, which is obtained by minimizing (4) in an RKHS, is specifically considered in Hirshberg et al. [2019], Kallus [2020]. Hazleu [2020], Singh [2024]. Thus, once-boosting with ridge provides a multiaccurate predictor, but the resulting point estimate 0(ima)) is waanerricall-equvvllleee to well-studied semiparamerrcaally efficient estimators. We leverage this connection to establish our main theoretical result.
Theorem (Informal). Given standard regularity assumptions and some conditions on the quality of fimis then for all f E O the ridge boosting plug-in in estimator is asymptotically Normal and its variance achieves the asymptotic variance lower bound Vi::
Vn /m(((()m)))) -(10)) - NO, Vē),, and V - Vg.
where V is the sample variance,
By
(m(fmma, -0(
n2p
See the Appendix for formal Theorem statement and proof. This result establishes that for any estimand whose Riesz representer belongs to the RKHS ball A, the ridge boosting estimator 0 (m)) is not just robust, it is semiparametrically efficient.
This estimator is also computationally convenient for practitioners interested in efficient inference. We simply take the initial predictor and run one step of boosting with kernel ridge regression, which has many aaadil--vaillbll implementations. And since plug-in estimator using this new predictor is semiparametricall) efficient for any estimand in O, we do not have to (explicitly) fit individual Riesz representer estimates åø for each 0 which can be expensive when there are many Os of interest. We also do not require specialized code for minimizing the Riesz loss (3), which can be a barrier for practitioners with less familiarity with Riesz representers.
4 Experiments
4.1
Simulation study
We now demonstrate in simulation that we simultaneously achieve robust and efficient inference by deploying multiaccurate predictor different estimands in different environments. To
8

---

1,0
1.0
1.0
Coverage
Coveroge
Cowerage
0.6
0.4
0.4
0.4
0.2
Naive
0.2
0.2
Boosted
100
200
300
400
500
100
200
500
400
500
100
200
300
400
500
Sample Size
Sample Size
Sample Size
Figure 1: The empirical coverage comparisons between single kernel ridge outcome regression and kemel--idge-boostee boossed estimator across sample-sizes and the three environments μ E {0, 1, -1). The blue lines with triangle markers plot the coverage for the base ridge model, and the orange line with circular markers for the once-boosted ridge model. The dotted line is at 0.95.
demonstrate the generality of the framework, we consider estimating an average derivative with correlated covariates its own, already difficult task - under distribution shift. We will fit both ridge and once-boosted ridge models a training sample, and then compute the usual 95% asymptotic Normal confidence interval for the average derivative across three test distributions, assessing empirical coverage over simulation draws.
Simulation Setup: We consider three-dimensional correlated covariates: X2, N N(, 1), and
X3 4- oXx1 - X2) + -
where (-) is the sigmoid, and (0,, For the training distribution, P. μ O; for the test distributions, we vary μ.. The outcomes are generated as:
Y f(X)+n, f(x) = X1 (0.2 + sin(Xi) (X() 0.2 Xx),, 17 ~ NO, 29).
The estimand is the average derivative of s(X) with respect to X1 under Q. Egl0s(x)/xxx] We consider three different distributions for Q, with the same setup but with μ {-1,0, 13. The dependence of X3 on both X1 and X2 makes the average derivative more challenging to estimate.
estimators. (1) Naive Kernel Ridge: standard kernel ridge outcome regression trained on the source data and plugged in for each target estimand. (2) Boosted Kernel Ridge: one-step kernel ridge boosting procedure applied the residuals of the initial kernel ridge regression. Both models are trained solely on the training (source) data and evaluated on each of the three test (target) distributions.
Methods: We compare
Monte Carlo Simulation: For each simulation, we draw a training sample of and Y from the source distribution, and fit the kernel ridge and boosted kernel ridge model. We then draw sample of from each the three test distributions, and estimate the average derivative with respect to X1 on that test distribution by symmetric differencing, along with the usual 95% asymptotic normal confidence interval. The whole process is repeated for sample sizes ranging from 50 to 500 and with 1,000 Monte Carlo replications. We report the empirical coverage of the confidence intervals for both methods across sample sizes and replications. The full simulation study is run on a four-core laptop.
The results are shown in Figure 1. Across all three test distributions, the naive confidence intervals using the base ridge model under cover. By contrast, confidence intervals from once- boosted ridge regression achieve good empirical coverage even with moderate sample size (n 300). The same pattern holds across all three randomly generated covariate shifts, showing the boosted ridge regression the same time. This reveals a new practical benefit of the multiaccurate estimator this setting previously unexplored in the multicalibration literature: we achieve valid uncertainty quantification under distribution shift.
Results:
achieve robustness toward covariate shifts and statistical efficiency at
4.2 Empirical application to retirement income
We now consider an empirical economics application: estimating the age profile of income throughout retirement. For an individual i, Y; be total income (including retirement income), let A, be age, and let X; be other covariates like education, race, and marital status. Define no (A, = E|Y [A, X].

---

Naive
Confidence Interval Width
1900
Boosted
1800
Average
54000
52000
50000
1400
65
75
85
70
Ages
Ages
Figure 2: The left panel shows point estimates for the age profile of income using naive kernel ridge regression plug-in estimator and our ridge boosting estimator. The dotted lines are the corresponding 95% confidence intervals and the width of the intervals are plotted in the right panel.
Our estimand is the age profile of income (for ages ......, 89), which is defined as the following counterfactual means:
0es == El/o(65,
0es 15/1 X)1,
0sg : 1/r (89, X)].
That is, for each 6.. we want the average value of (x,AAX)) but where we have counterfactually replaced the age of every individual with the fixed value a. This is key input into structural models of the macroeconomy- see, for example, Gourinchas and Parker [2002], Kaplan and Violante [2014]. Because the distribution the covariates X varies with A, this can induce very significant distribution shift, especially late in retirement. We emphasize that this is highly simplified example inspired by Bruns-Smith et al. [2025b], although applying our methodology to their setting is promising direction for future work.
State of the modeling here would construct separate semiparametric efficient estimators for each point in the age profile, necessarily requiring separate debiasing term at every age. In this application, we instead use single multiaccurate predictor estimate the 25 different target estimands one for each year of retirement age- demonstrating the utility of our procedure in practice. We estimate the age profile of retirement using data from 2018 American Community Survey as processed by the FolkTables package [Ding et al., 2021]. As in Section 4.1 we fit single kernel ridge and boosted kernel ridge model, and then compute plug-in estimates of 9₂ using these two models. The results are displayed in Figure 2. Whereas the naive estimate (without boosting) features steep decline of $1 lk from ages 65-89, our boosted estimates are substantially flatter better matching the theoretical model for pension and social security income from Kaplan and Violante [2014]. Furthermore, while the naive confidence intervals shrink for the highest ages, our boosted confidence intervals actually grow slightly, suggesting that the naive model may undercover for the oldest part of the age profile.
5 Conclusion
In this manuscript, we investigate the connection between multiaccuracy and semiparametric effi- ciency. Specifically, show that boosting initial predictor with kernel ridge produces estimator that is not only multiaccurate over estimands in an RKHS norm ball but also semiparametricall efficient for each target separately. This result can be understood through the lens of Riesz regression: ridge boosting implicitly performs Riesz regression, thereby yielding an augmented balancing weight estimator that attains the semiparametric efficiency bound. We demonstrate one practical benefit: valid confidence intervals across distributions under covariate-shine not seen at training time. How- ever, our results are limited to shifts described by an RKHS. And making this proposal fully practical requires additional investigation of appropriate hyperparameter tuning. We hope this initial work leads to further exchange between the robustness and semiparametrics literatures.
10

---

References
S. Athey, G. W. Imbens, and Wager. Approximate residual balancing: debiased inference of average treatment effects in high dimensions. Journal of the Royal Statistical Society Series B: Statistical Methodology, 80(4):597-623, 2018.
E. Ben-Michael, A. Feller, D. A. Hirshberg, and R. Zubizarreta. The balancing act in causal inference. arXiv preprini 2rXiv:2110. 14831, 2021.
P. J. Bickel, C. A. Klaassen, P. J. Bickel, Y. Ritov, J. Klaassen, J. A. Wellner, and Y. Ritov. Efficient and adaptive estimation for semiparametric models, volume Johns Hopkins University Press Baltimore, 1993.
D. Bruns-Smiin Smith, O. Dukes, A. Feller, and E. L. Ogburn. Augmented balancing weights as linear regression. Journal of the Royal Statistical Society Series Statistical Methodology, page qkaf019, 2025a.
D. Bruns-Smith E. Nakamura, and J. Steinsson. Disentangling age, time, and cohort effects in income inequality: A proxy machine learning approach. Working Paper 34380. National Bureau of Economic Research, October 2025b. URL https: httt://wwww...ww.///./p.//...... .bber..... org happppppe//3380000
V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duñlo, C. Hansen, W. Newey, and J. Robins. Double/debiased machine learning for treatment and structural parameters, 2018.
V. Chemozhukov, W. K. Newey, V. Quintas-Martinez, and V. Syrgkanis. Automatic debiased machine learning via riesz regression. arXiv preprint arXiv:2104. 1443:22704.1477, 2021.
V. Chernozhukov, W. K. Newey, and R. Singh. A simple and general debiased machine learning theorem with finite-sample guarantees. Biometrika, 10((1)257-2644 2023.
V. Cheroozhukov, C. Hansen, N. Kallus, M. Spindler, and V. Syrgkanis. Applied causal inference powered by ml and ai. arXiv preprint arXiv:2403.024677 2024.
B. Cho, Y. Mukhin, K. Gan, and I. Malenica. Kernel debiased plug-in estimation: Simultancous, automated debiasing without influence functions for many target parameters. Proceedings of machine learning research, 235:8534, 2024.
Z. Deng, C. Dwork, and L. Zhang. Happymap: A generalized multi-calibration method. arXiv preprint arXiv:2303.0433 04379, 2023.
F. Ding, M. Hardt, J. Miller, and L. Schmidt. Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems, 34:6478-6490, 2021.
S. Fischer and I. Steinwart. Sobolev norm learning rates for regularized least-squares algorithms. Journal of Machine Learning Research, 21(205):1-38, 2020.
D. J. Foster and V. Syrgkanis. Orthogonal statistical learning. The Annals of Statistics, 51(3):879-908, 2023.
I. Globus-Harris, D. Harrison, M. Kearns, A. Roth, and J. Sorrell. Multicalibration boosting for regression. In International Conference on Machine Learning, pages 11459-11492. PMLR, 2023.
P. Gopalan, M. P. Kim, M. A. Singhal, and S. Zhao. Low-degree multicalibration. In Conference on Learning Theory, pages 3193-3234. PMLR, 2022.
P.-.. Gourinchas and A. Parker. Consumption the life eycle. Econometrica, 70(1):47-89, 2002.
Hastings, C. Jung, C. Peale, and V. Syrgkanis. Taking a moment for distributional robustness. arXiv preprint 4rXhv:2405.05461, 2024.
C. Hazlett. Kernel balancing. Statistica Sinica, 30(3):1155-1189) 2020.
U. Hébert-Johnson, M. Kim, O. Reingold, and G. Rothblum. Multicalibration: Calibration for the (compiaationall--ddnnfffaaa masses. In International Conference on Machine Learning, pages 1948. PMLR, 2018.

---

D. A. Hirshberg and S. Wager. Augmented minimax linear estimation. The Annals of Statistics, 49 10:3206-3227, 2021.
D. A. Hirshberg, A. Maleki, and J. R. Zubizarreta. Minimax linear estimation of the retargeted mean. arXiv preprint arXiv: 19XX::9901110990 2019.
D. Johansson, U. Shalit, N. Kallus, and D. Sontag. Generalization bounds and representation learning for estimation of potential outcomes and causal effects. Journal of Machine Learning Research, 23(16)): 2022.
C. Jung, Lee, M. Pai, A. Roth, and R. Vohra. Moment multicalibration for uncertainty estimation. In Conference Learning Theory, pages 2634-2678. PMLR, 2021.
N. Kallus. Generalized optimal matching methods for causal inference. Journal of Machine Learning Research, 21(62)11-54 2020.
G. Kaplan and G. L. Violante. A model of the consumption response to fiscal stimulus payments. Econometrica, 199)):1199-1239 2014.
E. H. Kennedy. Semiparametric doubly robust targeted double machine learning: a review. Handbook of Statistical Methods for Precision Medicine, pages 207-236, 2024.
C. Kern, M. Kim, and Zhou. Multi-cate: Multi-aceurate conditional average treatment effect estimation robust to unknown covariate shifts. arXiv preprint arXiv:2405.18206. 2024.
M. P. Kim, A. Ghorbani, and Zou. Multiaccuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAT/ACM Conference on Al, Ethics, and Society, pages 247-254, 2019.
M. P. Kim, C. Kern, Goldwasser, F. Kreuter, and O. Reingold. Universal adaptability: Target- independent inference that competes with propensity scoring. Proceedings of the National Academy of Sciences, ( 2022.
Z. Lin and F. Han. On regression-adjusted imputation estimators of the average treatment effect, arXiv preprint arXiv:2212.05424. 2022.
C. X. Long, W. Alghamdi, A. Glynn, Y. Wu, and F. P. Calmon. Kernel multiaccuracy. 6th Symposium Foundations of Responsible Computing (FORC 2025), pages 7-1. Schloss Dagstuhl- Leibniz-Zentrum für Informatik, 2025.
W. K. Newey. The asymptotic variance of semiparametric estimators. Econometrica: Journal of the Econometric Society, pages 1349-1382, 1994.
N. Pimentel, A. Schuler, and M. van der Laan. Score-preserving targeted maximum likelihood estimation. arXiv preprint arXiv:2502.00200 2025.
J. M. Robins, A. Rotnitzky, and L. P. Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association, (94427):846-866 1994.
A. Schuler, A. Hagemeister, and M. van der Laan. Highly adaptive ridge. arXiv preprint arXiv:2410.026800 2024.
R. Singh. Kernel ridge riesz representers: Generalization, mis-specification, and the counterfactual effeectiv dimension. arXiv preprint arXiv:2102.11076:: 2024.
L. Van Der Laan, F. Ulloa-Pérez, M. Carone, and A. Luedtke. Causal isotonic calibration for heterogeneous treatment effects. In International Conference on Machine Learning, pages 34831- 34854. PMLR, 2023.
M. J. Van Der Laan and D. Rubin. Targeted maximum likelihood learning. The international journal of biostatistics, 2(1), 2006.
M. Van der Laan, S. Rose, et al. Targeted learning: causal inference for observational and experimental data volume 4. Springer, 2011.
12

---

J. Wainwright. High high-dimmessionl statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019.
Wu, J. Liu, Cui, and S. Wu. Bridging multicalibration and out-of-distribution generalization beyond covariate shift. arXiv preprint arXiv:2406.00661. 2024.
H. Ye and H. Li. Multicalibration for censored survival data: Towards universal adaptability in predictive modeling. arXiv preprint arXiv:240S. 15948, 2024.
R. Zubizarreta. Stable weights that balance covariates for estimation with incomplete outcome data. Journal of the American Statistical Association, 1106111::10-922,, 2015.
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have reflected the contributions and scope in both the abstract and intro- duction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
We provided assumptions in the manuscript and proof in the supplemental
material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide simulations to justify the efficiency of the proposed method in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code is provided in the Supplemental Material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
public/guides/ www.ee/ooeeeeiieiiiioooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss. wwtpcccppppppeeeeeeeccecccccceee tww///uiisee//ee /ew.eee/Coeeuummmiiii wwwsssssiiiiyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provide simulation details in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We reported the empirical coverage of the confidence intervals for different methods in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide these information Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurip::: wwwpeeurrpp.cppuulleeeepeeee hetpplic/Ethissuiiieetee
Answer: [Yes]
Justification: Our research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed this in the introduction.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode..ooo..... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ //tprr//eeu../eeeee Wwteurips.cc/oooffeee /0LLLLLM for what should or should be described.
20

---

A Proofs of Theoretical Results
A.1 Proof of Proposition
Proposition. Let è be some set of functionals such that Definition I and Assumption 3 hold. Let A be the corresponding set of Riesz representers:
A: fa: 30 E O s.t. 0()) (EI(())((X)) with B[J(X)²} < oof. Then if mma (A, mumiiacccce
- 0(70)) 10
Proof. For all 0 E.
10(70) (fmm))
|Erlao(X)o (X))o Er (a (mm) 7yma mmml (X))]
[X)a((X)(( (xm](
s sup [B(a(X)(Y -mmm S a.
aEA
A.2 Proof of Theorem
Assumption 2. We require the following boundedness conditions:
1. Assume that Ey [l B1.
2. Assume that Y, Timi X), and (xX) are bounded P-almost-surely
Theorem. For 9ma defined in (2), have:
X
MAE <
max 18SS o
MAE
where o3 are the eigenvalues of T pp Under standard regularity conditions, with probability
1/1
MAE 0
np
for 6, such that 8, - 0 as + co.
Proof. We begin by proving the first inequality - that the sample multiaccuracy of 9m always smaller than that of fimi- We have:
MAEA(Tim = sup aEAA
- (xxx)) (imin( TD,
Here A is Hilbert ball with norm 1. Since Hilbert spaces are self-dual, we can use the definition of the dual norm:
1 (X,) -rini (Xiin((p)))))))) 0p.33) - - Sinin (Xp) ..,.3111.
MAEA(fiml sup ae.A n
Similarly, the multi multi-accuracy error of Pinii is
A - -1 (V, Aiiil (X))) n
Using the closed-form for ridge regression, we have:
Broost = (L + Nla
1
- TT (Y, - Miii (X,)).
n p
n
21

---

Therefore:
MAEc(fma) = -1 1102 (Yp (Xp) - 4 2p 4/ 2p + Ald)
2/ -22 (Xp - -mnin (X)))())))1
1
II
T db
4 T D + (ld) (Yp Mimi ( (xp))
n
n2
Define
1 tT pp + Nla 11
-1
Pa
T n
Consider the singular value decomposition (SVD) of 1 Φ = -./ / we have
Pa VEp(Ep + Ald) -1 -yp
The cigenvalues of Px are thus
of + j= j=
where of are the eigenvalues of Φ 4 The eigenvalues of In Px are
j j= j=l,...,d
of + X
Thus we have
X isidd X of 10 +(xx))11
A S
max 15f5d À oj
Now the second inequality, that the multiaceuracy of Tm generalizes. We apply the same dual norm argument to the population multiaccuracy error:
MAF
aEA sup [E](a(X) (Y 7m (X))II
[[E) (Y Amm ( (X)
I EBiEP [(Y 9mm (mmm(() (
VB Simin (X)) - Nons (X))²
where the expression in the second line is well-defined when Assumption 4 holds, the third line uses Jensen's 's inequality for norms plus Cauchy-Sehie Schwarz and Assumption 4, and the last line uses the law of iterated expectations.
So ultimately, the expression depends on the mean squared error of kernel ridge regression for predicting Y finin (X) using X. Here we can use any off-the-shei bound to get our main result.
For example, when o(X) E IRd, as we assume for simplicity in the main text, and if is sufficiently small, apply the standard critical radius argument in Wainwright [2019] to get that with probability at least 1 - 7).
(V
1/n
EpI((Y - -rin - Tooos (X)) S
More generally, if H. an infinite dimensional Hilbert space, then given an effective dimension and certain standard smoothness assumptions, then by following Fischer and Steinwart [2020]. Singh [2024] we can show that taking = (log(n) (nn///n)/nnn (log(m)/n)//(001)) (b--1) the squared generalization error of kernel ridge regression converges rate:
o (log(n)/ni)//(tt))
22

---

A.3
Proof of Proposition 2 (Numerical equivalence of ridge regression and Riesz regression)
Ridge Regression:
Given the auditing function class M. ridge regression solves
min BERd
p + 111
with closed-form solution
B₂ = np
T7 + (lo) -1
D6 Zp.
Rp
The ridge boosting estimator is then given by
(B,)) 0(
T
Ald)
T 7pp
np
np
Riesz regression: The Riesz regression solves
min 11 "T " 20( TT + + fEF
np
The resulting Riesz representer estimator is
( = 1 = 4
T + -000)
np
The corresponding bias correction term is thus given by
1
(T) Zp = 0( nhp
of + A1)-1.
$ Zp. np
ny
This shows the numerical equivalence:
ax(xx)) Up = 04
nhp
Therefore, ridge regression estimator is numerically equivalent to the Riesz regression-basee aug- mented estimator.
A.4
Proof of Theorem
A.4.1 Notation
We first introduce notation that will be used in the proofs.
We denote the conditional excess risks of the nuisance estimators Ý and & given the observed covariates as follows:
R(9) = E8 (f(x) - rx [xp], R(a) = E [((()) (x))² |xp]
We denote the efficient influence function for ((70) as (X)) and its high-order moments
of Ex(()(X))),
x² = =oloo(()()))) 5² = Eq (X))))))
A.4.2 Assumptions
Assumption 3 (Continuity). We assume that Ourrget is a continuous linear functional. That is, there exists a constant C. > O such that:
8(f)² CEP(((()))
for all ff with (X)())) xc.
Assumption 4. We require the following boundedness conditions:
23

---

1. Assume that Ep (p < B1.
2. Assume that Y, (imi (X), and (xX) are bounded P- P-almost-surely.
Assumption 5. Ep [x)( < ao and ( = Ep((Y - (x X is bounded.
Assumption 6. The initial estimator 7 for "/ is consistent in the sense that R(S) +pp
Assumption 7. The moments of the efficient inftuence function (X) satisfy
(6 + 112 + 0.
A.4.3 Revisiting the theorems
Theorem 3. Assume that we have a source population P and Assumption 3-7 hold. Consider RKHS satisfying ag H and VTiR' ->y 0.
Then the ridge boosting estimator with X n 1/2 is asymptotically normal and its variance achieves the asymptotic variance lower bound V
(re(0((mma)) (()a)) + (0, and V(A) ->p Vg.
((A)
where
no ((A)
nQQ Σ
(0(f)n))
( 4 + - (Y, 9mmi (Xp)) 0(7mma)
A.4.4 Proof of Theorem 3
In Proposition 2, we established the numerical equivalence between ridge regression and Riesz regression. For any 0 E O, we have the following:
0(fmma) 0(Finit) + ((fhnoa)
= (((ini)) + क. 3x
1
0(%mma) +
a (X,)) TY, Aimi (Xp)),
np
where the last equality follows from applying Proposition 2 for Zp = Y, Tinir (Xp))
With this result, we then can derive the asymptotic normality using the Riesz regression formulation.
The asymptotic normality of the Riesz regression- bassdd augmented estimator follows from Corollary 5.1 in [Chemozhukov et al., 2023] (Lemmaa 1 in the Appendix).
The constants â and ā' are assumed to be bounded and are independent of sample size With sufficiently large sample size, the last three conditions in Theorem reduce to
{R(A)) (1()), (R(ax)) 1/2 (1()),
((ā()) 1/2 (1).
= (1) is guaranteed from the conditions in Theorem
Now it remains to study E (xx(X)- ax())
xpp Theorem 2 in [Singh, 2024] implies that
when À = n -
lim lim sus Px,~ EE -xx [xp > >n.n" --] 0. +-000 no0
Now show
[R(a))) = (1) and (nR()) (1).
Therefore, we can directly show the asymptotic normality of the ridge boosting estimator.
vn((((im)) 6) + 1
((A)
Finally, the consistency of the variance estimator follows from the law of large numbers.
24

---

A.4.5
Additional Lemma
Lemma 1 (Corollary 5.1 in [Chernozhukov et al., 2023]), Suppose the following regularity and learning rate conditions hold as - oo:
The conditional variance is uniformly bounded: B[YY (o)))) W - o².
The nuisance parameter and estimator are uniformly bounded:|laolles Sā,
llállx S a',
3 + + The moments of the efficient influence function controlled: ((==)²
n 1/ 0,
The excess risks decay appropriately:
(nl(()) 1/2
- -20)) ā (-(R()))/2=00)
(1)
00
Then the estimator 0 is consistent and asymptoticallyy normal.
= 8 + o,(),
0 1n Gb) ~ (),
B Random Forest Boosting
Our main "Universal Efficiency" argument for Ridge Boosting hinges on the fact that: For any O we have the following:
1 (ma) (fiini) + ((fbooot)
1 np
(((rmm))) +
(xp) Mimi (Xp)),
which has a form debiased estimator. Following the semiparametric efficiency theory results from Chernozhukov et al. [2023], long Aimit converges to "o as co, and à (Xp) - ap. and their product rate sufficiently fast, the resulting estimator efficient. Furthermore,
The implied weights that come from ridge boosting, ag (Xp) are equivalent to ridge minimizer of the Riesz loss. That allows us to use the fast rates proven in Singh [2024] for kernel Riesz representer estimates.
What about other linear smoothers that have the form: (((inmmooth) 1 (Xp)) Zp? Ridge regression is one example with the important property that the smoother weights implicitly estimate the Riesz representer, but there other examples?
"y
Remarkably, Lin and Han [2022] show that Random Forests (which can be written as a linear smoother) also have implied weights that estimate the Riesz representer. Furthermore, the rate is fast enough to secure semiparametric efficiency. Therefore, boosting with Random Forests will have exactly the same efficiency guarantees that we show for ridge boosting. This is important because Random Forests are much more flexible than ridge regression in a fixed basis indeed, one can think of Random Forests as kernel ridge regression but that learns adaptive kernel. That would mean that the robustness/universal efficiency properties would hold to a wider class estimands than just those expressible by single RKHS.
However, the proof in Lin and Han [2022] is specific ATE covariate shift type estimands. It would be interesting to see if this could be generalized in future work to apply to generic Riesz representers.
C Connection between multiaccuracy and TMLE: A unified view from boosting
C.1 Boosting to get multiaccurate estimator
The aulti-aceurate estimator can be derived by a boosting strategy, where you first have an initial estimator y of the outcome regression function, and then solve the following least square regression
25

---

problem:
min (Ap
Ep - -xx) ((X))
Σ As
6EH
Let (ho FOEH be the optimal solution, the first order optimality condition for this is exactly the definition of multiaccuracy with a = 0:
Ep(h(X) -x
Σ
As (x))] 0.
OEH
In this way, the final estimator ((X) + EPEn 2o:
((X) is multiaccurate.
In the context of ridge boosting, you could treat the above H. the basis of the Hilbert space.
C.2
TMLE
C.2.1
TMLE can be viewed as boosting in the direction of the estimated clever covariate
The boosting step get multi-accurati estimator share similar principle the targeting step of the TMLE. In the targeting step of TMLE, we first have an initial estimator (X)). Then, we specify parametric working submodel:
(x( ((X) + c. -(X). CE (-5,8).
Here o(X) is the estimated clever covariate, which is usually given in the explicitly derived efficient influence function. Then the targeting step in TMLE solves the following boosting-type regression:
arg min Epl( (x))² arg aigminEp(( ((X) - (
In the case of estimating ATE, solving the optimization once (one-step TMLE) will give you the semiparametrically efficient estimator under the standard assumptions, because it automatically solves the score equation
(y(x(X) (Y - ((x) - 2. (x((x))] 0.
and the update of clever covariate is needed.
C.3 Ridge boosting multiaccurate estimator can be seen TMLLE for multidimensional parameters with known clever covariate (Riesz representer)
C.3.1 Ridge boosting multiaccurate estimator
Consider auditing with the function class,
(dx)) Σ
C = {c(x)) ((x) |131| S B} = c(x)
3r (x( 1118 S B
k=1
We could implement the following boosting step:
BER i = arg min { E [lly 3(X) (xxx))((x"))) x(x) + +111112
to obtain multiaccurate estimator (XX) + +
C.3.2 Viewing this as TMLE for multidimensional parameters
From the TMLE perspective, the auditing function class here can be seen as a multidimensional (possibly infinite) parametric submodel in the targeting step. However, instead of targeting a single larget parameter, we simultancously target multiple parameters
[Eq, (Y), ..... Eq, (Y)
26

---

with dQv/dP = OR (X), where Φ& (X) is the kth dimension basis of C. Here dQv/dP can be seen as the clever covariate for the target parameter Eq, (Y).
With this multidimensional parametric submodel, unregularized TMLE simulateneously solves multiple score equations for these different parameters.
Ex)ox(X) (y - ((x) - 6(x) = 0, = I...,..,dd
By doing this, we get simultaneously semiparametrically efficient estimator for
Foq (
Moreover, if dQ/dP (Q is the target population) can be approximated by the linear combination of In other words,
dQ dP
min
cll2 + 0.
dEC
Then could still obtain semiparametrically efficient estimator for Eo[Y] by applying ((X) ((x)² 3 on X, the initial estimator Ý is good enough the sense that:
dQ cll2 p 0.
| - -olll + O, and vnllff 70ll2
min aEC dP
D Score-preserving TMLE
There is recent paper about Score-Preservive TMLE [Pimentel et al., 2025]. The general idea is that instead using the estimated clever covariate only, we add the basis functions in the nuisance parameter estimation into the targeting step. This paper mentioned that solving additional scores reduces the remainder term in the von-Mises expansion of our estimator because these scores may come close to spanning higher-order influence functions and result in estimator with better finite-sample performance.
In the case of estimating E[Y (1)), the authors use the following parametric submodel in the targeting step:
logit (W) = logit Q. + 60.
A
+ Σ E (w))
(W)
where (W) is the initial estimator, (W) is the estimated clever covariate and T (W) is the basis function in the nuisance parameter estimation. There is a deep connection between Score- preserving TMLE and ridge boosting: If we use ridge outcome regression and ridge boosting with function class C. it seems that are indeed performing version of score- poore-preeervngg.. TMLE because we include the basis functions of outcome regression in this submodel, and the clever covariate can also be represented using these basis functions. Right now, the Score- pooreppreerriig TMLE is studied in the setting of highly adaptive lasso. But there is a recent paper on highly adaptive ridge [Schuler al., 2024], where ridge regression might be good for score-preserving TMLE.
27


Reviews:
[{"review_id": "5LoRfYsB0H", "reviewer": "Reviewer_EDaD", "summary": "In this paper, the authors analyze properties of \"multiaccurate estimators\" in the covariate shift setting (labeled source data, unlabeled target data, the distribution of features $X$ changes). With the goal of estimating the expectation of Y (label) in the target distribution, and for a defined \"auditing function class\" (in their case a Hilbert ball), the authors analyze a previously derived multiaccurate model using a ridge boosting estimator (an additive term that corrects a model fit to the labeled source data using, e.g., ERM). The authors show the ridge boosted multiaccurate estimator is semiparametrically efficient for any member of the Hilbert ball (and equivalently, any corresponding covariate shift-correcting density ratio). This means that despite needing to estimate the source outcome regression function (\\gamma), and the auditing functions, the boosted multiaccurate estimator of $E_Q[Y]$ is asymptotically $\\sqrt{N}$-consistent and achieves the optimal variance bound. In a simple linear regression simulation, the authors show that the boosted ridge regression estimator achieves correct confidence interval coverage as opposed to undercoverage by standard ridge regression.", "questions": "Q1: If, as the authors state, one lets the auditing function class be a \"collection of density ratios\", how does multiaccuracy differ from Distributionally robust optimization (DRO) (e.g., [1, 2]) (Equations 3 and 4 in [1] might be most relevant.)? In DRO we might be interested in minimizing the maximum error across distributions in a bounded density-ratio ball. If this worst-case error is below $\\alpha$, and the density ratio ball is the same as the auditing function class C, is the solution to this DRO problem a multiaccurate estimator? If so, discussion and comparison to DRO approaches seems interesting and relevant. If they are related, how does your efficiency result compare to the statistical properties of DRO estimators under covariate shift?\n\n\n[1] Duchi, J., Hashimoto, T., & Namkoong, H. (2023). Distributionally robust losses for latent covariate mixtures. Operations Research, 71(2), 649-664.\n\n[2] Duchi, J. C., & Namkoong, H. (2021). Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics, 49(3), 1378-1406.", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "For context, I reviewed this as someone who has done a lot of work on model training and evaluation under distribution shifts, and who is familiar with the causal inference literature. Notably, I did not have prior familiarity with multiaccuracy (though I had heard of multicalibration).\n\n### Clarity:\n- I think the biggest weakness of the paper is that it is written in a way where it heavily depends on the reader being intimately familiar with multiaccuracy literature. This may be the simplest presentation of the material, since it demonstrates a property of the ridge boosting estimator from Kim et al. (2022), but this means that the paper does not stand well on its own without having read the original paper first.\n\n### Originality/Significance:\n- Because the paper is written to heavily rely on familiarity with Kim et al. (2022), the potential significance and impact of this paper is much lower than it might otherwise be. To grossly oversimplify, it seems the main result is to show that, for a particular class of auditing functions, the estimator from Kim et al. (2022) achieves semiparametric efficiency \"for free.\" I will let a more theoretically knowledgeable reviewer speak to the significance of this result, but speaking from the perspective of a reader interested in theory + applications of distribution shift who is not as well-steeped in this literature, the paper does not speak to why this result is surprising, or what this result makes possible that could not be done before.\n- I think the authors could better motivate the choice of $E_Q[Y]$ as their estimand, since $E_Q[Y]$ is not usually the quantity of interest in machine learning. That said, I can think of examples where it is useful. For example, estimating performance on new data (under covariate shift) in settings where labels can't be easily obtained in a timely manner (such as post-deployment monitoring for long term risk predictors). Or estimating the average treatment effect under a covariate shift. \n- The form of multiaccuracy in Definition 1, and its relation to a \"collection of density ratios\" (Ln 144) makes it look quite similar to the form of Distributionally Robust Optimization. For readers familiar with DRO but not with multiaccuracy, it would be useful for the authors to include a discussion of the similarities and differences.\n\n### Quality:\n- I don't have much to comment on re: quality. I leave the more technical details to other reviewers.\n- The simulated experiment confirms the asymptotic normality result of Theorem 2 in finite samples for a very simple linear regression setup.", "was_revised": true, "final_justification": "The authors addressed my concerns (not readable without Kim et al (2022); engaged in discussion about DRO) in their rebuttal. As such I have raised my score.", "rebuttal": "# General Response\n\n## Expanded Results:\n\nWe appreciate reviewers' thoughtful engagement with our submission. To streamline exposition in our initial submission, we characterized the results in terms of linear functions of $X$ and focused on the estimand $\\mathbb{E}_Q[Y]$. In revisiting our technical results, we realized that we can substantially generalize both the class of functions and target estimands we consider --- with essentially no changes to the proofs at all. We believe that these revised results are responsive to reviewer questions and greatly expand the scope of our paper.\n\n* **Beyond linearity in $X$.** First, our results hold exchanging linear functions of $X$ for linear functions in features $\\phi(X)$, which is a remarkably broad class of functions. This class includes when $\\phi$ are the infinite-dimensional features from an RKHS; in this case the resulting algorithm would be once-boosted kernel ridge regression. This class also includes adaptive bases selected via sample splitting, such as honest random forests or last-layer embedding of a pre-trained neural network. \n\n     An important limitation is that the results using our proof techniques are not fully general and do not hold for adaptive bases without sample splitting. That said, there are other examples where boosting also implicitly estimates a density ratio that do not rely on ridge regression or sample splitting. One prominent instance is random forests, which Lin and Han [1] show implicitly estimate a density ratio. In our revised manuscript, we adapt their results to show that, under certain restrictions, random forests satisfy the requirements in our Theorem 2; and therefore that boosting with random forests enjoys the same properties as boosting with ridge. We think this is a very exciting direction that opens up many promising research avenues. \n\n* **More general estimands** Second, we show that our results continue to hold when we replace the estimand $\\mathbb{E}_Q[Y]$ with any linear functional of $\\mathbb{E}[Y|X]$, including average derivatives, average impulse responses, average treatment effects and more. To do so, we modify the class $\\mathcal{C}$ to contain the Riesz representers for these functionals: a special case is the density ratio $dQ/dP$, which is the Riesz representer for $\\mathbb{E}_Q[Y]$. Again, this extra generality does not change the proofs. \n\n     One important upshot of this generalization is that we can simultaneously consider a vector of estimands if each has a Riesz representer in the set $\\mathcal{C}$. Many applied problems estimate a vector of estimands, and existing semiparametric estimators require fitting a separate debiasing model for each element of the vector. Our once-boosted ridge procedure only requires fitting a single model, which can then be used to construct the whole vector of estimates. This is practically very appealing, although in future work we would need to secure valid joint inference.\n\n[1] Lin, Zhexiao, and Fang Han. \"On regression-adjusted imputation estimators of the average treatment effect.\" arXiv preprint arXiv:2212.05424 (2023).\n\n## Difficult Average Derivative Simulation:\n\nTo demonstrate our results in a more general setting, we consider a non-linear simulation where we estimate the average derivative across multiple distributions (combining distribution shift and a new estimand). We retain a synthetic setup because we would like to show coverage of the confidence intervals. We choose a setting with 3 covariates: $X_1, X_2$ are iid $N(\\mu,1)$, $X_3 = 4 \\cdot \\sigma(X_1 - X_2) + \\epsilon - 2$, where $\\sigma(\\cdot)$ is the sigmoid, and $\\epsilon \\sim N(0,2)$. For the training distribution $\\mu = 0$.\n\nThe outcomes follow: $Y \\sim f(X) + \\eta$, where $f(X) = X_1 \\cdot( 0.2 + \\sin(X_1) + \\sigma(X_2) - 0.2 \\cdot X_3)$ and $\\eta \\sim N(0,2)$. The estimand is the average derivative under $Q$, $\\mathbb{E}_Q[ \\partial f(X) / \\partial X_1 ]$. We consider three different distributions for $Q$. Each is identical to the setup above but changes the value of $\\mu \\in \\{-1,0,1\\}$. The dependence of $X_3$ on both $X_1$ and $X_2$ makes the average derivative very difficult to estimate.\n\nWe fit kernel ridge regression (with an RBF kernel) of $Y$ on $X$ in the training set plus one boosting step with kernel ridge. For each we compute estimates of the average derivative of the model under the three Q distributions. The nominal (not boosted) krr estimate *badly* undercovers: with $n = 500$, the $95\\\\%$ coverage for the three target estimands is $49\\\\%$, $70\\\\%$, and $31\\\\%$ respectively. Our boosted estimator achieves coverage $93\\\\%$, $95\\\\%$, $92\\\\%$ respectively. The improvement for the boosted estimator is actually larger than in the linear setting. (The final submission will include full figures for this experiment, which we cannot include in the rebuttal.)\n\n## Real Data Experiment\n\nFollowing reviewer feedback, we also applied this approach to a real data example using American Community Survey (ACS) data, where the goal is to estimate counterfactual outcome means across a vector of ages. In this setting, $Y$ is income, $A$ is age, and $X$ are covariates like occupation, marriage status, and place of birth. Let $f(A,X) = \\mathbb{E}[Y|A,X]$. Our estimands are $\\mathbb{E}[f(65, X)], \\mathbb{E}[f(66, X)], ... \\mathbb{E}[f(90, X)]$. This vector of estimands represents the \"age profile\" of total income in retirement keeping $X$ fixed. This profile is an important input in many macroeconomic models (see, for example, the seminal paper from Kaplan and Violante [2]), especially for modeling consumption in retirement. \nEach of these estimands has substantial covariate shift when focusing on ages over 65. \nTypical semiparametric estimators would require constructing a separate debiasing term at every age. We can instead use our theoretical results to motivate a common debiasing procedure for the entire vector of estimands: in our example we fit kernel ridge regression to estimate $f(A,X)$ and then use one round of boosting. Finally, we use the boosted $\\hat{f}$ to compute a semiparametric-efficient estimate of every point on the age profile with corresponding confidence intervals.\n\n| Age | Naive Estimate | Boosted Estimate | Naive C.I. Width | Boosted C.I. Width |\n| --- | --- | -- | -- | ---- |\n| 65 | 61.1k | 59.5k | 1.7k | 1.8k |\n| 71 | 60.6k | 58.8k | 1.7k | 1.8k |\n| 77 | 58.5k | 57.6k | 1.6k | 1.8k |\n| 83 | 55.1k | 56.5k | 1.5k | 1.8k |\n| 89 | 50.8k | 55.9k | 1.4k | 1.9k |\n\nWe give the results (in thousands of US dollars) for 5 evenly-spaced points on the age profile in the table above. Whereas the naive estimate (without boosting) features a steep decline of \\$11k from ages 65-89, our boosted estimates are substantially flatter --- better matching the theoretical model for pension and social security income from Kaplan and Violante. Furthermore, while the naive confidence intervals shrink for the highest ages, our boosted confidence intervals actually grow slightly, suggesting that the naive model may undercover for the oldest part of the age profile. \n\n[2] Kaplan, Greg, and Giovanni L. Violante. \"A model of the consumption response to fiscal stimulus payments.\" Econometrica 82.4 (2014): 1199-1239.\n\n# Reviewer-Specific Response\n\nWe appreciate the reviewer's push to engage more critically with DRO and to make this work more accessible to a broader audience. \n\nFirst, we agree that the exposition in the original submission relied too heavily on Kim et al. (2022). In the revision, we have expanded the background and setup so that readers unfamiliar with the earlier multiaccuracy results can nonetheless engage with the current work. \n\n**Question 1**: We thank the reviewer for this insightful question. Indeed, there is a close connection between **multiaccuracy** and **distributionally robust optimization (DRO)**, as also discussed in [3]. To clarify this connection, we first recall the standard DRO objective:\n$$\n\\min_{f \\in \\mathcal{F}} \\sup_{w \\in \\mathcal{C}} \\mathbb{E}_P[w(X) \\cdot \\ell(f(X), Y)],\n$$\nwhere $w(X) = \\frac{dQ}{dP}(X)$, $\\mathcal{C}$ is a bounded density-ratio ball, and $\\ell(f(X), Y)$ is the loss function of interest.\n\nIn the special case where $\\ell(f(X), Y) = Y - f(X)$, and the worst-case error is bounded by $\\alpha$, as you noted, the solution to the DRO problem satisfies the multiaccuracy condition with respect to the function class $\\mathcal{C}$ and the distribution $P$.\n\nUnder the squared loss $\\ell(f(X), Y) = (f(X) - Y)^2$, there is also a connection between multi-accuracy and the first-order optimality condition for DRO. Let $w^* \\in \\mathcal{C}$ denote the adversarial density ratio that attains the supremum. If $\\mathcal{F}$ is a Hilbert space (e.g., an RKHS), the first-order optimality condition for the DRO objective becomes:\n\n$\\mathbb{E}_P[w^*(X) \\cdot (f(X) - Y) \\cdot h(X)] = 0, \\quad \\forall h \\in \\mathcal{F}.$\n\nThis is equivalent to the multiaccuracy condition under a reweighted distribution $\\tilde{P}$, where $d\\tilde{P}/dP = w^*$, i.e.,\n$$\n\\mathbb{E}_{\\tilde{P}}[(f(X) - Y) \\cdot h(X)] = 0, \\quad \\forall h \\in \\mathcal{F}.\n$$\n\nHence, under these assumptions, the DRO solution satisfies multiaccuracy with respect to the class $\\mathcal{F}$ and the reweighted distribution $\\tilde{P}$.\n\nRegarding efficiency, while many estimators can satisfy multiaccuracy, our proposed estimator is just one example. The DRO solution may take a different form. Our theoretical efficiency guarantees, which are specific to our construction, do not directly apply to general DRO estimators. Nonetheless, we agree that this connection is highly interesting and provides a promising direction for further exploration.\n\nWe appreciate the reviewer highlighting this connection, and we will incorporate a discussion into the revised manuscript.\n\n[3] Hastings, J., Jung, C., Peale, C., $\\&$ Syrgkanis, V. (2024). Taking a moment for distributional robustness. arXiv preprint arXiv:2405.05461."}, {"review_id": "1qvad8CyGw", "reviewer": "Reviewer_cMtW", "summary": "This paper connects two subareas of machine learning, multiaccuracy and semiparametric efficiency. It focuses on the case where the class of tests C (for multiaccuracy) is a Hilbert space. With some additional niceness assumptions, the ridge boosting estimator is multi-accurate and semiparametrically efficient. Together, these properties imply that the estimator has small bias and asymptotically efficient variance on unseen distributions Q (i.e., not used for training), for Q induced by the set of tests C. \n\nThis paper is in a line of very recent papers (last couple years) connecting multicalibration and causal inference. For example, Kim et al. 2022 showed that multiaccurate predictors can achieve low bias across certain (unseen) target populations, even when the distribution shift is unknown. However, multiaccuracy alone only guarantees bounds on the bias, not also the variance. In this paper, the Hilbert ball assumption is used to help bound the variance. Theorem 1 bounds the multi-accuracy error, and Theorem 2 bounds the bias and variance. \n\nThe paper also includes an experiment with synthetic data, comparing the coverage of single ridge outcome regression and ridge regression with auditing. On the synthetic data, with the same tuning parameter lambda, the boosted ridge has better coverage per sample size.", "questions": "---With respect to the assumptions 1-5 in the paper:\n-Why is assumption 5 (supplemental pg 2) necessary?\n-On pg 7, main body, assumption 3 and 4 are claimed to be standard in the semiparametric efficiency literature. What is an example of a problem for which some of the assumptions fail, and this argument cannot be applied? Are there \"natural\" extensions (e.g., slightly different problem settings) that are not captured by this approach (and perhaps worthy of future investigation)?\n\n\n---What other auditing function classes (besides Hilbert balls) are plausible candidates for connecting multiaccuracy (or multicalibration definitions more generally) to the semiparametric efficiency literature?\n\n---Minor typos:\nLine 228: derive -> deriving\nLine 319: are results -> our results\n---How would this algorithm perform on a \"real-world\" data set? Which applications could plausibly satisfy (or be close to satisfying) the 5 assumptions necessary for Theorem 2? To which applications is the estimator in this paper particularly well-suited? (e.g., If you had to pick one \"real-world\" application, what would it be?)", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Quality:\nThe claims are primarily theoretical and supported by proofs. The proofs appear to be complete and correct. The paper makes some claims about the practicality of multiaccurate learning algorithms (in general), but the experiment that appears in this work is only on synthetic data. \n\nClarity:\nThe paper is nicely organized, and the background sections set the stage well for the technical content of the paper. Additional context in the exposition of the scope of the work's results would be appreciated, as discussed by my review and other reviewers. \n\nSignificance:\nThis paper attempts to connect the multicalibration and semiparametrics literatures. In the special case when the multiaccuracy class is the Hilbert ball, ridge regression leads to an estimator that implicitly performs Riesz regression. It shows that the notions from multiaccuracy and semiparametric efficiency can coincide, and that there may be other possible connections between these two subareas. This could lead to future projects by researchers from either areas. \n\nOriginality: \nMany comments about significance above also apply for originality. The paper draws heavily from both multicalibration and semiparametrics, as it should for a paper attempting to combine ideas from both areas. Relevant citations are provided to distinguish this work from previous work.", "was_revised": true, "final_justification": "Copying the following from my comment to the authors, after their initial rebuttal to my review:\n\nHaving read the other reviewers' evaluations, comments, and rebuttals, my overall evaluation of the paper has not significantly changed. I am positive towards this paper bridging two subareas (multiaccuracy and semiparametric efficiency) and leading to further interesting research.\n\nWhile the high-level contributions seem impressive and above the bar for acceptance, admittedly, the technical content is difficult for me to follow. Therefore, it is hard for me to confidently suggest an evaluation for this submission. I am willing to defer to reviewers EDaD and wM69's positive reviews, trusting that their domain knowledge and understanding of this submission's merits greatly outstrips my own. Moreover, at the time of writing this comment, it appears that the negative comments of Reviewer 6txf have not been resolved, and I do not currently have a nuanced understanding of their disagreements with the authors.\n\nNevertheless, I do not feel personally comfortable increasing my confidence or rating of the submission", "rebuttal": "# General Response\n\n## Expanded Results:\n\nWe appreciate reviewers' thoughtful engagement with our submission. To streamline exposition in our initial submission, we characterized the results in terms of linear functions of $X$ and focused on the estimand $\\mathbb{E}_Q[Y]$. In revisiting our technical results, we realized that we can substantially generalize both the class of functions and target estimands we consider --- with essentially no changes to the proofs at all. We believe that these revised results are responsive to reviewer questions and greatly expand the scope of our paper.\n\n* **Beyond linearity in $X$.** First, our results hold exchanging linear functions of $X$ for linear functions in features $\\phi(X)$, which is a remarkably broad class of functions. This class includes when $\\phi$ are the infinite-dimensional features from an RKHS; in this case the resulting algorithm would be once-boosted kernel ridge regression. This class also includes adaptive bases that are selected via sample splitting, such as honest random forests and ridge regression on the last-layer embedding of a pre-trained neural network. \n\n     An important limitation is that the results using our proof techniques are not fully general and do not hold for adaptive bases without sample splitting. That said, there are other examples where boosting also implicitly estimates a density ratio that do not rely on ridge regression or sample splitting. One prominent instance is random forests, which Lin and Han [1] show implicitly estimate a density ratio. In our revised manuscript, we adapt their results to show that, under certain restrictions, random forests satisfy the requirements in our Theorem 2; and therefore that boosting with random forests enjoys the same properties as boosting with ridge. While we leave a thorough discussion of this to future work, we think this is a very exciting direction that opens up many promising research avenues. \n\n* **More general estimands** Second, we show that our results continue to hold when we replace the estimand $\\mathbb{E}_Q[Y]$ with any linear functional of $\\mathbb{E}[Y|X]$, including average derivatives, average impulse responses, average treatment effects and more. To do so, we modify the class $\\mathcal{C}$ to contain the Riesz representers for these functionals: a special case is the density ratio $dQ/dP$, which is the Riesz representer for $\\mathbb{E}_Q[Y]$. Again, this extra generality does not change the proofs. \n\n     One important upshot of this generalization is that we can simultaneously consider a vector of estimands if each has a Riesz representer in the set $\\mathcal{C}$. Many applied problems estimate a vector of estimands, and existing semiparametric estimators require fitting a separate debiasing model for each element of the vector. Our once-boosted ridge procedure only requires fitting a single model, which can then be used to construct the whole vector of estimates. This is practically very appealing, although a thorough assessment of this is beyond the scope of the current paper, especially with respect to valid joint inference.\n\n[1] Lin, Zhexiao, and Fang Han. \"On regression-adjusted imputation estimators of the average treatment effect.\" arXiv preprint arXiv:2212.05424 (2023).\n\n## Difficult Average Derivative Simulation:\n\nTo demonstrate our results in a more general setting, we consider a non-linear simulation where we estimate the average derivative across multiple distributions (combining distribution shift and a new estimand). For these initial results, we retain a synthetic setup because we would like to show coverage of the confidence intervals. We choose a setting with 3 covariates: $X_1, X_2$ are iid $N(\\mu,1)$, $X_3 = 4 \\cdot \\sigma(X_1 - X_2) + \\epsilon - 2$, where $\\sigma(\\cdot)$ is the sigmoid, and $\\epsilon \\sim N(0,2)$. For the training distribution $\\mu = 0$.\n\nThe outcomes follow: $Y \\sim f(X) + \\eta$, where $f(X) = X_1 \\cdot( 0.2 + \\sin(X_1) + \\sigma(X_2) - 0.2 \\cdot X_3)$ and $\\eta \\sim N(0,2)$. The estimand is the average derivative under $Q$, $\\mathbb{E}_Q[ \\partial f(X) / \\partial X_1 ]$. We consider three different distributions for $Q$. Each is identical to the setup above but changes the value of $\\mu \\in \\{-1,0,1\\}$. The dependence of $X_3$ on both $X_1$ and $X_2$ makes the average derivative very difficult to estimate, as we will show.\n\nWe fit kernel ridge regression (with an RBF kernel) of $Y$ on $X$ in the training set. Then we also fit a once-boosted kernel ridge regression on the training set. For each we compute estimates of the average derivative of the model under the three Q distributions. The nominal (not boosted) krr estimate *badly* undercovers: with $n = 500$, the $95\\\\%$ coverage for the three target estimands is $49\\\\%$, $70\\\\%$, and $31\\\\%$ respectively. Our boosted estimator achieves coverage $93\\\\%$, $95\\\\%$, $92\\\\%$ respectively. In this challenging non-linear setting the gap in performance is actually much larger than in the linear ridge setting we originally submitted. (The final submission will include full figures for this experiment, which we cannot include in the rebuttal.)\n\n## Real Data Experiment\n\nFollowing reviewer feedback, we also applied this approach to a real data example using American Community Survey (ACS) data, where the goal is to estimate counterfactual outcome means across a vector of ages. In this setting, $Y$ is income, $A$ is age, and $X$ are covariates like occupation, marriage status, and place of birth. Let $f(A,X) = \\mathbb{E}[Y|A,X]$. Our estimands are $\\mathbb{E}[f(65, X)], \\mathbb{E}[f(66, X)], ... \\mathbb{E}[f(90, X)]$. This vector of estimands represents the \"age profile\" of total income in retirement keeping $X$ fixed. This profile is an important input in many macroeconomic models (see, for example, the seminal paper from Kaplan and Violante [2]), especially for modeling consumption in retirement. \nEach of these estimands has substantial covariate shift when focusing on ages over 65. \nTypical semiparametric estimators would require constructing a separate debiasing term at every age. We can instead use our theoretical results to motivate a common debiasing procedure for the entire vector of estimands: in our example we fit kernel ridge regression to estimate $f(A,X)$ and then use one round of boosting. Finally, we use the boosted $\\hat{f}$ to compute a semiparametric-efficient estimate of every point on the age profile with corresponding confidence intervals.\n\n| Age | Naive Estimate | Boosted Estimate | Naive C.I. Width | Boosted C.I. Width |\n| --- | ----- | ----- | ----- | ---- |\n| 65 | 61.1k | 59.5k | 1.7k | 1.8k |\n| 71 | 60.6k | 58.8k | 1.7k | 1.8k |\n| 77 | 58.5k | 57.6k | 1.6k | 1.8k |\n| 83 | 55.1k | 56.5k | 1.5k | 1.8k |\n| 89 | 50.8k | 55.9k | 1.4k | 1.9k |\n\nWe give the results (in thousands of US dollars) for 5 evenly-spaced points on the age profile in the table above. Whereas the naive estimate (without boosting) features a steep decline of \\$11k from ages 65-89, our boosted estimates are substantially flatter --- better matching the theoretical model for pension and social security income from Kaplan and Violante. Furthermore, while the naive confidence intervals shrink for the highest ages, our boosted confidence intervals actually grow slightly, suggesting that the naive model may undercover for the oldest part of the age profile. \n\n[2] Kaplan, Greg, and Giovanni L. Violante. \"A model of the consumption response to fiscal stimulus payments.\" Econometrica 82.4 (2014): 1199-1239.\n\n# Reviewer-Specific Response\n\nWe are grateful for your thoughtful engagement with our paper and for the constructive feedback. \n\n**Assumptions.**  First, we agree that our initial submission did not provide sufficient intuition for the underlying assumptions. We give some additional intuition here, and will include longer responses in the updated manuscript. \n\nAssumption 5 essentially requires that the higher-order moments of the efficient influence function are bounded. This is a minimal condition on the higher-order moments of the efficient influence function such that the variance $\\hat{V}$ asymptotically converges to the true variance and therefore provides a valid confidence interval. Most central limit theorem-based confidence intervals require a similar condition. This condition is mostly like to be violated in heavy-tailed data.\n\nAssumption 3 bounds variances for the outcome. These conditions are very likely to be satisfied unless the $X$ distribution or conditional noise distribution have very heavy tails (like a Cauchy distribution). Assumption 4 is also a relatively weak requirement. This assumption merely states that given an infinite amount of data, the estimate of the density ratio will eventually converge to the truth (and that this can happen at an arbitrarily slow rate). This assumption is very plausible using the flexible basis functions that we describe in the general response above.\n\nOverall, Assumptions 3-5 will be satisfied in most empirical applications except when dealing with heavy tailed data (such as on some stock return data sets, weather data with extreme events, etc).\n\nThe key substantive assumptions are unconfoundedness and overlap, and these do not hold for every application. These common observational causal inference assumptions are more likely to hold (approximately) in settings like electronic medical records with rich covariate information and enough overlap in the $X$ distributions from varied treatment assignment. In terms of other applications, the ``age profile'' example that we detailed above is inspired, although in extremely simplified form, by a real empirical application (currently unpublished ongoing work) where fitting a debiasing nuisance for each point on the age profile would be computationally-intensive and add additional variance. Using this boosting procedure in that setting would be quite helpful. \n\nFinally: thank you for catching those typos!"}, {"review_id": "HyaFGEhNgf", "reviewer": "Reviewer_SMVd", "summary": "This paper bridges fairness-inspired multiaccuracy estimation with semiparametric efficiency from causal inference. It shows that when the auditing class is a Hilbert space, a simple ridge-boosting step applied to the residuals of a base estimator yields a multiaccurate predictor that is numerically equivalent to an augmented balancing weight (ABW) estimator. As a result, this estimator inherits both universal adaptability (robustness under unknown covariate shifts) and semiparametric efficiency (minimum asymptotic variance), even without access to target distribution data. The authors provide theoretical justification, draw connections to Riesz regression and TMLE, and validate their claims through simulations demonstrating improved confidence interval coverage under distribution shift.", "questions": "Have you evaluated the proposed estimator on real datasets with covariate shift? If not, do you anticipate any challenges in doing so?\n\nYour simulation assumes a linear-Gaussian data-generating process. How do you expect performance to change in more complex, nonlinear settings?", "limitations": "This is a theoretical contribution that adds conceptual clarity and unifies ideas across fairness, robustness, and causal inference. Its practicality could be enhanced by empirical validation on real datasets and extending beyond linear function classes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper establishes a link between multiaccuracy and semiparametric efficiency, uniting fairness and causal inference methodologies.\n\nThe estimator does not require access to target distribution data, making it highly applicable in privacy-sensitive or resource-constrained settings.\n\nThe method is simple, which implements a practical, one-step ridge-boosting procedure that is easy to use and analyze.\n\nAt last, the paper provides empirical evidence that the once-boosted estimator achieves valid confidence interval coverage under covariate shifts.\n\nWeakness:\nOnly one synthetic simulation setting is presented, with no real-world benchmark or application to support claims of practical effectiveness.\n\nKey efficiency results rely on assumptions such as consistent outcome regression and the density ratio being well-approximated within the Hilbert space. In practice, these assumptions may not hold, and there is no empirical test or sensitivity analysis of robustness to assumption violations.\n\nWhile the connection to existing debiasing methods is insightful, the novelty lies more in interpretation than in introducing a new estimator. The paper might be perceived as a reinterpretation of known techniques rather than a fundamentally new method.", "was_revised": true, "final_justification": "Thank you for the detailed response. The expanded theory, broader applicability, and practical demonstrations address my concerns. I am revising my score to borderline accept.", "rebuttal": "# General Response\n\n## Expanded Results:\n\nWe appreciate reviewers' thoughtful engagement with our submission. To streamline exposition in our initial submission, we characterized the results in terms of linear functions of $X$ and focused on the estimand $\\mathbb{E}_Q[Y]$. In revisiting our technical results, we realized that we can substantially generalize both the class of functions and target estimands we consider --- with essentially no changes to the proofs at all. We believe that these revised results are responsive to reviewer questions and greatly expand the scope of our paper.\n\n* **Beyond linearity in $X$.** First, our results hold exchanging linear functions of $X$ for linear functions in features $\\phi(X)$, which is a remarkably broad class of functions. This class includes when $\\phi$ are the infinite-dimensional features from an RKHS; in this case the resulting algorithm would be once-boosted kernel ridge regression. This class also includes adaptive bases that are selected via sample splitting, such as honest random forests and ridge regression on the last-layer embedding of a pre-trained neural network. \n\n     An important limitation is that the results using our proof techniques are not fully general and do not hold for adaptive bases without sample splitting. That said, there are other examples where boosting also implicitly estimates a density ratio that do not rely on ridge regression or sample splitting. One prominent instance is random forests, which Lin and Han [1] show implicitly estimate a density ratio. In our revised manuscript, we adapt their results to show that, under certain restrictions, random forests satisfy the requirements in our Theorem 2; and therefore that boosting with random forests enjoys the same properties as boosting with ridge. While we leave a thorough discussion of this to future work, we think this is a very exciting direction that opens up many promising research avenues. \n    \n\n* **More general estimands** Second, we show that our results continue to hold when we replace the estimand $\\mathbb{E}_Q[Y]$ with any linear functional of $\\mathbb{E}[Y|X]$, including average derivatives, average impulse responses, average treatment effects and more. To do so, we modify the class $\\mathcal{C}$ to contain the Riesz representers for these functionals: a special case is the density ratio $dQ/dP$, which is the Riesz representer for $\\mathbb{E}_Q[Y]$. Again, we achieve this generality without changing the proofs. \n\n     One important upshot of this generalization is that we can simultaneously consider a vector of estimands if each has a Riesz representer in the set $\\mathcal{C}$. Many applied problem estimate a vector of estimands, and existing semiparametric estimators require fitting a separate debiasing model for each individual point in the vector. Our once-boosted ridge procedure only requires fitting a single model, which can then be used to construct the whole vector of estimates. This is practically very appealing, although a thorough assessment of this is beyond the scope of the current paper, especially with respect to valid joint inference. In particular, in follow up work we will better articulate conditions under which the 95\\% confidence band over the whole vector is simultaneously valid. \n\n[1] Lin, Zhexiao, and Fang Han. \"On regression-adjusted imputation estimators of the average treatment effect.\" arXiv preprint arXiv:2212.05424 (2023).\n\n## Difficult Average Derivative Simulation:\n\nTo demonstrate our results in a more general setting, we consider a non-linear simulation where we estimate the average derivative across multiple distributions (combining distribution shift and a new estimand). For these initial results, we retain a synthetic setup because we would like to show coverage of the confidence intervals. We choose a setting with 3 covariates: $X_1, X_2$ are iid $N(\\mu,1)$, $X_3 = 4 \\cdot \\sigma(X_1 - X_2) + \\epsilon - 2$, where $\\sigma(\\cdot)$ is the sigmoid, and $\\epsilon \\sim N(0,2)$. For the training distribution $\\mu = 0$.\n\nThe outcomes follow: $Y \\sim f(X) + \\eta$, where $f(X) = X_1 \\cdot( 0.2 + \\sin(X_1) + \\sigma(X_2) - 0.2 \\cdot X_3)$ and $\\eta \\sim N(0,2)$. The estimand is the average derivative under $Q$, $\\mathbb{E}_Q[ \\partial f(X) / \\partial X_1 ]$. We consider three different distributions for $Q$. Each is identical to the setup above but changes the value of $\\mu \\in \\{-1,0,1\\}$. The dependence of $X_3$ on both $X_1$ and $X_2$ makes the average derivative very difficult to estimate, as we will show.\n\nWe fit kernel ridge regression (with an RBF kernel) of $Y$ on $X$ in the training set. Then we also fit a once-boosted kernel ridge regression on the training set. For each we compute estimates of the average derivative of the model under the three Q distributions. The nominal (not boosted) krr estimate *badly* undercovers: with $n = 500$, the $95\\\\%$ coverage for the three target estimands is $49\\\\%$, $70\\\\%$, and $31\\\\%$ respectively. Our boosted estimator achieves coverage $93\\\\%$, $95\\\\%$, $92\\\\%$ respectively. In this challenging non-linear setting the gap in performance is actually much larger than in the linear ridge setting we originally submitted. (The final submission will include full figures for this experiment, which we cannot include in the rebuttal.)\n\n## Real Data Experiment\n\nFollowing reviewer feedback, we also applied this approach to a real data example using American Community Survey (ACS) data, where the goal is to estimate counterfactual outcome means across a vector of ages. In this setting, $Y$ is income, $A$ is age, and $X$ are covariates like occupation, marriage status, and place of birth. Let $f(A,X) = \\mathbb{E}[Y|A,X]$. Our estimands are $\\mathbb{E}[f(65, X)], \\mathbb{E}[f(66, X)], ... \\mathbb{E}[f(90, X)]$. This vector of estimands represents the \"age profile\" of total income in retirement keeping $X$ fixed. This profile is an important input in many macroeconomic models (see, for example, the seminal paper from Kaplan and Violante [2]), especially for modeling consumption in retirement. \nEach of these estimands has substantial covariate shift when focusing on ages over 65. \nTypical modeling here would construct separate semiparametric efficient estimators for each point in the age profile, necessarily requiring a separate debiasing term at every age.\nWe can instead use our theoretical results to motivate a common debiasing procedure for the entire vector of estimands: in our example we fit kernel ridge regression to estimate $f(A,X)$ and then use one round of boosting. Finally, we use the boosted $\\hat{f}$ to compute a semiparametric-efficient estimate of every point on the age profile with corresponding confidence intervals.\n\n| Age | Naive Estimate | Boosted Estimate | Naive C.I. Width | Boosted C.I. Width |\n| --- | -------------- | ---------------- | ---------------- | ------------------ |\n| 65 | 61.1k | 59.5k | 1.7k | 1.8k |\n| 71 | 60.6k | 58.8k | 1.7k | 1.8k |\n| 77 | 58.5k | 57.6k | 1.6k | 1.8k |\n| 83 | 55.1k | 56.5k | 1.5k | 1.8k |\n| 89 | 50.8k | 55.9k | 1.4k | 1.9k |\n\nWe give the results (in thousands of US dollars) for 5 evenly-spaced points on the age profile in the table above. Whereas the naive estimate (without boosting) features a steep decline of \\$11k from ages 65-89, our boosted estimates are substantially flatter --- better matching the theoretical model for pension and social security income from Kaplan and Violante. Furthermore, while the naive confidence intervals shrink for the highest ages, our boosted confidence intervals actually grow slightly, suggesting that the naive model may undercover for the oldest part of the age profile. \n\n[2] Kaplan, Greg, and Giovanni L. Violante. \"A model of the consumption response to fiscal stimulus payments.\" Econometrica 82.4 (2014): 1199-1239.\n\n# Reviewer-Specific Response\n\nThank you for your feedback. We hope that the discussion above addresses some of your concerns about extending beyond linear function classes and demonstrating practical usage on a real dataset. \n\nWe agree that this is not a fundamentally new method --- boosting with ridge regression is well-studied in the multicalibration literature. Our result instead establishes a new property for boosting with ridge regression: that it can be simultaneously semiparametrically efficient for all estimands described by the set $\\mathcal{C}.$ This is in contrast to usual algorithms for semiparametric efficiency, which require estimating a different nuisance parameter specific to each estimand of interest. Previous work on multiaccurate estimators like ridge boosting have focused on robustness or fairness considerations, so it is surprising that the same procedure could achieve efficiency.\n\nFinally, we appreciate the question about a specification test or sensitivity analysis regarding violations of our assumptions. The core assumption required for our once-boosted ridge estimate to achieve Theorem 2 for a particular $Q$ is that $dQ/dP$ must belong to the Hilbert space ball $\\mathcal{C}$. An infinite-dimensional Hilbert space can be extremely flexible, so this is not so much a question of model specification. However, it does crucially hinge on the radius of the Hilbert space ball. For a $Q$ with intense distribution shift, $dQ/dP$ could be very large, and so the radius must be very large. Practically speaking, this corresponds to making the regularization parameter for the boosting step very small. As a result, the variance of the estimate would grow substantially, resulting in a very large and even uninformative confidence interval, accurately reflecting the uncertainty of transferring to a very different $Q$. It would be useful to build a specification test/sensitivity analysis that allows us to check, how small we need to make the regularization parameter for the boosting step. Developing a feasible test for this condition would be an important topic for future work."}, {"review_id": "JZ3XciAAdl", "reviewer": "Reviewer_6txf", "summary": "This paper studies the problem of estimating $\\mathbb{E}_Q[Y]$ from labeled samples $(X_i, Y_i) \\sim P$ under covariate shift, where $\\mathbb{E}_P[Y|X] = \\mathbb{E}_Q[Y|X]$. The goal is to construct an estimator that generalizes well to the target distribution.\n\nThe main idea of this work is to apply multiaccuracy: a predictor $\\hat{f}$ is said to be $(\\mathcal{C}, \\alpha)$-multiaccurate if\n$$\n\\sup_{c \\in \\mathcal{C}} \\left| \\mathbb{E}_P\\left[ (Y - \\hat{f}(X)) \\cdot c(X) \\right] \\right| \\leq \\alpha.\n$$\n\nIf $dQ/dP \\in \\mathcal{C}$, then this guarantees that the expected bias  $\\mathbb{E}_Q[Y - \\hat{f}(X)]$ is at most $\\alpha$. \n\nThe estimator studied in the paper is a ridge-boosted predictor of the form\n$$\n\\hat{\\psi}(X) = \\hat{\\gamma}(X) + X^\\top \\hat{\\beta},\n$$\nwhere $\\hat{\\gamma}$ is any initial predictor trained on $(X_p, Y_p)$ and $\\hat{\\beta}$ is obtained by ridge regression on the residuals $Y - \\hat{\\gamma}(X)$ using source data.\n\n**Theorem 1** shows that this ridge-boosted predictor has lower multiaccuracy error than the initial predictor. That is, the worst-case residual correlation with functions in $\\mathcal{C}$ is reduced by the boosting step. In the case where the $P$ has a diagonal covariance matrix, they can show that multiaccuracy tends to 0 as the regularization, $\\lambda$ goes to 0. \n\nIn their main result, **Theorem 2**, they  show that the average prediction of the ridge-boosted estimator over target samples is asymptotically normal and achieves the semiparametric efficiency bound for estimating $\\mathbb{E}_Q[Y]$, provided that $\\hat{\\gamma}$ is consistent and $dQ/dP$ is well-approximated in $\\mathcal{C}$.\n\nFinally, they experimentally validate their results over synthetic data.", "questions": "Please see the list of weaknesses I provided. In general, I think this paper could do with significant revision. It also needs to justify its setting (where $\\mathcal{C}$ is just a Hilbert ball more).", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: this work studies an interesting connection between multiaccuracy and OOD generalization. It also provides a relatively thorough analysis of the proposed algorithm (showing that the estimator achieves the lower bound on variance) which is a nice guarantee. \n\nWeaknesses: \n\n1. The setting where $\\mathcal{C}$ only comprises of linear functions is rather limited. It doesn't feel very plausible to me that dQ/dP would follow this form in almost any situation. If the authors don't agree here, I would find some examples very helpful. At the very least, arguing why this is an important theoretical step would be appreciated.\n\n2. Theorem 1 feels a bit weak. Merely showing that the multiaccuracy is 'smaller' than the previous one doesn't feel like much of a result on its own. This is only helpful in the context of multiaccuracy being a goal (from some kind of fairness perspective). However, for the OOD setting (where the goal is to just achieve a good loss over Q) I don't really see how this is relevant.  \n\n3. Theorem 2 feels like a relatively straightforward consequence of the Corollary from Chernozhukov et. al. 2023. This wouldn't be an issue if Theorem 2 were characterizing a more relevant or practical problem, but given that Theorem 2 is restricted to a very limited theoretical setting (weakness 1) i do feel like there is a higher onus for providing new technical innovatinos. \n\n4. The presentation of Theorem 2 is sloppy. what does $\\min_{c \\in C} ||\\frac{dQ}{dP} - c|| \\to_p 0$ mean? What are we taking the limit with respect to? None of those quantities vary with the sample size. Furthermore, insepcting the proofs, it seems that this work effectively just plugs this in all over the place to achieve other limits going to 0. What this means is that simply stating $\\frac{dQ}{dP} \\in \\mathcal{C}$ would have equivalently sufficed. As far as I can tell, $\\frac{dQ}{dP} \\in \\mathcal{C}$ is the only reasonable interpretation of this statement, but I'm happy to be corrected. \n\n5. The general presentation can be significantly improved. There are several passages where algebra is done to provide intuition, but I think that this can be mostly omitted -- the manipulations are relatively standard and mostly take up space. Instead, this space could be used to argue more effectively for the relevance of this setting and to provide more proof details. The supplementary material of this paper is quite short suggesting that some of the proofs could be included in the main body.\n\n6. Additionallly the notation feels cumbersome and often poorly defined. For example, $\\Psi(Q)$ is never defined and appears for the first time in Theorem 2. I'm assuming that $\\Psi(Q)$ simply means $\\mathbb{E}_Q[Y|X]$.", "was_revised": true, "final_justification": "See comment to authors.", "rebuttal": "# General Response\n\n## Expanded Results:\n\nWe appreciate reviewers' thoughtful engagement with our submission. To streamline exposition in our initial submission, we characterized the results in terms of linear functions of $X$ and focused on the estimand $\\mathbb{E}_Q[Y]$. In revisiting our technical results, we realized that we can substantially generalize both the class of functions and target estimands we consider --- with essentially no changes to the proofs at all. We believe that these revised results are responsive to reviewer questions and greatly expand the scope of our paper.\n\n* **Beyond linearity in $X$.** First, our results hold exchanging linear functions of $X$ for linear functions in features $\\phi(X)$, which is a remarkably broad class of functions. This class includes when $\\phi$ are the infinite-dimensional features from an RKHS; in this case the resulting algorithm would be once-boosted kernel ridge regression. This class also includes adaptive bases that are selected via sample splitting, such as honest random forests and ridge regression on the last-layer embedding of a pre-trained neural network. \n\n     An important limitation is that the results using our proof techniques are not fully general and do not hold for adaptive bases without sample splitting. That said, there are other examples where boosting also implicitly estimates a density ratio that do not rely on ridge regression or sample splitting. One prominent instance is random forests, which Lin and Han [1] show implicitly estimate a density ratio. In our revised manuscript, we adapt their results to show that, under certain restrictions, random forests satisfy the requirements in our Theorem 2; and therefore that boosting with random forests enjoys the same properties as boosting with ridge. While we leave a thorough discussion of this to future work, we think this is a very exciting direction that opens up many promising research avenues. \n    \n\n* **More general estimands** Second, we show that our results continue to hold when we replace the estimand $\\mathbb{E}_Q[Y]$ with any linear functional of $\\mathbb{E}[Y|X]$, including average derivatives, average impulse responses, average treatment effects and more. To do so, we modify the class $\\mathcal{C}$ to contain the Riesz representers for these functionals: a special case is the density ratio $dQ/dP$, which is the Riesz representer for $\\mathbb{E}_Q[Y]$. Again, we achieve this generality without changing the proofs. \n\n     One important upshot of this generalization is that we can simultaneously consider a vector of estimands if each has a Riesz representer in the set $\\mathcal{C}$. Many applied problem estimate a vector of estimands, and existing semiparametric estimators require fitting a separate debiasing model for each individual point in the vector. Our once-boosted ridge procedure only requires fitting a single model, which can then be used to construct the whole vector of estimates. This is practically very appealing, although a thorough assessment of this is beyond the scope of the current paper, especially with respect to valid joint inference. In particular, in follow up work we will better articulate conditions under which the 95\\% confidence band over the whole vector is simultaneously valid. \n\n[1] Lin, Zhexiao, and Fang Han. \"On regression-adjusted imputation estimators of the average treatment effect.\" arXiv preprint arXiv:2212.05424 (2023).\n\n## Difficult Average Derivative Simulation:\n\nTo demonstrate our results in a more general setting, we consider a non-linear simulation where we estimate the average derivative across multiple distributions (combining distribution shift and a new estimand). For these initial results, we retain a synthetic setup because we would like to show coverage of the confidence intervals. We choose a setting with 3 covariates: $X_1, X_2$ are iid $N(\\mu,1)$, $X_3 = 4 \\cdot \\sigma(X_1 - X_2) + \\epsilon - 2$, where $\\sigma(\\cdot)$ is the sigmoid, and $\\epsilon \\sim N(0,2)$. For the training distribution $\\mu = 0$.\n\nThe outcomes follow: $Y \\sim f(X) + \\eta$, where $f(X) = X_1 \\cdot( 0.2 + \\sin(X_1) + \\sigma(X_2) - 0.2 \\cdot X_3)$ and $\\eta \\sim N(0,2)$. The estimand is the average derivative under $Q$, $\\mathbb{E}_Q[ \\partial f(X) / \\partial X_1 ]$. We consider three different distributions for $Q$. Each is identical to the setup above but changes the value of $\\mu \\in \\{-1,0,1\\}$. The dependence of $X_3$ on both $X_1$ and $X_2$ makes the average derivative very difficult to estimate, as we will show.\n\nWe fit kernel ridge regression (with an RBF kernel) of $Y$ on $X$ in the training set. Then we also fit a once-boosted kernel ridge regression on the training set. For each we compute estimates of the average derivative of the model under the three Q distributions. The nominal (not boosted) krr estimate *badly* undercovers: with $n = 500$, the $95\\\\%$ coverage for the three target estimands is $49\\\\%$, $70\\\\%$, and $31\\\\%$ respectively. Our boosted estimator achieves coverage $93\\\\%$, $95\\\\%$, $92\\\\%$ respectively. In this challenging non-linear setting the gap in performance is actually much larger than in the linear ridge setting we originally submitted. (The final submission will include full figures for this experiment, which we cannot include in the rebuttal.)\n\n## Real Data Experiment\n\nFollowing reviewer feedback, we also applied this approach to a real data example using American Community Survey (ACS) data, where the goal is to estimate counterfactual outcome means across a vector of ages. In this setting, $Y$ is income, $A$ is age, and $X$ are covariates like occupation, marriage status, and place of birth. Let $f(A,X) = \\mathbb{E}[Y|A,X]$. Our estimands are $\\mathbb{E}[f(65, X)], \\mathbb{E}[f(66, X)], ... \\mathbb{E}[f(90, X)]$. This vector of estimands represents the \"age profile\" of total income in retirement keeping $X$ fixed. This profile is an important input in many macroeconomic models (see, for example, the seminal paper from Kaplan and Violante [2]), especially for modeling consumption in retirement. \nEach of these estimands has substantial covariate shift when focusing on ages over 65. \nTypical modeling here would construct separate semiparametric efficient estimators for each point in the age profile, necessarily requiring a separate debiasing term at every age.\nWe can instead use our theoretical results to motivate a common debiasing procedure for the entire vector of estimands: in our example we fit kernel ridge regression to estimate $f(A,X)$ and then use one round of boosting. Finally, we use the boosted $\\hat{f}$ to compute a semiparametric-efficient estimate of every point on the age profile with corresponding confidence intervals.\n\n| Age | Naive Estimate | Boosted Estimate | Naive C.I. Width | Boosted C.I. Width |\n| --- | -------------- | ---------------- | ---------------- | ------------------ |\n| 65 | 61.1k | 59.5k | 1.7k | 1.8k |\n| 71 | 60.6k | 58.8k | 1.7k | 1.8k |\n| 77 | 58.5k | 57.6k | 1.6k | 1.8k |\n| 83 | 55.1k | 56.5k | 1.5k | 1.8k |\n| 89 | 50.8k | 55.9k | 1.4k | 1.9k |\n\nWe give the results (in thousands of US dollars) for 5 evenly-spaced points on the age profile in the table above. Whereas the naive estimate (without boosting) features a steep decline of \\$11k from ages 65-89, our boosted estimates are substantially flatter --- better matching the theoretical model for pension and social security income from Kaplan and Violante. Furthermore, while the naive confidence intervals shrink for the highest ages, our boosted confidence intervals actually grow slightly, suggesting that the naive model may undercover for the oldest part of the age profile. \n\n[2] Kaplan, Greg, and Giovanni L. Violante. \"A model of the consumption response to fiscal stimulus payments.\" Econometrica 82.4 (2014): 1199-1239.\n\n# Reviewer-Specific Response\n\nWe appreciate your detailed feedback on our submission. We hope that the generalization from linear functions to more flexible functional forms addresses your concern over the class C. We provide responses to some of your specific comments below.\n\nRegarding Weaknesses 2 and 3, we agree that the proof techniques are not fundamentally new. One of our main goals in the paper is to connect the two very separate literatures on multiaccuracy/multicalibration and semiparametric efficiency.  We believe this has lead to some exciting and novel research directions.\n\nFor Theorem 1, our goal was to establish that once-boosted ridge satisfies the multi-accuracy criterion. While the broader multi-accuracy literature often uses ridge boosting, it is usually performed with multiple boosting iterations and with an exponential weighting-type update. We wanted to verify that our simpler ridge boosting procedure satisfies the same multi-accuracy property. We will update the text to further clarify this point.\n\nFor Theorem 2, the result indeed follows from applying Chernozhukov et al. (2023) together with Singh (2024): the novelty lies in recognizing that a single model (once-boosted ridge) satisfies the requirements of Chernozhukov et al. (2023) simultaneously for many different estimands. In particular, the debiased estimator outlined in Chernozhukov et al. (2023) requires estimating two nuisances: one for the outcome and one for the particular estimand of interest. It then guarantees semiparametric efficiency for that one estimand. Our Theorem then shows that once-boosted ridge provides semiparametric efficiency for *every* estimand in which the corresponding $dQ/dP$ or Riesz representer is in the Hilbert space ball $\\mathcal{C}$.\n\nFinally, we appreciate you pointing out the issues in Weaknesses 4-6. We have focused on tightening notation in our revision. The probabilistic limit on $dQ/dP - c$ and $\\Psi(Q)$ in Theorem 2 are both typos that we have fixed."}, {"review_id": "8QdA2KzUdB", "reviewer": "Reviewer_wM69", "summary": "The paper considers the problem of estimating the mean outcome $\\mathbb{E}_Q[Y],$ in a target population $Q$, denoted given labeled data $(X_i, Y_i)$ from a source distribution $P$. The authors adopt the \\emph{target-independent learning} setup introduced by Kim et al. (2022). In particular, Kim et al. (2022) establish an interesting connection between multi-accurate predictors (a concept in fair machine learning) and this distribution shift problem, showing that multi-accurate predictors are ``universally adaptable'' meaning that they achieve low bias for any target distribution $Q$ for which the auditing class $\\mathcal{C}$ that well approximates $dQ(x) / dP(x)$. \n\nThis paper expands upon the connections between multi-accurate predictors and causal inference. The main result establishes a numerical equivalence: when the auditing class $\\mathcal{C}$ is a Hilbert ball, then the multiaccurate predictor can be implemented as a boosted ridge regression $\\hat{\\psi}(X) = \\hat{\\gamma}(X) + X^\\top \\hat{\\beta}$ and consequently it is numerically equivalent to the automatic debiased machine learning estimator (e.g., Chernozhukov et al. 2022). Consequently, the multi-accurate predictor is not only universally adaptable, but it is also asymptotically normal for any target distribution and semi-parametrically efficient. One practical upshot of this connection is that we can construct valid confidence intervals for $\\mathbb{E}_Q[Y]$ based on this multi-accurate predictor using standard techniques as illustrated in their Monte Carlo simulations.", "questions": "Please see my earlier discussion of strengths/weaknesses of the paper.", "limitations": "Please see my earlier discussion of strengths/weaknesses of the paper.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I found the paper to be very well-written, effectively presenting its main result and describing how it builds upon the connections between fair machine learning and causal inference that were initiated in Kim et al. (2022). The paper's main result is ex-post obvious once it is shown, but I certainly had not noticed this nice connection before despite having familiarity with both multi-accuracy and the auto-DML framework. In this sense, the paper provides a nice bridge between fair machine learning and causal inference/semi-parametric theory, which have two active areas at the intersection of statistics and machine learning.\n\nI have one main concern / point of clarification for the authors' main result. The central assumption in the paper is that the density ratio $dQ/dP$ can be well approximated by a linear function of the covariates $X$ (the first condition in Theorem 2). However, as stated in the main text, this assumption appears slightly stronger than those typically made in the Auto-DML literature (e.g., Chernozhukov et al. 2022). In particular, the Auto-DML framework typically assumes that the density ratio lies in the span (or is well approximated) by a (possibly growing) class of nonlinear transformations of the covariates $\\frac{dQ}{dP}(x) \\approx \\sum_{j=1}^K \\delta_j \\phi_j(x), \\quad \\phi_j \\in \\mathcal{F}$, where $\\mathcal{F}$ is a dictionary of basis functions (e.g., polynomials, splines, neural networks). I was thinking through the simple example where $X \\sim N(0, 1)$ on $P$ and $X \\sim N(1, 1)$ on $Q$ in which case the density ratio is $dQ / dP(x) = \\exp(x - 0.5)$, which is clearly not linear in $x$.\n\nOf course, the feature space $X$ in this paper could be re-interpreted in a broader view -- for example, we could think of $X$ as already the non-linear bases typically used in the Auto-DML framework. Indeed, I looked in the Appendix and the authors already noted this, and so I would strongly encourage them to move this into the main text as a remark. Of course, this resolution also highlights a limitation of this connection --- for this construction, the researcher must pre-commit to the feature map ex-ante, rather than trying to learn a sufficiently expressive representation.", "was_revised": true, "final_justification": "The paper provides a nice connection between multi-accuracy and semi-parametric statistics. The authors addressed my main concerns during the rebuttal.", "rebuttal": "# General Response\n\n## Expanded Results:\n\nWe appreciate reviewers' thoughtful engagement with our submission. To streamline exposition in our initial submission, we characterized the results in terms of linear functions of $X$ and focused on the estimand $\\mathbb{E}_Q[Y]$. In revisiting our technical results, we realized that we can substantially generalize both the class of functions and target estimands we consider --- with essentially no changes to the proofs at all. We believe that these revised results are responsive to reviewer questions and greatly expand the scope of our paper.\n\n* **Beyond linearity in $X$.** First, our results hold exchanging linear functions of $X$ for linear functions in features $\\phi(X)$, which is a remarkably broad class of functions. This class includes when $\\phi$ are the infinite-dimensional features from an RKHS; in this case the resulting algorithm would be once-boosted kernel ridge regression. This class also includes adaptive bases that are selected via sample splitting, such as honest random forests and ridge regression on the last-layer embedding of a pre-trained neural network. \n\n     An important limitation is that the results using our proof techniques are not fully general and do not hold for adaptive bases without sample splitting. That said, there are other examples where boosting also implicitly estimates a density ratio that do not rely on ridge regression or sample splitting. One prominent instance is random forests, which Lin and Han [1] show implicitly estimate a density ratio. In our revised manuscript, we adapt their results to show that, under certain restrictions, random forests satisfy the requirements in our Theorem 2; and therefore that boosting with random forests enjoys the same properties as boosting with ridge. While we leave a thorough discussion of this to future work, we think this is a very exciting direction that opens up many promising research avenues. \n    \n\n* **More general estimands** Second, we show that our results continue to hold when we replace the estimand $\\mathbb{E}_Q[Y]$ with any linear functional of $\\mathbb{E}[Y|X]$, including average derivatives, average impulse responses, average treatment effects and more. To do so, we modify the class $\\mathcal{C}$ to contain the Riesz representers for these functionals: a special case is the density ratio $dQ/dP$, which is the Riesz representer for $\\mathbb{E}_Q[Y]$. Again, we achieve this generality without changing the proofs. \n\n     One important upshot of this generalization is that we can simultaneously consider a vector of estimands if each has a Riesz representer in the set $\\mathcal{C}$. Many applied problem estimate a vector of estimands, and existing semiparametric estimators require fitting a separate debiasing model for each individual point in the vector. Our once-boosted ridge procedure only requires fitting a single model, which can then be used to construct the whole vector of estimates. This is practically very appealing, although a thorough assessment of this is beyond the scope of the current paper, especially with respect to valid joint inference. In particular, in follow up work we will better articulate conditions under which the 95\\% confidence band over the whole vector is simultaneously valid. \n\n[1] Lin, Zhexiao, and Fang Han. \"On regression-adjusted imputation estimators of the average treatment effect.\" arXiv preprint arXiv:2212.05424 (2023).\n\n## Difficult Average Derivative Simulation:\n\nTo demonstrate our results in a more general setting, we consider a non-linear simulation where we estimate the average derivative across multiple distributions (combining distribution shift and a new estimand). For these initial results, we retain a synthetic setup because we would like to show coverage of the confidence intervals. We choose a setting with 3 covariates: $X_1, X_2$ are iid $N(\\mu,1)$, $X_3 = 4 \\cdot \\sigma(X_1 - X_2) + \\epsilon - 2$, where $\\sigma(\\cdot)$ is the sigmoid, and $\\epsilon \\sim N(0,2)$. For the training distribution $\\mu = 0$.\n\nThe outcomes follow: $Y \\sim f(X) + \\eta$, where $f(X) = X_1 \\cdot( 0.2 + \\sin(X_1) + \\sigma(X_2) - 0.2 \\cdot X_3)$ and $\\eta \\sim N(0,2)$. The estimand is the average derivative under $Q$, $\\mathbb{E}_Q[ \\partial f(X) / \\partial X_1 ]$. We consider three different distributions for $Q$. Each is identical to the setup above but changes the value of $\\mu \\in \\{-1,0,1\\}$. The dependence of $X_3$ on both $X_1$ and $X_2$ makes the average derivative very difficult to estimate, as we will show.\n\nWe fit kernel ridge regression (with an RBF kernel) of $Y$ on $X$ in the training set. Then we also fit a once-boosted kernel ridge regression on the training set. For each we compute estimates of the average derivative of the model under the three Q distributions. The nominal (not boosted) krr estimate *badly* undercovers: with $n = 500$, the $95\\\\%$ coverage for the three target estimands is $49\\\\%$, $70\\\\%$, and $31\\\\%$ respectively. Our boosted estimator achieves coverage $93\\\\%$, $95\\\\%$, $92\\\\%$ respectively. In this challenging non-linear setting the gap in performance is actually much larger than in the linear ridge setting we originally submitted. (The final submission will include full figures for this experiment, which we cannot include in the rebuttal.)\n\n## Real Data Experiment\n\nFollowing reviewer feedback, we also applied this approach to a real data example using American Community Survey (ACS) data, where the goal is to estimate counterfactual outcome means across a vector of ages. In this setting, $Y$ is income, $A$ is age, and $X$ are covariates like occupation, marriage status, and place of birth. Let $f(A,X) = \\mathbb{E}[Y|A,X]$. Our estimands are $\\mathbb{E}[f(65, X)], \\mathbb{E}[f(66, X)], ... \\mathbb{E}[f(90, X)]$. This vector of estimands represents the \"age profile\" of total income in retirement keeping $X$ fixed. This profile is an important input in many macroeconomic models (see, for example, the seminal paper from Kaplan and Violante [2]), especially for modeling consumption in retirement. \nEach of these estimands has substantial covariate shift when focusing on ages over 65. \nTypical modeling here would construct separate semiparametric efficient estimators for each point in the age profile, necessarily requiring a separate debiasing term at every age.\nWe can instead use our theoretical results to motivate a common debiasing procedure for the entire vector of estimands: in our example we fit kernel ridge regression to estimate $f(A,X)$ and then use one round of boosting. Finally, we use the boosted $\\hat{f}$ to compute a semiparametric-efficient estimate of every point on the age profile with corresponding confidence intervals.\n\n| Age | Naive Estimate | Boosted Estimate | Naive C.I. Width | Boosted C.I. Width |\n| --- | -------------- | ---------------- | ---------------- | ------------------ |\n| 65 | 61.1k | 59.5k | 1.7k | 1.8k |\n| 71 | 60.6k | 58.8k | 1.7k | 1.8k |\n| 77 | 58.5k | 57.6k | 1.6k | 1.8k |\n| 83 | 55.1k | 56.5k | 1.5k | 1.8k |\n| 89 | 50.8k | 55.9k | 1.4k | 1.9k |\n\nWe give the results (in thousands of US dollars) for 5 evenly-spaced points on the age profile in the table above. Whereas the naive estimate (without boosting) features a steep decline of \\$11k from ages 65-89, our boosted estimates are substantially flatter --- better matching the theoretical model for pension and social security income from Kaplan and Violante. Furthermore, while the naive confidence intervals shrink for the highest ages, our boosted confidence intervals actually grow slightly, suggesting that the naive model may undercover for the oldest part of the age profile. \n\n[2] Kaplan, Greg, and Giovanni L. Violante. \"A model of the consumption response to fiscal stimulus payments.\" Econometrica 82.4 (2014): 1199-1239.\n\n# Reviewer-Specific Response\n\nThank you for your very helpful feedback on our submission. Following your advice, we have updated the main text to work in a basis expansion $\\phi(X)$, which could possibly be infinite dimensional. We think this has made the paper much stronger, and many other reviewers were also concerned about modeling density ratios as linear. In fact, we originally used the multivariate Gaussian example with different means in the main text --- so like you say, this is technically a mis-specified example (and maybe this explains why we actually over-cover slightly in Figure 1). Our new simulation setting with kernel ridge boosting avoids this mis-specification angle. Thank you for pointing this out!\n\nWhen applying our ridge-regression-specific arguments, we believe that your observation highlights the core split between when our results hold or not: the basis must be chosen in advance. This can include representations that are learned in separate samples, but then we must run once-boosted ridge regression on that learned representation.\n\nInterestingly, while revising our main argument to apply to basis expansions $\\phi(X)$, we noticed that the Lin and Han (2023) paper mentioned in the general response shows a new avenue for achieving the same result we show in Theorem 2 but via once-boosting with a random forest. In practice, researchers in the multicalibration literature seem to prefer auditing with tree classes, and so this is a possibly important generalization for connecting the two literatures. Their argument involves characterizing random forests as a linear smoother, and shows that the implied weights of that smoother implicitly approximate the density ratio from the training distribution to the test distribution on which the random forest is applied. In future work we hope to explore whether or not the argument holds for more general classes of linear smoothers. \n\nAgain, we are grateful for your thoughtful engagement with our paper and believe that the revised version is stronger as a result."}]

Author rebuttal:

Meta-review: The authors illustrate how boosting estimators for improving multiaccuracy can be viewed as semiparametric efficient estimators of the subpopulation means across an auditing class, under the assumption that the auditing class is a Hilbert space. Reviewers found the connection established in this work between multiaccuracy and semiparametric theory/double machine learning to be insightful. As a reviewer mentioned, the connection can be obvious after the fact, but far from obvious initially.

I congratulate the authors on this interesting work and recommend acceptance of its work. I urge the authors to take into account the feedback from reviewers, particularly the suggestion to include additional empirical evaluations using real-world datasets.
Decision: Accept (spotlight)