Multi-Accurate CATE is Robust to Unknown Covariate Shifts
Authors: Christoph Kern
Michael P. Kim
Angela Zhou
Venue: Accepted by TMLR ()

Abstract: Estimating heterogeneous treatment effects is important to tailor treatments to those individuals who would most likely benefit. However, conditional average treatment effect predictors may often be trained on one population but possibly deployed on different, possibly unknown populations. We use methodology for learning multi-accurate predictors to post-process CATE T-learners (differenced regressions) to become robust to unknown covariate shifts at the time of deployment. The method works in general for pseudo-outcome regression, such as the DR-learner. We show how this approach can combine (large) confounded observational and (smaller) randomized datasets by learning a confounded predictor from the observational dataset, and auditing for multi-accuracy on the randomized controlled trial. We show improvements in bias and mean squared error in simulations with increasingly larger covariate shift, and on a semi-synthetic case study of a parallel large observational study and smaller randomized controlled experiment. Overall, we establish a connection between methods developed for multi-distribution learning and achieve appealing desiderata (e.g. external validity) in causal inference and machine learning.

Multi-Accurate CATE is Robust to Unknown Covariate Shifts
Christoph Kern"
Department of Stalistics Mariiiiimaimmiiiins-. Universily of Munich Munich Center for Machine Learning (MCMT.
Michael Kim'
mpk@ess.ornelllett
Department of Computer Science
Cornell University
Angela Zhou"
zhoua
Department Data Sciences and Operations University of Southern California
Reviewed on OpenReview: https: // openreview. net/ forum? id- VOGLT6270b
Abstract
Estimating heterogeneous treatment effects is important to tailor treatments to those indi- viduals who would most likely benefit. However, conditional average treatment effect pre- dictors may often be trained on one population but possibly deployed on different, possibly unknown populations. We use methodology for learning multi-accurate predictors to post- process CATE T-learners (differenced regressions) to become robust to unknown covariate shifts the time of deployment. The method works in general for pscudo-outcome regres- sion, such the DR-learner. We show how this approach can combine (large) confounded observational and (smaller) randomized datasets by learning a confounded predictor from the observational dataset, and auditing for multi-accuracy We show improvements in bias and mean squared error in simulations with increasingly larger covariate shift, and on a semi-synthetic case study of a parallel large observational study and smaller randomized controlled experiment. Overall, we establish a connection be- tween methods developed for multi-distribution learning and achieve appealing desiderata. (e.g. external validity) in causal inference and machine learning.
the randomized controlled trial.
1 Introduction
Causal inference studies how to make the right decision, at the right time, for the right person. Extensive recent. literature on helerogeneous treatment effects, also called conditional average teetmentt effects (CATE), studies the estimation of personalized causal effects, rather than only population-level average treatment effects. Estimating CATE can inform better triage of resources to those who most benefit in healthcare, social services, e-commerce, and many other domains.
In these consequential domains, many fimms/decision-makers face treatment decisions where other firms also need to make the same decision, although perhaps each with slightly different data distributions. For example, problems of clinical risk prediction, such as risk of heart disease medication treatment guidelines, are
*Atthors listed alphabetical order.

---

shared widely across hospitals, but each has its own distribution of patients in addition to idiosyncratic reporting, testing, and treatment patterns that can hinder external validity (Caruana et al. 2015). Indeed, off-tho-shelf, relatively simple clinical risk calculators developed on one population are often broadly deployod as a decision support tool in many locations, without the ability to share the originating individuua-level data, or with data drift over time. In social settings, the Arnold Public Safety Assessment (PSA), trained a proprictary dataset and used in hundreds of jurisdictions (Goel et al. 2021), is example of a widely deployed tool. Its accompanying (leciionnmakking matrix is another example of treatment recommendation rule made more widely available (Laura and John Arnold Foundation 2016). which can introduce disparities and poor treatment efficacy (Zhou 2024). Other examples include the design of algorithmic profiling in active
labor market programs (Crépon and Van Den Berg 2016 Bach et al. 2023 Körtner and Bonoli 2028): many
different jurisdictions run different active labor market programs, and policymakers face questions about how to learn from what works elsewhere and how to scale up programs across heterogeneous locations.
A key challenge in these settings is to certify valid predictive performance of personalized causal effects for unknown deployment settings, each of which could have a different largel covariate distribution. For example, predictive risk calculators, such as those for chronic heart discase, lcarned on a specific population might induce biased estimation for different locales with different populations. As one example, the widely used Framingham risk score overestimates risk for Asian populations (Badawy et al. 2022]. This problem is not limited to earlier risk scores, but also affects modern ones: a sepsis predictive risk score provided by Epic, a major healthcare IT provider, fell short in a study of external validity on another population (Habib et al.
2021
External validity, generalizabilltyy and transportability are also important questions for causal inference Tip-
2014 Tipton and Hartman 2023 Barcinboim and Pearl 2013 Heterogeneous causal effect estimates might also be similarly learned on one population, but made more widely available, hence vulnerable to unknown covariate shifts. [Spini 2022 studies the potential impacts of shifts in population for generalizing
ton
results from the Oregon Health Insurance Experiment, while Shyr et al. (2024) studics potential shifts in effect heterogeneity across multiple cancer studies.
On the other hand, we do want to leverage predictive information when it is available. How can we develop methods for heterogeneous treatment effect estimation so that new hospital, without its own large database or in-house machine learning team, is still assured guarantees of low predictive bias on its own population, that might differ in unknown ways from a proprietary risk score that does not publish the original data? Importantly, while some methods guarantee validity of prediction models for a known target distribution, our previous examples highlight important cases when we want to guarantee low prediction bias under mony unknown target distributions.
In this paper, we show how methods from multi-accurate learning Hebert-Johnson et al. 2018 Kim et al. 2019) can endow conditional average treatment effect estimation with robustness to unknown covariate shifts. Indeed, the problem of confounding itself is a covariate shift problem, from the treated or control popula- tion to some target population (Johansson et al. 2022). ) Multi-accurate lcarning is a powerful and flcxible framework that, by ensuring low predictive bias over a test function class, is also robust to combinations of these covariate shifts: those induced by confounding or unknown covariate shifts in the target popu- lation. Although multi-calibrated and accurate learning originated from fairness motivations for ensuring calibration //ow prediction bias over rich subgroups, in this work we show how the adversarial test functions in the formulation also confer broad robustness against covariate shift. To highlight this flexibility, we use multi-accurate calibration on an extremely small clinical trial to correct a predictor from an confounded observational study.
Though there is extensive work on establishing external validity and transportability of causal effects, most of this work assumes information about target population. Drawing inspiration from Kim et al. (2022). which studied "universal adaptability" of averaged outcome estimators with bias robust to unknown covariate shifts, we learn CATE estimates that will maintain unbiased predictions under unknown target populations.
2

---

Although causal inference and machine learning has witnessed significant methodological innovation either in orthogonal/statiisicc. learning or other machine learning adaptations (Kenndyy 2023 Nie and Wager
2020
Chernozhukov ct al., 2018 Wager and Athey 2018 Shalit et al. 2017 Hill 2011). to name just a fow,
muliiaccurate learning (Kim et al. 2019) introduces a different met hodologial loolkit related to boost-
ing/ /ogaaverraaral formulations of conditional moment conditions (Dikkala et al. 2020; Bennett and Kallus
Ghassami al. 2022 We conduct thorough empirical study comparing finite- and largo-sample perlormance of multi-accurate learning and other causal machine learning techniques more specifically tai- lored for causal structure. To summarize, we find that multi-accurate methods grant additional robustness against unknown covariate shifts while being competitive with more advanced causal machine learning meth- ods in finite-samples. There is a robustuess- tradcoff: the latter methods are designed to exploit in-distributtin efficiency, which mulli-accurate learning "off-the-shell" does not. Nonetheless, our work con- nects these two previously unrelated lines of work and shows how multi-accurate learning "off-the-shelf can address the problem of robust CATE estimation. Multi-accurate learning reduces prediction bias from model misspecification, just as is required for conditional average treatment effect estimation.
2023
In our thorough empirical study we find that our proposed multi-accurate T- and DR-lcarner perform well under unobserved covariate shift. Although our work does not suggest multi-accurate learning as a replacement for state-of the--rtttt--ttttoe causal machine-learning for in-distribution estimation, due to differences in variance reduction or hyperparameter selection subroutines, does provide evidence that could inform further methodological improvements and variance reduction of multi-accurate learning for CATE estimation. In summary: Multi-accurate learning can be used "off-the-shell" to post-process CATE estimates based on differenced outcome regressions to endow them with robustness to unknown covariate shift. Such post- processing can be appcaling because it can work with only black-boz access to predictors and original data. (Our earlier examples indicate many situations where only black-box access to predictors is available). Alternative approaches to robustness against unknown shifts, like distributionally robust optimization, could change the robust-optimal predictor to a risk-sensitive one rather than the true CATE, but multi-accurate learning does not.
The contributions of our work are the following.
We propose multi-accurate post-pro peat-procesinn of differenneedregrresionn (T-learner) and psoudo-outcome (the doubly-robust score in the DR-learner) based CATE estimation to obtain unbiased prediction on unknown deployment populations. This approach can also flexibly adapt to a variety of covariate shifts from confounding adversarial/ estimator that combines large observational/smal randomized data. We theorctically establish identification.
unngarooill unnnown shifts: we illustrate by postprocessing a CATE
In Proposition 3 we show that multi-accurate post-processing of simple CATE estimates (T-learner) with a richer test function class can approximate a les--mll-aacurrde//sssstoos but more- advanced CATE estimator, i.c. a multi-accurate DR-lcarner under a simpler test function class. This establishes theoretically favorable estimation properties: that our method not only provides robust bias control, but is also comparable to (weaker implementations of) CATE estimators that pursue efficient estimation. This is an important contribution since the multi-accuracy framework alone only guarantees robust bias control.
We show in extensive experiments with simulations and real-world observational and randomized data from the Women's Health Initiative how our approach achieves finite-sample gains in ensur- ing robust bias control (and correspondingly, MSE) under unknown distribution shifts, Hence we empirically verify our theoretical contributions.
we introduce the formal problem setup. In Section g we describe the background context and most closely related work to our methodological developments (other related work is discussed in Sec- tion G). We develop and analyze our methodology in Section In Section 5. we provide extensive empirical
In Section 2
3

---

comparisons in simulated data and a case study of the Women's Health Initiative parallel clinical trial and observational study.
2 Problem setup
We overview the problem setup and describe directly related prior work on multicallbratoon/mmliaaaaaaa See Scction for discussion of other methodological approaches.
The dataset D ((X,, T,, (((T))) comprises of covariates, treatment. {0, 1}, and (potential) outcomes (T).
Data.
In different applications it will satisfy different assumptions, so we will later define different variants of D. We first assume it arises from a randomized controlled trial or observational study under the assumption of weak ignorability, so that the following assumption about selection on observables holds. Assumption (Unconfomndedness (ignorability)).
(I), (0)) 1
Assumption is generally untestable assumption that permits causal identification. For example, it holds in randomized trials by design, and in observational studies if the observed covariates are fully informative of selection into treatment. Later on, we will jointly consider access to both large-scale observational study (with potential violations of unconfoundednes) and a small randomized trial.
We denote treatment-conditiona. outcome regressions, and the propensity score as:
(x) E|y X = T, = ex)) P(T t X = x)
These are the typical so-called nuisance estimation functions used in common estimators. We use A,, & denote estimates.
Throughout also assume standard assumptions of consistency, SUTVA, and overlap.
Assumption 2 (Consistency, SUTVA, and overlap). We assume that Y; = Y; (T;) (consistency and SUTVA), and that there erists > such that v S (x) < 1 11-v.
Finally, we have a mild assumption that outcomes are bounded. Assumption (Bounded outcomes). We assume that (((I)).(Y()) S B.
Many prior works have also assumed bounded outcomes, including but not limited to Kennedy (2020); Dudik et al. (2014). Relaxing the assumption of bounded outcomes is mild technicality: it is used in the convergence analysis of multi-accuracy
sufficient but not necessary condition and as condition for Chernoff inequalities, which could be replaced with stronger concentration inequalities under weaker assumptions. See also Devroye and Wisc (1980): Gayraud (1997 for canonical estimators for estimating the support of a distribution.
Estimands. The common estimand in causal inference is the average treatment effect, (ATE) (1((1)- Y()).. In regimes with posited heterogeneous treatment. effects, that predictable given co- variates X, (functional) estimand of interest is the conditional average treatment effect (CATE)
7(X) F
x].
Performance assessment. The convention for benchmarking estimation of CATE is the mean-squared error (MSE) with respect to the true T(X) CATE function:
E[(((x) T(X))31.

---

Setting 1: External Validity.
Trained on covariate distribution P(X)
CATE estimation (T / DR-Learnrr)
as-if randomized (X,T.))
MC-Boost Auditing
Different (unknown) deployment
covariate distributions Q
Setting 2: Observational Randomized Data
small RCT (X, (X, data
((X)
biased CATE estimation (T DR-Learner)
observational (X, TX,Y
MC-Boost Auditing
Unknown Q(X)
Figure 1: Schematic of setting 1 (external shift), and setting 2 (learning from large observational and small RCT data). We propose multi-accuracy (MC-Boost) auditing as an "off-the-shelf" procedure to improve the downstream robustness of CATE learners to unknown covariate shifts both settings.
Further, estimators for CATE will all eventually involve different regressions that implicitly minimize pre- dictive error marginalized over the dataset's distribution of ~ Px-
Later on, our work will focus on providing guarantees on conditional bias achieved by a CATE estimate marginalized under a target covariate distribution X Qx that can be different from the distribution X upon which the CATE estimate was trained:
(Bo((f(X--7XX))
(1)
The bias is of course component of the MSE: multi-accuracy methods provide guarantees on the absolute bias; later in Section 5 extensively empirically evaluate the mean squared error well.
We write Qx(x)) Px. Px,, Px. for the marginal distribution of X under Q, the marginal distribution of X under P, and Px,, Px. under X T 1, T = on the observed data, respectively. We also denote Ep, [t], E.. to denote marginalization over X in the training data, X T = 1, or X T = 0, respectively. For brevity we write Eql:] to denote expectations under the unknown target distribution Qx on X (which can be extended to accommodate shifts beyond the typical covariate shift assumption in slight abuse of notation). For example, 44 arg min, Frxx -a--i((X---- 9(X))²) e.g. by default, regression in each treated minimizes the MSE under the covariate distribution of each treatment arm.
Notation conventions. When, for example describing the multi-accuracy criteria without reference the dataset's distribution, we write E[:] when referring the distribution of the training data.

---

We next introduce the shift scenarios (i.c. combinations of assumptions) under which we seek guarantees on CATE estimation. See Figure for an informal illustration.
2.1
Robustness to unknown deployment shifts
Setting 1 (Unknown external covariale shifls)). Suppose Assumplion 1 that unconfoundeehness holds, and
Assumption 2 Consider valid likelihood ratios with respect to the marginal distribution of X in observational data, Px:
C1 = dQx(x) PP
dQx(=)
VE s.t. P(x) > 0) = dQx(X)
= 1
(x)
[T)
< <0,,
(X),
Lo = dQx(a) [xxx(=) xx)x(())
dQx(m)
< <0, 8.1. P(x) > 0} = 1, FI dQx(x)
27xx (x) = =1}
We seek an estimator t(X) with low bias under Q [E] x(X))l c.
L, is the set of valid covariate shift likelihood ratios, such that Qx is absolutely continnous w.r.t Px,, ie. the likelihood ratio is finite whenever Px, has nonzero support (almost surely, with probability 1 under the mcasure Px,). and marginalizes to 1. We will call this type of unknown deployment shift an "external shift." This is analogous to the unknown shift setting si udied in Kim el al. (2022). ; as well as other literature on unknown covariate shifts (Jeong and Namkoong, 2020 Subbaswanyy et al. 2021 Hatt et al., 2021). In contrast to an extensive litcrature on transportability and external validity, we focus on the case of a-priori unknoun deployment shifts.
If suitably nonparametric CATE estimation indeed recovered the Bayes-optimal predictor in finite samples, there would be no issue of unknown deployment shifts. But because in finite samples it generally does not, modifying estimation to protect against unknown deployment shifts can protect against misspecification and finite-samlle issues. For example, misspecified CATE estimation is vulnerable to unknown covariate shift. Furthermore, the conventional mean-squared error MSE can be nonzero for the Bayes-optimal predictor Pi (X)= (1) X]. If the conditional bias variance in Y heteroskedastic (i.e. varies in æ), the predic- tion MSE changes as external shifts change the marginalizing covariate distribution. Under misspecification and nontrivial residual variance of the Bayes regressor, different weighting functions can indoed change the population minimizer (Shimodaira, 2000: Buja et al., 2019 Later we will use multi-accurate learning to post-process CATE estimates to ensure robustness against covariate shifts represented by function class of likelihood ratios.
2.2 Unobserved confounding: observational data with RCT
We consider different setting where unknown covariate shifts may arise: large observational dataset and small randomized trial. The observational study may be subject to unobserved confounding, On the other hand, the sample size of the randomized data may be small, so that learning conditional causal effects solely from randomized data is unsupported. This regime is common in clinical settings, such as the parallel Women's Health Initiative observational study and clinical trial (Machens and Schmidt-Gollwitzer 2003 see also [Colnet et al. 2020 Yang et al., 2020 and (Bareinboim and Pearl 2013) for identification results for the related setting of data fusion.
The data sctting is follows. The observational datasct may have been collected under unobscrved con- founders, Dis, = (X, U, T,Y), but we only observe Dobe (,, T, Y). Hence unbiased causal estimation is not possible from the observational dataset alone. On the other hand, we also have a randomized controlled study, Dres- = (Xr, Tr, Yz). We summarize our assumptions about the sample size, and shift regimes of snteraationa/raandmizze
data below. The punchline is that multi-accuracy provides robustness against these potentially unknown shifts, under assumptions well-specification of the test function class. (All shifts are in the "causal", rather than "anti-causal" setting).

---

In the below setting, we aim to learn valid CATE estimator (1 X of the observational study or additional unknown covariate shifts.
for the covariate distribution
g Suppose an obsercational dataset Dobs, collected under violations of Assumption [ (the observational data were collected under unobserved confounders), and a randomized dalasel Dyet, where Assumplion 7 holds (the randomized dala are uncon- founded). For covariate shift, consider likelihood ratio functions with respect to a target population Q. Again we seek an estimator F(X) with low bias under Q: [Eol((())- - -(X))l S.
Setting 2 (Observational and randomized study). Assume Assumption
Assumption 4 (Small RCT, large observational study). [Dona| > [Dretl
Assumption is not necessary for identification, but it describes the relevant regime where the method is helpful: if instead [Dret] >> [Dosel, unbiased CATE estimation is possible from the randomized data alone.
3 Background and related work
Conditional average treatment effect estimation. We briefly discuss a few options for estimating т, upon which we will build multi-calibrated approaches in differing shift scenarios. The T-learner differences two regressions for the conditional mcans of Y for treated and untreated:
(x) (1(x)) - (o(x))
Implicitly in the definition of these methods, both of these basic approaches for CATE estimation learn predictive models pe (x) by minimizing the mean-squared error, evaluated over some distribution of covariates X. Namely, for the T-learner,
(x(x) E arg min Ep - (xx) (0.1)
s(x)eg
Although many other advanced machine learning and causal inference methods have been developed based advanced estimating equations (Nie and Wager 2020 Kennedy 2023 Oprescu et al. 2019: Wager and
Athey 2018 Semenova and Chernozhukov 2021] D, or other machine- maachnn-learning adaptations Shalit et al. 2017 Shi et al. 2019), we will first instantiate our post- post-processing method with the T-learner and describe how
it can be used with more advanced methods based on pseudo-outcome regression (such as the DR-learner; Kennedy 2020 Semenova and Chernozhukov 2021).
Our meta-algorithm based post-processing CATE estimation (the T- DR-learner) with algorithms for multi-calibration. Next, we describe multi-calibration and its prior use for universal adaptability.
Universal Adaptability via Multicalibration. Recent work of Kim et al. 2022 introduced the concept of universal adaptability. Much work on inference under (external) covariate shift assumes that the shift is known al the time of estimation. Instead, universal adaptability builds a prediction function from source dataset whose average outcome estimate incurs small bias on any downstream covariate distribution, within a broad class of unknown shifts. The work of Kim et al. (2022) ostablishes the fcasibility of universal adaptability via a connection to the notions of muuli-callibrttonn mulllcaalbraaiionmmmliaaaaaa mat.o//mmmmiiiaccrrcyy. originally introduced in the literature on algorithmic fairness (Hébert-Johnson al. 2018). Following this line of research, we show how multi aacuicc can be used, off-the-shelf, to address unknown (external and internal) shifts in the context of CATE.
The multi-calibration criterion was originally motivated to provide guarantees over a variety of subpopu- lations, such as valid calibration over arbitrary subgroups (Hébert-Johnson et al. 2018). The related, but somowhat weaker, notion of multi-accuraccy ensures low prediction bias within ar arbitrary subgroups (Kim) et al. 2019). Throughout this paper, we focus on multi-accuracy (although analogous results hold for the stronger criterion of multi-calibration).

---

Definition 1 (Multt-accuuracy). For c(X) in set of functions C, predictor 5 X - [-B, is (C, multi-accurate if
max <EC
E(Y ((x) So,
Interpretations depend on the specification of the function class C. When C class of subgroup indicator functions, = {I[z E C]: C' C}. with C' a set of subsets of X, then the multi-accuracy criterion ensures low prediction bias over rich set of subpopulations. The class C could indicate sublevel sets of functions with a finite VC-dimension. For example, if C is the space of all decision trees of depth 4, it has a finite VC-dimension and can describe complex subpopulations.
A growing line of work has developed algorithms with guarantees to (approximately) satisfy multi-calibration and multi-accuracy criteria (Hébert-Johuson et al. 2018 Kim et al. 2019 Gopalan et al.) 2022b; Pfisterer can be viewed
et al. 2021) via boosting. When initialized from scratch, learning algorithm, but it can also be used to post-process given predictor, we do in this paper. Our meta-algorithms leverage these existing algorithms for obtaining multi-accurate prodictors by post-proccssing.
Specifically, we use the MCBoost algorithm (Pfisterer et. al. 2021 pseudocode included in Algorithm 4 MCBoost (Piiterrer et al. 2021) takes as input a given initial predictor p, test-function class C, approzima- tion parameter a, and post- proc-processing datasets Doost for calibration and validation. For historical reasons, because the codebase was developed initially for multi-calibration which requires discretization of the in- terval 1], we standardize the prediction range, under assumption without loss of generality. Later on in our meta-algorithmss to be concise will refer to this running MCBoost(p, C, a. Dpos). As a brief summary, MCBoost is a boosting procedure that proceeds via series of anditing steps: given the initial predictor p; it then solves a least-squares problem over the calibration dataset to find a c E that maximizes correlation with the residuals. The process iterates until the total miscalibration or accuracy error drops below a stopping criterion. Next, we discuss the auditing step of this procedure in more detail.
This pre-processing is completely
Auditing. The auditing step solves subproblem of finding subgroup with worst-case multi-accuracy error (bias).
Definition 2 (Multiaccuracc auditing). Let a > 0 and H. be a test class of functions. Suppose D Dpost ~ D is a set of independent samples. A predictor p : X + B- B] passes (a)-multiaccuraayy auditing if for c' E arg mineen ((((p()))(()) Y)- 1(x))?,
IE[(Y (())) a
In practice, evaluation of the multi-calibration or multi-accuracy criterion over discrete subgroups is im- plemented via regression, reminiscent of twicing (Tukey al.] 1977). "Twicing" is a one-step boosting procedure: given predictor function p(xr), fit. the residual errors c* arg mincec F[(((Y p(X)) c()))² and return p(x) + +c* cx) as the final prediction. (In the conditional moment literature, these audit test func- tions would be called instrument functions). That is, the algorithm often audits over real-valued functions c(x): X' + R. These can be connected to the subpopulation motivation by viewing c(x): X' + [0, 1] as a relaxation of indicator functions; and real-valued functions as a rescaling of the former. (Later on, we will relate the real-valued weight functions directly to IPW weight functions (i.e. Riesz representers) in causal inference estimators).
The MCBoost algorithm incorporates these worst-case subgroup error functions to improve the predictor. Given a predictor Pε(x)) at some iteration & of the algorithm, the auditing step learns a test function that best correlates with the residual function p=( y. For auditing over regression prediction functions, for step k this is a regression problem: the auditing step solves mineec E[((Y pply-ppxXY (X)) - c(X)) The predictor is then updated with a multiplicative- weights update based on the worst-case test function c²(x), that is, the next predictor Pk+1 (x) is upweighted or downweighted based on the predicted error e (x(x), Pa+ (x) x A (u)/2 (c)/2.p((xx), where Ag the multi-accuracy error computed on a validation set, Ev[c" (X)cc(X)(YY (X))]
8

---

Therefore the MCBoost procedure adjusts the original predictor based on the predicted errors (c* adjusting more strongly in regions of higher error, as well as if the predictor incurs higher error on average. Auditing and postprocessing occurs in a different held-out datasct: we will refer to this as the post-processing dalasel, including both calibration and validation sets. If the multi-accuracy criterion is not met for this test function c c C, the algorithm takes boosting step and adds a multiplicative update with this test function. If the multi-accuracy criterion is met, the algorithm terminates.
Remark 1 (Relation to conditional moment, restrictions). A render in causal inference or economebrics may notice connections to conditional moment formulations. We expect that our later analysis, which is focused on cuutti-calirratonnaaaeea /accuracccccccyycccooooooo algorithms. also hold for adversarial formulations of conditional moments. (For ezample, Greenfeld and Shalit (220)) observes that adversarial moment conditions, in their case HSIC for independence of residuals, imply robustness to covariate shift)). Recend advances in machine learning for conditional moment equations (Dikkala et al.) 2020; Bennett and Kallus, 2023 Ghassami et al.) (022) typi- cally develop min-maz estimation algorithms that are unstable in practice; besides, the theory of conditional moment restrictions typically identifies finite-dimensional poramelers rather than enlire functions like the CATE. (See Section 2 for more extensive discussion of related work.)
Multi-accuracy vs. multi-calibration. Throughout this paper, we have focused on a weaker multi- accuracy criterion (Kim et al. 2019), which nonetheless sufficient for the robustness guarantees we seek for external validity and robustness to shifts. On the other hand, the stronger criterion of multi-calibration is somewhat more often studied in the literature. In this paper, we do not investigate the calibration properties of heterogeneous treatment effects, focusing instead on robustness to eaternal shift. On the other hand, these are not completely unrelated: with slightly different notions, Wald et al. (2021) | studies connections wetween multi-domain calibration and out out-of-dde of-distribution generalization (without a focus on causal estimation). Other papers have investigated calibration for heterogencous treatment effects (Xu and Yadlowsky 2022). including
orthogonalized isotonic regression
VVan Der Laan et al., 2023).
4 Method
In this section, we develop and analyze our meta-algorithm that uses multi-accurate post- post-processing on the black-box regression models for a T-learner of the CATE. We first describe the meta-algorithm. To justify its use, we start with basic results focusing on ATE estimation. First we show how regression adjustment with multi-accurate outcome models approximates an augmented doubly-robust estimator (AIPW), in the absence of unobserved confounding. (This relates to estimation properties of our final approach). Second, we discuss the inget-independeet identification properties of ATE estimation, under the potential presence of unobserved confounding, which relates to our guarantees under unknown covariate shifts.
Identification implies we can write our target causal estimand in terms of functionals of the observational distribution alone. Estimating the nerresion-aajistmen--ddi indetiiiinide functional with malti-accuratee post- processing will result in certain equivalences with other estimation approaches, such as double robustness, with corresponding function classes. We establish results for the different settings we consider by combining these properties.
We then discuss the specific settings of unknown covariate shifts (external validity), observa- Hional/randonied data, and CATE estimation therein. Finally, natural question remains: we established the properties of a multi-accurate T-learner, but what about more complex approaches? We provide exten- sions to recently studied pscudo-outcome regressions and prove approximate equivalence of a multi-accurate T-learner with a "weaker" multi-accurate DRllearner, suggesting that expanding the test. function class can capture some of the improvements of a more complex estimation target.
9

---

Algorithm Multi-accuracy for CATE estimation for Setting unknown covariate shifts
1: Input: D = {(X,, T, Y,) unconfounedd data, F auditor function class, G function class for outcome
functions.
2: Split D into Dest and Posst
Fit trealment-conditionaa outcome functions from the observational dataset Dasti
pe(x) t arg min (l(g(X) - Y)² T= t), for t {0,1]
(())E9
4: Post-process (X( for t E {0.1} by multi-accuracy: [+(x)) + MCB,ost(ftt 0t, Di Poss), where Ppost is the subset of D post where I[T = 1).
so that ja [Eplf(X) (Y ((()) = Sa.
Return f(x) رo (x) (x)
4.1 (Meta)- Alggotth::: Multi-accurate post-processed T-learner
We describe the meta-learner in Algorithm 1 We learn CATE estimates based on the T-learner (i.c. dif- ferencing outcome regression models). We post-process the outcome models with multi-accuracy. Then the multi-accurate CATE estimate is: 7(X) Tm (X) (X(
(2)
It also admits a natural regression-adjjstment estimate for the ATE:
E[+(X) = Elū (o(X)).
(3)
This represents estimating the ATE by imputing potential outcomes via regression: hence its name regression adjustment in the statistics literature, the "direct method" in the computer science literature, etc. Notably, the estimator does not explicitly use propensity scores. Nonetheless, we show how the robust test functions in multi-accurate auditing can nonetheless approximate inverse propensity score functions.
4.2 Warmup: Multicalibration, universal adaptability, and the ATE
We introduce propertics of multticallbratiionmmltiti /mmttimacuuauacyyy algorithms for estimation of the CATE and ATE. We begin by establishing that regression adjustment estimation with multi-accurate outcome predictors can approximate an AIPW estimator with certain nuisance functions. Later on, we combine these estimation properties with robust identification of the ATE and CATE under external covariate shifts.
Robust estimation of the ATE under unconfoundedness: regression adjustment with suffi- ciently post-processed multi-accurate outcome predictors = doubly-robust estimation. As a warm-up, we first consider estimation when unconfoundodness holds. We show in the causal context, that can be viewed as finding a boosted predictor whose marginalization satisfies estimating equations for the average treatment effect. In addition, multi-calibe culliicalibration/ tattsi/o/mmutiiccee mmulti-acccrrayy as an algorithmic scheme exponds the functional complexity of the original predictor it is initialized with. Inter- wwwaiiaattemmlliiaaaaaeoooo
estingly, regression adjustment with a robust estimator for the ATE, and hence is consistent. if either the original predictor is well-specified, the inverse propensity score is within the auditor function class, or if the prediction function is within the exponded function class output by multi-calibration/ htwpa//rrattooommllllaaccrraaaaa mmutii-accuuacy.
predictor approximates the doubly-
The doubly-robust augmented inverse-propensee weighting estimator (AIPW) a canonical estimator high- lighting improved estimation opportunities for causal inference (Robins al. 1994). It has the following
10

---

form, for a given outcome and propensity model p,
IT i| ex())
(L - Y()
E E (E(0,1)
(Y -x(()) + mi(x)
It enjoys improved estimation properties, such as the mixed-bias property (only requiring one of outcome or propensity model to be consistent for consistent cstimation of the ATE) or rate double-robustness.
The approximation relies on conducting multi-accurate learning with an auditor function class containing the inverse propensity score. As note, the below statements hold up an additional misspecification error (as shown in Kim et al. 2022 ). Because the auditor function class is typically large (i.c. contains functions
beyond the inverse propensity score), this is "robust" way to conduct doubly-robust estimation.
Proposition (Multi-accuracy implies robust estimation of the ATE via regression adjustment). Suppose 7 to 3 Consider an auditor class H. that is closed under affine transformation. Assume unconfoundedness holds. Consider the estimator [[f(X)] where F(x) is the output of Algorithm ! with auditor class H. approzimation parameter a, initial outcome model estimators (r)(x), fo()), and Dostt from the same distribution as the data.
Assumptions
If at least one of the following is true: (1) the original outcome models (z(x),, (z(x) are consistent estimators, (2) (1 ex)) H. or (3) if using multi- accuracy, the true (1)),,, po(x) are in the linear span
of 9 + conu(H), then
E E (o(X)) (1) - Y 20,
i.e. we obtain 2a-consistent estimation of the ATE
Although identification of the ATE with doubly-robust. estimator is standard, note that our estimator that uses multi-accurate outcome models does not explicitly estimate the propensity scores. Nonetheless, the novelty of Prop. is that averaging multi-accurate outcome models, because multi-accuracy implics E((Y- - The (x()))((X)) < vh E H leads to direct approximation of a doubly-robust estimator with the mulli-accuraaee regression- based estimator E (X) - (x()) under conditions on how the test function class relates to inverse propensity score functions. Because this novel perspective, we establish it for ATE estimation first before showing how it applies the robust covariate shifts that motivate our method.
This proposition connects the of multi-aceurate estimation to doubly-robust estimates and therefore establishes variance reduction properties, which is important because the multi-accuracy criterion itself is characterized via bias reduction on subgroups alone, without directly discussing the mean-squared error or estimation variance.
Although the beneficial estimation properties AIPW are well-established (Robins et al. 1994 Kennedy 2016), recall that our multi-accurate T-learner was not explicitly constructed as an AIPW estimator. It is therefore an example of outcome-regression based approach that nonetheless satisfies doubly-robust estimating equation properties, in the spirit of (Bang and Robins 2005), but algorithmically different. This additional perspective on how multi-accurate learning may robustly approximate doubll--obusi estimator is new in our paper, beyond previous analysis of universal adaptability. It also relates in spirit to recent work studying connections between regression adjustment and IPW methods, as (Chattopadhyay and Zubizarreta, 2023) does for linear regression. We build on this characterization in Proposition 3 to relate our later developments to doubly-robust CATE estimators.
Robust farget-independeet identification and estimation of the ATE under universal adapt- ability. Next we show how estimation with multi-accurate learning can robustly identify and estimate the ATE under potential violations unconfoundedness. This is a re-interpretation of "universal adaptabil- ity" in Kim et al. 2022)) which studied missing data under unknown shifts, which directly implies robust identification for causal inference.
11

---

If we had known the true propensity score function 1/e, U), we would obtain identification with respect to the observable marginalization of E(1/e1(X,U) x]:
E 1] X, X, XX, = E|YIT- = 1X]T(X))
(5)
where in the first equality we apply ignorability conditional (X,U) and iterated expectations obtain identification via the observable marginalization of
= (X))e(XXU) x].
(Note that 1/e1 + (X X] due to Jensen's incquality). Robust identification of the ATE follows from Equation 5 i.e. that H contains (approximately) (X, U) X]. Essentially, assuming that the auditor function class contains the (unknown) identifying weight, we can re-interpret the multi-accurate criterion as approximation of adversarial IPW. We summarize this in the following corollary.
Corollary 1. Suppose Assumptions [ and Suppose that (x)),, (X) H. Run Algorithm up to (a)-multiaccuracy on D (possibly with unobserved confounders) over auditor function class H and outcome function class G to oblain F(X) M1 (X) (X)) Then
[[]] (1) Y()) 2a
(6)
The result follows from the multi-accuracy criterion, which implies that |E[
S a
which obtains identification as in eq- (5) and the triangle inequality.
Of course, we have not gained identification for free: we cannot verify the assumption that (x:)., (1) H from observational data alone, just as we connot test the unconfoundedness assumption from data alone. However, multi-accuracy/ httpr/uraaa/mmntiaaaaiiiaaa /multi-calibration methods already work with quite flexible function classes, which could be nouparametric (RKHS, etc).
This is how multi-accuracy confers general robustness to distribution shift, whether from the data generating process such unobserved confounders, from external covariate shifts at the time of deployment.
4.3 External validity: unknown deployment shift
Identification under Setting 1 The robust identification argument for "universal adaptability" re-
interprets the test functions c(X) E C as potential adversarial likelihood ratios for distribution shift.
However, the same properties of multi-accuracy also imply robustness to external shift. In this subsection, we indeed suppose Assumption D unconfoundedness. Recall that our goal was to control the predictive bias on a target covariate distribution Qx, potentially unknown, [Eq[(+(X)= -(X))] Note that cach fr, Po are learned on a treatment conditional distribution, so we have that the valid likelihood ratio, which we denote wy (z), is defined as:
dxx(x) dQx(x) P(T= (x(x) dPx (x) (x) ex(x)
Obtaining robust identification for "universally adaptable" CATE function instead interprets adversarial test functions as a product function class F= C x H for both the subpopulations that identify CATE, and the adversarial likelihood ratio function. Our next proposition gives conditions on the weight functions w/ E H,t E 13 to satisfy robust CATE estimation under unknown covariate shifts.
Proposition 2. Suppose Assumptions 1 3 Let C denote a lest function class for subgroup membership and H test function class for likelihood ratios. Run Algorithm D for a-multi--ccan accurate postprocessing of the outcome models of T-learner estimate. Then, for all target covariate distributions Q such that the likelihood ratios W1, Wo E H.
Ve(z) C, (FllF(()-(X00)) (()()))(())(())) 2a.
12

---

Algorithm 2 Multi-accuracy for CATE estimation for calibrating CATE on small Randomized Controlled Trial data
Imput: Dobe (X) confounded observational data, Diat (X.T,Y) unconfounded randomized
data, F auditor function class, G function class for outcome functions
2: Fit traatment-conditional outcome functions from the obscrvational dataset:
(e(x) < arg min (x Y)² T f), for EE {0, g(x)EG
For t E {0,1}, multi-accurate learning with Pect validation set, i.e. [z(z)) &- (z)) Dra).
4: Return F(x)= i (x) (T)
Because the guarantee holds for all functions f E F, holds for complex subpopulations c(X) and vacuous likelihood ratios with h(X)= 1, as well as the inverse: complex h(X) and vacuous subgroups (i.e. c(z) = 1). Our assumption is that F is sufficiently well-specified to cover the product of these relevant functions, but are generally agnostic to the precise complexity of its constituent classes C, H. And, in practice, following the algorithmic implementation of MCBoost, we work with auditor function classes such as ridge regression, rather than direct products of subpopulations and other test functions.
Observe that although similar arguments apply, obtaining conditional guarantees for CATE estimation re- quires a richer test function class than for universal adaptability of the ATE alone. This illustrates that the case of learning CATE is indeed statistically harder than that of "universal adaptability" of the ATE, For CATE estimation, need to choose richer auditor function class than would for ATE estimation.
4.4 Observational and randomized data (Setting 2)
(leta)-Aloorithm. In this setting, learn confounded outcome regressions from the observational data. We use the smaller randomized controlled trial data post-processing datasets in MCBoost (the boosting paradigm for multi-calibrated and multi-accurate predictors). In Algorithm 2 describe the meta-algorithm.
Identification and estimation of CATE. Identification and estimation for the CATE follows by in- terpreting the auditing functions c(X) as subpopulations. Achieving multi-accuracy on the RCT data hence identifies the CATE. That is, multi-accuracy assures us that
F[((Y (e(X)))(X) T 1]I S a, Ve(x) EC
and we can evaluate this criterion the unconfounded RCT data. On the unconfounded RCT data, we indeed have that E[Y X,7 = EtY X] so that the T-learner identifies CATE.
The intuition for why our mela-algorithh improves upon directly running the T-learner on the randomized data alone is that we can learn a low-variance, high-bias (due to unobserved confounding) estimate of the true outcome model E[Y (t) X] by outcome modeling on the observational data to obtain Eobs / T = 1, X]. On the other hand, although randomized data is available, the finite-sample estimate of Era(Y be high-variance (though unbiased) under Assumption 4 We do note that the analysis of the boosting algorithm in Hébert-Johnson et al. (2018) is not tight enough to provably show faster convergence from warm-starting on the confounded regressions on the observational data, relative to multi-calibrating on the randomized data alone. However, we show benefits in later experiments.
1,
Identification of target-independent CATE.
In complete analogy to the external shift setting, chang- ing our interpretation of the target functions allows us to infer robustness to external shifts. Multi-accuracy ensures that, for all target covariate distributions Qx such that the likelihood ratios w, (x) E H, e {0,
13

---

running Algorithm 2 with auditor function class F = C x H. results in multi-accurate and deployment-shin robust CATE estimate:
A (()) EC, [Fo((F(X) - ((0)))(()))( 2a.
4.5 Extension to CATE ascudo-outcome regression
A natural question given our work on the T-learner is whether we can provide similar guarantees for estimation-improeed CATE learner, since the T-learner generally does not enjoy any improved estimation properties in causal inference. The causal inference and machine learning literature has developed many improved orthogonllssmmppaaanttinn
efficient procedures such as (but not limited to) the R-learner (Nic and Wager )020), DR-learner (Kennedy 2023), or other machine-learning adaptations (Wager and Athey 2018 Shalit et al. 2017).
Namely, some CATE estimation procodures give pscudo-outcome (O; 1i), where O denotes data tuples, i.e. 0 YY) such that E[ X] = r(X). (It is designated a pseudo-outcome because regressing upon it identifies the CATE or functional of interest, although it is not exactly an outcome itself).
One such pscudo-outcome is the doubly-robust score. Pseudo-outcome regression of it as a CATE estimator was recently studied in Semenova and Chernozhukov
(2021); Kennedy 2020)
(O);,,)) T eX) (1x))(1e((X)) (Y-Ar(X)) + (1(X) (x)
(8)
Regressing upon pseudo-outcooess with favorable properties like orthogonality therefore confers these favor- able properties to the estimated functional, such as improved statistical rates of convergence. Our arguments for external validity can naturally be extended for pseudo-outcome based CATE regression, so long as the pseudo-outcome' conditional expectation is the CATE function.
We multi-accurately postprocess the pscudo-outcome regression step. That is, we learn f such that:
EH(G(O;E - =()))
Next, we instantiate such procedure when the pseudo-outcome is the doubly-robust score.
Multi-accurate DR-learner. We give the algorithm for obtaining multi-accurate DR-learner estimate in Algorithm 3 To summarize: we do need four folds of data (Dia, Di,, D2, D3); the first three for sample- splitting of the muisance estimates and pseudo-outcome evaluation and the last for validation/calibee /calibration for MCBoost. Estimate the nuisance functions on the first two folds Dia; Dis and on D2, evaluate the pseudo- Finally, we conduct post-processing via
outcome value ΦfO;; è, fi) and regress f(x) = E, (@(O; é, A)) == x]. multi-accurate learning upon the DR-lcarner estimate f, obtain a multi-accurate 7.
Again we will interpret the input auditor function class J = C H as product function class of subgroup envelope functions C and likelihood ratios H. (Likelihood ratios are assumed to transport from the marginal distribution of X to the new distribution). Then, (robust) identification of the predictions follows exactly as in Proposition 2
Proposition cstablishes that under specification assumptions, the multi-accurato regression adjustment estimator (robustly) equivalent to the doubly-robust estimator up to approximation error, connecting multi-calibration with doubly-robust estimation. This implies basic (robust) doubly-robust properties of the multi-accurate T-lcarner. We now strengthen this connection by showing that multi-accurate post-processing of the T-learner over richer function class (containing the true propensity score, and additional functions) implies that iA, is also multi-accuralle estimate of the DR-learner over the additional functions.
Proposition 3. Suppose A sssupttions to Suppose that (x)) +- MCBoostl F, Dpost), over anditor
function class FF such that Fe C F, where F. {1, IT=1 x C x H. That is, pr foo comprise an
14

---

Algorithm 3 Mulli-accurate DR-learner (Equation for unknown covariate shift.
Input: (Dia: Dis D2, D3) four independent samples of observations of O, (X,, Tu Y₁) (Dg can be smaller). Auditor function class F, approximation paramcter a.
Learn nuisance functions: Estimate propensity scores on D Estimate outcomes (i) Dis-
3: Pseudo-outcome regression: Construct the pseudo-outcome which takes as input observation O =
A) and muisance functions ft
T (X) ((O:e,p) e1 (()((1-e(X) (Y -Y-Ar(()) + +x(()) - (x(
and regress it on covariates X in the test sample D2. Post-process pseudo-outcome regression: run MCBoost(far; D3) to obtain multi-accurate such that
5: Cross-fitting (optional)
a-multi-accurate T learner. Then
max Fl(s(e,;; ((X)):))))((X)(()))))) c
max ([(F(X)- (x(x))()(xXx)))(X)) s 2a chECxH
That is, the multi multi-accurate T-learner ji Po is, up to 2a additive approzimation error, a multi-accurate DR-learner with outcome model ù. post-processed over the function class C x H.
Proof. Consider a function class richer than that needed for Proposition 2 Define
F = {1 IT= xCxH
ex(X)
Consider a multi-accurate T-learner 7 f01 io where each jú a-multi-accurate over an auditor function class F, that F, F.
Note that
max EHHT( - (x())) T[T (C
= T[T==] (x + + - (o(()) T[7 =0 0) + (x( - - (XX))((X)((X
cx(X)
max exhE(CxH) (X) To( - (XX))((X)(XXII
E (((r-m((x)) (x)) =T= =t c(x))(x)
max (xH(((xH))
ex(X)
(E(0.1)
< 20
by the triangle inequality and multi-accuracy of jú, over the richer function class.
The interpretation is that post-processing simple T-learner for multi-accuracy over richer (yet well- specified) function class can approximate DR-learner that post-processed for multi-aceuracy over weaker function class. The population criterion for multi-accuracy confers some nonparametric robustness to bias over the specified test function class. Although this is a different estimation approach than causal machine learning estimates, we relate them formally here, and investigate empirically and thoroughly in Section 5 So, although multi-accurate post-proccssing of a T-learner appears on its face as a basic CATE estimator, fact, the judicious choice of richer function class for post-processing can approximate a more advanced estimator.
15

---

Interestingly, concurrent with the preparation of this work, Bruns-Smith et al. (2023) study augmented balancing weights and find a certain target-independent property of the augmented estimator related to the universal adaptability of Kim et al. (022). Studying connections further would be an interesting direction for future work.
5 Experiments
We previously provided identification arguments and meta-algorithms for leveraging multi-accurate learning to learn CATE subject unknown covariate shifts. To be sure, modern estimation of CATE prescribes nonparametric estimation that, in the infinite-dtta limit, is immune to external covariate shifts if CATE estimation recovers the Bayes-optimal predictor. Of course, real-world datasets are often smaller so that our methods can improve robustness in finite samples. To illustrate this, we conduct extensive empirical studies, testing our proposed multi-calibrated CATE estimation algorithms in comparison to other CATE learners in simulation scenarios that follow the previously introduced settings unknown deployment shifts (Setting T and observational data with RCT (Setting) 2
5.1 Simulations
For both settings, we simulate data according to pre-specified propensity score functions, true outcome functions and external shift functions, with different degrees of complexity. For the external shift setting (simulation and 1b), we assume access to training data from an observational study without unobserved confounding and a small auditing sample from the same distribution. The test data used for evaluation, however, is externally shifted by deliberately sampling with weights given by the shift function (and different shift intensities) from the original distribution. In this setting, we implement two simulations that differ in the complexity of the true CATE and propensity score functions (simulation la: linear CATE, beta confounding, setting, we assume
simulation 1b: full linear CATE, logistic confounding). In the joint observational access to both a large observational training data set and a small RCT; and an external shift between both data sources. We implement two simulations in this setting that differ whether both data sources (simulation 2a: confounded observational data and RCT) or only the observational training data (simulation 2b: total shift between observational data and RCT) are affected by unobserved confounding. The test data used for evaluation follows the covariate distribution of the observational training data. In simulation 2a, the problem is that of covariate shift alone; while in simulation 2b, the underlying conditional model (true (() X] also changes. Simulation 2b illustrates the usefulness of the framework to simultaneously handle a variety of shifts. We present the simulation framework and comparisons to a broader set of baseline methods in supplementary material D.2).
Methods. In both simulation settings, we use causal forests (CForest- OS) and random forest-based T- learner (T-learner-OS) and DR-learner (DR-learner-OS) trained in the observational training data bench- mark methods. T-learner-OS and DR-learner-OS also serve as the input for post-processing with MCBoost using ridge regression in the auditing data in simulation la and 1b (T-learner-MC--Ridge. DR-learner-MC- Ridge). In simulation 2a and 2b, post- post-processing is implemented with ridge regression-based auditing in the RCT data. In these simulations, we present causal forcsts trained in the RCT data (CForest-CT) as an additional baseline. To prevent overfitting with small auditing data, we regularized multi-accuracyy boosting using small learning rates and a limited fixed number of boosting iterations (see Table 2 in D.2
Comparing to T-lcarner-OS establishes the robustness benefits of our methods. On the other hand, our do- no-harm property holds with respect to the MSE of the best-in-class T-learner Comparing to CForest-OS allows us to assess robustness-effcciency tradeoffs. CForest-OS is representative state-of-the-art method that leverages the causal structure and modifies the estimation procedure of random forests; it is a very strong comparison point, but also very data-hungry. In contrast, our post-processing approach does not modify the estimation procedure. An interesting direction for future work is to achieve the robust bias guarantees of multi-calibration with other rariande-reduced CATE estimators.
16

---

Results. We evaluate the outlined methods with respect to MSE of the estimated CATE in the test data in Figure 2 (external shift) and Figure 3 (observational data with RCT), over different sizes of initial training datasots and different intensitics of covariate shift. The results of simulation Ia (Figure 2n) highlight how post-processing robustifies the initial T-learner and consistently improves over T-learner-OS in scenarios with moderate and strong external shift. When the observational training data is small, the multi-accurate T-learner also outperforms causal forests in these scenarios. With small training data, we see similar im- provements of the multi-accurate DR-learner over DR-learner-OS, As the training data size increases, the naive DR-learner becomes more competitive and post-processing yields smaller gains.
In simulation 1b (Figure 2), the more complex CATE function leads to higher MSE overall, while the previously observed pattern persists: The multi-accurate T-learner consistently improves over the naive T- learner, particularly under distribution shift. Our approach, DR-learner-CC-Ridgge is best settings with strong unknown external shift and small dataset size: it then outperforms both T-learner and causal forest, Larger dataset sizes permit estimation over richer function classes and methods become asymptotically equivalent. We compare our approach to additional baselines, including shift-reweighted causal forests, T- and DR-lcarner in the supplementary material D.2.2
Figure 3i shows that in simulation 2a, learning from both the observational training data and a small RCT via multi-accuracy boosting is beneficial across scenarios. The multi-accurate T-learner and DR- learner considcrably improve over T-lcarner-OS and DR-lcarner-OS and in particular T-learner-MC-Ridee is competitive with CForest-OS. The improvement from multi-accuracy boosting can also be observed when post-processing was conducted with externally shifted RCT data and is similarly prominent for both T- and DR-learner in the "total shift" setting (Figure Bb). In both simulations, learning directly in the RCT data (CF) is only a viable option in the absence of a shift in the covariate distribution in the evaluation distribution (i.e. when only deploying on the smaller RCT population), and can incur considerable error otherwise. For results of additional CATE estimation techniques, see supplementary material D.2.3
5.2 WHI data application
We next present, a case study that draws on data from the Women's Health Initiative (WHI) studies (aachens and Schmidt- Goolwitzelr 2003). The WHI includes a large observational study and clinical trial data to in- vestigate the offectiveness of hormone replacement therapy (HRT) preventing the onset of chronic discases. As the observations study has been suspected lo suffer from various (unobserved) confounding phenomena (Kallus and Zhou, 2018), study how utilizing both data sources in combination via multi-accuracy boost- ing compares to learning CATE from the observational or clinical trial data only. We focus the effect of HRT treatment on systolic blood pressure and use age and ethnicity as covariates. Implementation details and results with an extended set of covariates is presented in the supplementary material D.3
Methods and Results. We subsample the observational data. to train causal forests (CForest-OS) and initial T-learner (T-learner-OS) and DR-learner (DR-learner-OSS, We further sample from the clinical trial data to create CT training data with different sample sizes to post-process the initial T- and DR-learner with DR-learner-
MCBoost using ridge regression ---learnerMMC-Riide solely on the CT training data as an additional (strong) bascline (CForcst-CT). Another sample from the CT data serves as the test set, with which we infer the (unobserved) "true" CATE using elastic net-based R-learner (Nie and Wager 2020) and estimate the ATE as evaluation benchmarks.
We also train causal forests
We evaluate bias of the estimated ATE and MSE of the estimated CATE in Figure 4 Figure +l shows how post-processing an initial T- and DR-learner with CT data. can reduce bias, even if the auditing data is small. We similarly see improvements in MSE when comparing the multi-cccurate learner to T-learner-Os additionally improves over CForest-OS. Training only in the CT data leads to ATE estimates with low bias, but the MSE of CForest- CT is not competitive when model training based on CT data with small sample sizes. Further results are presented in supplementary material D3
and DR-learner-OS in Figure
L-HearnerMMC-Ricee
17

---

modersse
strong
100
MSE (CATE)
60
75
CForest-OS
T-leemer-DS
T-kaamrr MC Aldge
DR DR-Iearmer-0S
20
DR-Hearer-MCC-
2000
3500
5000
500
3500
2000
3500
5000
Train size
(a) Simulation la (linear CATE, beta confounding)
no shitt
modarccee shift
strong shirt
100
MSE (CATE)
75
200
200
CForest-OS
T-icamar-05
50
T-kemmr-MC-BB iigg
1.00
1.00
DR DR-teamer-OS
25
DR Hlarner NC Ridde
500
2000
2500
5000
500
2000
3500
5000
500
2000
3500
5000
Train size
Simulation (full linear CATE, logistic confounding)
Figure 2: Average MSE of CATE estimation by shift intensity and training set size for post-processed (multi-calibrated) T- and DR-learner and benchmark methods in simulation studies (external shift setting).
6 Related work: further discussion
A popular approach for handling unknown shifts is to enforce robustness against family of covariate shifts (c.g. unknown shifts parametrized by unknown covariate shift functions) (Liu and Ziebart 2014 Wen et al. 2014 Chen et al. 2016). The goal is to find a robust hypothesis that maximizes the worst-case prediction risk (for example, squared error) evaluated with respect to unknown shifts within some class of covariate shifts. Parametrizations include distributionally robust optimization or linear basis functions. The work of Greenfeld and Shalit (2020) is motivated differently and penalizes with a Hilbert-Schmidt Independence Criterion loss; they show this implies some robustness to covariate shift. While most of this work in the generic prediction setting, recent work also assesses ATE under covariate shift via distributionally robust oplimization (Subbaswany et al. 2021 use of the marginal sensitivily model for external shifts (Hatl et al. 2021), or variational characterizations of coherent risk measures (Jeong and Namkoong, 2020). Methodolog- ically, some of this work is similar to work in causal inference that studies unobserved confounding under the lens robust optimization adversarial likelihood ratios over some ambiguity set (Kallus et 201 Sa Kallus
and Zhou 2021 2020 Dorn al.] 2021 Zhao al. 2019 Bruns-Smith and Zhoul 20233 Yadlowsky et al.
2018 Tan 2006
Bruns-Smith and Zhou 2023). This highlights the broad simultaneous interpretations of
adversarial weight functions for handling unobserved confounding (in the generation of the data) in addition
to robust. adversarial covariate shifts (in the deployment of the predictor).
The approach based on multi-accuracy boosting, although it can be stated as a similar optimization problem in the abstract, differs from the previously mentioned works in a few important ways: (1) boosting couples the functional complexity of the post-processed predictor and the covariate shift function, and (2) under well-specification of the auditor function class and other conditions, boosting's asymptotic limit the Bayes-
18

---

shift
stong shit
200
200
CForest-OS
200
MSE (CATE)
T-itemer-05
T-iaaer Rigge
DR-Iemmer-OS
DR-Heamer--CC-B
CForettCCT
500
2000
3600
5000
500
2000
5000
Train size
(a) Simulation 2a (confounded observational data and RCT)
shitt
modorate shift
strong shift
200
200
200
Chorest-OS
150
150
150
MSE (CATE)
T-leamer
T-leamerMMC
100
100
100
DR-Ieamer-OS
DR-Heamer-MC-RiGG
50
50
CForest-CT
500
2000
3500
5000
500
2000
3500
5000
500
2000
3500
5000
Train
Simulation 2b (total shift between observational data and RCT)
Figure 3: Average MSE of CATE estimation by shift intensity and training size for post-processed (multi- calibrated) T- and DR-learner and benchmark methods in simulation studies (observational data with RCT setting).
Slas (ATE)
MSE (CATE)
CForsst-S5
T-lemmr-DD
T-leamer Mahaaarr-C--ae -C--id
Di-iemme-05s
CFors:-CT
250
300
150
1000
25c
100
750
1000
CT size
Figure 4: Average absolute bias and MSE by clinical trial sample size in WHI data application
optimal predictor, whereas robust optimization changes the asymptotic limit: typically coherent risk measure. In this sense, we expect that approaches based on multi-aceuracy less conservative within distribution. To the best of our knowledge, the only prior discussion of connections between boosting-style algorithms and distributionally robust oplimization is Blanchet et al. (2019).
Approaches based on multi-calibratiin or multi-accuracy inherently couple the specification of the (expanded) hypothesis class of multi-calibrated predictors along with the specification of covariate shift functions, i.c. the boosting-type algorithm returns a predictor the sum class of the original predictor and the classes of shifts. Approaches for robust covariate shift, to reduce the complexity of the adversary, require additional
19

---

moment constraints satisfied by valid likelihood ratios, i. specifying a sharp set C of only valid covariate shifts. In robust optimization-based approaches to covariate shift, the hypothesis class and class of weight functions can be independently varied. But for multi-calibration, restricting the auditor function classes also simultaneously reduces the functional complexity of the hypothesis class of predictors. Distributionally robust objectives are equivalent variance regularization or control of the tail risk, which couples statistically more difficult control of tail behavior with the control of ambiguous shift functions. Another important point of difference is that the Bayes-optimal predictor satisfies the multi-accuracy criterion, while a Bayes- optimal predictor with heteroskedastie noise may not. satisfy desiderata, of uniform performance implied by distributional robustness. For example, (Duchi and Namkoong, 2018 Example 2) discusses the example of lincar well-specified models where the distributionally robust predictor coincides with the Bayes-optimal predictor; but in cases of model misspecification peaapeeeeccsioon/eeeerreeeaddddd / noise, this may not. be the case. We leave a fmer-grained comparison for alternative work.
Our discussion of the hybrid observational and randomized setting is more to highlight an "off-the-shelf" application of multi-accuracy, rather than the tightest analysis in this setting. Other works use more structure of this hybrid setting, or more heavily modify algorithms (i.e. learning shared representations) (Hatt et al. 2021 Yang et al.) 2020: Kallus et al. 2018b); analogous adaptations with multi-accuracy are interesting directions for future work. See also Bareinboim and Pearl (2013) for a survey on transportability and external validity, Colnet et al. (2020) for a survey on learning from observational and randomized data. In this literature, Wu and Yang| (2023) 2022) among others consider reweighting towards shifted distributions.
Other approaches include Cheng et al. (2023).
7 Conclusion
In this work, we connect multi-accurate learning and conditional average treatment effect (CATE) estimation and show how off-the-shelf multi-accurate learning can be used for CATE estimation that is robust to unknown covariate shift. Although we empirically compare to more "state of the art" cansal machine learning, these methods were designed for different purposes. Important directions for future work include "best- of- both-woddds" guarantees on both robustness and efficiency by improving variance reduction properties of Multi-CATE, A fincr-grained analysis of the statistical implications of algorithmic implementations of boosting could also be relevant, in addition improving hyperparameter luning in the causal setting. In our work, we focus on establishing robustness properties.
We thank the Simons Institute and the Simons Institute Program Causality, where much of the work of this paper was conducted. AZ thanks the Foundations of Data Science Institute.
Acknowledgments
References
Ruben L. Bach, Christoph Kern, Hannah Mautner, and Frauke Kreuter. The impact of modeling decisions in statistical profiling. Data & Policy, 5:e32, 2023. doi: 20.1017/dap.2223.20
Mohammed Abd ElFattah Mohammed Darwesh Badawy, Lin Naing, Sofian Johar, Sokking Ong, Hanif Abdul Rahman, Dayangku Siti Nur Ashikin Pengiran Tengah, Chean Lin Chong, and Nik Ani Afiqah Tuah. Evaluation of cardiovascular diseases risk caleulators for evds prevention and management: scoping review.
BMC Public Health, 22(1):1742, 2022.
Heejung Bang and James M Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, 61 (4):9622-973) 2005.
Elias Bareinboim and Judea Pearl. general algorithm for deciding transportability of experimental results. Journal of causal Inference, 1(1):107 134, 2013.
Andrew Bennett and Nathan Kallus. The variational method of moments. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(3):810 841, 2023.
20

---

Jose Blanchet, Fan Zhang, Yang Kang, and Zhangyi Hu. A distributionally robust boosting algorithm. In 2019 Winter Simulation Conference (WSC), pages 3728- 3739. IEFE, 2019.
David Bruns-Smith and Angela Zhou. Robust eittatin and iteration under sequentially exogenous unobserved confounders. arXiv preprint wxXi:2332.006602 2023.
David Bruns-Smith, Oliver Dukes, Avi Feller, and Elizabeth L. Ogburn. Augmented balancing weights linear regression. arXiv preprint arXiv:2304. 14545, 2023.
Andreas Buja, Lawrence Brown, Richard Berk, Edward George, Emil Pitkin, Mikhail Traskin, Kai Zhang, and Linda Zhao. Models as approximations Statisticol Science, 34(4):523 544, 2019.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1721-1730, 2015.
Ambarish Chattopadhyay and José R Zubizarreta. On the implied weights of linear regression for causal inference. Biometrika, 110(3):613- 629, 2023.
Xiangli Chen, Mathew Monfort, Anqi Liu, and Brian D Ziebart. Robust covariate shift regression. In Arlificial Intelligence and Statistics, pages 1279. PMLR, 2016.
Yuwen Cheng, Lili Wu, and Shu Yang. Enhancing treatment effect estimation: A model robust approach integrating randomized experiments and external controls using the double penalty integration estimator. In Uncertainty in Artificial Intelligence, pages 381 390. PMLR, 2023.
Victor Chernozhukov, Mert Demirer, Esther Duflo, and Ivan Fernandez-Wi Val. Generic machine learning infer- ence on heterogeneous treatment effects in randomized experiments, with application to immunization in india. Technical report, National Bureau of Economic Research, 2018.
Bénédicte Colnet, Imke Mayer, Guanhua Chen, Awa Dieng, Ruohong Li, Gaël Varoquaux, Jean-Philippp Vert, Julie Josse, and Shu Yang. Causal inference methods for combining randomized trials and observa- tional studies: a review. aixiv preprint arXiv:2011.088 100847, 2020.
Bruno Crépon and Gerard J Van Den Berg. Active labor market policies. Annual Review of Economics, 8: 521-546, 2016.
Lue Devroye and Gary I. Wise. Detection of abnormal behavior via nonparametric estimation of the support. SIAM Journal on Applied Mathematics, 38(3):480 488, 1980.
Nishanth Dikkala, Greg Lewis, Lester Mackey, and Vasilis Syrgkanis. Minimax estimation of conditional moment models. Advances in Neural Information Processing Systems, 33:12248 12262, 2020.
Jacob Dorn, Kevin Guo, and Nathan Kallus.
sensitivity analysis for causal inference with unmeasured confounding, arXiv preprint orXiv:2112.11449, 2021.
John Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. arXiv preprint arXiv:18100 08700,, 2018.
Miroslav Dudik, Dumitru Erhan, John Langford, and Lihong Li. Doubly robust policy evaluation and optimization. 2014.
Ghislaine Gayraud. Estimation of functionals of density support. Mathematical Methods of Statistics, 6(1): 26-46, 1997.
AmirEmad Ghassami, Andrew Ying, Iya Shpitser, and Eric Tchetgen Tchetgen. Minimax kernel machine learning for a class of doubly robust functionals with application to proximal causal inference. In Inter- national conference on artificial intelligence and statistics, pages 7210 7239. PMLR, 2022.
21

---

Sharad Goel, Ravi Shroff, Jennifer Skeem, and Christopher Slobogin. The accuracy, equity, and jurisprudence of criminal risk assessment. In Research handbook on big data law, pages 9 9-28. Edward Elgar Publishing, 2021.
Parikshit Gopalan, Lunjia Hu, Michael P Kim, Omer Reingold, and Udi Wieder. Loss minimization through the lens of outcome indistinguishabiltty, arXiv preprint arXiv:2210.0800 08649,, 2022a.
Parikshit Gopalan, Michael P Kim, Mihir A Singhal, and Shengjia Zhao. Low-degree multicalibration. In Conference Learning Theory, pages 3193 3234 PMLR, 2022b.
Daniel Greenfeld and Uri Shalit. Robust learning with the hilbert-schmidt independence criterion. In International Conference on Machine Learning, pages 3759 3768. PMLR, 2020.
Anand R Habib, Authony Lin, and Richard W Grant. The epic sepsis model falls short the importance of external validation. JAMA Internal Medicine, (81(0):1040-100) 1041, 2021.
Tobias Hatt, Daniel Tschernutter, and Stefan Feuerriegel. Generalizing off-policy learning under sample selection bias. arXiv preprint arXiv:2112. 013887, 2021.
Ursula Hébert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the ccomptaatioally--dettfffllllll masses. In International Conference on Machine Learning, pages 1948. PMLR, 2018.
Jennifer T, Hill. Bayesian nonparametric modeling for causal inference. Journal of Compulational and Graphical Statistics, 20(1):217 240, 2011.
Sookyo Jeong and Hongseok Namkoong. Robust causal inference under covariate shift via worst-case sub- population treatment effects. In Conference on Learning Theory, pages 2079 2084. PMLR, 2020.
Fredrik Johansson, Uri Shalit, Nathan Kallus, and David Sontag. Generalization bounds and represen- (ation learning for estimation of potential outcomes and causal effects. Journal of Machine Learning Research, 23166)111-50, 2022.
Nathan Kallus and Angela Zhou. ronfounding-robusst policy improvement. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Biancia and R. Garnett, editors, Advances in Newral Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL hattlhhtps:///. http::///pooceeens.eeeeeerre
pdf
Nathan Kallus and Angela Zhou. ronfoundinggrobust policy evaluation in anfinite-horizon reinforcement learning. Advances in Neural Informalion Processing Systems, 33:22293 22304, 2020.
Nathan Kallus and Angela Zhou. Minimax-optimal policy learning under unobserved confounding. Manage- ment Science, 67(5):2870-22)) 2890, 2021.
Nathan Kallus, Xiaojie Mao, and Angela Zhou. Interval estimation of individual-level causal effects under unobserved confounding. arXiv preprint arXiv:1810.028)) 2018a.
Nathan Kallus, Aahlad Manas Puli, and Uri Shalit. Removing hidden confounding by experimental ground- ing. Advances in newral information processing systems, 31, 2018b.
Edward H Kennedy. Semiparametric theory and empirical processes in causal inference. In Statistical Causal Inferences and their Applications in Public Health Research, pages 141-167. Springer, 2016.
Edward Н Kennedy. Optimal doubly robust estimation of heterogeneous causal effects. arXiv preprint arXiv:2004.1440 144977 2020.
Fdward H Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics, 17(2):3008 3049, 2023.
22

---

Michael P Kim, Amirata Ghorbani, and James Zou. Multiaccuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAT/ACM Conference on AT, Ethics, and Society, pages 247-254, 2019.
Michael P Kim, Christoph Kern, Shafi Goldwasser, Frauke Kreuter, and Omer Reingold. Universal adapt- ability: Target-independent inference that competes with propensity scoring. Proceedings of the National Academy of Sciences, 14)( 2022.
John Körtner and Giuliano Bonoli. Predictive algorithms in the delivery of public employment services, In Handbook of Labour Market Policy in Advanced Democracies, pages 387-398, Edward Elgar Publishing, 2023.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79--8) 86, 1951.
Sören R. Künzel, Jasjeet S. Sekhon, Peter J. Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences, 116(10); 4156-4165, 2019. doi: (0.1073/pmas.180455771166 1804597116.
Laura and John Arnold Foundation. Public safety assessment decision making framework cook county, il [effective march 2016]. https::/news wwt.:wttwwtccoooo.. http://www//eeeeeee//aaaa//////e /ttp/////eeeeeeeee////aa/aaaeeee
P512DDDeciisaa/////aaannn///rrae pdf 2016.
Daniel Lewandowski, Dorota Kurowicka, and Harry Joc. Generating random correlation matrices based on vines and extended onion method. Journal of Mullinariate Analysis, 100(9):1989 2001, 2009. ISSN 0047-259X. doi: http:///ddooor/1/...1/////////// hww.::/.........000000.... 00..000.. URL actarr///rrtcccee///////75766669
com/
Anqi Liu and Brian Ziebart. Robust classification under sample selection bias. Advances in neural information processing systems, 27, 2014.
K. Machens and K. Gooniitt-oolwiizer.. Issues to debate on the Women's Health Initiative (WHI) study, Hormone replacement therapy: an epidemiological dilemma? Human Reproduction, 18(10):1992-1999 2003. ISSN 0268-1161.
doi: 10.1008/hunrrep/deed000
Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108 (2):299-33)) 319, 2020. doi: /a009///imet////// asaa076. URL attps:///oi..rrg/1 http://d....rr/1....11.0//////..
Miruna Oprescu, Vasilis Syrgkanis, and Zhiwei Steven Wu. Orthogonal random forest for causal inference. In International Conference on Machine Learning, pages 1932-4941. PMLR, 2019.
Florian Pfisterer, Christoph Kern, Susanne Dandl, Malthew Sun, Michael P Kim, and Bernd Bischl. meboost: Multi-calibration boosting for r. Journal of Open Source Software, 6(64):3453, 2021.
R. Core Team. R: A Language and Environment for Statistical Computing. R. Foundation for Statistical Computing, Vienna, Austria, 2020. URL htttt:::/ R t-pproeect.oog org/
James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors not always observed. Journal of the Americon Statistical Association, 89(427):846 866, 1994.
Vira Semenova and Victor Chernozhukov. Debiased machine learning of conditional average treatment effects and other causal functions. The Econometrics Journal, 24(2):264-28)) 289, 2021.
Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In International conference on machine learning, pages 3076 3085. PMLR, 2017.
Claudia Shi, David Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effects. Advances in neurol information processing systems, 32, 2019.
23

---

Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelhhood function. Journal of statistical planning and inference, 90(2):227 -244,4 2000.
Cathy Shyr, Boyu Ren, Prasad Patil, and Giovanni Parmigiani. Multi-study r-learner for estimating hetero- geneous treatment effects across studies using statistical machine learning, 2024.
Pietro Emilio Spini. Robustness, heterogeneous treatment effects and covariate shifts. arXiv preprint arXiv:2112.0925: 2021.
Adarsh Subbaswamy, Roy Adams, and Suchi Saria. Evaluating model robustness and stability to dataset shift. In International Conference on Artificial Intelligence and Statistics, pages 2611 2619. PMLR, 2021.
Zhiqiang Tan. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, 101(476):1619-1637 2006.
Julie Tibshirani, Susan Athey, Erik Sverdrup, and Stefan Wager. grf: Generalized Random Forests, 2021. R. package version 2.0.2.
URL https://cRAA www.:ain.-projettccoom www.jject..gg//cccciicc wwwpg///aaaaaggriaaa.
Elizabeth Tipton. How generalizable your experiment? an index for comparing experimental samples and populations. Journal of Educational and Behavioral Statistics, 39(6):478 501, 2014.
Elizabeth Tiplon and Erin Hartman. Generalizability and transportability. In Handbook of Matching and Weighting Adjustments for Cansal Inference, pages 39-60 Chapman and Hall/CRC 2023.
John Wilder Tukey et al. Exploratory data analysis, volume 2. Springer, 1977.
Lars Van Der Laan, Ernesto Ulloa-Pérez, Marco Carone, and Alex Luedtke. Causal isotonic calibration for heterogeneous treatment effects. In International Conference on Machine Learning, pages 14831-34855 PMLR, 2023.
Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113((23):1228 1242, 2018. doi: 10.1080/01621459. 2017.1319839.
Yoay Wald, Amir Feder, Daniel Greenfeld, and Uri Shalit. On calibration and out-of-domain generalization. Advances in neural information processing systems, 34:2215 22052,22772222 2021.
Junfeng Wen, Chun-Nam Yu, and Russell Greiner. Robust learning under uncertain test distributions: Relating covariate shift to model misspecification. In International Conference on Machine Learning, pages 631 639. PMLR, 2014.
Marvin N. Wright and Andreas Ziegler. ranger: A fast implementation of random forests for high dimensional data in C++ and R. Journal of Statistical Software, 77(1):1 17, 2017. doi: 10.18633/jss.v077..00
Lili Wu and Shu Yang. Integrative r-learner of hetcrogencous treatment effects combining experimental and observational studies. In Conference on Causal Learning and Reasoning, pages 904 11-9926. PMLR, 2022.
Lili Wu and Shu Yang. Transfer learning of individualized treatment rules from experimental to real-world data. Journal of Computational and Graphical Statistics, 32(3):1036-100 2023.
Yizhe Xu and Steve Yadlowsky. Calibration error for heterogeneous treatment effects. In International Conference on Artificial Intelligence and Statistics, pages
PMLR, 2022.
Steve Yadlowsky, Hongseok Namkoong, Sanjay Basu, John Duchi, and Lu Tian. Bounds on the conditional and average treatment effect with unobserved confounding factors. arXiv preprint Xiv: 09521, 2018.
Shu Yang, Chenyin Gao, Donglin Zeng, and Xiaofei Wang. Elastic integrative analysis of randomized trial and real-world data for treatment heterogeneity estimation. arXiv preprint arXio:2005.1057 2020.
24

---

Qingyuan Zhao, Dylan S Small, and Bhaswar B Bhattacharya. Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrap. Journal of the Royal Statistical Sociely: Series B (Statistical Methodology), (144) 2019.
Angela Zhou. Optimal and fair encouragement policy evaluation and learning. Advances in Neurol Infor- mation Processing Systems, 36, 2024.
25

---

A Notation summary
Notation/O Objjct
Description
X
Covariates
T
Treatment, T 0,
((T)
Potential outcomes
Y
Observed outcome, Y Y
U
Unobserved confounders
e(x)
Propensity scorc, P(T = x
(x)
Outcome regression, E x
T(x)
Conditional average ment effect (CATE), (T((1) Y(0) x x]
C
Class of subsets of X Function classes
F.9.H
Multi-accuracy parameter
Dobs
Observational dataset.
Dres
Randomized controlled trial dataset
f(x)
Estimated CATE function
F(x)
Multi-cccurate/calirrteed CATE estimator
Table 1: Notation used in the paper.
B Details algorithms
For completeness describe the MCBoost algorithm for multi-calibration. See (Hébert-Johnson et al. 2018 Kim et al. 2019 2022 Pfisterer et al. 2021) for more details, including thcoretical analysis and implementation details. We describe the algorithm for generic (z, y) dataset (without reference to causal inference). See (Kim et al. 2019) for more details on the variant. that achieves multi-accuracy (although ideas at a high level are similar.)
The key imputs include regression algorithm for the boosting procedure, approximation parameter a which is stopping condition (although in practice finite limit the number of iterations is used), and set. When developing methods for Setting Setting[[[nkkown (unknown covariate shifts), the calibration and validation set are drawn from the observational distribution. Our method for Setting 2 uses the (assumed small) RCT data as calibration interattionvaliaatiinnn validations sets.
26

---

Algorithm MCBoosT
Given:
Po X - 00.1]
// initial predictor // regression algorithm for functions in C // approximation parameter
A: (X x 1), C
a >>0 0
$ = {(11,))) (12, v2), (Jims Vmn } = (L01,81) (22,82),... (Ies y0)}
// calibration set // validation set
Returns:
(C, .CCaa--altt- predictor j
Scale outcomes y [0, 1]
Repeat: h: = 0, 1, 2...
Sy + {(I1, 1/1 P& (x(x1)) ...., (Im, Um Ph (2())))
update labels calibration set // regression St // compute miscalibration over V, validation set
A(Sa)
A. V E d(x) - -p)(x))
(v.))EV
A >> a then
(2) x Acccs//2 (x)
// miltiplicative weights update return when miscalibration small
elsereturn p = Pk rescaling the outcomes
end if
27

---

C Proofs
Proof of Proposition 7 (la) Suppose (2),, (o(r) are consistent estimators and e E H. Then Equation
immediately implies e-consistency.
(1b) Suppose i (a), (o()) are consistent estimators but e & H. If f (2),) are consistent, they will asymptotically satisfy the multi-calibrated multi-accurate criterion. See Hébert-Johnson al. (2018) for related M-no--ham properties in this setting. Let 1 E[y X,
denote the true conditional expectation; it satisfies μ E arg min E px()))² T and that E[Y (X) T 1, 0, a.s.
Hence Vf(X) E F, E((Y PT (x))) | = 1] O. Therefore μt (X) is feasible. Since the additive iterates of boosting approaches like MCBoost for multi-accuracy are commutative, (Gopalan et. al. 2022a) characterizes multi-accuracy via a global optimization of squared loss over additive basis functions of H. Since 14 (X) is optimal solution for the unconstrained problem, and feasible for the constrained problem, it is also optimal for the constrained problem.
(2) Suppose any of Ao are not consistent estimators and e (x)) e H. The implications of multi-accuracy with respect to H. relate to the classical doubly-robust estimator:
Eietaay
E
TT= (Y (a(()) + [x(x)] - (X)] s 2a ex
(9)
By properties of AIPW, the left hand term is consistent due to model double-robustness. By multi-accuracy, the CATE estimator is 2a-close to AIPW under well-specification.
(3) This follows via the same arguments given in (1b).
28

---

Experiments
D.1 Data and Software
We provide code of the simulation studics and the real data application for replication purposes the following public OSF repository:
https://osf..oo/z/////
Data preparations, model training and evaluation are conducted in R (3.6.3) (R Core Team 2020) using the packages ranger (0.13.1) (Wright and Ziegler 2017), grf (2.0.2) (Tibshirani et al. 2021) and rlearner
(1.1.0) Nie and Wager 2020). The simulation studies heavily draw on the causal experiment simulator of the causalToolbox (0.0.2.000) Künzzl et al. 2019) package.
In all experiments, (initial) T-learner and DR-learner are post-processed using the MCBoost algorithm implemented in the mcboost (0.4.2) (fisterrr et al. 2021 package. More concretely, make use of boosting for 2 multi-calibratton, (slightly) stronger notion than multi-accuracy, but computationally less demanding than full multi-calibe calibraiili Gopalan et al. (2022b). The hyperparameter settings used for post-processing are listed as part of the following detailed presentation of the experiments (Table and 13
D.2 Simulations
D.2.1
Setup
Data We follow the simulation setup of Künzel et al. (2019) in designing our experiments. Each of the following simulations is initialized by specilying the following components: Propensity score e, outcome functions M and μi. and external shift function 2. We then simulate the following components:
10-dimensional feature vector,
X X N),
with modest correlations in (governed by alpha of the vine method Lewandowski al. 2009 which is set to 0.1).
Potential outcomes simulated according to the pre-specified covariate-conditioaa. outcome func- tions H and μi,
Y((0) -0(()) + Y(L) = (i())) + +
where & 1).
Treatment assignment is simulated given the pre-specified propensity score e,
T. ~ Bern(e e (X))
and the observed outcome set to Y. = -T.).
set of sampling weights is constructed given the external shift function z (and shift intensity s),
w(()(x) = (1 =(x)
2
and used to simulate externally shifted observational data shift or shifted randomized control
trial (RCT) data, Drat (where (X) 0.5), depending on the simulation scenario.
29

---

We vary the shift intensity E (0,0.25.... 2 and training set size {500, 2000, 3500, 5000), and run experi- ments for each combination 25 times. The size of the (audit/RCT) data used for multi-calibration boosting (500 observations) and the (test) data used for model evaluation (5000 observations) is fixed.
Evaluation We compare and evaluate various techniques with respect to bias in ATE and MSE CATE estimation. Bias is assessed based on the true ATE and the average of the estimated f(x) in the test data.
Bias = T Σ f(x)
We further evaluate the true CATE r(x) against f(x)) of the respective CATE estimation method.
MSF = - Σ (-(x) - A(x)))
To quantify the external shift, we compute the multivariate Kullback-Leible Leibler (KL) divergence (Kullback and Leibler 1951p between the covariate distribution (I1 (11)...10)) of the training set (simulation la, 1b)/ the RCT (simulation 2a, 2b) (NG) and the test set (Ag))
1KL())) = (ur((2)))) + (#1 Bu) 14) -p - In Eol + In E
2
where = E-
MCBoost
Multi-calibration boosting conducted using the hyperparameter settings listed Table 2
Hyperparameter settings of the baseline CATE learner are shown in Table
D.2.2 External Shift
In this initial setting, we simulate data that emulates an observational study with (observable) confounding. We additionally consider an external shift between the observational data that is available for initial model training. Dos: and the distribution of the test (or deployment) data, Dos-shift- We further assume access auditing sample from the original training distribution. The task is to estimate the true CATE function evaluated the shifted test set, using models that either learned in the observational training data only made additional use of the auditing data.
(Xtrains Trrains Ytrain) ~ Does (Xaudit, Taudit, Yaudit) Dany (Xrest, Trest, Ytest) ~ Das- shift
Simulation la (external shift, linear CATE, beta confounding)
I'3 if # < -0.4
(((x)
I'pm if - 0.4 S ø10 S 0.4
3' if 0.4 < 310
with 3r mnff([-5,5]10) 8m unif([- 5115), B_ 5m.
() (x) 3m 5
e1 (x) (1 B (z) (x1,
where 2,x4) is the beta distribution with parameters and 4.
z(x) = 1 el- 0.5) - 2( 0.5
30

---

Table 2: Hyperparameter settings for post-processing using MCBoost. Default settings used for param-
eters not listed.
(a) T-karnnr MC
Method
Implementation
Hyperparameter
Value
Ridge
mcboost
max_iter
alpha
1e-06
0.5
weight_degree
2
glmnet
alpha
Tree
mcboost
max_ _ttr
alpha
1e-06
eta
0.5
rpart
maxdepth
(b) DR-lcarner MC
Method
Implementation
Hyperparameter
Value
Ridge
mcboost
max iter alpha
1e-06
eta
0.1 1
weight_degres
2
glmnet
alpha
0
s
1
Tree
mcboost
max_iter
5
alpha
1e-06
0.1
weight weight_degres
2
rpart
maxdepth
3
Note: eta 0.01 in simulation and D.2.3
Table 3: Hyperparameter settings of (baseline) CATE learners. Default settings are used for parameters not listed.
Method
Implementation
Hyperparamoter
Value
CForest
grf
num. trees
2000
mtry sample.. fraction
sp)
0.5
honesty fraction
0.5
min. size
S-TT,
ranger
num.
500
DR-learner
mtry
sqrt (p)
min. size
replace
TRUE
sample. fraction
31

---

CATE estimation We use the following methods for estimating the CATE based on the observational training data. Shift-reweighting conducted by training a logistíc regression to predict sample membership in the obsorvational training versus shifted test data and calculating propensity weights 1-P based on the predicted probability of membership the training data p.
f
(CForet-OOS) Causal forest. (Waeer and Athey 2018) trained in the observational training data.
WOF) Causal forest trained in the shift-reweighted observational training data.
(S-learner-OS) S-learner using random forest trained in the observational training data.
(S-eearner-wOS) S-learner using random forest trained in the shift-reweighted observational train- ing data.
(DR-learner-OS) DR-learner (ennedy 2023 using random forest trained in the observational training data.
OT-learner-OS) T-learner using random forest trained in the observational training data.
(T-learner-wOS) T-learner using random forest trained in the shift-reweighted observational train- ing data.
We further estimate DR-learner and T-learne using multi-calibration boosting with small set of auditing data.
DR-learner-MC-RRige)) DR-learner using random forest in the observational training data is post-processed with MCBoost using ridge regression the auditing data.
WRR-learner-MC-Tree DR-learner using random forest in the observational training data is post- processed with MCBoost using decision trees in the auditing data.
(T-learner-MC-Ridge) T-learner using random forest in the observational training data is post- processed with MCBoost using ridge regression in the auditing data.
(T-learner-MC-Tree T-learner using random forest in the observational training data post- processod with MCBoost using docision trees in the auditing data.
Evaluation We evaluate bias in ATE and MSE in CATE estimation in the externally shifted test data.
Results We show the bias of the estimated average treatment effect (ATE) by shift intensity (column panels) and training set size (row panels) for each CATE estimation method in Figure (see also Table which includes quantifications of the distribution shifts using KT. divergence). The results show that in the present setting all methods are able to produce unbiased estimates of the ATE in the non-shifted test data (first column). Introducing an external shift (sccond and third column), however, incurs bias across all methods with the shifl-reweighted causal forest and shift-reweighted T-learner performing best. The ridge regression- bagee multi-accurate DR- and T-learner perform best among the shift-blind methods that had no access to the shifted test distribution.
We show the corresponding results for the MSE of the CATE estimation by shift intensity and training set size in Figure 6 (and Table 5). In the present setting, causal forest achieve the smallest MSE in the non-shifted test data as well as settings with large initial training data (first column and third and last row). With increasing shift, however, ridge- multi-accurate T-learner perform best in settings with small to moderately sized training data (upper right quadrant).
32

---

no shift
moderate shift
strong shift
000
CForest-OS
2000
CForest-wOS
S-tearner-OS
S S-learner-wOS
DR DR-learner-OS
DR -R-eearmer-MCRRide
DR Me-eeeeer-MC-Teee
T-leamer-OS
00SE
T-leamer-wOS
T-leamer-MC-Riin
T-leamer-
0005
-15
-10
-5
10 -15
-10
-5
5
10
-15
-10
-5
0
5
10
Bias (ATE)
Figure Bias of ATE cstimation by shift intensity and training set size for different CATE estimation methods (Simulation la (external shilt, linear CATE, beta conlounding). The distribution of bias scores over simulation runs is shown. Given an external shift between training and test data, -edeereee-----iiieeee and T-learner-MC-Riddee perform best among the shift- blind methods that had no access to the shifted target distribution.
33

---

no shift
moderate
strong shift
CForest-OS
0002
CForest-wOS
S-learner-OS
5 S-learner-wOS
DR DR-learner-05
DR OR-terrner-MC-Ridee
DR Me-eeeeer-MC-TTn
T-teamer-OS
ODSE
T-leamer-wOS
T-leamer-MC-Ruse
T-eamer-MMCTree
0005
20
40
50 100
150
100
200
300
MSE (CATE)
Figure 6: MSE of CATE estimation by shift intensity and training set size for different estimation methods (Simulation la (external shift, linear CATE, beta confounding). The distribution of MSE scores over simulation runs is shown. F-Jearner-MC-Riddee performs best in settings with small to moderately sized training data and shifted test data.
34

---

Simulation 1b (external shift, full linear CATE, logistic confounding)
(z) 3.1 + 5.2
(x(x) (x)(x) x 3, with 3 ~ uni
6i (X) + el 2(2) 0.5) 1 (1 0.5))
=(x) 1 (1)
Evaluation Bias in ATE and MSE CATE estimation evaluated in the externally shifted test data.
Results The results for bias of the ATE estimation in Figure (and Table including KL divergence) show that in absence of external shift (first column), causal forest-based estimators perform best and are able to achieve unbiasedness. Introducing an external shift between the observational training and test data (second and third columns) amplifies bias such that only the shift-rewiighted causal forest is able to approximate the true ATE on average, given sufficient training data. The ridge regression- based multi-accurate DR-learner improve over the initial DR-learner and are competitive with shift-reweighted causal forest in settings with small initial training data and strong shift. Again note that, cont.rast to the shift-reweighted methods, the multi-accurate learner had no access to the shifted test distribution during model training.
Figure (and Table shows results for the MSE of the estimated CATE. The ridge-based multi-accurate DR-learner consistently improve over the initial DR-learner and achieve the lowest MSE among all methods in the initial, non-shifted setting. With increasing shift, ridge-based multi-accurate and shift-reweighted
learner perform well in sel tings with small to moderately sized training data. Causal forest is competitive in all settings and particularly as the training set size increases.
35

---

no shift
moderate shift
strong shift
005
CForest-OS
2002
CForest-wOS
S-tearner-OS
S S-learner-wOS
DR DR-learner-05
DR DR-eearner-MCRRide
DR MR-eeaner-MC-Teee
T-leamer-OS
O0SE
T-leamer-wOS
T-leamer-MC-Rlie
T-leamer-MC
0005
-20
-10
10 -20
-10
10 -20
-10
10
Bias (ATE)
Figure Bias of ATE cstimation by shift intensity and training set size for different CATE estimation methods (inullation 1b (exlernal shift, full linear CATE, logistic confounding]. The distribution of bias scores over simulation runs is shown. Given an external shift between training and test data, DR-learne- MC-Ridge performs best among the shift-blind methods, particularly in settings with small initial training data.
36

---

no shift
moderate shift
strong shift
008
CForest-OS
2000
CForest-wOS
S-tearner-OS
S S-learner-wOS
DR DR-learner-05
DR -R-eearmer-MC-Ridee
I
DR Meeeeeeer-M--Teee
T-leamer-OS
OOSE
T-leamer-wOS
T-leamer-MC-Rlin
T-leamer-MC
000S
20
40
160 200
300
100
200
300
400
MSE (CATE)
Figure 8: MSE of CATE cstimation by shift intensity and training set size for different estimation methods (Simulation 1b (external shilt, full linear CATE, logistic confounding].. The distribution of MSE scores over simulation runs is shown. DR-learner-MC-Ridee and "-learerrMCC-Ridee consistently improve over DR-learner-OS and T-learner-OS. DR-llarrer--C--Ridgge performs best overall in settings with small to moderately sized training data.
37

---

D.2.3 Observational study and RCT
We simulate a setting in which we have access training data from an observational study (OS) and from small randomized control trial (RCT). We consider covariate/ external shift. between the observational study, Dosi and the RCT, Dret- We further assume unobserved confounding either in both data sources (D.2.3) in the observational training data only (D..3). The task is to estimate the true CATE using models that learned either in the observational (training) data or in the RCT, or by using both data sources in combination.
(Xtrain; Train Yeroin) ~ Dosi (Xaudit, Toudit, Yandit) ~ Dret, (Xtest: Trests Ytest) ~ Dos
That is, the randomized controlled trial data is crucial to obtain identification, but ultimately we seek a predictor with good performance on the covariate distribution of the observational data.
Simulation 2a (confounded observational data and RCT) the first simulation, we consider co- variate shifts from the observational to the RCT setting alone. Assumption (Covariate shift from observational to RCT).
P(Xobs) + rot)
P(Y obs = y X, U, A) P(Y,a = = X,
In addition the sctup in Appendix D.2 we introduce unobsorved confounding. The specification
follows:
The unobserved confounder U is correlated with Z1:
0.8 if x > œ1
u(x) = 0.2 in
U ~ (ern(((())
Mo(z) = 3x1 + 532
M1(x) = (1(x)) + =3x1 + 502
16 u) [o(x) u vi(x, w) in (x:) + 3u
6 (x, u)
+ e(2 3u+ (-2(21 0.5) (:22 0.5)))
end = 0.5
=(x) 1+ el2 (0)))
CATE Estimation We the following methods for estimating the CATE based on training sets of simulated observational data.
(CForest-OS) Causal forest WWaerr and Athey 2018 trained in the training set of the observational data.
(S-learner-OS) S-leaneer using random forest trained in the training set of the observational data.
(DR-learner-OS) DR-learner (Kennedy 2023 using random forest. trained the training set of the observational data.
38

---

(T-learner-OS) T-learnr using random forest trained in the training set of the observational data.
We estimate DR-learner and T-learner using multi-calibration boosting with simulated RCT data.
DD-llerner-MC-Ridee) DR-learner using random forest in the training set of the observational data is post-processed with MCBoost using ridge regression in the randomized control trial.
DR-learner-MC-Tree DR-learner using random forest in the training set of the observational data is post-processed with MCBoost using decision trees in the randomized control trial.
TTlearner-MC--Ridges T-learner using random forest in the training set of the observational data post-processed with MCBoost using ridge regression in the randomized control trial.
TTlearmer-MMC-reee T-learner using random forest in the training set of the observational data is post-processed with MCBoost using decision trees the randomized control trial.
We further compare to CATE learner that are solely based on the simulated RCT data. Shift-reweighting is conducted by training a logistic regression to predict sample membership in the observational versus RCT data and calculating propensity weights 1- based on the predicted probability of membership in the RCT data p.
(CForest-CT) Causal forest trained in the randomized control trial.
(CForest-wCT)) Causal forest trained in the shift-reweighted randomized control trial.
(S-learner-CT) S-learner using random forest trained in the randomized control trial.
(S-learner-WCT) S-learner using random forest trained in the shift-rewoighted randomizod control trial.
(DR-lcarner-CT) DR-lcarner using random forest trained in the randomized control trial.
(T-learner-CT) T-learer using random forest trained in the randomized control trial.
(T-learner-wCT) T-learner using random forest trained the shift-reweighted randomized control trial.
We evaluate bias in ATE and MSE in CATE estimation on a test set drawn from the ob- servational data. In calculating the true ATE and T(x), we marginalize over U and compute [[{]( ((11) + +3E and E[Y[(0)]X] Y:(0) X:).
Evaluation
Results We plot the bias of the estimated ATE for each method by shift intensity (column panels) and training set size (row panels) in Figure (see also Table g including KL divergence). In the absence of covariate shift (first column), naive learning in the observational data results in biased estimates of the ATE, Utilizing both data sources in combination via multi-calibration boosting allows to improve over the initial DR- and T-learner. Introducing a covariate shift between the observational data and the RCT (second column) degenerates the performance of the RCT-based estimators and the best results are achieved by multi-accurate DR- and T-learner, especially for strong shifts (third column).
Results for the MSE of the estimated CATE are shown in Figure 10 (Table g In the absence of covariate shift (first column), the RCT-based est.mmaooss outperform the istimators that Tearned from the observational data. Introducing shift between the observational study and the RCT (second and third column) increases the MSE the RCT-based learners considerably such that the best results can now be observed for the tree- based multi-accurate T-learner, followed by the ridge regression-based multi-accurate T-learner and causal forests learned in the observational data.
39

---

no shift
moderate
strong shift
-0-
005
CForest-OS
S-learner-OS
DR -R-aenne---O
0002
DR UR-lerrerrMC-RRdee
DR DR-teamer-MMC-Tee
T-leamer--05
T T-leamer-MC-RRide
T T-leamer-MC-Tre
CForest-CT
CForest-WCT
00SE
S-learnr-CCT
S S-learner-WCT
DR -n-tearner-CT
T-leamer-CT
T-lemmerwWCT
10
0005
-00
-20
-10
-20
-10
10
20
-20
-10
10
20
Bias (ATE)
Figure 9: Bias of ATE estimation by shift intensity and training set size for different CATE estimation methods (Sinulation 2a (confounded observational data and RCT]). The distribution of bias scores over simulation runs is plotted. Given moderate to strong covariate shift between the observational data and RCT, multi-aceurate learner achiove the best results.
40

---

no
moderate
strong shift
0DS
CForest-OS
S-learner-OS
DR -RR-earner-OS
2002
DR-MeamnerMMC-Aidde
DR DR-teamer--MCTTee
T-learmer-05
T-leamer-MC-Rlde
T-leamer-MC-Tre
CForest-CT
CForest-WCT
COSE
S-learer-CCT
S S-tearne--WCT
DR -e-eeener-CT
T-teamer-CT
T-leamer-WCT
000S
-0
50
100
150
200
s0
100
150
200
100
200
300
MSE (CATE)
Figure 10: MSE of CATE estimation by shift intensity and training set size for different estimation meth Simulation 2a (confounded observational data and RCT).. The distribution of MSE scores over simulation runs is shown. T-learner-M-Tree outperform other methods in settings with shifted RCT data.
41

---

Simulation 2b (total shift between observational data and RCT) In simulation 2b, we consider potentially stronger distribution shifts beyond covariate shift alone.
Assumption 6 (Total shift from observational to RCT).
P(Xons) + P(Xret)
P(Yoss = A) & P(Y y vIX,A)
The difference between Assumption G and Assumption is whether we allow the marginal distribution of U to shift. Assumption 5 is a "conditional model invariance" assumption between the data-generating process and the RCT. sufficient condition for this to hold is that P(Unts) P(Uret) and the invariant conditional probability assumption above. On the other hand, the total shift of Assumption could arise from shifts in the distribution of U. Both of these additional covariate shifts.
The specification is follows:
(o(x)= = 1ord(1) (x) = 3z + 5x2
(1( = (x) = -|0((>) + 3.1 + 5.02
40 108 (x, u) /2(()) -u
Mi 0o0 (z, u) #n (1(x) 3u
e (x,
1 el2 3a+( 2( 0.5) 1 0.5)))
end(x) (x) = =.5
=(x) 1 c(2(12 (13
Evaluation We evaluate bias ATE and MSE CATE estimation on lest set that follows the covariate distribution of the observational data, Doz However, in constructing the true ATE and r(x) we use (o()) and aa (1) specified in the RCT, i.c. without unobserved confounders.
Results The bias of the estimated ATE for each method by shift intensity (column panels) and training set size (row panels) is presented in Figure 11 (and Table 10 including KL divergence). The first set of results (first column) indicate that under unobserved confounding in the observational data only and without external covariate shift, based estimators are, as expected, unbiased. The multi-accurate DR- and T-learner that draw on both data sources are able to reduce the bias of the naive DR- and T-learner. As the external shift between the observational data and the RCT increases (sccond and third column), learning only the RCT incurs bias and shift- thifl-reweighted methods as well (tree-based) multi-accurate DR- and T-learner achieve the best results.
The results for the MSE of the CATE are shown in Figure 12 Table 11
Similar to the results for bias, the RCT-- estimators perform best under the no external covariate shift setting (first column). Among the estimators based on the observational data, the ridge- and tree-based multi-accurate T-learner and causal forests perform best. Tree- based post-processing performs best among all methods scenarios with strong covariate shift (third column).
42

---

no shift
moderate
strong shift
006
CForest-OS
S-learner-OS
DR -R-aenne---O
0002
DR -R-lerrerr-C-RRdie
DR DR-teamer-MMC-Tee
T-leamer--05
T T-leamerr-C-RRide
T T-leamer-MC-Tre
CForest-CT
CForest-WCT
00SE
S-learnr-CCT
S S-Jearner-WCT
DR -n-tearner-CT
T-leamer-CT
T-leamerwWCT
0005
CD-
-20
-10
-20
-10
20
-20
-10
10
20
Bias (ATE)
Figure 11: Bias of ATE estimation by shift intensity and training set size for different CATE estimation methods (Simulation 2b (total shift, between observational data and RCT)). The distribution of bias scores over simulation runs is shown. As the external shift between the observational data and the RCT increases, multi-accurate DR-learner and T-learnrr-CC MC-Tre are competitive with shift-rewoighted learning.
43

---

no shift
moderate
strong shift
006
CForest-OS
S-learner-OS
DR -R--eeane--OSS
0002
DR DR-learner-MC-Ridg
DR DR-teamer-MMC-Tee
T-leamer--05
T-leamerr-C-RRide
T-leamer-MC-Tiee
CForest-CT
CForest-WCT
ODSE
S-learnr-CCT
S S-Jearner-WCT
DR -n-tearner-CT
T-leamer-CT
T-lemmerwWCT
0005
50
100
150
200
50
150 MSE (CATE)
100
200
100
200
Figure 12: MSE of CATE estimation by shift intensity and training set size for different estimation methods (Simulation 2b (total shift. between observational data and RCT)). The distribution of MSE scores over simulation runs is shown. T-Jearner-MC-111 performs best among all methods in scenarios with strong covariate shift.
44

---

Table Bias of ATE estimation by shift intensity and training size for different CATE estimation methods, averaged simulation runs Simulation la (external shift, linear CATE, beta confounding) For each setting, method achieving best performance printed in bold (second best in italic).
Train size
Shift degree
KL-
CForest
S-learner
DR-learner Ridge
T-learner
Div.
OS
wOS
OS
wOS
OS
Tree
OS
wOS
Ridge
Tree
0.08
0.04
-0.02
-0.02
0.76
0.35
0.34
-0.09
-0.12
-0.15
-0.11
0.25
0.23
1.62
-0.11
2.13
1.87
0.83
1.19
1.39
1.17
0.61
0.64
0.84
0.5
0.65
3.17
-0.09
4.2
3.39
1.92
1.88
1.88
2.34
1.34
1.57
1.8
0.75
1.15
4.46
-0.55
5.96
4.69
2.43
2.65
3.4
3.41
2.17
2.49
2.45
500
1.62
5.26
1.52
7.05
3.69
3.28
3.55
2.77
3.01
3.45
1.25
1.91
5.87
-2.09
7.65
5.92
3.89
3.36
4.42
4.4
2.55
2.77
3.08
1.5
2.09
6.29
-2333
8.1
6.38
2.59
3.68
3.93
5.09
2.46
3.4
3.69
1.75
2.22
6.8
-2.73 7.73
8.45
7.42
4.28
4.91
4.93
5.13
2.26
3.01
4.21
2.40
6.88
-4.08
8.6
7.1
5.01
4.01
4.31
5.17
1.69
3.48
3.58
0.02
0.04
0.04
0.01
0.31
-0.01
0.21
-0.01
-0.01
0.16
0.2
0.25
0.17
0.79
0..4
1.79
1.35
0.46
0.61
0.6
0.74
0.33
0.5
0.58
0.5
0.60
1.8
-0.65
2.42
1.41
1.21
1.35
0.98
1.19
1.24
0.75
1.11
2.52
1.41
5.14
2.87
1.8
1.4
1.92
2.39
1.43
1.77
2
2000
1.52
2.97
-2.08
3.27
1.82
1.91
1.58
-2.11
2.18
1.25
1.86
3.35
-2.12
6.65
3.96
2.35
1.92
2.1
3.26
1.99
2.2
2.52
1.5
2.05
3.63
-2.84
7.07
4.21
2.38
2.19
2.17
3.49
1.92
2.56
2.89
1.75
2.21
3.81
33.27
7.29
4.5
2.29
2.08
2.27
3.59
1.86
2.54
2.84
2.32
3.93
-3.59
7.48
4.55
2.67
2.32
2.52
3.85
1.71
3.01
3.59
0.02
0.03
0.02
-0.05
-0.05
0.28
0.14
0.14
-0.07
-0.07
-0.18
-0.09
0.25
0.16
0.62
-0.27
1.65
1.15
0.39
0.53
0.43
0.66
0.3
0.48
0.57
0.5
0.60
1.31
-0.57
0.94
0.99
0.84
1.03
1.37
0.75
1.10
2.08
-1.22
1.77
2.57
1.5
1.13
1.37
2.25
1.38
1.62
1.81
3500
1.52
2.38
5.64
2.83
1.88
1.54
1.62
1.55
2.29
1.25
1.82
2.59
-2.1
6.19
3.03
1.7
1.51
1.87
2.91
1.64
2.16
2.38
1.5
2.05
2.72
-3.61
6.45
3.28
2.13
1.59
1.98
3.05
1.61
2.19
2.62
1.75
2.20
2.94
-3.18
6.7
3.67
2.18
1.69
1.85
3.23
1.67
2.45
2.73
2.29
3
-3.97
6.87
3.79
1.9
2.28
2.26
3.35
1.49
2.4
2.87
0.01
-0.06
-0.05
-0.11
-0.11
-0.06
0.02
-0.01
-0.13
-0.13
-0.14
-0.14
0.25
0.16
0.52
0.25
1.57
1.07
0.32
0.36
0.41
0.62
0.27
0.49
0.71
0.5
0.59
1.1
-0.54
1.82
0.76
0.7
0.96
1.14
0.75
1.10
1.65
-1.38
4.56
2.28
1.11
1.11
1.39
2.07
1.2
1.56
1.66
5000
1.50
2.02
5.38
2.55
1.4
1.59
1.83
2.24
1.25
1.81
2.34
-1.66
5.91
2.97
1.58
1.69
1.47
2.76
1.69
2.02
2.4
1.5
2.05
2.37
-3.08
6.2
2.94
1.93
1.97
1.54
2.89
1.62
2.3
2.23
1.75
2.21
2.55
-3.24
6.39
3.26
1.91
1.74
1.45
3.06
1.68
2.49
3.14
2.30
2.63
-3.56
6.53
3.7
1.83
1.9
1.98
3.2
1.7
2.38
2.62
45

---

Table 5: MSE of CATE estimation by shift intensity and training set size for different estimation methods, averaged over simulation runs (Simulation la (external shift, linear CATE, beta confounding) For each setting, method achieving best performance printed in bold (second best in italic).
Train size
Shift degree
KL
CForest
S-learner
DR-learner Ridge
T-learner
Div.
OS
wOS
OS
wOS
OS
Tree
OS
wOS
Ridge
0
0.08
18.52
12.5
19.64
19.69
39.11
35.83
34.71
14.99
15.01
23.65
23.34
0.25
0.23
16.22
14.87
25.4
24.31
39.71
38.62
33.19
18.39
17.96
13.76
23.99
0.5
0.65
23.97
17.31
38.24
30.95
49.65
53.33
34.47
22.98
20.9
15.71
28.8
0.75
1.15
33.76
22.6
53.54
39.98
56.69
43.87
48.8
29.32
25.36
21.3
31.88
500
1.62
39.1
33.58
63.77
46.94
58.16
59.52
49.9
34.82
30.83
22.51
36.65
1.25
1.91
44.13
50.24
69.13
47.81
57.9
58.83
60.97
34.24
26.97
22.24
32.55
1.5
47.91
55.52
74.24
51.85
82.21
66.16
56.16
40.65
25.54
24.22
37.26
1.75
2.22
53.07
164
78.5
63.76
60.46
64.53
62.66
39.19
29.42
27.65
41.72
2.40
100.17
80.29
58.83
68.18
63.73
56.24
40.18
24.46
22.7
37.35
0.02
5.06
5.1
14.97
14.92
15.75
15.83
15.31
9.56
9.52
8.5
15.34
0.25
0.17
6.13
5.98
19.28
17.02
16.96
15.4
15.32
10.43
10.35
8.83
15.77
0.5
0.60
10.47
9.33
29.76
20.21
20.28
18.6
17.6
14.83
14.49
11.65
18.63
0.75
1.11
13.19
14.76
40.88
21.47
19.86
18.81
19.15
17.24
15.84
13.61
22.23
2000
1.52
15.44
23.4
47.64
23.43
23.09
21.72
25.05
19.44
17.07
541.18
22.88
1.25
1.86
17.09
30.8
53.67
27.69
25.46
22.39
21.78
21.68
19.04
14.56
25.03
1.5
2.05
18.42
40.07
57.9
28.71
26.72
26.79
24.99
23.04
17.78
16.42
25.81
1.75
2.21
19.23
12.16
59.87
29.85
26.8
22.7
22.47
23.04
16.93
15.57
27.11
2.32
19.99
62.18
30.1
24.72
27.07
23.7
16.63
19.02
29.57
0
0.02
3.2
3.19
12.91
12.87
12.56
11.02
11.25
7.64
7.63
7.24
12.44
0.25
0.16
4.22
4.17
16.63
14.13
12.04
11.28
10.98
9
9.05
7.34
13.86
0.5
0.60
6.57
6.46
26.72
16.98
15.43
13.4
11.72
12.47
12.03
9.51
16.99
0.75
1.10
9.78
12.25
36.13
18.59
14.68
14.92
14.56
15.35
14.25
11.21
17.79
3500
1.52
10.65
14.68
42.75
19.42
17.16
15.14
15.14
16.8
15
12.84
21.89
1.25
1.82
10.83
23.62
47.2
19.39
16.18
16.57
16.01
17.78
14.77
12.98
20.17
1.5
2.05
11.61
32.26
49.25
20.51
18.26
16.88
17.81
18.68
14.49
13.39
23.67
1.75
2.20
12.69
38.13
51.6
22.77
17.92
16.71
16.32
19.75
15.28
15.07
21.18
12.73
56.72
53.34
23.36
21.69
19.04
20.17
15.18
14.76
22.09
0.01
2.62
2.63
12.45
12.36
9.31
8.73
9.23
6.97
6.99
6.14
10.51
0.25
0.16
3.19
3.01
15.53
13.18
9.32
8.49
8.95
7.77
7.73
7.74
12.25
0.5
0.59
4.95
4.58
23.96
14. 14
10.56
9.48
10.36
10.57
10.09
8.2
14.69
0.75
1.10
6.66
9.45
33.15
16.08
12.62
11.07
12.24
13.68
12.92
11.27
16.6
5000
1.50
7.89
11.84
38.96
16.78
13.9
12.46
12.89
14.89
13.63
11.27
18.55
1.25
1.81
9.41
16.77
43.57
18.71
15.24
12.35
13.52
16.02
13.83
22.07
20.03
1.5
2.05
9.06
23.63
45.78
17.95
14.93
13.22
13.17
16.68
13.61
13.18
19.91
1.75
2.21
9.82
29.24
47.46
19.6
14.82
13.09
14.68
17.55
13.89
14.38
22.76
2.30
10.36
40.95
48.85
22.73
15.09
14.9
15.18
18.62
14.32
14.42
19.83
46

---

Table 6: Bias of ATE estimation by shift intensity and training set size for different CATE estimation meth- ods, averaged over simulation runs
Simulation 1b (external shift, full linear CATE, logistic confounding)
For each setting, method achieving best performance printed in bold (second best in italic).
Train size
Shift degree
KL
CForest
S-Jearner
DR- DR-learner
T-learee
Div.
OS
wOS
OS
wO$
OS
Ridge
Tree
OS
wOS
Ridge
0
0.07
-0.59
-0.66
-2.25
-2.27
-2.28
-2.32
-2.42
-4.59
-4.6
-3.73
-4.4
0.25
0.22
-2.94
-1.19
-4.51
-3.91
-3.96
-3.38
-3.88
-6.21
-5.58
-4.39
-5.64
0.5
0.60
-5.21
-1.94
-6.68
-5.78
-5.14
-5.16
-5.36
-7.94
-6.79
-6.71
-8.1
0.75
1.05
-7.23
-2.63
-8.54
-7.44
-6.94
-5.77
-6.43
-9.59
-7.92
-8.42
-9.96
500
1.47
-8.57
-4.95
-10.07
-8.73
-8.58
-6.96
-7.66
-11.09
-9.13
-9.26
-10.79
1.25
1.76
-9.07
-5.13
10.771
-9.44
-6.83
-6.66
-7.65
-11.62
-9.41
-10117
-11.92
1.5
-9.56
-6.54
-1.14
-9.84
-8.59
-6.72
-8.72
-12
-9.49
10.49
12.25
1.75
2.21
-9.71
-5.74
-1.54
-10.19
-8.4
-7.83
-8.57
-12.48
-9.76
-11.2
-2,58
2.35
-00044
-7.48
-11.7
10.16
-9.39
-7.1
-8.24
-2.66
-9.29
10.75
-12.02
0.02
0.18
0.17
2.05
2.06
-1.62
-1.63
-1.8
-3.51
-3.52
-3.3
-3.46
0.25
0.16
-2.01
-022
-393
-3.77
2249
-2877
-3.04
-4.83
-4.28
-4.3
-4.78
0.5
0.53
-3.88
-0.64
-5.6
-4.8
-9.66
-3.81
-3.81
-6.53
-5.42
-6.2
-7.06
0.75
0.99
-3.55
-1.76
-7.44
-6.77
-4.39
-3.9
-4.55
-8.12
-6.72
-7.29
-8.26
2000
1.39
-6.5
-1.8
-8.64
-7.27
-5.98
-5.41
-5.71
-9.15
-7.28
-8.18
-9.26
1.25
1.73
-7.18
2.12
99.55
-8.04
-6.11
-5.29
-5.73
-9.95
-7.68
-8,98
10.53
1.5
-7.6
-2233
88.77
-6.02
-6.41
10.26
-7.87
-0.63
1.75
2.09
-7.8
-3.98
10.05
-8.81
-6.2
-5.67
-6.62
10.55
-8.44
-9.95
-11.77
-7.87
10.19
-8.6
-5.42
-6.29
10.63
-7.7
-9.79
-1.07
0
0.01
0.13
0.16
2.01
-2
-1.41
-1.46
-1.54
-3.17
-3.17
-3.02
-3.15
0.25
0.15
-1.64
-0.01
-3.64
-3.09
2.29
-2.4
-235
-4.37
-3.86
-4.1
-4.39
0.5
0.52
-3.5
-0.4
-5.33
-453
-3.6
-3.77
-3.77
-6.08
-5.04
-5.9
-6.55
0.75
0.98
-4.77
-0.37
-6.87
-5.61
-4.15
-4.01
-4.16
-7.27
-5.73
-6.72
-7.8
3500
1.38
-5.83
-0.64
-8.12
-6.73
-5.24
-4.64
-4.55
-8.57
-6.61
-7.76
-8.58
1.25
1.70
6.39
-0.74
-8.74
-7.25
44.78
-5.2
-6.17
-9.08
-7.02
-8.4
-9.62
1.5
1.92
-0.03
-9.06
-7.42
-6.11
-5.35
-5.37
-9.5
-7.08
-8.98
1.75
2.10
-7
-0.6
99.99
-7.63
-6.05
-5.85
-5.81
-9.68
-7.09
-8.8
10008
-7.17
-9.58
-7.77
-6.13
-6.14
-6.2
10.07
-7.46
-9.93
0.01
0.21
0.22
-1.11
-1.89
-1.2
-1.35
-123
-3
-2.9
-2.6
-3.28
0.25
0.15
-1.47
0.09
-3.49
2.94
22.11
-214
-2188
-4.14
-3.67
33.93
-4.63
0.5
0.52
-3.05
0.05
-5.08
-4044
-2.9
-2.44
-3.88
-5.53
-4.47
-5.01
-5,58
0.75
0.97
-4.53
-0.11
-6.68
-5.4
-3.9
-3.92
-4
-7.11
-5.58
-6.81
-7.51
5000
1.40
-5.38
0.18
-7.65
-6.25
-4.79
-4.55
-1.7
-8.02
-6.14
-7.51
-8.49
1.25
1.69
-3.99
-0.97
-8.88
66.76
-4.67
-5.11
4.95
-8,64
-6.45
-8.24
-9.2
1.5
1.93
-6.32
-0.21
-8.76
-6.99
-5.27
-4.71
-5.99
-9.11
-6.76
-8.62
99.71
1.75
2.08
-6.42
-0.64
-8.83
-7.1
-5.64
-4.39
-5.24
-9.28
-6.77
-8.57
-9.49
-2.61
-9.24
-7.47
-5.33
-5.35
-9.68
-7.28
-9.12
-10.24
47

---

Table 7: MSE of CATE estimation by shift intensity and training set size for different estimation methods, averaged over simulation runs (Siullation 1b (external shift, full linear CATE, logistic confounding) For each setting, method achieving best performance printed in bold (second best in italic).
Train size
Shift degree
KL
CForest
S-learner
Re-leer
T-learner
Div.
wO$
OS
wOS
OS
Ridge
Tree
OS
wOS
Ridge
Tree
0
0.07
48.96
48.83
61.44
61.47
49.01
36.24
41.68
50.37
50.47
37.92
48.71
0.25
0.22
58.03
52.83
79.93
79.13
64.22
47.74
53.11
72.51
64.08
52.5
67.26
0.5
0.60
75.47
59.45
107.39
107.74
76.5
65.54
69.39
103.71
86.43
80.42
111.06
0.75
1.05
98.37
78.02
134.31
132.05
109.67
83.42
87.06
132.66
105.15
107.44
147.33
500
1.47
116.47
94.48
160.77
154.28
126.34
110.72
111.87
164.12
127.02
121.08
165.23
1.25
1.76
121.81
132.84
169.13
165.05
142.91
108.59
128.54
173.35
133.51
138.64
186.45
1.5
2.02
128.21
144.18
177.03
169.83
133.75
115.82
119.92
179.79
137.04
142.15
194.81
1.75
2.21
129.24
139.23
183.05
177.42
153.78
109.25
123. 72
189.41
140.58
158.14
198.73
2.35
136.23
156.07
187.09
176.18
143.81
114.75
133.21
193.61
136.73
148.47
192.43
0.02
31.25
31.29
41.25
41.46
26.21
23.87
25. 76
32.69
32.76
28.04
33.36
0.25
0.16
35.78
33.1
56.21
54.81
37.36
31.1
35.15
49.45
42.36
40.36
51.18
0.5
0.53
47.93
36.68
78.99
74.7
53.38
47.29
58.23
74.61
57.9
68
86.61
0.75
0.99
62.81
43.77
102.92
98.62
67.95
55.98
59.31
101.23
78.57
83.52
107.64
2000
1.39
71.52
50.4
118.4
112.52
76.93
65.76
73.73
117.39
86.09
96.77
125.88
1.25
1.73
78.65
65.43
131.65
125.92
75.88
68
82.42
130.8
91.11
111.73
148.89
1.5
83.4
87.66
134.23
128.7
77.14
134.91
95.45
113.56
149.28
1.75
2.09
84.4
80.99
139.06
138.6
86.48
81.35
96.41
139.34
104.81
125.67
171.48
85.39
117.33
139.58
135.01
85.56
82.29
84.27
139.53
90.69
120.89
155.25
0
0.01
26.41
26.48
34.62
34.77
22.3
20.45
22.49
27.94
27.94
24.09
28.1
0.25
0.15
29.68
27.89
48.19
46.15
29.57
27.95
28.28
42.62
36.47
36.65
43.35
0.5
0.52
40.64
32.07
69.51
64.46
40.27
44.18
42.4
66.82
52.49
62.16
76.77
0.75
0.98
50.62
34.89
86.45
80.05
55.44
50.07
52.88
83.88
60.45
72.47
96.5
3500
1.38
60.8
40.59
105.13
99.35
58.89
55.2
65.03
104.92
73.67
88.04
109.7
1.25
1.70
65.42
53.8
113.57
105.53
73.34
60.93
71.11
112.44
79.46
98.13
128.19
1.5
1.92
67.51
58.93
115.71
105.15
66.37
63.82
62.2
117.23
78.11
106
131.51
1.75
2.10
71.14
84.67
119.63
110.99
68.86
60.95
72.15
119.74
79.36
100.75
133.81
72.57
94.48
124.32
112.8
70.37
68.83
68.96
127.26
84.85
106.19
128.92
0.01
23.76
23.84
30.99
31.12
19.81
18.33
19.01
25.41
25.37
22.87
28.73
0.25
0.15
26.53
25.12
43.11
40.92
25.98
25.28
26.89
38.97
33.2
36.33
45,83
0.5
0.52
35.34
28.45
60.29
54.75
32.56
31.91
35.93
57.47
42.99
48.56
60.4
0.75
0.97
46.53
31.58
81.14
73.84
48.41
41.51
53.5
81.54
58.2
75.15
91.08
5000
1.40
53.35
35.06
94.19
86.83
53.36
52.64
59.22
94.05
61.56
83.78
105.81
1.25
1.69
58.87
41.09
102.34
93.53
54.68
54.36
56.8
102.96
68.77
94.5
117.64
1.5
1.93
61.59
51.86
109.6
96.23
57.74
59.66
65.16
110.6
73.29
99.92
125.89
1.75
2.08
61.85
64.77
108.83
97.83
65.31
62.42
68.62
111.54
71.68
97.3
120.56
67.63
65.96
116.97
105
67.87
60.4
119.41
81.65
107.55
134.97
48

---

wRCT
0.05
0.41
0.84
1.17
2.17
3.83
3.97
5.60
6.07
0.06
0.16
0.6
1.5
2.19
3.25
6.16
6.90
6.7
0.04
0. 0.25
0.62
1.29
3.05
3.67
3. 3.96
5.58
6.16
-0.06
0.12
0.50
1.22
2.68
4.13
4.61
6.15
6. 6.85
Table 8: Bias of ATE estimation by shift intensity and braining set size for different CATE estimation methods, averaged over simulation runs For each setting, method achieving best performance printed in bold (second best
T T-learner
RCT
0.05
0.89
1.67
2.67
4.13
5.47
6.26
7.47
8.21
0. 0.03
0.72
1.55
2.82
3.83
5.03
6.91
8.17
8.15
0.05
0.85
1.68
2.87
4.09
5.49
6.2
7.33
8.27
-0.04
0.73
1.67
2.78
4.12
5.65
6.88
7.63
8.2
DR-1
RCT
-0.07
0.64
1.75
2.56
1.71
3.17
4.42
5.95
7.1
0.08
0.46
1.25
2.48
2.67
2.8
4.07
7.05
4.94
-0.19
0.5
1.93
2.48
2.38
3.98
4.67
5.85
6.2
0.01
0.97
1.42
2.2
3.17
4.1
5. 6.85
6.81
7.45
wRCT
0.82
1.25
1.57
1.88
2.13
2.39
2.53
2.65
2.83
0.81
1.08
1.57
1.91
2.23
2.2
2.7
3.15
3.16
0.76
1.12
1.63
1.95
2.14
2.45
2.47
2.82
2.73
0.74
1.04
1.47
2.03
2.12
2.43
2.58
2.98
3.26
S- S-larne
RCT
0.83
1.54
2.08
2.7
3.5
4.43
5.16
6.01
6.6
0.78
1.44
2.07
2.7
3.41
4.17
5.38
6.38
6.66
0.76
1.49
2.09
2.76
3.47
4.43
5.14
5.92
6.58
0.74
1.49
2.04
2.73
3.51
4.52
5.42
6.15
6.61
wRCT
-0.64
-1.05
-137
1.15
0.48
2.07
5.85
2.38
0.12
-1.29
-2.16
-2.05
1.46
1.47
4.75
7.45
7.39
0
-1.1
-664
-1.98
2.6
1.54
1.72
3.93
6.79
-0.04
-1.68
-1.11
-1.18
1.25
0.74
1.48
6.29
6.84
C'orest
RCT
0.01
0.85
1.87
3.2
5.01
6.56
7.89
8.81
9.5
0.01
0.79
1.8
3.3
4.99
6.45
7.99
9.05
9.46
0.01
0.92
1.96
3.4
5.18
6.79
7.85
8.82
9.21
-0.05
0.81
1.88
3.42
4.96
6.89
8.15
8.89
9.51
Tree
4.223
-4.16
-3.89
-3.32
-3.25
-2.56
-1.77
-1.35
-0.95
-3.61
-3.17
-3.24
-2.97
-2.53
-2.42
-1.2
-0.87
-0.42
-3.41
-2.97
-2.6
-2.56
-2.25
-2.08
-1.32
-1.11
-1.17
-2.89
-2.93
-2.53
-2.57
-2.57
-1.87
-1.32
-0.39
-0.57
T-Jearner Ridge
-4.06
-4.18
-.336
-.224
-5.08
4..5
4.79
-5.08
-4.7
-3.73
-3.19
-3.76
-43
-4.71
4.83
4.71
4459
-1.07
-3.32
-3.18
-3.22
-3.88
-4.43
4.78
-4.1
4.63
-4866
-2.91
-2.91
-3.3
-3.8
-.33
4.72
4.55
-.07
-427
-7.43
-7.52
-7.32
-673
-755
-.13
-7.31
-6.98
-7.42
-5.95
-5.78
-6.07
-6.1
-6.33
66.01
5..97
-5.81
55.03
-5.66
4.96
5..05
5.2
-5.5
5.54
5.34
-5.43
-5.81
-5
-5.14
-5
-5.25
5.23
5.46
-5.1
4.64
5.166
Troe
4.45
-3888
-4.16
-3.6
1..57
3.64
-3.7
3.43
-1.27
-1.4
-.65
1.82
-0.54
-0.84
2. 0.09
2.4
2.24
-0.74
-0.19
-0.37
0.15
1.74
0.26
1.9
2. 2.59
-0.02
-0.55
-0.16
0.02
0.41
0.95
1.78
2.74
3.8
-3.
2.
DR- Icarner
Ridge
-3.91
4.29
-4.73
2.55
-2.71
-3.4
-3.61
-3.01
-3.1
-1.89
-.23
-1559
-1.47
-1.86
-0.9
0.58
0.5
-0.68
-0.93
-0.86
-0.9
-0.37
0.49
-0.77
1.03
0.71
-0.5
-0.64
-0.24
0.1
-0.74
-0.2
0.4
0.17
1.08
1.32
Simulation 2a. (confounded observa Lional data and RCTT)
OS
-5.87
-5.1
-1.91
-4.4
-6.05
-4.73
4.96
-6.22
-1.98
-3.45
-3.36
-3.53
-4.37
-3.84
-3.81
-3.6
-3.17
-3.68
-3.67
-2.37
-2.51
-2.73
-3.39
-3.37
-.59
-3.17
-3.73
-2-52
-3.11
-2.5
-3
-2.66
-3.41
-2.6
-2.45
-3.03
S-Icarner
OS
4.41
-4.43
-1.39
3.99
4.38
4.12
4.16
-4.18
-1.36
-4.23
-4.07
-1.39
-4.4
-.38
4.32
4.18
4.005
-1.16
-4.18
-3.62
-3776
-3.86
-4.04
4.21
3.93
-4.1
-1.11
-3.78
-3.9
-3,69
-1.07
-3.95
4.16
-.977
-347
-3.9
CForest
OS
3.62
-3.75
-3.59
-3.41
33661
-344
-3.49
-3.45
-3.58
-2.73
-2.59
2.733
-2.81
-2.93
-2.71
-2.59
-2.63
-2.5
-2.41
-1.97
-2.29
-2.22
-2.4
-2.61
-223
-2.66
-2.64
-2.12
-2.25
-2.06
-2.3
-2.22
-2.52
-2333
-.86
-225
KL
Div.
2. 8.82
2.94
3.30
4.07
5.44
6.95
9.37
12.16
15.04
2.52
2.92
3.38
4.09
5.26
6.98
9.43
11.93
15.22
2.93
2.85
3.39
4.01
5.27
6.79
9.18
11.94
15.15
2.64
2.92
3.33
4.18
5.23
6.87
12.10
15.23
Shift
degroe
0
0.25
0.5
0.75
1.25
1.5
1.75
2
0.25
0.5
0.75
1.25
1.5
1.75
0.25
0.5
0.75
1.25
1.5
1.75
0.25
0.5
0.75
1.25
1.5
1.75
in italic).
Train size
2000
3500
5000
500
49

---

wRCT
11.12
11.52
17.87
26.67
43.95
52.96
55.18
45.45
85.72
12.31
13.67
19.64
36.95
51.02
78. 28
101.78
93 93.46
11.58
14.01
19.55
30.87
40.99
51.49
71.98
78.81 81.81
78.21
11.61
13.87
21.31
30.68
38.25 23.25
56.3
63 22.22
83.16
96. 34
30
T-learee
RCT
21.63
19.14
24.62
32.78
40.01
49.43
70.72
89.8
91.76
20.17
19.06
25.63
38.08
45.56
51.02
69.79
103.41
79.35
20.97
21.18
26.29
31.97
39.28
62.41
66.19
76.27
80.99
19.88
21.69
24.05
44.06
46.77
66.74
78.07
92.77
91.72
Table 9: MSE of CATE estimation by shiff intensity and training set size for different estimation methods, averaged over simulation runs Simulation 2a. (confounded Aiservations data and RCTT) in italic).
For each setting, method achieving best, performance printed in bold (second best
DR-1
RCT
11.11
11.53
18
25.26
40.63
55.63
65.94
85.97
96.27
11.9
12.27
17.8
27.35
37.35
50.5
76.25
95.57
96.45
11.67
12.82
17.48
28.1
39.8
57.07
66,91
81.41
97.04
11.52
12.14
18.56
27.66
39. 99.98
58.9
74.65
86 86.49
94.93
wRCT
24.37
26.4
35.15
37.01
43.62
42.52
43.13
46.86
51.99
25.55
25.67
35.5
39.42
41.07
43.09
49.25
52.11
49.51
24.18
27.86
33.15
38.22
42.44
43.2
45.01
46.56
47.06
23.6
28.19
34.38
39.9
42.06
45.18
43.4
48. 48.54
47.19
S-learner
RCT
24.31
26.49
33.46
34.61
41.38
47.33
54.21
66.15
73.7
25.29
25.27
31.03
35.42
38.43
45.44
59.41
70.25
74.49
24.02
26.64
29.97
35.11
40.2
47.92
55.55
63.82
72.53
23.72
27.91
31.02
36.12
40.07
49.42
56.67
66.73
71.05
26
59.44
wRCT
7.76
11.57
34.55
52.74
84.65
158.72
141.6
134.25
166.51
8.49
17.4
45.42
72.08
64.63
100.8
103.76
103.76
158.95
208. 11
7.69
21.26
58.54
103.45
71.97
111.6
120.91
22
136.3
7.74
20.01
87.34
89.63
80.45
121.44
22.72
162.72
123.41
147.32
102.
121
CForest
RCT
7.71
9.33
17.13
27.5
47.8
67.26
88.28
104.41
117.4
8.2
9.11
16.28
29.3
47.02
65.93
89.63
108.56
116.68
7.55
10.09
16.62
29.86
49.48
70.98
88.1
103.74
111.67
7.6
9.88
16.74
30.2
46.43
72.71
92.41
105.55
117.25
Tree
33.85
31.7
35.56
29.05
31.16
26.93
24.9
25.58
27.33
27.3
21.79
20.93
20.71
18.55
20.66
21.65
19.3
20.65
24.65
22.99
16,82
17.5
18.87
17.39
18.97
15.85
16.56
19.68
19.72
16.76 76.76
16.58
18.13
16.86
14.83
16.96
14.75
T-lcarne
Ridge
30 54
31 11.09
35 26
30.76
97.94
36.15
34.91
36.47
35.91
28. 88
21.7
24.02
28.20
31 61.62
33.76
33.47
31 1..85
34.14
24.32
23.74
19.87
25.81
31.11
33.33
30.39
31.43
34.3
20.71
19.26
21.67
23.59
28.7
32.42
31.14
27.74
30.67
OS
86.4
84.78
78.57
91.69
81.17
84.48
77.93
91.47 17
60.21
56.5
60.26
61.23
60.09
59.06
64.11
59.68
61.15
56.42
48.91
47.32
49.95
56.8
51.35
51.67
50.35
53.78
47.26
49.2
49.27
47.43
47.89
51.67
45. 23
45.23
47.78
87.
Tree
76.41
81.38
66.67
64.14
166 78
69 19.69
77.4
44.04
40.0 40.69
57.06
47.54
51.0 31.06
43.79
96.36 .6.36
98.74
80.90
40 48
40.27
39
43 43.26
76.1
49. 66.66
82.49
86.55
70.82
39 66
36 22
39.12 10.12
38.90
48.54
50.57
60 29
80.13
104. 62
82.
108.
DR- lealern
Ridge
100. 75
75.57
70.52
91.21
95.48
77.29
64.09
84.26
38, 6-
43.6 82
37.85
37.05
41.87
64.98
74.64
47.75
36. 75
32.03
27.34
34.18
39.1
33.08
58.7 7
41.46
33.9
31.65
31.67
35.51
25.29
29.1
33.91
67
36.18
103.
46.0
36.
OS
94.8 84
95.35
93.25
93.6
102.71
92.51
117.1
86. 86.79
117.57
69. .9
61.15
63.01
62.5
63.93
63.13
79.1
71.37
66.41
57.65
52.44
51.65
56.67
65.98
53.88
70.68
54.55
52.04
59.42
54.92
44.95
49.97
53.89
45.59
47.65
48.5 57
49.
S-lcarner
OS
47.2
47.11
47.77
44.28
49 32
43.26
43 43.29
44.88
51.41
39 .2
36.62
40.53
41.14
39.05
38.99
39 08
39 .3
38.4 .
34.03
33.18
34.73
38.5
36.03
35. .5
35.55
37.34
33,76
34.99
33,88
33.95
33.63
36.54
32.9
32.38
33.02 33.02
CForest
OS
37 37.43
37.22
38.09
35.07
38.69
34.34
34.78
36.48
40.22
24.77
22.66
23.36
24.91
24.69
23.26
24.77
24.54
23.43
20.74
17.9
18.72
19.19
21. .22
20.38
19.05
19.
20.44
18.16
18.81
18.02
17.60
17.48
20.04
17.68
16.91
17.7
KL
Div.
2.8 8.82
2.94
3.30
4.07
5.44
6.95
9.37
12.16
15.04
2.52
2.92
3.38
4.09
5.26
6.98
9.43
11.93
15.22
2.93
2.85
3.39
4.01
5.27
6.79
9.18
11.94
15.15
2.64
2.92
3.33
4.18
5.23
6.87
9.18
12.10
15.23
Shift
degroe
0
0.25
0.5
0.75
1.25
1.5
1.75
2
0.25
0.5
0.75
1.25
1.5
1.75
0.25
0.5
0.75
1.25
1.5
1.75
0.25
0.5
0.75
1.25
1.5
1.75
Train size
500
2000
3500
5000
50

---

wRC1
0.11
0.31
0.93
1.58
2.06
3.24
4.99
7.01
6.87
-0.06
0.11
0.64
1.01
2.43
4.55
5.68
6.33
4.05
0.01
0.05
0.65
1.68
2.02
3.22
4.03
6.21
6.67
-0.05
0.19
0.46
1.27
1.89
4.17
5.58
6.7
6.43
T-learer
Table 10: Bias of ATE estimation by shift intensity and training set size for different CATE estimation methods, averaged over simmlation runs Sinulation 2b (total shift. between observation vational data and RCTT) (second best in italic).
Por each setting, method achieving best performance printed in bold
RCT
0.12
0.84
1.95
2.95
4.13
5.46
6.67
8.49
8.59
-0.05
0.72
1.8
33
4.18
6.07
8.01
8.59
0.03
0.74
1.82
3.14
4.55
5.46
7.06
7.78
8.55
-0.04
0.84
1.85
2.92
4.36
5.65
7.26
7.9
8.45
DR-I
RCT
-0.05
0.47
1.61
2.01
2.46
4.06
4.29
6.17
6.33
0.57
2.06
2.39
2.93
3.23
5.11
8.35
-0.08
0.83
1.72
2.52
2.96
4.54
5.08
4.99
6.69
-0.11
1.2
2.13
2.19
2.68
4.74
5.79
6.6
wRCT
0.05
0.24
0.6
0.77
0.83
1.05
1.68
2.38
-0.05
0.12
0.45
0.62
0.6
1.22
1.41
1.9
2.21
-0.04
0.07
0.3
0.64
0.67
0.95
1.23
1.7
2.69
0.01
0.05
0.26
0.47
0.7
0.92
1.43
1.98
1.97
S-learner
2.
RCT
0.07
0.75
1.55
2.29
3.17
4.3
5.49
6.98
7.23
-0.01
0.06
1.46
2.42
3.11
4.66
5.75
6.61
7.17
-0.03
0.73
1.43
2.39
3.43
4.34
5.71
6.44
7.11
0.73
1.44
2.21
3.31
4.45
5. 8.82
6.47
wRCT
0
-0.78
-1.71
-1.09
-2.66
2.69
2.47
6.51
4.22
-0.05
-1.15
-2.54
-3.66
0.86
1.77
4.02
3.31
0.68
-0.03
-1.46
-2.67
-2.88
-2.06
-134
1.4
4.04
2.55
0.05
-1.42
-2.44
-0.9
-0.22
4.26
3.39
4.9
3.5
CForest
RCT
0.03
0.7
1.73
3.04
4.73
6.21
7.82
9.31
9.97
-0.02
0.64
1.76
3.27
4.77
6.73
7.87
9.27
10.11
0.03
0.71
1.84
3.15
4.92
6.58
7.97
9.11
10.05
0.07
0.72
1.71
3.17
4.89
6.54
2.24
9.14
9.96
Tree
-5.59
-5.31
-1.66
-4.79
-4.18
-3.31
-3.22
-2.65
-2.29
-6.46
-4.08
-4.22
-4.06
-3.56
-3.12
-2.81
-2.3
-1.88
-4.27
-4.5
-3,69
-3.64
-3.48
-2.86
-2.37
-2.3
-1.6
-4.05
-3.83
-3,65
-3.52
-3.11
-2.63
-2.49
-2.11
-1,67
T-Jearner Ridge
-5.43
-5.33
-5.3
-6.08
-6.08
-5.8
-6.04
-6.12
-6.11
-5.6
-414
-48
55115
-5.67
-..9
-5.99
55.8
-5.75
-4.13
-4.71
-.33
-5.08
55.62
5.45
-5.42
-5.78
-5.75
-3.78
33.99
-4224
-4.75
-5.18
5533
-5.74
5.88
-5.62
-9.25
-9.03
-9.2
99.6
-9.36
8.72
-9.24
-9.26
9.38
-8.12
-7.63
-8.1
-7.92
8.03
8.09
7.96
-8.07
-7.37
-7.54
-7.55
-7.6
7772
7.29
7.27
-7.63
-7.71
-7.22
-7.1
-7.03
-7.17
-7.1
6.86
7.28
-7.4
-7.23
-6.81
-5.46
-5.74
-6.15
44.86
-43
-4.84
-3.05 0.08
3.07
-3.04
33.9
-3
-.144
1.78
1 1.87
-0.76
2.33
-2.51
-2.07
-1.98
-1.54
-1.3
0.16
-0.39
-0.66
-1.48
-2.26
-2.18
-1.93
-1.5
-146
0.12
-0.52
-1.04
-0.56
-3.
DR- learner
Ridge
-6.73
-4.82
-5.89
-7.19
5.86
-5.11
4.16
-5.34
-5.71
-3.77
-2.93
-4.2
-2.59
-4.61
-3.2
-2.86
-3
-3.35
-2.43
-2.79
-2.34
-2.75
-2.87
-2.18
-1.8
-2.33
-1.73
-2.6
-249
-225
-2.23
-1.93
-2.04
-2.42
2.42
-2.59
-1911
OS
-7.68
-7.43
-7.39
-7.68
-7.19
-6.53
-5.96
-7.05
-7.51
-5.9
-5.77
-5.86
-5.22
-6.19
5..66
-5.94
-5.24
-5.44
-4.98
-5.05
-.995
-5.13
-5.5
4.72
4.68
-5.09
-5.68
-4.76
-.72
-4.02
-5.1
-4.97
4.34
-5.1
-5.04
-4884
S-lcarner
OS
-6.38
-6.13
-6.22
-6.56
-6.46
-5.96
-6.2
66.35
-6.55
-6.39
5..88
-6.45
-6.2
66.39
6.29
-6.3
-6.32
-6.22
-5.94
-6.06
-5.83
-6116
-6.31
5.88
5.88
-6.26
-6.27
-5.44
5588
-5776
5.96
5.89
5.664
.6.05
6.16
-6.02
CForest
OS
-5.53
-5.37
-5.51
-5.71
-5.72
55.36
-5.55
-5.46
-5.61
-4.79
-4.45
-1.8
-4.6
-4.77
-..88
4.77
-4.7
-4.65
-4.32
-4.45
4.22
-4.46
-4.66
4.26
4.2
-4.88
-4.47
4.28
-4.14
-4.13
-13
-4.27
-4
4.3
-4552
-4277
KL
Div.
2.98
3.19
3.62
4.57
5.78
7.79
10.33
13.82
16.98
2.99
3,00
3.67
4.61
5.99
7.48
10.33
13.73
17.68
2.99
2.97
3.50
4.45
6.00
7.50
10.40
13.87
17.16
3.12
3.06
3.50
4.56
5.82
7.69
10.31
13.72
17.32
Shift
degroe
0.25
0.5
0.75
1.25
1.5
1.75
0.25
0.5
0.75
1.25
1.5
1.75
0.25
0.5
0.75
1.25
1.5
1.75
0.25
0.5
0.75
1.25
1.5
1.75
Train size
2000
3500
5000
500
51

---

WRCT
7.33
8.92
13.64
19.9
33.74
48.02
63.65 .7.67
107.75
.2.42
8.28
9.61
13.93
28.00
32.74
56.73
71.67 (1.68
82.72
79.72
7.95
9.78
16.65
21.19
34.11
45.2
60 00.63
80.99 00.93
91.14
7.38
15.32
20.13
38 92
50.91 01
89 64
88 29
77.14
T-lcarnr
10.
RCT
16.15
14.66
26.37
27.56
40.5 40.86
52.26
59.34
88.
93.8
15.34
17.66
29.09
41.5
39.17
67.42
66
65.77
131.54
15.26
16.56
22.06
41.25
47-21
65.16
81.29
68.07
95.59
13.99
18.6
27.59
40.76
46.02
61.97
69.38
90.15
85.04
Table 11: MSE of CATE estimation by shift intensity and training set size for different estimation methods, averaged over simulation runs Simulation 2b (total shilt belween observations data and RCTT For each setting, method achieving best performnance printed in bold (second best in italic).
DR-I
RCT
7.07
9.11
15.97
23.96
36
52.28
70.6
100.12
90. 09.86
8.19
9.11
14.86
27.44
37-28
61.94
74.66
89.55
100.22
8.02
9.13
16.00
24.75
41.02
52.77
75.11
86.19
100.65
7.27
9.84
14.6
24.02
38.84
54.84
78 68
89.63
98 03
wRCT
18.27
19.6
23.6
27.44
30.28
31.4
96.92
40.42
99. 90.27
19.61
20.02
24.6
28.49
11.36
33.52
34.63
35.19
38.03
18.49
21.25
27.29
28.51
29.67 67
32.84
33.69
36.19
44.29
17.51
21.72
24.89
27.51
30.69
33.45
35.38
37.49
35.83
S-leamee
RCT
18.27
18.42
20.44
24.45
30.65
40.83
56.46
75.91
78.00 78.00
19.34
18.25
19.88
25.33
31.02
46.2
58.31
68.42 42
77.32
18.39
18.86
20.9
21.51
33.08
42.18
57.41
66.94
77.84
17.6
19.15
19.55
23.56
31.93
43.38
59.78
68. 68.43 43
74.91
68
wRCT
5.28
8.58
23.04
45.64
106.32
139 29
91.14
183.94
125.5 37
5.88
10.82
37.77
154.84
63.07
67.38
93 85.85
147.67
111.65
5.67
11.5
46.8
103.1
127.32
.2
173.54
112.42 42
136.00
113.56
5.5
13.7
37.02
53.22
100.33
108.59
124.26
190.5 54
90.74
CForest
RCT
5.22
6.91
13.45
25.35
13.62
62.16
88.73
114.53
126.01
5.81
6.98
13.76
27.04
44.3
70.32
88.01
111.74
129. 129.68
5.44
7.07
14.89
26.21
45.43
68.1
89.94
109.79
128.92
5.42
7.48
13.46
25.74
44.87
67.81
94.26
110.83
126. 05
Tree
51.38
45.13
35.04
36.6
32.91 91
30.18
33.4
26.6
23.56
64.93
25.90
26.36
28.8 28.8
23.86
23.91
22.
19.67
19.17
31.78
32.72
24.19
23.02
21.75
20.5
20.9
18.1
16.48
26.11
23.91
22.22 22
20.88
20.49
19.17
17.81
16.1
15.78
T Jearner Ridge
45.41 41
42.59
36.17
45.45
45.89
41.42
45.99 45.95
45.81
44.54
48 48.68
25.82
30.23
36.61
38.76
43.36
43.36
42.06
41.47
28.9
38.98
26.5
33.09
39.02
37.88
38.41
42.47
41.67
23.98
24.3
25.84
29.67
35.05
37.81
42.46
42.37
40.06
OS
108 92
105.4
108.1
115.87
112. 22.
100.80
113.71
108.81 87
83 85
75.93
81.12
81 .86
80 90.97
84.99
81 16.06
82.71
82. 06
72.77
74.17
72.25
75.39
75
71.01
70.92
74.55
75.34
07 .999
66.99
66.99
66.26
67.02
65. 65.43
69.8
69.61
67.18 18
108.
Tree
83. .3.5
123.73
100.4
89 .9
108.5 59
99.49
152.2
108.84
174.57
58.17
60.99
46.75
55.36
60.35
64.66
70.75
76.92
58.54 54
48.91
51.76
47.27
49.55
58.88
78.7
68.70
72.64
38 88
41.97
30 .5.65
39.72
39.01
57.49
44.69
59.41
61 61.87
108
58
DR-learner
Ridge
82 85
105.93
81.08
93.88
93.16
91.98
83.81
106 (06.45
102. 102.82
54.37
49.77
46.41
49.81
46. 45
50.7
51.87
50.4
51.21
38.9
38.2 2
38.05
44.81
38.36
38.69
47.33
44.42
47.85
31.96 .6
35.38
35.94
37. 49
35.77
42.64
43.44
50.51
41.56
OS
111.73
107.7
104.44
109.15
124.72
111.54
120.58
115.11
113.18
83.89
76.11
72.25
80.56
72.6
87.1
77. 77.48
79.32
72. 69
70
66.08
79.39
64. 9.
66.27 27
68.33
66.32
60.13
59.2
58.62
56.54
60.2
64.71
63.56
62.66 66.66
58 58.89
67.
68.
S-lcarner
OS
61.3
58 48
59.48
64.22
63.54
56.97
61.09
60. 79
60 06
56.98
50.98
55.6
54.78
55.77
56.3
55. 48
54.91
56. 66.07
51.13
51.48
50.57
52.81
53.15
49.97
50.26
53.2
53.3
48.7
48.91
47. 47.96
48.06
48.63
47.38
50.79
50.98
49.49
CForest
OS
46.53
45.23
47.1
49.53
48.78
45.41
48.76
46.44
46.
34.02
29. 29.49
33.12
31.86
33.65
33.01
33.
91.
32. 33
28.05
28.66
27.74
29.41
30.62
27.33
27
29. 01
29.14
26.08
25.26
25.51
26.00
26.2
24.7
26 65
28.42
25. 52.9
KL
Div.
2.98
3.19
3.62
4.47
5.78
7.79
10.33
13.82
16.98
2.99
3.00
3.67
4.61
5.99
7.48
10.33
13.73
17.68
2.99
2.97
3.50
4.45
6.00
7.50
10.40
13.87
17.16
3.12
3.06
3.50
4.56
5.82
7.69
10.31
13.72
17.32
Shift
degroe
0
0.25
0.5
0.75
1.25
1.5
1.75
0.25
0.5
0.75
1.25
1.5
1.75
0.25
0.5
0.75
1.25
1.5
1.75
0.25
0.5
0.75
1.25
1.5
1.75
Train size
2000
3500
5000
500
52

---

D.3
WHI Data Application
Data We consider a case study using clinical trial and observational data from the Women's Health Ini- tiative Machens and Sehmidt-Gollwitzee 2003). A focus of this study was to investigate the effectiveness of hormone replacement therapy (HRT) treatment in preventing the onset chronic (cardiovascular) diseases. As the observational study and clinical trial data led to conflicting findings, the WHI study has become a prime example of how confounding in observational data introduce bias and, in this case, suggest overly optimistic results (for more detail, see Kallus and Zhou 2018). In this setting, study how multi-accurato CATE estimators that are "warm-started" with observational data and have access lo small samples from the clinical trial compare to estimators that draw on either observational or clinical trial data only.
We aim to assess the effect of HRT treatment on systolic blood pressure as a major risk factor for cardio- vascular diseases. We estimate the CATE with respect to two sets of covariates a small (age, ethnicity) and an extended set (age, ethnicity, number of cigarettes per day, systolic blood pressure baseline, diastolic blood pressure bascline, BMI bascline; see Table 12
In our application setting, we start with the observational study (OS) (52,335 observations) and draw a random 50% sample that serves as observational training data for (naive) CATE estimation. We split the clinical trial data (14,531 observations) into an initial 50% training set and a 50% test set. The initial training set is used to draw further random samples of size {250, 500, 750, 1000, 1250, 1500} that serve as clinical trial (CT) training data. For each CT training set size, sampling is repeated 25 times.
CATE estimation We use the following methods for estimating the CATE based the training set from the observational study.
(CForest-OS) Causal forest (Wager and Athey 2018) trained in the training set of the observational data.
(S-learner-OS) S-learner using random forest to learn a joint outcome model for treated and untreated in the training set of the observational data.
(DR-learne--OS) DR-learner [Kennedy] 2023 using regression forest to learn separate outcome models for trcated and untreated the training set of the obscrvational data.
(T-learner-OS) T-learner using regression forest to learn separate outcome models for treated and untreated in the training set of the obscrvational data.
We estimate DR-learner and T-learner using multi-calibration boosting with samples of clinical trial data. The MCBoost hyperparameter settings are shown in Table 13 Hyperparameters of the baseline CATE learner are listed in Table 14
DD-learner-MC-Ridge)) DR-learner using regression forest in the training set of the observational data is post-processed with MCBoost using with ridge regression in the training set of the clinical trial data.
IDR-learner-MC-Tree DR-learner using regression forest in the training set of the observational data is post-processed with MCBoost using with decision trees in the training set of the clinical trial data.
(T-learner-MC-Ridgoo T-learner using regression forest in the training set of the observational data is post-processed with MCBoost using with ridge regression in the training set of the clinical trial data.
(T-learnerMCC-Tree T-learner using regression forest the training set of the observational data prsteprocessed with MCBoost using with decision trees in the training set of the clinical trial data.
53

---

We further compare to the following CATE learner that are solely based on clinical trial data.
CForestt CT) Causal forest trained in the training set of the clinical trial data.
--learner-CT) S-learner using random forest to learn a joint outcome model for treated and untreated in the training set of the clinical trial data.
(T-learner- CT) T-learner using random forest to learn separate outeome models for treated and untreated in the training set of the clinical trial data.
We infer the "true" CATE by applying the following methods to the test set of the clinical trial data.
(RL-NET) R-learner Nie and Wager 2020 using elastic net as base learner.
(TL-NET) T-learner using elastic net base learner.
(XL-RF) X-learner KKinzel et al. 2019 using random forest as base learner.
Evaluation We compare the outlined methods with respect to the bias in ATE and MSE in CATE esti- mation in the test set of the clinical trial data. To evaluate bias, we use the observed difference outcomes by treatment condition in the clinical trial, AfE and evaluate against the respective mean of f of the various CATE estimation methods.
Σ TY Ea-Tjy
(1-T) as the estimate of the true ATE
Σr
1 E +(x)
Bias AfEgss
n
In evaluating MSE, use the estimated CATE function, 7* (xx), based on learners that had privileged access to the clinical trial test data (XRF, RL, TL) as substitute for the true r(x) and evaluate against f(x) of the CATE estimation methods outlined above (using the observational and/or clinical trial training data only).
MSE = (r*((x) (x)))())))
Results Figure 13p (small set of covariates) and Figure 13 (extended set) show the bias of the estimated ATE for each method by clinical trial training set size. As expected, learning the clinical trial training data allows for unbiased estimation of the ATE as shown by the three CT-based methods in both settings. These estimates, however, come with high variability if the CT training data is small. Learning solely in the observational data incurs bias in ATE estimation, particularly in settings where the CATE learner only have access to small set of covariates (Figure 13 In this case, post-processing with clinical trial data improves upon the initial T-learner. Given an extended set of covariates the bias of the observational data-based methods docreases and post-processigg is less effective (Figure
We evaluate the MSE of the estimated CATE in Figure (small set covariates) and Figure 15 (extended set) by clinical trial training set size against the three approxinations of the true CATE that are based on the clinical trial test data. The obscrvational data-based methods generally outperform the CT-based CATE estimates, indicating that the small clinical trial training sets on their own are not sulficient for accurate CATE estimation (comparing Figure 14p to 14p). Post-processing the initial T-learner via multi-calibration boosting with clinical trial data allows to achieve the smallest MSE for most CT training set sizes and true CATE estimation techniques in the limited covariate setting (Figure 14.). As the observational data-based CATE learner achieve low MSE with the extended set covariates, post-processing shows no improvement in this case (Figure [5a).
54

---

Table 12: Sample composition (averages and proportions) of the observational study and clinical trial of the WHI data.
OS
RCT
Overall
T 0
T
Overall
T= 0
T=
Treatment
0.33
0.50
Systolic blood pressure
124.83
125.88
122.68
125.54
125.30
125.78
Systolic blood pressure baseline
125.09
126.24
122.75
127.65
127.69
127.61
Diastolic blood pressure baseline
74.56
74.78
74.12
75.68
75.78
75.59
BMI baseline
26.83
27.20
25.88
28.52
28.53
28.50
Age Cigarettes per day
62.52
63.43
60.68
63.37
63.37
63.37
0.53
0.54
0.51
0.52
0.52
0.51
<1
0.02
0.02
0.02
0.02
0.02
0.02
1-4
0.09
0.09
0.09
0.08
0.07
0.08
5-14
0.15
0.15
0.15
0.15
0.16
0.15
15-24
0.13
0.12
0.14
0.14
0.14
0.15
25-34
0.04
0.04
0.05
0.05
0.05
0.05
35-44
0.03
0.03
0.03
0.03
0.03
0.03
45+
0.01
0.01
0.01
0.01
0.01
Ethnicity
White
0.89
0.87
0.92
0.84
0.84
Black
0.05
0.07
0.02
0.07
0.07
0.06
Hispanic
0.03
0.03
0.02
0.05
0.05
0.05
American Indian
0.00
0.00
0.00
0.00
0.00
0.00
Asian/Pacific Paccific Islander
0.02
0.02
0.03
0.02
0.02
0.02
Unknown
0.01
0.01
0.01
0.01
0.01
0.01
Forrstb-S
CFerest-O5
5-burman-O
S-burran-Os
D8-1camer-05
D6 Di-izamen-O5
D1-4eamme--ee
Mecaamme-MC--tinn -MC-liig
D1--4amm--MC Mi-keam--MM--iiig
13-4esme--C--hn -Te
3R-4eamer-te MC MC--Twe
T-lesmer-55
T-lemme-DS
T-lemme-MMG--
T-lemme-MC-
COOL
I-lemmr-tC--lww
T-lumen-C-Toe
Chome-CC
Chowe-CT
S-leomen-CT
5-lemmer-CT
T-lemme-CC
T-leeme-CC
35
2
50
Blas (ATE)
Blas (ATE)
(a) Small set covariates
(b) Extended covariates
Figure 13: Bias by clinical trial training set size WHI Data Application.. The distribution of bias scores over sampling repetitions is plotted. Post-processing initial T-learner with clinical trial data improves over T-learner-OS in the limited covariate setting.
55

---

Table 13: Hyperparameter settings for post-processing using MCBoost. Default settings are used for param- eters not listed.
(a) T-kcarner MC
Method
Implementation
Hyperparameter
Value
Ridge
mcboost
max_iter _ex_
10
alpha
1e-06
eta
0.1
geight_degree degree weight alpha
glmnet
Tree
mcboost
max_ _ttr
10
alpha eta
1e-06
0.1
deight_degree
rpart
maxdepth
(b) DR-leamner MC
Method
Implementation
Hyperparameter
Value
Ridge
mcboost
max max_ited iter
alpha
1e-06
eta
0.1
beight_degres
2
glmnet
alpha
0
s max_ iter
1
Tree
mcboost
5
alpha eta
1e-06
0.1
weight weight_degree
2
rpart
maxdepth
3
Note: 0.01 extended set of covariates setting.
56

---

Table 14: Hyperparameter settings of (baseline) CATE learners. not listod.
Default settings are used for parameters
Method CForest
Implementation
Hyperparameter
Value
grf
num. trees
2000
mtry
sprt
sample fraction
0.5
honesty. fraction
0.5
min.node.
5
T-.DR-learner
regression_foreet
num. trees
2000
mtry
(p)+())+20
sample fraction
0.5
honesty.fraction fraction
0.5
min. size
S-learner
ranger
num. trees
500
mtry
sqrt
nin..ode.sszz
5
replace
TRUE
sample. fraction
1
57

---

RL-NET
TL-NET
XL-RF
005
CForest-OS
S-learner-OS
050
DR-tearne--OS
DR-tearner-MC-Rinn
DR-teamer-MMC-Tee
DOOL
T-Heamer-SS
T-teamer-MC-Rlige
T-leamer-MC-Tree
05:00
0051
10
10
15
10
15
20
MSE (CATE)
(a) Observational data- and multi-accurate CATE estimation
RL-NET
TL-NET
XL-RF
000
054
CForest-CT
S-lemmer-CT
0000
T-learerr-CT
O54L
0051
10
20
30
10
20
30
40
10
20
30
40
50
MSE (CATE)
(b) Clinical trial data-based CATE estimation
Figure 14: MSE by 'true' CATE estimation method and clinical trial training set size with small set of covariates [WI Data Application.. The distribution of MSE scores over sampling repetitions is plotted.
T-learner-MC-Riddee outperforms other methods for most CT training set sizes and true CATE estimation techniques RI-NET and TL-NET.
58

---

RL-NET
TL-NET
XL-RF
250
005
CForest-OS
S-learner-OS
050
DR-tearne--OS
DR-tearner-MC-Ridg
DR-teamer-MMC-Tee
000L
T-Heamer-SS
T-teamer-MC-Rlige
T-leamer-MC-Tree
15:00
0051
10
20
10
20
10
20
30
MSE (CATE)
(a) Observational data- and multi-accurate CATE estimation
RL-NET
TL-NET
XL-RF
0
o5c
000
054
CForest-CT
S-lemmer-CT
0OO'
T-learnrr-CT
OSCL
0051
20
40
60
20
40
60
20
60
MSE (CATE)
(b) Clinical trial data-based CATE estimation
Figure 15: MSE by "true" CATE estimation method and clinical trial training set size with extended set of covariates (WH Data Application). The distribution of MSE scores over sampling repetitions is plotted. Multicalibration boosting yields little improvement as CForest-OS and T-learner-OS already achiove low MSF.
59


Reviews:
[{"review_id": "sMalF3E1Uk", "reviewer": "Reviewer_Gt65", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The robustness of unknown covariates shifts in CATE estimation is important and challenging, and it also has promising applications.\n2. The proposed method is novel, combining the strengths of multi-accurate learning to ensure a low estimation bias.\n3. The experiment evaluations are extensive, including a simulation study, a semi-synthetic case study, and an RCT experiment.\n4. The authors employed a lot of examples to introduce their motivation, which is easy for readers to understand.\n \n\nWeaknesses:\n1. The related work discussion in the introduction is somewhat scattered.\n2. It is advised to list contributions separately in the introduction.\n3. It is suggested that the definition of the propensity score be moved to before the overlap assumption.\n4. I suggest authors put the background in an individual section, rather than the Method section.\n5. Since the paper aims to improve the robustness of CATE under covariate distribution shifts, I suggest the authors discuss the quantified criterion of distribution shift (e.g. Kullback-Leibler Divergence) in the simulations section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a novel method for conditional average treatment effect estimation that is robust to unknown covariate shifts. The extensive experiments show the efficiency and excellent bias control of the proposed method under distribution shifts. This approach can also adapt to multiple kinds of covariate distribution shifts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the Weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "S4fEtRFpbF", "reviewer": "Reviewer_wUVC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nApplying multi-calibration technique to the estimation of CATE is an interesting research direction.\n\n### Weaknesses\n\nUnfortunately, the proposed method is based on misunderstandings of the original work (Kim et al. 2022) and is fundamentally flawed.\n\nThe writing is very sloppy, and the paper reads like cursory notes based on a quick read and application of the previous work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper applies recent advances in algorithmic fairness to the estimation of CATE. Two settings are considered: 1) a CATE estimator is trained on an unconfounded dataset and the goal to deploy it on another unconfounded dataset; and 2) a CATE estimator is trained on a large but confounded dataset and the goal is to deploy it on a small RCT dataset.  In both cases, it is expected that the estimators are valid on the target datasets, after post-processing estimators on target datasets by the multi-calibration technique from the algorithmic fairness literature.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the whole project should be reworked, and a whole new paper should be written. So please take my following comments as suggestions for future work.\n\n### Method \n\nThe current paper apparently says multi-calibration can deal with arbitrary covariate shifts. However, the problem addressed by Kim et al. (2022) is roughly the adaptation of the averaged outcome estimator from one of the treatment/control groups to the other group. That is, the difference between the two groups (populations) can be *completely specified* by the true propensity score, although the true propensity score is *unknown*. \n\nPerhaps the most obvious mistake in the current paper is that the multi-calibration technique is applied to the conditional outcome $\\mu_t(X)$. However, *what is multi-calibrated should be a propensity score* which, as stated in the 1st point, lies at the core of the technique. \n- The domain of the function $\\tilde{p}$ in Def 1 and 2, taken from the previous work, also shows this clearly. This is an example of taking cursory notes from the previous work without much thinking (or it can be noticed that multi-calibration should be applied to propensity scores).\n\nThe identification results in the current paper make no sense. In practice, Kim et al. (2022) address the problem of ATE error caused by the error of propensity score estimation, so that the former has a known bound if the latter is bounded (Universal Adaptability). That is, *the original work deals with the problem of estimation but not identification*, and, in fact, the identification, including under those settings in the current paper, is just textbook results of identification under unconfoundedness. \n\n### Writing \n\nThe paper does not use consistent terminology. For example, reference population and target population refer to the same thing.\n\nAnother general problem is unexplained terminology and symbols. For example, \n\n- reference/target population is not really explained in the Intro and their meaning can only be guessed from later parts; \n- \"<<\" and \"P-a.s.\" in Setting 1.\n\nSome explanations are not in the right place. Examples: \n\n- “pseudo-outcome regression” is mentioned in the Intro and Background, but only defined as late as page 11; \n\n- “MC-Boost” and “Auditing” are in Figure 1, but are explained nearly 3 pages later, and this makes Figure 1 not very meaningful.\n\nInsufficient/misleading explanations. Examples: \n\n- why it is required that the conditional expectation equals to 1 in the Setting 1? \n\n-  in Figure 1 for Setting 2, \"unknown $Q$\" is mentioned, while it is only used in the main text for Setting 1 but not Setting 2;\n\n- at the beginning of Sec 3.1, the eqs do not show the difference between S and T learners because $\\mu_t(x)$ is an equivalent way to write $\\mu(x, t)$; and it puzzles me why S-learner is mentioned here because S-learner is never mentioned again in the main text. \n\n- the paper spends quite some space on MCBoost and auditing while it adds no intuition but confusion, e.g., \"residuals\" and \"twisting\" are not clearly explained.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0P8EPlq4Qq", "reviewer": "Reviewer_Hf3b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The method works in general for pseudo-outcome regression, such as the DR-learner. They show how this approach can combine (large) confounded observational and (smaller) randomized datasets by learning a confounded predictor from the observational dataset, and auditing for multi-accuracy on the randomized controlled trial. \n\n\nThey show improvements in bias and mean squared error in simulations with increasingly larger covariate shift, and on a semi-synthetic case study of a parallel large observational study and smaller randomized controlled experiment. \n\n\nOverall, the authors establish a connection between methods developed for multi-distribution learning and achieve appealing desiderata (e.g. external validity) in causal inference and machine learning.\n\nWeaknesses:\n\nWhat are the hyperparameters setup?\n\nTheoretically speaking, what are the advantages of the proposed method?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a method to learn multi-accurate predictors to postprocess CATE T-learners (differenced regressions) to become robust to unknown covariate shifts at the time of deployment.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: First, let me thank the reviewers (particularly wUVC) and authors for the fruitful discussion, which has notably improved the submission.

After reviewing the submission, the reviewers' comments, and the authors' responses, I recommend acceptance with minor revision.
The authors revised their submission according to the reviewers' comments, convincingly addressed concerns raised during the response period, and improved their submission. Any remaining issues can addressed in the final revision. 

Specifically, given the lively discussion, I request the authors

* have one more pass over the manuscript to ensure any remaining inconsistencies are resolved (e.g. just after Assumption 4 a $\mathcal{D}_\text{o}$ has snuck in instead of $\mathcal{D}_\text{obs}$);
* add an outline to the paper in §1 and also an outline to the Method section (§4) to give an overview of their exposition;
* have another pass over the paper to ensure that concepts are explained or referenced for general readers in machine learning (estimation vs identification, etc) and assumptions are clear.

I'm looking forward to the camera-ready version of the paper. It makes a valuable contribution with practical implications and theoretical insights that will be of interest to the TMLR audience.
Decision: Accept (poster)