Selecting Treatment Effects Models for Domain Adaptation Using Causal Knowledge
Authors: Trent Kyono
Ioana Bica
Zhaozhi Qian
Mihaela van der Schaar
Venue:  (2021)
Keywords: causal inference
treatment effects
healthcare

Abstract: Selecting causal inference models for estimating individualized treatment effects (ITE) from observational data presents a unique challenge since the counterfactual outcomes are never observed. The problem is challenged further in the unsupervised domain adaptation (UDA) setting where we only have access to labeled samples in the source domain, but desire selecting a model that achieves good performance on a target domain for which only unlabeled samples are available. Existing techniques for UDA model selection are designed for the predictive setting. These methods examine discriminative density ratios between the input covariates in the source and target domain and do not factor in the model's predictions in the target domain. Because of this, two models with identical performance on the source domain would receive the same risk score by existing methods, but in reality, have significantly different performance in the test domain. We leverage the invariance of causal structures across domains to propose a novel model selection metric specifically designed for ITE methods under the UDA setting. In particular, we propose selecting models whose predictions of interventions' effects satisfy known causal structures in the target domain. Experimentally, our method selects ITE models that are more robust to covariate shifts on several healthcare datasets, including estimating the effect of ventilation in COVID-19 patients from different geographic locations.

Under review as a conference paper at ICLR 2021
SELECTING TREATMENT EFFECTS MODELS FOR DOMAIN ADAPTATION USING CAUSAL KNOWLEDGE
Anonymous authors Paper under double-blind review
ABSTRACT
Selecting causal inference models for estimating individualized treatment effects (ITE) from observational data presents a unique challenge since the counterfactual outcomes are never observed. The problem is challenged further in the unsupervised domain adaptation (UDA) setting where we only have access to labeled samples the source domain, but desire selecting model that achieves good performance on a target domain for which only unlabeled samples are available. Existing techniques for UDA model selection are designed for the predictive setting. These methods examine discriminative density ratios between the input covariates in the source and target domain and do not factor in the model's 's predictions in the target domain. Because of this, two models with identical performance on the source domain would receive the same risk score by existing methods, but in reality, have significantly different performance on the test domain. We leverage the invariance of causal structures across domains to introduce a novel model selection metric specifically designed for ITE models under the UDA setting. In particular, we propose selecting models whose predictions of the effects of interventions satisfy known causal structures in the target domain. Experimentally, our method selects ITE models that are more robust to covariate shifts on several synthetic and real healthcare datasets, including on estimating the effect of ventilation in COVID-19 patients from different geographic locations.
INTRODUCTION
Causal inference models for estimating individualized treatment effects (ITE) are designed to provide actionable intelligence as part of decision support systems and, when deployed on mission-critical domains, such as healthcare, require safety and robustness above all (Shalit et al., 2017: Alaa & van der Schaar, 2017). In healthcare, it is often the case that the observational data used to train an ITE model may come from a setting where the distribution of patient features is different from the one in the deployment (target) environment, for example, when transferring models across hospitals or countries. Because of this, it is imperative to select ITE models that are robust to these covariate shifts across disparate patient populations. In this paper, address the problem of ITE model selection in the unsupervised domain adaptation ( (UDA) setting where we have access to the response to treatments for patients on source domain, and we desire to select ITE models that can reliably estimate treatment effects on a target domain containing only unlabeled data, i.c., patient features.
UDA has been successfully studied in the predictive setting to transfer knowledge from existing labeled data in the source domain to unlabeled target data (Ganin et al., 2016; Tzeng et al., 2017). In this context, several model selection scores have been proposed select predictive models that are most robust to the covariate shifts between domains (Sugiyama et al., 2007; You et al., 2019). These methods approximate the performance a model on the target domain (target risk) by weighting the performance on the validation set (source risk) with known (or estimated) density ratios.
However, ITE model selection for UDA differs significantly in comparison to selecting predictive models for UDA (Stuart et al., 2013). Notably, we can only approximate the estimated counterfactual error (Alaa & van der Schaar, 2019), since we only observe the factual outcome for the received treatment and cannot observe the counterfactual outcomes under other treatment options (Spirtes et al., 2000). Consequently, existing methods for selecting predictive models for UDA that compute a weighted sum of the validation error as a proxy of the target risk (You et al., 2019) is suboptimal for

---

Under review as a conference paper at ICLR 2021
selecting ITE models, as their validation error in itself is only an approximation of the model's ability to estimate counterfactual outcomes on the source domain.
To better approximate target risk, we propose to leverage the invariance of causal graphs across domains and select ITE models whose predictions of the treatment effects also satisfy known or discovered causal relationships. It is well-known that causality is a property of the physical world, and therefore the physical (functional) relationships between variables remain invariant across domains (Schoelkopf et al., 2012; Bareinboim & Pearl, 2016; Rojas- Carulla et al., 2018; Magliacane et al., 2018). As shown in Figure 1, we assume the existence of an underlying causal graph that describes the generating process of the observational data. We represent the selection bias present in the source observational datasets by arrows between the features (X1, X2). and treatment T. In the target domain, we only have access to the patient features, and we want to estimate the patient outcome (Y) under different settings of the treatment (intervention). When performing such interventions, the causal structure remains unchanged except for the arrows into the treatment node, which are removed.
Inweront underigng caval mechertems
Cresal graph G deseritinz state generating precess
Interventional causal graph Cip
Intervene
T
doeT f
Amoe Blasi predectiin
Select carce Informce model ribrut to diertthutionl target
DD Dere I nh Pp(
Train
Cancidans
Digt I th Pr(
Votecvaticall daraedt
Px(X) (x(
datset
Conacitt dalls betwees denains
Sourse domaln
Tareet demain
Figure 1: Method overview. We propose selecting ITE model whose predictions of the treatment effects the target domain satisfy the causal relationships in the interventional causal graph Gy-
Contributions. To the best of our knowledge, we present the first UDA selection method specifi- cally tailored for machine learning models that estimate ITE. Our ITE model selection score uniquely leverages the estimated patient outcomes under different treatment settings on the target domain by incorporating measurement of how well these outcomes satisfy the causal relationships in the interventional causal graph GT. This measure, which we refer to as causal risk, is computed using a log- logliikelhhood function quantifying the model predictions' fitness to the underlying causal graph. We provide a theoretical justification for using the causal risk, and we show that our proposed ITE model selection metric for UDA prefers models whose predictions satisfy the conditional independence relationships in G7 and are thus more robust to changes in the distribution of the patient features. We also show experimentally that adding the causal risk to existing state-of-the-art model selection scores for UDA results in selecting ITE models with improved performance on the target domain. We provide an illustrative example of model selection for several real-world datasets for UDA, including ventilator assignment for COVID-19.
2 RELATED WORKS
Our work is related causal inference and domain adaptation. In this section, we describe existing methods for ITE estimation, UDA model selection in the predictive setting, and domain adaptation from causal perspective.
ITE models. Recently, large number of machine learning methods for estimating heterogeneous ITE from observational data have been developed, leveraging ideas from representation learning (Johansson et al., 2016; Shalit et al., 2017: Yao et al., 2018), adversarial training, (Yoon et al., 2018). causal random forests (Wager& & Athey, 2018) and Gaussian processes (Alaa & van der Schaar, 2017; 2018). Nevertheless, no single model will achieve the best performance on all types of observational data (Dorie et al., 2019) and even for the same model, different hyperparameter settings or training iterations will yield different performance.
ITE model selection. Evaluating ITE models' performance is challenging since counterfactual data is unavailable, and consequently, the true causal effects cannot be computed. Several heuristics for estimating model performance have been used in practice (Schuler et al., 2018; Van der Laan & Robins, 2003). Factual model selection only computes the error of the ITE model in estimating the factual patient outcomes. Alternatively. inverse propensity weighted (IPTW) selection uses the estimated propensity score weigh each sample's 's factual error and thus obtain unbiased estimate (Van der Laan & Robins, 2003). Alaa & van der Schaar (2017) propose using influence functions to approximate ITE models' error in predicting both factual and counterfactual outcomes. Influence
2

---

Under review as a conference paper at ICLR 2021
function (IF) based validation currently represents the state-of-the thet method in selecting ITE models. However, existing ITE selection methods are not designed to select models robust to distributional changes in the patient populations, i.c., for domain adaptation.
UDA model selection. UDA is a special case of domain adaptation, where we have access to unlabeled samples from the test or target domain. Several methods for selecting predictive models for UDA have been proposed (Pan & Yang. 2010). Here we focus on the ones that can be adapted for the ITE setting. The first unsupervised model selection method was proposed by Long al. (2018), who used Importance-Weigee Weighted Cross-Validatioe Validation (IWCV) (Sugiyama et al., 2007) to select hyperparameters and models for covariate shift. IWCV requires that the importance weights (or density ratio) be provided or known ahead of time, which is not always feasible in practice. Later, Deep Embedded Validation (DEV), proposed by You et al. (2019), was built on IWCV by using a discriminative neural network to learn the target distribution density ratio to provide an unbiased estimation of the target risk with bounded variance. However, these proposed methods do not consider model predictions on the target domain and are agnostic of causal structure.
Causal structure for domain adaptation. Recently, Kyono & van der Schaar (2019) proposed Causal Assurance (CA) domain adaptation selection method for predictive models that leverages prior knowledge in the form of a causal graph. Because their work is centered around predictive models, it is suboptimal for ITE models, where the edges into the treatment (or intervention) will capture the selection bias of the observational data. Furthermore, their method does not allow for examining the target domain predictions, which is key novelty of this work. We leverage do-calculus (Pearl, 2009) to manipulate the underlying directed acyclical graph (DAG) into an interventional DAG that more appropriately fits the ITE regime. More recently, researchers have focused on leveraging the causal structure for predictive models by identifying subsets of variables that serve as invariant conditionals (Rojas-Carulla Carulla et al., 2018; Magliacane et al., 2018).
PRELIMINARIES
3.1 INDIVIDUALIZED TREATMENT EFFECTS AND MODEL SELECTION FOR UDA
Consider a training dataset Dere- = {(ufre ,tre wfrr) J i= consisting of Ngre independent real- izations, one for each individual of the random variables T, Y) drawn from the source joint distribution p (X, T, Y). Let (X) be the marginal distribution of X. Assume that we also have
access to a test dataset Dige = {-r ]==1" from the target domain, consisting of Negs independent realizations of X drawn from the target distribution P* (X), where p., (X) & (X') Let the random variable X E X represent the context (e.g. patient features) and let 7 E describe the intervention (treatment) assigned to the patient. Without loss of generality, consider the case when the treatment is binary, such that T = {0, 1). However, note that our model selection method is also applicable for any number of treatments. We use the potential outcomes framework (Rubin, 2005) to describe the result of performing an intervention t E T as the potential outcome (t) E y Let Y(1) represent the potential outcome under treatment and Y (0) the potential outcome under control. Note that for each individual, we can only observe one of potential outcomes (0) or Y (1). We assume that the potential outcomes have a stationary distribution py (Y X)= = (Y (1) X) given the context X; this represents the covariate shift assumption in domain adaptation (Shimodaira, 2000).
Nage
Observational data can be used to estimate E| X = xx T = t] through regression. Assumption I describes the causal identification conditions (Rosenbaum & Rubin, 1983), such that the potential outcomes are the same as the conditional expectation: Et)(t) | X = x] = E X = x, T=T==t =f=], t).
Assumption (Consistency, Ignorability and Overlap). For any individual (wnit) i, receiving treatment t,. we observe Y, Yt,,)- Moreover, (Y, Y(1) and the data generating process P(X,T,Y) satisfy strong ignorability Y (0), (1) IL T X and overlap Vx P(T X = x) > O.
The ignorability assumption, also known the no hidden confounders (unconfoundedness) assump- tions, means that we observe all variables X that causally affect the assignment of the intervention and the outcome. Under unconfoundedness, X blocks all backdoor paths between Y and A (Pearl, 2009).
Under Assumption 1, the conditional expectation of the potential outcomes also be written the interventional distribution obtained by applying the do- operator under the causal framework of Pearl (2009): E[Y X x] = ElY X = sr, do t)). This equivalence will enable us to reason
3

---

Under review as a conference paper at ICLR 2021
about causal graphs and interventions causal graphs in the context of selecting ITE methods for estimating potential outcomes.
Evaluating ITE models. Methods for estimating ITE learn predictors f X'xxx T - such that f( t) approximates E|y X = x, T = t| = E X = x = E|y X = I, do(T = t)}. The
goal is to estimate the ITE, also known the conditional average treatment effect (CATE):
7 (z) E (1) X = x] - (0) X x]
(1)
E X = z, do = 1)] - B|Y X = 2, do = 0)]-
(2)
The CATE is essential for individualized decision making as guides treatment assignment policies. A trained ITE predictor f(x, t) approximates CATE as: f(z) = faz 1) - f(z, 0). Commonly used
to assess ITE models the precision of estimating heterogeneous effects (PEHE) (Hill, 2011):
PEНE- = Bezp(=)) [(T(x) - {2)))
which quantifies model's estimate of the heterogencous treatment effects for patients in population.
UDA model selection. Given a set F {f,,... fm] of candidate ITE models trained on the source domain Dercs our aim is to select the model that achieves the lowest target risk, that is the lowest PEHE on the target domain Digt. Thus, ITE model selection for UDA involves finding:
f arg min Eaap, (m)(((()) - (z))² fEF
arg min Ex-p, (z) [[[(2) (f(x 1) f(z 0))) (4)
fer
For this purpose, we propose using the invariance of causal graphs across domains to select ITE predictors that are robust to distributional shifts in the marginal distribution of X.
3.2 CAUSAL GRAPHS FRAMEWORK
In this work, we use the semantic framework of causal graphs (Pearl, 2009) to reason about causality in the context of model selection. We assume that the unknown data generating process in the source domain can be described by the causal directed acyclic graph (DAG) G, which contains the relationships between the variables = (X, Y) consisting of the patient features X, treatment T, and outcome Y. We operate under the Markov and faithfulness conditions (Richardson, 2003; Pearl, 2009), meaning that any conditional independencies in the joint distribution of p P,,(X, Y) are indicated by d-separation in G and vice-versa.
In this framework, an intervention on the treatment variable T E V is denoted through the do- operation do(T 1) and induces the interventional DAG Cr. where the edges into T are removed. The interventional DAG Gy corresponds to the interventional distribution p (X,Y, Y do(T = ()) Pearl (2009). The only node on which perform interventions in the target domain is the treatment node. Consequently. this node will have the edges into it removed, while the remainder of the DAG is unchanged. We assume that the causal graph is invariant across domains (Schoelkopf al., 2012; Ghassami et al., 2017; Magliacane et al., 2018) which we formalize for interventions as follows:
Assumption (Causal invariance). Let V = (X,T,Y Y) be a set of variables consisting of patient features X, treatment T, and outcome Y. Let & be set of domains, Ps(X, Y do(TT = =)) be the corresponding interventional distribution on V in domain 5 E A, and (V)) denote the set of all conditional independence relationships embodied in (VV).. then
vō,, 5j E
Y do(T = ())) = T(ps, (X do(T = 1)))
(5)
4 ITE MODEL SELECTION FOR UDA
Let F= [fi, f2.... fmm be set of candidate ITE models trained on the data from the source domain Dare. Our aim is to select the model f E J that achieves the lowest PEHE on the target domain Digt, as described in Equation 4. Let G be causal graph, either known or discovered, that describes the causal relationships between the variables in X, the treatment T and the outcome Y. Let Gy be the interventional causal graph of G that has edges removed into the treatment variable T.
Prior causal knowledge and graph discovery. The invariant graph G can be arrived at in two primary ways. The first would be through experimental means, such randomized trials, which
4

---

Under review as a conference paper at ICLR 2021
does not scale to large number of covariates due financial or ethical impediments. The second would be through the causal discovery of DAG structure from observational data (for a listing of current algorithms
refer to (Glymour et al., 2019b)), which is more feasible in practice. Under the assumption of no hidden confounding variables, score-based causal discovery algorithms output a completed partially directed acyclical graph (CPDAG) representing the Markov equivalence class (MEC) of graphs, i.c., those graphs which are statistically indistinguishable given the observational data and therefore share the same conditional independencies. Provided CPDAG, it up to an expert (or further experiments) to orient any undirected edges of the CPDAG to convert it into the DAG (Pearl, 2009). This step is the most error-prone, and we show in our real data experiments how subgraph (using only the known edges) can still improve model selection performance.
Improving target risk estimation. For the trained ITE model f. let O(0) = [(xr 0) and let (1) = (xr, 1) be the predicted potential outcomes for x ~ (r:)... We develop a selection method that prefers models whose predictions on the target domain preserve the conditional independence relationships between X, I and Y in the interventional DAG GT with edges removed into the treatment variable T. We first propose a Theorem, which we later exploit for model selection.
Theorem 1.
Let py (X, TYY be a source distribution with corresponding DAG G. If f(X,T), ie., is an optimal ITE model, then
Io(G,) (( ((X)) do(T'=
(6)
where Pm (X. ((X,())) () | do(T = []) is the interventional distribution for the target domain and IG(G)) and (p (X, fX) do(T t))) returns all the conditional independence relationships in GT and Pa (X, f(X,t) do(T = f)). respectively.
For details and proof of Theorem see Appendix B. Theorem 1 provides an equality relating the predictions of f in the target domain to the interventional DAG G-. Therefore we desire the set of independence relationships in Gy to equal (X, f(Xtt) do(T = t)))- In our case, we do not have access to the true interventional distribution p= (X, fXX, t) do(T t)), but we can approximate it from the dataset obtained by augmenting the unlabeled target dataset Dege with the
( model's predictions of the potential outcomes: Digs ((x 00, o/at (0)), (1/00 1, g. (1)) s=1
Nugt
where " (t) f( .t), for 2/ E Digt. We propose improve the formalization in Eq. by adding a constraint on preserving the conditional independencies of G7 as follows:
mig Rf( s.t. B[NCI(G7 Degg() = 00,
fer
where RT(() is a function that approximates the target risk for model S. NCT Digt) is the number of conditional independence relationships in the graph G7 that are not satisfied by the test. dataset augmented with the model's 's predictions of the potential outcomes Digt-
Interventional causal model selection. Consider the schematic in Figure 2. We pro- pose an interventional causal model selec- tion (ICMS) score that takes into account the model's 's risk on the source domain, but also the fitness to the interventional causal graph Gy on the target domain according to Eq. 4. A score that satisfies this is provided by the Lagrangian method:
[teff bopp. 14 (oWW
trcelasce diects
madd
Butiont
11 Gpt -ccic 12ga
Time
Prifictions Parta
Valibatian Lose
D+
Tarpet
Deleut
Target Risk Felleate
Dept
Density
W. Diy)
ICMS
Figure 2: ICMS is unique in that it caleulates a causal risk (green) using predictions on target data. Purple arrows denote pathways unique to ICMS.
C=
Rr (f) + XE|NCI Dige)l- (8)
The first term RT is equivalent to the expected test PEHE which at selection time can be approx- imated by the validation risk (either source or target risk), which we represent as e, (f, D.., Dege). The second term, [[N Drge)), which is derived from Theorem 1, evaluates the number of conditional independence relationships resulting from d-separation in the graph Gy that are not satisfied by the test dataset augmented with the model's predictions of the potential outcomes Degr- However, this term may never equal 0 and directly minimizing NCI(G Dige) involves evaluating conditional independence relationships, which is a hard statistical problem, especially for continuous variables (Shah et al., 2020). Because of this approximate it by using a causal fitness score
5

---

Under review as a conference paper at ICLR 2021
that measures the likelihood of a DAG given the augmented dataset Digt. which we rewrite as cf Digt, GT). This represents an alternative and equivalent approach, also used by score- score-based causal discovery methods (Ramsey et al., 2017b; Glymour et al., 2019c). Consider partitioning the source dataset Darc= (argro t. y ) j=l"" into a training dataset Dtr and validation dataset Dr such that Dare = Der U D.. From Eq. we define our ICMS score r as follows:
Definition 1 (1CMS score). Let f be an ITE predictor trained on Dtr- Let Digt = ) Nuge be test dataset and let Gr be the interventional causal graph. We define the following selection score: r(f, De, Digt, GT) = vr Dv. Degt) + Xxx (f, Deges GT)
(9)
where Ur measures the validation risk on the validation set Dr and er is a scoring, function, which we call causal risk, that measures the fitness of the interventional causal graph G7 to the dataset
Digt- {(x]0 O, y." (0)), (x/ .1.0% (.00(9((1)) (1)) where g (t) = f( .t) x E &Digt-
The validation risk (f D.. Degs) can either be (1) source risk where we use existing model selection scores for ITE (Alaa & van der Schaar, 2019; Van der Laan & Robins, 2003), or (2) approximation of target risk using the preexisting methods of IWCV or DEV (Sugiyama et al., 2007; You et al., 2019). We describe in the following section how to compute the causal risk or Degt: G..). À is a tuning factor between our causal risk term and validation risk ür- We currently set X = 1 for our experiments, but ideally, would be proportional to our certainty in our causal graph. We discuss alternative methods for selecting X, as well as a X sensitivity analysis in Appendix F. We provide ICMS pscudocode and graphical illustration for calculating ICMS in Appendix C. We provide additional practical considerations and experiments regarding computational complexity, a subgraph analysis, causal graph misspecifications, ICMS selection on tree- rree-baed methods, ICMS selection on causally invariant features, noisiness of fitness score, and additional further discussion in Appendix H.
Assessing causal graph fitness. The causal risk term e Digtt GT) as part of our ICMS score requires assessing the fitness of the dataset Degt to the invariant causal knowledge in Gr- Some options include noteworthy maximum- likelihood algorithms such as the Akaike Information Criterion (AIC) (Akaike, 1998) and Bayesian Information Criterion (BIC) (Schwarz, 1978). Both the BIC and AIC are penalized versions of the log-likelihood function of a DAG given data, e.g., LL(GT Digt).. In score based causal discovery, the DAG that best fits the data will maximize the LL(C, Dege) subject to some model complexity penalty constraints. In this work, we are not searching between candidate causal graphs and only care about maximizing our DAG to dataset fitness. Thus, we use the negative lioeliill of given Degt, i.e., LC((G Degt). for our causal risk term c,. The -CC(GT| Dege) has a smaller value when G is closer to modeling the probability distribution in Digt, i.e., the predicted potential outcomes satisfy the conditional independence relationships in G.
In score-based causal discovery, the Bayesian Information Criterion (BIC) is a common score that is used to discover the completed partially directed acyelic graph (CPDAG), representing all DAGs in the MEC, from observational data. Under the Markov and faithfullness assumptions, every conditional independence in the MEC of G' is also in DD. The BIC score is defined as:
(log2 N
BIC = - (GGD) +
|G]]]
(10)
2
where N is the data set size and ||G|| is the dimensionality of G. For our function f in Eq. 9, we use the BIC score. However, since N and ||G|| are held constant in our proposed method our function f x LD).. To find the LL(G)D) we use the following decomposition:
LL(G -N Σ (X,)))AA),
(11)
X,PA:
where N is the dataset size, PA are the parent nodes of X; in G, and H is the conditional entropy function which is given by (Darwiche, 2009) for discrete variables and by (Ross, 2014) for continuous or mixed variables.
Limitations of UDA selection methods In the ideal scenario, we would be able to leverage labeled samples in the target domain to estimate the target risk of machine learning model. We can express the target risk Regs in terms of the testing loss as follows:
Rege~
Σ (((ro0())) y (o)) (f(atot, 1) ((='0t,0)3
(12)
Negt
6

---

Under review as a conference paper at ICLR 2021
However, in general, we do not have access to the treatment responses for patients in the target set and, even if we did, we can only observe the factual outcome. Moreover, existing model selection methods for UDA only consider predictions on the source domain and do not take into account the predictions of the candidate model in the target domain. Specifically, DEV and IWCV calculate density ratio or importance weight between the source and target domain as follows:
p(d = 1/x) Nore
w (x)
(13)
p(d = 0/r)) Nieat
where d designates dataset domain (source is O, target is 1), and p(d=1la) p(d=0ja) can be estimated by discriminative model to distinguish source from target samples (You et al., 2019). Both calculate their score as a function of Á as follows:
Σ wy (x)()))) (Et) 0), S(,
(14)
where ((-..-)) is a validation loss, such as influence-uuncion based validation (Alaa & van der Schaar, 2019). Note that the functions I and are only defined in terms of validation features x! from the source dataset. Such selection scores can be used to compute the validation score V, (,, Dr, Dtge) part of the ICMS score.
However, our ICMS score also computes the likelihood of the interventional causal graph given the predictions of the model in the target domain as a proxy for the risk in the target domain. By adding the causal risk, we the improve the estimation of target risk. Additionally, we specifically make use of the estimated potential outcomes on the test set (x*99 0) and (= 1) to calculate our selection score as shown in Eq. 9. Fig. 2 depicts how we use the predictions of the target data to calculate our ICMS score.
EXPERIMENTS
We evaluate methods by the test performance in terms of the average PEHE of the top 10% of models in the list returned by the model selection benchmarks. We will refer to this as the PEHE-10 test error. We provide additional metrics for our results in Appendix G.I.
Benchmark ITE models. We show how the ICMS score improves model selection for state-of- the-art ITE methods based on neural networks: GANITE (Yoon et al., 2018), CFRNet (Johansson et al., 2018 8), TARNet (Johansson et al., 201 8), SITE (Yao et al., 2018) and Gaussian processes: CMGP (Alaa & van der Schaar, 2017) and NSGP (Alaa van der Schaar, 2018). These ITE methods use different techniques for estimating ITE and currently achieve the best performance on standard benchmark observational datasets (Alaa & van der Schaar, 2019). We iterate over each model multiple times and compare against various DAGs and held-out test sets. Having various DAG structures results in varying magnitudes of test error. Therefore, without changing the ranking of the models, we min-max normalize our test error between 0 and 1 for each DAG, such that equal weight is given to each experimental run, and a relative comparison across benchmark ITE models can be made.
Benchmark methods. We benchmark our proposed ITE model selection score ICMS against. each of the following UDA selection methods developed for predictive models: IWCV (Long et al., 2018) and DEV (You et al., 2019). To approximate the source risk, i.e., the error of ITE methods in predicting potential outcomes on the source domain (validation set D.), we use the following standard ITE scores: MSE on the factual outcomes, inverse propensity weighted factual error (IPTW) (Van der Laan & Robins, 2003) and influence functions (IF) (Alaa & van der Schaar, 2019). Note that each score (MSE, IPTW, etc.) can be used to estimate the target risk the UDA selection methods: IWCV, DEV, or ICMS. Specifically, we benchmark our method in conjunction with each combination of ITE model errors [MSE, IPTW, IF) with validation risk (9. IWCV, DEV). We include experiments with 0, to demonstrate using source risk an estimation of validation risk.
5.1 SYNTHETIC UDA MODEL SELECTION
Data generation. In this section, we evaluate our method in comparison to related selection methods on synthetic data. For each of the simulations, we generated a random DAG, G, with
7

---

Under review as a conference paper at ICLR 2021
Table 1: PEHE-10 performance (with standard error) using ICMS on top of existing UDA meth- ods. ICMS( means that the was used as the validation risk 2, in the ICMS. For example, ICMS(DEV(*)) represents (EV(+) selection used as the validation risk e, in the ICMS. The indicates the method used to approximate the validation error on the source dataset. Our method (in bold) improves over each selection method over all models and source risk scores (Src.).
SELECTION METBOD
GANITE
TAR
CMGP
NSGP
MSE
0.395 (0.051)
0.363 (0 (42)
0.391 (0.050)
0.157 (0.055)
0.131 (V.046)
0.282 (0.049)
ICMS(MSE)
0.222 (0.049)
0.212 (0.036)
0.264 (0.034)
0.126 (0.027)
0.120 (0.050)
0.210 (0.047)
IWCV(MSE)
0.348 (46)
0.393 044044
0.364 (0 052)
0.185 (0:053)
0.201 (0.041)
0.209 (41)
ICMSIVCV(MSEE)
0.212 (0,043)
0.220 (0.051)
0.256 (0.039)
0.149 (0.033)
0.183 (0.055)
0.172 (0.043)
DEV(MSH)
0.398 056)
0.414 042)
0.427 049)
0.198 (0.0:3)
0.239 (0. (.0)
0.183 (0 (1.42)
ICMS(DE
0.224 (0.042)
0.210 (0.0399
0.269 (0.035)
0.120 (0.0400
0,160 (0,047)
0.169 (0.042)
IPTW
0.381 (499)
0.355 (0. 046)
0.394 (0.052)
0.357 (0.045)
0.182 (0. D46)
0.292 (0.045)
ICMSAPTW)
0.220 (0.049)
0.217 (0.039)
0.272 (0.082)
0.228 (0.031)
0.149 (0.050)
0.207 (0.047)
[WCVUPTW]
0.269 055)
0,518 (.49)
0,453 0138)
0.416 (0.053)
0.417 (0.043)
0.475 (0.053)
ICMPIWCY(PTW)
0,053 (0.028)
0.121 (0.1134)
0.119 (0.035)
0.207 (0.139)
0.304 (0.059)
0.328 (0.058)
DEV(PTTW
0.302 (0.072)
0.472 (0.056)
0.114 (0. 041)
0,400 (0.057)
0.441 (0.071)
0.193 056)
ICMSDEV(PTTV)
0.087 (0.035)
0.194 (0.052)
0.120 (0.027)
0.220 (0.031)
0,282 (0.041)
0.355 (0.050)
(414)))
0.255 (0. 050)
0.250 (0. (4.0466)
0.321 (0.059)
0.392 (0.051)
(0.057)
(CMSOF)
0.127 (0.039)
0.166 (0.042)
0.190 (0.044)
0.215 (0.056)
0.212 (0.053)
0,250 (0.054)
IWCV(F)
0.180 059)
0.364 (0. (0.051)
0.286 D41)
0.293 (0.043)
0.415 (0. (0.048)
0.437 (0 057)
(CM))
0.058 (0.018)
0.104 (0.025)
0.108 (0.033)
0.173 (0.028)
0,292 (0.062)
(0.051)
DEV(TF)
0.193 (0 (55))
0.415 (0. 045)
0.292 (D 0446
0.214 (D) (0.058)
0,490 (0. (43)
0.544 (0.053)
ICMS(DEV(II)
0.069 (0.026)
0.191 (0.048)
0.107 (0.029)
0.147 (0.025)
0.229 (0.054)
0.364 (0.056)
vertices and up n(n 1)/2 edges (the asymptotic maximum number of edges in a DAG) between them. We construct our datasets with functional relationships between variables with directed edges between them in G and applied Gaussian noise (0 mean and 1 variance) each. We provide further details and pseudocode in Appendix G.1. Using the structure of G, we synthesized 2000 samples for our observational source dataset Derc. We randomly split Dare into a training set Der and validation set DD with 80% and 20% of the samples, respectively. To generate the testing dataset Digt, we use G to generate 1000 samples where half of the dataset receives treatment, and the other half does not. For D Digt, we randomly shift the mean between I and 10 of at least one ancestor of Y in G, whereas in Dare& a mean of 0 is used. It important to note that the actual outcome or response is never seen when selecting contains selection bias into the treatment node, whereas the synthetic test set Dige does not, since it was generated by intervention at the treatment node. Our algorithm has only access to the covariates X in Digt-
1)/2
models. Furthermore, the training dataset Dare is observational and
Improved selection for all ITE models. Table I shows results of ICMS on synthetic data over the benchmark ITE models. Here, we evaluate three different types of selection baseline methods: MSE, IPTW, and IF. We then compare each baseline selection method with UDA methods: IWCV, DEV, and ICMS (proposed). We repeated the experiment over 50 different DAGs with 30 candidate models for each model architecture. Each of the candidate algorithms was trained using their published settings and hyperparameters, as detailed in Appendix E. In Table 1, we see that our proposed method (ICMS) improves on each baseline selection method by having lower testing error in terms of PEHE-10 (and inversion count in Appendix G) over all treatment models.
5.2 APPLICATION TO THE COVID-19 RESPONSE
ICMS facilitates and improves model transfer across domains with disparate distributions, i.e., time, geographical location, etc.. which we will demonstrate in this section for COVID-19. The COVID-19 pandemic challenged healthcare systems worldwide. At the peak of the outbreak, many countries experienced a shortage of life-saving equipment, such as ventilators and ICU beds. Considering data from the UK outbreak, the pandemic hit the urban population before spreading to the rural areas (Figure 3). This implies that if we reacted in a timely manner, we could transfer models trained on the urban population to the rural population. However, there is significant domain shift as the rural population is older and has more preexisting conditions (Armstrong et al., 2020). Furthermore, at the time of model deployment in rural areas, there may be no labeled samples available. The characteristics of the two populations are summarized in Figure 3. We provide detailed dataset details and patient statistics in Appendix J.
8

---

Under review as a conference paper at ICLR 2021
Dally masyraa cramaaae)
Urban
Rural
weeks
Age (median)
65
Cocaity
Cihooki
Chrono
Sec: male
65%
61%
Age
Chronic Resp.
Averila
Chronic
3.5%
Obesity
Hypertension
Asdimaa
Hyperiese
Clronic
APm
Figure 3: Left: COVID-19 pandemic hit urban areas before spreading to rural areas. Middle: Feature subset showing there exists significant covariate shift between urban and rural populations with the urban population younger and with fewer preexisting conditions. Right: Discovered COVID-19 DAG.
COVID-19 Ventilation UK (urban) UK (rural). Using the urban dataset, we performed causal discovery on the relationships between the patient covariates, treatment, and outcome. The discovered graph (Figure 3) agree well with the literature (Williamson al., 2020; Niedzwiedz al., 2020). To be able to evaluate the ITE methods on how well they estimate all counterfactual outcomes, we created a semi- semi-synthetic version of the dataset with outcomes simulated according to the causal graph. Refer to Appendix J for details of the semi-synthetic data simulation. Our training observational dataset consists of the patient features, ventilator assignment (treatment) for the COVID-19 patients in the urban area, and the synthetic outcome generated based on the causal graph.
For each benchmark ITE model, we used 30 different hyperparameter settings and trained the various models to estimate the effect ventilator use on the patient risk of mortality. We used the same training regime in the synthetic experiments and the discovered COVID-19
Improving modal selectin for GANITE
Patentteehn win improved aulcomee
causal DAG (using FGES Ramsey al. (2017a) shown in Figure 3. We evaluated the best ITE model selected by each model selection method in ventilator assign- ment task. Using each selected ITE model, we assigned 2000 ventilators to the rural area patients that would have the highest estimated benefit (individualized treatment effect) from receiving the ventilator. Using the known synthetic outcomes for each patient, we then computed how many patients would have improved outcomes using each selected ITE model for assigning ventilators. By
(CNPP))
Figure 4: Performance of model selection methods in terms of the additional num- ber of patients with improved outcomes compared to selecting models based on the factual error on the source domain.
considering selection based on the factual outcome (MSE) on the source dataset as a bascline, in Figure 4, we computed the additional number of patients with improved outcomes by using ICMS on top of existing UDA methods when selecting GANITE models with different settings of the hyperparameters. We see that ICMS (in blue) identified the GANITE models that resulted in better patient outcomes in the UK's rural areas without access to labeled data. We include additional experimental results in Appendix J.
Additional experiments. On the TWINS dataset (Almond et al., 2005) (in Appendix I), we show how our method improves UDA model selection even with partial knowledge of the causal graph (i.e., using only a known subgraph for computing the ICMS score). Note also that in the Twins dataset, we have access to real patient outcomes. Moreover, we also provide additional UDA model selection results for transferring domains on prostate cancer dataset and the Infant Health and Development Program (IHDP) dataset (Hill, 201 in Appendix I.
CONCLUSION
We provide a novel ITE model selection method for UDA that uniquely leverages the predictions of candidate models on a target domain by preserving invariant causal relationships. To the best of our knowledge, UDA. We provide a theoretical justification for using ICMS and have shown on a variety of synthetic, semi-synthetic, and real data that method can improve on existing state-of-the-art UDA methods.
have provided the first model selection method for ITE models specifically for

---

Under review as a conference paper at ICLR 2021
REFERENCES
Hirotogu Akaike. Information Theory and an Extension of the Maximum Likelihood Principle, pp. 199-213. Springer New York, New York, NY, 1998. ISBN 778-1-4612-1694-00 doi: 10.1007/ 978-1-4612- 11112--694-0-1..-
Ahmed Alaa and Mihacla van der Schaar. Limits of estimating heterogeneous treatment effects: Guidelines for practical algorithm design. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 139-138, 2018.
Ahmed Alaa and Mihacla van der Schaar. Validating causal inference models via influence functions. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 Proceedings of Machine Learning Research, pp.
191-201, 2019.
Ahmed M Alaa and Mihaela der Schaar. Bayesian inference of individualized treatment effects using multi-task gaussian processes. In Advances in Neural Information Processing Systems, pp. 3424-3432, 2017.
Ahmed M Alaa, Michael Weisz, and Mihaela Van Der Schaar. Deep counterfactual networks with propensit-droopou. arXiv preprint vxXiv:1706.05966 2017.
Douglas Almond, Kenneth Y Chay, and David S Lee. The costs of low birth weight. The Quarterly Journal of Economics, 1003))1031-1083 2005.
Jacob Armstrong, Justine K. Rudkin, Naomi Allen, Derrick W. Crook, Daniel J. Wilson, David H. Wyllie, and Anne Marie O'Connell. Dynamic linkage of covid-19 test results between public health england's second generation surveillance system and uk biobank. Microbial Genomics, 6(7):e000397, 2020. doi: http://dbb.or//11.00999///oooo.o URL https: www. mi wtt.://www..ooo....aa....ooooooa http://ottttttt/ouuuaaaa//////// 1099 /mgen./g.....//mm 0. 000397.
Elias Bareinboim and Judea Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences, 113327):7345-3352 2016. ISSN 0027-8424. doi: 10. 10.1073/pnas.
1510507113.
Hugh A. Chipman, Edward I. George, and Robert E. McCulloch. Bart: Bayesian additive regression trees. Annals of Applied Statistics, ((1):266-295, 2010.
Adnan Darwiche. Modeling and Reasoning with Bayesian Networks. Cambridge University Press, New York, NY, USA, 1st edition, 2009. ISBN 0521884381, 9780521884389.
Vincent Dorie, Jennifer Hill, Uri Shalit, Mare Scott, Dan Cervone, et al. Automated versus do-it- yourself methods for causal inference: Lessons learned from data analysis competition. Statistical Science, 34(1):43-68 68, 2019.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17()):096-2030, 2016.
AmirEmad Ghassami, Saber Salehkaleybar, Negar Kiyavash, and Kun Zhang. Learning causal structures using regression invariance. In Guyon, U. V. Luxburg. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp- 301 1-3011. Curran Associates, Inc., 2017.
Clark Glymour, Richard Scheines, Peter Spirtes, and Joseph Ramsey. Tetrad, 2019a. URL http: //www. phil. cmu.
html.
Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based on graphical models. Frontiers in Genetics, 10:524, 2019b.
Clark Glymour, Kun Zhang, and Peter Spirtes. Review causal discovery methods based on graphical models. Frontiers in genetics, 10:524, 2019c.
10

---

Under review as a conference paper at ICLR 2021
Jennifer L Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217-240, 2011.
Fredrik Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual
inference. In International conference on machine learning, pp. 3020-3029, 2016.
Fredrik Johansson, Nathan Kallus, Uri Shalit, and David Sontag. Learning weighted representations for generalization across designs. arXiv preprint arXiv: 2018.
Trent Kyono and Mihaela van der Schaar. Improving model robustness using causal knowledge. CoRR, abs/191 1. 12441, 2019.
Mingsheng Long, ZHANGJIE CAO, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In S. Bengio, H. Wallach, H. Larochelle, Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 1640-1650. Curran Associates, Inc., 2018.
Sara Magliacane, Thijs Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 10846-10856. Curran Associates, Inc., 2018.
Claire L Niedzwiedz, Catherine A O' D'Dnnell, Bhautesh D Jani, Evangelia Demou, Frederick K Ho, Carlos Celis-Morales, Barbara Nicholl, Frances Mair, Paul Welsh, Naveed Sattar, Jill Pell, and Srinivasa Vittal Katikireddi. Ethnic and socioeconomic differences in sars-cov-2 in- fection: prospective cohort study using uk biobank. medRxiv, 2020. doi: 10.1101/2020.04.222 20075663. URL https: //tp://wwwwwww. htt/:/wwwwwxxxx////////////ttttt org/ http:///nneeeeeee//e/ee//ee 04 02.22.2 20075663.
J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering,
1359, Oct 2010. ISSN 1041-4347. doi: 10.1 10.1109/KDE.2000..99
Pearl. Causality. Causality: Models, Reasoning, and Inference. Cambridge University Press, 2009.
ISBN 9780521895606.
Joseph Ramsey, Madelyn Glymour, Ruben Sanchez-Romerd and Clark Glymour. A million variables and more: the fast greedy equivalence search algorithm for learning nigh-dimensional graphical causal models, with an application to functional magnetic resonance images. International Journal of Data Science and Analytics, 3(2):121-129, Mar 2017a. ISSN 2364-4168. doi: 10.1007/
s4 1410600001-0032--1
Joseph Ramsey, Madelyn Glymour, Ruben Manchez-Roment and Clark Glymour. A million variables and more: the fast greedy equivalence search algorithm for learning high nigh-dimensional graphical causal models, with an application to functional magnetic resonance images. International journal of data science and analytics, 3(2):121-129, 2017b.
Thomas Richardson. Markov properties for acyclic directed mixed graphs. Scandinavian Journal of Statistics, 30(1):145-15) 2003.
Mateo Rojas-Carulla Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. Journal of Machine Learning Research, 19(36):1334 2018.
Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41 1--55,, 1983.
Brian C. Ross. Mutual information between discrete and continuous data sets. PLOS ONE, 9(2):15, 02 2014. doi: 10.
Donald B Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469):322-331 2005.
11

---

Under review as a conference paper at ICLR 2021
Bernhard Schoelkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa. Kun Zhang, and Joris Mooij. On causal and anticausal learning. Proceedings of the 29th International Conference on Machine Learning, ICML 2012, 2, 06 2012.
Alejandro Schuler, Michael Baiocchi, Robert Tibshirani, and Nigam Shah, A comparison of methods for model selection when estimating individual treatment effects. arXiv preprint arXiv:1804.0511 06,46, 2018.
Gideon Schwarz. Estimating the dimension of model. The Annals of Statistics, 6(2):461 1978. ISSN 00905364.
Rajen Shah, Jonas Peters, et al. The hardness of conditional independence testing and the generalised covariance measure. Annals of Statistics, 48(3): 2020.
Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: general- ization bounds and algorithms. In Proceedings of the 34th International Conference on Machine Learning-Volee Volume 70, pp. 3076-3085. JMLR. org, 2017.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
Ilya Shpitser and Eli Sherman. Identification personalized effects associated with causal pathways. Conference on Uncertainty in Artificial Intelligence, 2018, 8, 08 2018.
Peter Spirtes, Clark Glymour, Scheines N., and Richard. Causation, Prediction, and Search. Mit
Press: Cambridge, 2000.
Elizabeth A Stuart, Eva DuGoff, Michael Abrams, David Salkever, and Donald Steinwachs. Esti- mating causal effects in observational studies using electronic health data: challenges and (some) solutions. Egems, 1()), 2013.
Masashi Sugiyama, Matthias Krauledat, and Robert Müller. Covariate shift adaptation by importance weighted cross validation. J. Mach. Learn. Res., 8:985-1005, December 2007. ISSN 1532-4435.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 7167-7176, 2017.
Mark J Van der Laan and James M Robins. Unified methods for censored longitudinal data and causality. Springer Science & Business Media, 2003.
Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):1228-1222 2018
Elizabeth Williamson, Alex J Walker, Krishnan J Bhaskaran, Seb Bacon, Chris Bates, Caroline E Morton, Helen J Curtis, Amir Mehrkar, David Evans, Peter Inglesby. Jonathan Cockburn, Helen I Medonald, Brian MacKenna Laurie Tomlinson, Ian J Douglas, Christopher T Rentsch, Rohini Mathur, Angel Wong, Richard Grieve, David Harrison, Harriet Forbes, Anna Schultze, Richard Croker, John Parry, Frank Hester, Sam Harper, Rafael Perera, Stephen Evans, Liam Smeeth, and Ben Goldacre. Factors associated with covid- 19-related death using opensafely. Nature, 584, 2020. doi: 10.1038/s41586-000 2521444
Liuyi Yao, Sheng Li, Yaliang Li, Mengdi Huai, Jing Gao, and Aidong Zhang. Representation learning for treatment effect estimation from observational data. In Advances in Neural Information Processing Systems, pp. 2633-2643, 2018.
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Ganite: Estimation of individualized treatment effects using generative adversarial nets. International Conference on Learning Repre- sentations (ICLR), 2018.
Kaichao You, Ximei Wang, Mingsheng Long, and Michael Jordan. Towards accurate model selection in deep unsupervised domain adaptation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7124-7133, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
12

---

Under review as a conference paper at ICLR 2021
A WHY USE CAUSAL GRAPHS FOR UDA?
To motivate our method, consider the following hypothetical scenario. Suppose we have X1, X2, T. and Y representing age, respiratory comorbidities, treatment, and COVID-19 mortality, respectively, and the causal graph has structure X1 X2 + Y + T. Suppose that each node was simple linear function of its predecessor with i.i.d. additive Gaussian noise terms. Now consider we have two countries A and B, where A has already been hit by COVID-19 19 and B is just sceing cases increase (therefore have no observed outcomes yet). B would like to select a machine learning model trained on the patient outcomes from A. However, A and B differ in distributions of age X1. Consider the regression Y on X1, X2 and T, i.e., Y = e1 X1 + 02 + C3 by two models fi and f2 that are fit on the source domain and evaluated for 02 and cg. but differ in C1, where C₁ 0 for fi and C₁ & 0 for f2. We know that Y is a function of only X, and T. Thus in the shifted test domain, f1 must have lower testing eπror than f2, since the predictions of f2 use X1 (since # 0) and f1 does not. Furthermore the predictions of fi have the same causal relationships and conditional independencies
the target domain. Suppose that f1 and f2 have the same value
Y, such as fi (X1, X2, T) IL X2 X1- This is not the case for f2, where (2(1,, X2. 7) A. X2 X1. Motivated by this, we can use metric of graphical fitness of the predictions of Ji to the underlying graphical structure to select models in shifted domains when all have unlabeled samples. As added bonus, which will highlight later, unlike existing UDA selection methods our method can be used without needing to share data between A and B, which can help overcome patient privacy barriers that are ubiquitous in the healthcare setting.
B PROOF OF THEOREM
In this section, we present proof for Theorem 1.
Proof. In the source domain, by the Markov and faithfullness assumptions the conditional indepen- dencies in G the same in Py (X, T, Y). such that
Ic(G) (X,
(15)
To estimate the potential outcomes (t),, we apply the do-operator to obtain the interventional DAG
T and interventional distribution P/ (X, Y dT f)). such that:
lc(GT) I (X, do = tt)).
(16)
Since assume Y f(X,T) T) obtain:
Ic(G7) ([]GT)==(p((X,,,))))) [(pp(X, do(T== =))).
(17)
By Assumption 2, know that the conditional independence relationships in the interventional distribution are the same in any environment, so that
(X f(x,t) do(T = t))) = (X, do(T = +))),
(18)
such that we obtain:
Ic(G7)= - 1X,, do(T =)))).
(19)
C ICMS ADDITIONAL DETAILS
To clarify our methodology further we have provided pseudocode in Algorithms I and 2. Algorithm I calculates the ICMS score (from Eq. 9) from given model. The values for c, and e, are min-max normalized between and across all models. Algorithm 2 returns a ranked list of models by ICMS score from set of ITE models F. It takes optional prior knowledge in the form of causal graph known connections.
In Figure 5, provide graphical illustration for calculating NCT.
13

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Calculate ICMS
Input: ITE model f; source validation dataset D,; unlabeled target test set Digt (ar
tgt Nus.
J==1
interventional DAG Gr; scale factor A.
Output: ICMS score: r(f, Du, Degt: GT)
Function: ICMS(f. , Du, Digt. Gr: )):
g (t) - f t). for 2; E Digtt
tgt
!gt
Digt + ((
gt O, gt (0)), (xt 11 1. (1))
Neg
Cr t Measure of Digt to DAG GT fitness. 0r f- Validation risk of on D. and Digt-
return + λu, (from Eq. 9).
Algorithm 2 ICMS Selection
Input: Source dataset Drr {(rrre, trre, ")""") Nore split into a training set Dt, and validation
set Da; set of ITE models FF trained Drri unlabeled test set Digt's optional prior knowledge in the form of a DAG G scale factor A.
Output: A list F' of models in _F ranked by ICMS score.
Function: TCMS_se (F, Drr, De, Degt, X, G. 0):
Ga causal discovery Drr
G assumed invariant DAG from G3 or Ga
G interventional DAG of G (remove edges into T')
F + Sort J by ICMS(f, Dv, Degt, Ggg A) ascending
return FF
G
Intervene on
NCI D(s)
f(X,t)
Dus (lat* o. (0)). .1.6/= (1))) 24
Dut (x/th
the her
Figure 5: Schematic demonstrating calculation of NCI.
D CAUSAL DISCOVERY ALGORITHM DETAILS
In this section we discuss our causal discovery algorithms used. For real data, where we did not know all of the connections between variables, we discovered the remaining causal connections from the data using the Fast Greedy Equivalence Search (FGES) algorithm by (Ramsey et al., 201 on the entire dataset using the red software package (Glymour et al., 2019a). FGES assumes that all variables be observed and there is linear Gaussian relationship between each node and its parent. retrad allows prior knowledge to be specified terms of required edges that must exist, forbidden edges that will never exist, and temporal restrictions (variables that must precede other variables). Using our prior knowledge, we used the FGES algorithm in Tet to discover the causal DAGs for each of the public datasets. Only the directed edges that were output in the CPDAG by FGES were considered as known edges in the causal graphs. The Tet software package automatically handles continuous, discrete, and mixed connections, i.c., edges between discrete and continuous variables. If not using Tet rad for mixed variables, the method from (Ross, 2014) can be used.
14

---

Under review as a conference paper at ICLR 2021
E HYPERPARAMETERS FOR ITE MODELS
E.1 GANITE
We used the publicly available implementation of GANITE', with the hyperparameters set as indicated in Table 2:
Hyperparameter
Value
Optimization
Adam Moment Optimization
Batch size
Q
Number of iterations
5000
Number of units per hidden layer
Number hidden layers
Table 2: Hyperparameters used for GANITE. 8 represents the number of input features.
E.2 CFR AND TAR
For the implementation of CFR and TAR (Johansson et al., 2018), used the publicly available code?, with hyperameters set as described Table Note that for CFR we used Wasserstein regulatization, while for TAR the penalty imbalance parameter is set O.
Hyperparameter
Value
Optimization
Adam Moment Optimization
Batch size
100
Num, representation layers
Num. of hypothesis layers
Dim. of representation layers
200
Dim. of hypothesis layers
100
Table 3: Hyperparameters used for CFR and TAR.
E.3 SITE
For the implementation of SITE (Yao al., 2018), used the publicly available code", with hyperameters set as described in Table 4.
Hyperparameter
Value
Optimization
Adam Moment Optimization
Batch size
100
Num. of representation layers
3
Num. of hypothesis layers
Dim. of representation layers
200
Dim. of hypothesis layers
100
X
10-4
Table 4: Hyperparameters used for SITE.
oaafffo0rroo11113331003333333313 983/ /atp////gggg//////////////aa////
15

---

Under review as a conference paper at ICLR 2021
E.4 CMGP AND NSGP
CMGP (Alaa et al., 2017) and NSGP (Alaa & van der Schaar, 2018) are ITE methods based on Gaussian Process models for which we used the publicly available implementation". Note that for these ITE methods, the hyperparameters associated with the Gaussian Process are internally optimized.
F LAMBDA
We base our choice of À to be proportional to our belief in our causal DAG that we use for UDA selection. If are given prior knowledge the form of causal graph G G optional and can be empty graph well. In either case can use causal discovery our observational dataset to discover a DAG Ca. Determining the edges that are truthful (and therefore invariant), in practice comes down using human/expert knowledge to select the DAG that most copacetic with existing beliefs of the natural world (Pearl, 2009). We refer to the selected truthful DAG as G, and we define À follows:
E(G) |E( U E(G))
(20)
where E(G) represents the set of edges of G and [EG is the cardinality or number of edges in G. Intuitively, as the number of edges our truthful dag decreases relative our prior knowledge and what is discoverable from data, the less belief we have in our truth causal DAG. In the event that all causal edges are known ahead of time and discoverable from data appropriately, then 1.
PEHE-10
0.150
0.125
0.075
DEV(IF)
0.050
(WCVIDEV(I))
0.0
0.2
0.4
0.6
0.8
10
A
Figure 6: sensitivity analysis.
Lambda sensitivity. We analyze the sensitivity of method the parameter À in Eq. 9. We used the same experimental set-up used for the synthetic experiments. Figure shows the sensitivity of our method À for GANITE using DEV and IF for calculating the validation risk u,.
G SYNTHETIC DATA GENERATION
Here we describe our synthetic data generation process (DGP). Algorithm generates observational data according given invariant DAG G. Algorithm 4 generates interventional or treatment data according to a given invariant DAG G, where the treatment node is binarized and forced to have the value of 0 for half of the samples and 1 for the remainder.
'https: 70a6f6 30113000990003330000000 ha org.a//eeeee...aaaaaaaaaaaaaaaaa processes_ited
org
16

---

Under review as a conference paper at ICLR 2021
Algorithm 3 Generate Observational Data
Input: A Graphical structure G, a mean μ, standard deviation edge weights w and dataset
size n.
Output: An observation dataset according to G with samples.
Function: gen_obs_datate μr, o, w)n):
edges of G
G Gaorted + topological_sort((
ret empty list
for node E G do
Append to ret[node] list of Gaussian (1r and σ) randomly sampled list of size end for
for node E Gsorted do
for par {parentssnode) do ret[node] += ref perr * w wpaar, node), where w(par, node) the edge weight from par to node. end for end for Apply s sigmoid function to the treatment node and binarize.
return ret.
G.1 ADDITIONAL METRICS FOR SYNTHETIC EXPERIMENTS
We use an inversion count over the entire list of models, and provides a measure of list "sortedness". we normalize this between the maximum number of inversions n(n - 1)/2, where is the number of models in the list, then a completely sorted list in ascending order will have value of O. Similarly, monotonically descending ordered list will have value of We provide additional synthetic results in terms of inversion count in Table 5.
Table 5: Inversion count using ICMS on top of existing UDA methods. ICMS means that the was used as the validation risk v, in the ICMS. For example, ICMS(DEV(+)) represents DEV(*) selection used as the validation risk u, in the ICMS. The * indicates the method used approximate the validation error on the source dataset. Our method (in bold) improves over each selection method over models and source risk scores (Src.).
SELECTION
GANITE
CMGP
MSE
0.395 (0.071)
0.363 (0.042)
0.391 (0.050)
0.157 (0.035)
0.131 (0.046)
0.282 (Ó (69)
ICMS(MSE)
(0.069)
0.212
(0.034)
(0.027)
(0.050)
(0.067)
IWCV(MSE)
0.348 (0.056)
0.393 (0.064)
0.364 (0.052)
0.185 (0.033)
0.191 (0.08T)
0.209 (0.060)
ICMS/ICCV(MSL))
(0.063)
(0.033)
(0.063)
DEVIMSE)
0.398 (0.076)
0.414
0.427 (0.049)
0.198 (0.038)
0.239 (0.078)
0.163 (O. 068)
(CMS(DEV(MSEE)
(0.062)
(0.049)
(0.035)
(0.040)
(0.067)
(0.062)
IPTW
0.395 (0.071)
0.355 (0.046)
0.391 (0.050)
0.157 (0.035)
0,182 (0. (4.046)
0.292 (0.0 075)))
ICMS(PPTW)
0.373 (0.069)
0.217 (0.039)
0.272 (0.032)
0.128 (0.031)
0.140 (0.050)
0.207 (0.067)
IWCV(PTW)
(0.075)
(0.059)
(0.058)
(0.053)
(0.063)
CCVPPI))
0.073 (0.028)
0.121 (0.034)
0.119 (0.035)
0.207 (0.039)
0.304 (0.079)
0.328 (0.078)
DEVIPTW)
0.302
(0.057)
(0.071)
(0.086)
(CMPSDE
0.087 (0.035)
0.194 (0.052)
0.120 (0.027)
0.220 (0.031)
0.282 (0.041)
0.355 (0.077)
IF
0.222 (0.041)
0.255 (0.050)
0.250 (0. 046)
0.321 (0.059)
0.392 (0.091)
0. 0.376 (0.097)
ICMS(IF)
0.127 (0.039)
(0.042)
(0.044)
0.215 (0.076)
(0.073)
0.250 (0.084)
IWCV(I)
0.18 (0.059)
0.364 (0.051)
0.256 (0.061)
0.293 (0.043)
0.415 (0.058)
11.377 (0.087)
ICC
(0.018)
(0.025)
(0.033)
(0.028)
(0.077)
DEVILE
0.193 (0.058)
0.415 (0.075)
0.292 (0.056)
0.214 (0.038)
0.450 (0.063)
U.544 (U (9.)
ICMS(DEV(E)
0.069 (0.026)
(0.048)
0.107 (0.029)
0.147 (0.025)
0.229 (0.074)
0.364 (0.076)
17

---

Under review as a conference paper at ICLR 2021
Algorithm 4 Generate Treatment Data with perturbation
Input: A Graphical structure G, mean μ,, standard deviation edge weights w, a dataset size list of perturbation nodes p. a perturbation mean M4p and a perturbation standard deviation op. Output: An treatment dataset according to G with samples and perturbation applied at nodes p.
Function: gen_ lexeat_dattace μ4, 0, w, n, Pp: op):
edges of G
Gsorted + copoooogie potpoooogicallsor(((
ret empty list for node E G do
if node p then
Append to ret list of Gaussian (Up and op) randomly sampled list of size n.
else
Append to ret (nod]] list of Gaussian (μ and σ)) randomly sampled list of size n.
end if end for for node E 2orted do for par (parents(node) do if node & treatment or response node then
ret [node] += ret(par] w(par, node), where w(par, node) is the edge weight from par to node. end if end for
end for Binarize ret (trea]] into 50% with O value and the rest with value. ret[response]] < incoming edges G multiplied by edge weights w. return ret.
H PRACTICAL CONSIDERATIONS
Here provide discussion on some practical considerations.
Computational complexity. The computational complexity of ICMS as shown in Algorithm and 2 scales linear with the number of models in F. Specifically,, the computational complexity is ON,, x Q D)), where Ny is the number of candidate models in F and Q(G, D) is the computational complexity of calculating the fitness score of dataset DD to G. In our case, we use the log-likeli le-likelihhod score, which requires calculating the conditional entropy between each parent node and child. In the worst case, this has a computational complexity of (VV2), where Ve is the number of vertices (or variables) in G since DAG with Ve vertices will have asymptotic number of edges Vc(Ve-1) 2
0.20
PEHE-10
0.15
0.10
(CMD(DEV(IF) DEV(IF)
20
40
60
80
100
Percentage of known edges into outcome
Figure 7: Performance gain in terms of known edges into the outcome node.
18

---

Under review as a conference paper at ICLR 2021
Utilization of subgraphs. In practice, we will likely not know the true underlying causal graph completely. Due to experimental, economical or ethical limitations, we often can not determine the orientation of all edges completely. Additionally, the process of causal discovery is not perfect and likely will result in unoriented, missing, or spurious edges that result from noisiness and biases in the observational dataset used. In Figure 7, we plot the performance of our ICMS method when selecting GANITE models as we increase the percentage of known edges into the outcome node in the causal subgraph used. We indeed prefer subgraphs that contain information about the parents of the outcome node. We conclude that it is perfectly admissible to use our methodology with a subgraph as input with the understanding that edges are missing, performance degrades. However, the performance is still better than without using our ICMS score.
(PEHE-10 G PEHE-10 GT)
DEV(IF)
0.30
INCV(EVVIFI Gr (oracle)
(NCVIDEVII))
APEHE-10 Error
0.25
0.20
0.15
0.10
0.05
0.00
0% (Identical)
20%
40%
60%
80%
100%
Percentage difference between G and G7
Figure 8: Performance of ICMS on incorrect graphs using IWCVDEV((F)). APEHE-10 error is the difference of the PEHE-10 error of T and GT using ICMS versus the percentage of graphical distance (in terms total edges). G is the oracle causal graph and is held static across the r-axis.
Analysis of causal graph correctness. We investigate our method's 's sensitivity to incorrect causal knowledge. Here, we maliciously reverse add spurious edges ICMS. We used our same synthetic experimental setup, except we mutilate our oracle DAGs to form incorrect DAGs. We set À to since we assume the graph is truth (even though it incorrect). We use GANITE with DEV and IF as our validation risk metric and show our results in Fig. 8, which shows the APEHE-10 10 error, i.e., the difference in PEHE-10 10 eтror of the erroneous DAG Gy and the oracle
causal DAG when calculating
DAG Gy, versus the percentage graph difference (between Gy and Gr). The graphical difference is calculated in terms of the percentage of edges that are mutated or removed. Fig. 8 shows the correlation between the correctness of the causal graph and the relative model selection improvement. This correlation testifies to the validity of ICMS, where counterexample of our method would be incorrect DAGs leading to ICMS selecting better models (which is not the case).
Noisiness of fitness score or graphs. We would like to point out that there is noisiness in the fitness score that we use. The likelihood requires estimating the conditional entropy between each variable given their parents. This step is not perfect and there are many permutations of graphical structures that could have scores that are very close. We hypothesize that improving our fitness scores will likely improve the efficacy of our approach in general.
Application: towards personalized model selection. In some instances, various target do- mains may be represented by different underly- ing causal graphs (Shpitser & Sherman, 2018). Consider the following clinical scenario. Sup- pose that we have two target genetic populations A and B that each have their own unique causal graph. We have a large observational dataset with genetic information about each patient. At inference time assuming that we know which genetic group a patient belongs to (and corresponding causal graph), we hypothesize that we can
Table 6: Additional PEHE-10 (with standard error) results for BART and Causal Forest using DEV and IF as validation risk.
SEL. METHOD
BART
COREOT
IF ICMS(F))
0.205 (0.032) 0,253 (0.036)
0.098
0.175(0.033)
IWCV(IF)
0.297 (0.039)
0.288 (0. (0.036)
ICMS(WCV((F))
0.094
0.189(0.029)
(EV(IF) (CMSDEV(EF)
0.214 (0.036)
0.308 (0.038)
0.082(0.023,
0.194(0.000 (0.02)
19

---

Under review as a conference paper at ICLR 2021
select the models that will administer the more appropriate treatment for each genetic population using our proposed ICMS score.
Tree-based methods. Here we provide brief experiment showing that ICMS improves non-deep neural network approaches of Bayesian additive regression tree (BART) (Chipman et al., 2010) and Causal Forest (Wager & Athey, 2018) well. Replicating our synthetic experiments, evaluated BART and Causal Forest using ICMS with DEV, IWCV, and IF for a validation risk, In Table 6, we see that even for tree-based methods our ICMS metric is still able select models that generalize best to the test domain.
Model selection on causally invariant fea- tures. Here we provide brief experiment show- ing that ICMS can be used selection method for the causal feature selection algorithms of Rojas-Carull Carulla et al. (2018); Magliacane et al. (2018). It is important to note that model selec- tion is still important for models that are trained on invariant set of causal features. These models can still converge to different local min- the tar-
Table 7: Additional PEHE-10 (with standard error)
results for Rojas- et al. (2018) (R.C. (2018)) and Magliacane et al. (2018) (Mag. (2018)) perfor- mance (with standard error) using DEV and as validation risk.
SEL. METHOD
R.C. (2018)
MAG. (2018)
IF ICMS(IF) IWCV(IF)
0.312 (0.033)
0.381 (0.022)
0.192(0.021)
(..5800.30))
0,240 (0.029)
0.292 (0 041)
ima and have disparate performances get domain. Replicating our synthetic experi- ments, used Rojas-Carulla et al. (2018) and Magliacane et al. (2018) to select causally in- variant features, which we use for training and testing our model. We then selected models using ICMS and compared against our standard benchmarks using GANITE. In Table 7, we see that even for these feature selection methods our ICMS metric is still able to select models that generalize best to the test domain (in comparison to DEV, IWCV, and IF).
ICMS(WCV(F))
0.136
0.183(0.037
DEV(IF) ICMS(DEV(IF)
0.257 (0.025)
0.212 (0.035)
0.110(0.024) 0.127(0.044)
20

---

Under review as a conference paper at ICLR 2021
1 EXPERIMENTAL SET-UP FOR SEMI-SYNTHETIC DATASETS AND ADDITIONAL RESULTS.
In this section, we highlight additional experiments performed on real datasets with semi semi-synthetie outcomes. Since real-world data rarely contains information about the ground truth causal effects, ex- isting literature uses semi-synthetic datasets, where either the treatment or the outcome simulated (Shalit et al., 2017). Thus, we evaluate our model selection method on a prostate cancer dataset and the IHDP dataset where the outcomes are simulated and on the Twins dataset (Almond al., 2005) where the treatments are simulated. Furthermore, we provide UDA selection results on the prostate cancer dataset for factual outcomes as well.
Table 8: Results on IHDP, prostate cancer, and TWINS datasets. IF validation is used to compute the source risk. We report the PEHE-10 test error (with standard error) of various selections methods on ITE models. Our method (in bold) improves in terms of PEHE-10 over all methods and ITE models.
DATASET
METHOD
GANITE
TAR
SITE
CMGP
NSGP
IF
0,186 (0.040)
0.448 (0.052)
0.444 (0.066)
0.430 (0.050)
0.461 (0.038)
0.473 (0.066)
ICMS(FF)
(0.031)
0.386 (0.045)
(0.045)
0.342 (0.051)
0.380 (0.053)
0.462 (0.053)
IHDP
IWCV(F)
0.134 (0,059)
0.493 (0.055)
0.412 (0.057)
0.491 (0.057)
0.519 (0.072)
0.647 (0.090)
(CM)UWCV(P)
0.106 (0.023)
0.447 (0.036)
0.360 (0.047)
0.488 (0.073)
0.372 (0.095)
0.576 (0.019)
DEV
D.174 (0.050)
0.462 (0.036)
0.403 (0.046)
0:458 (0.023)
0,550 (0.174)
0.654 (0.097)
ICMSDDEV(P)
0.095 (0.025)
0.438 (0.036)
0.427 (0.065)
0.405 (0.049)
0.475 (9.199)
0.583 (0.026)
IF
0.298 (0.053)
0.377 (0.054)
0.419 (0.0544)
0.194 (0.048)
0.771 (0.042)
0.679 (0.061)
ICMS(F)
(0.057)
0.143 (0.026)
(0.054)
(0.039)
(0.028)
(0.051)
PC(UK)+ SEER(US)
IWCV(F)
0.125 (0.058)
0,340 (0.060)
0,366 (0.035)
0.39% (0.073)
0.238 (0.051)
0.481 (0.032)
ICMS(WCYOr)
0.018 (0.011)
0.146 (0.054)
0.218 (0.051)
0.331 (0.055)
0.161 (0.038)
0.329 (0.049)
DEVUF)
0.239 (0.068)
0.308 (0.037)
0,361 (0.064)
0.348 (0.078)
0.253 (0.065)
0.150 (0.011)
ICMS(DEVAP)
0.036 (0.013)
0.120 (0.038)
0.168 (0.057)
0.318 (0.062)
0,203 (0.032)
0.254 (0.057)
IF
0.286 (0.027)
0.527 (0.054)
0.164 (0.067)
0.168 (0. (0.102)
0.223 (0.082)
0. 0.488 (0.087)
ICMS(F)
(0.022)
0.370 (0.065)
(0.040)
0.299 (0.097)
(0.039)
(0.029)
TWINS- TWINS(SEMI)
IWCV(F)
0.495 (0.054)
0.538 (0.051)
0,574 (0.066)
0.611 (0.075)
0.438 (0.069)
0.444 (0.077)
ICMSOWCVArD
0.288 (0.059)
0,497 (0.074)
0.508 (0.048)
0.500 (0.077)
0.218 (0.055)
0.375 (0.099)
DEVUF)
0.135 (0.059)
0.584 (0.084)
0,518 (0.101)
0.184 (0.115)
0.351 (0.074)
0.180 (0.089)
ICMS(DEVAP)
0.277 (0.054)
0.512 (0.086)
0.475 (0.040)
0.447 (0.074)
0.227 (0.043)
0.411 (0.104)
Table 9: Results on predicting the outcome of prostate cancer given a treatment from models trained on the Prostate Cancer UK (PCUK) dataset and tested on the SEER dataset (United States). IF validation used to compute the source risk. Here we show the factual error (of the top 10% of selected models) in terms of MSE of various selections methods on ITE models. Our method (in bold) improves in terms of test error over all methods and ITE models. The standard error is shown in parentheses.
DATASET
METHOD
GANITE
CFR
TAR
SITE
CMGP
NSGP
(0.061)
0.183 (0.078)
(0.078)
0.521 (0.013)
0.305 (0.074)
(0.082)
ICMS(IP)
0.105 (0.015)
0.127 (0.052)
0.311 (0.031)
0.243 (0.080)
0.258 (0.078)
0.294 (0.053)
POUK-- PC(US)
IWCV(F)
(0.081)
(7.1400066))
0.395 (0.043)
0.545 (0.051)
0.297 (0.032)
(0.057)
(REAL OUTCOMES)
IVGE))
0.230 (0.014)
0.361 (0.035)
0.518 (0.049)
0.287 (L.037)
0.282 (0.042)
0.493 (0.019)
DEVTIE)
0.361 (0.129)
0.448 (0.162)
0.471 (0. 172)
0.379 (0.)
0. (0.163)
ICMS(DEVIP))
0.123 (0.017)
0,313 (0.047)
0.396 (0.052)
0.326 (0.029)
0.332 (0.032)
0.412 (0.141)
IHDP dataset. The dataset was created by (Hill, 2011) from the Infant Health and Devel- opment Program (IHDP)* and contains informa- tion about the effects of specialist home visits on future cognitive scores. The dataset contains 747 samples (139 treated and 608 control) and 25 covariates about the children and their moth- ers. We use set-up similar to the one in (Dorie et al., 2019) to simulate the outcome, while at the same time building the causal graph G.
Head Crommbenece
Weeks
Noonatal health
pre-iemm
Treatment
Outcome
Body weight
Figure 9: Interventional DAG for computing ICMS score on IHDP dataset.
'The dataset can be found as part of the Supplementary Files at doi / /10 01..1.01. 11 / jogs. 201 10110.00. 088 62?scra1 11 1-top
21

---

Under review as a conference paper at ICLR 2021
Since we do not have access to any real outcomes for this dataset, build the DAG in Figure 9, such that a subset of the features affect the simulated outcome. Let I represent the patient covariates and let be the covariates affecting the outcome in the DAG represented in Figure 9. We build the outcome for the treated patients f(x;, 1) and for the untreated patients f(x 0) as follows: f(x,, 0) = exp(S(e+ 2)) + and [xx 1) Bu++ where 3 consists of random regression coefficients uniformly sampled from [0.1, 0.2, 0.3, 0.4] and < ~~ (0, 1), 1 N(0, 1) are noise terms.
To create target dataset with covariate shifts for the IHDP, we hold out the samples where the continuous variables neonatal health, head circumference and mom age have extreme val- ues (either in the top 20% or the lowest 20%).. We again ran 20 experiments and for each ex- periment we trained 30 candidate models for each model architecture. We use IF validation to approximate the source risk and we report the PEHE-10 10 test error. Table 8 illustrates the results on the IHDP dataset.
Smokkng
Gestation
Weight
Treatment
1-yeer mortalty
Drinkin
AnOTiO
TASTIN
RON
Interventional DAG for computing ICMS score on Twins dataset. The DAG contains a subset of the features available in the dataset for which we discovered causal relationships with the outcome indicated by the probability of 1 1-year mortality of the twin.
Figure 10:
TWINS dataset. The TWINS dataset con- tains information about twin births in the US be- tween 1989-1991 (Almond et al., 2005)* 6. The treatment Ê = 1 is defined as being the heavier twin and the outcome corresponds to the 1-year mortality. Since the dataset contains information about both twins we can consider their outcomes as being the potential outcomes for the treatment of being heavier at birth. The dataset consists of 11,400 pairs of twins and for each pair we have information about 30 variables related to their parents, pregnancy and birth.
We use the same set-up as in (Yoon et al., 2018) to create an observational study by selectively observing one of the twins based on their features (therefore inducing selection bias) as follows: t x Bernoulli (iigooiis x + n)) where w ~ =(((-0.1, 300) and n ~ (N 0.1).
Since we have access to the twins outcomes, we perform causal discovery to find causal relationships between the context features and the outcome. However, due to the fact that we do not have prior knowledge of the relationships between all 30 variables, we restrict the causal graph used to compute the causal risk to only contain a subset of variables, as illustrated in Figure 10.
Table 8 illustrates the results for the Twins dataset. Note that in this case, we use real outcomes and we also show the applicability of our method when only a subgraph of the true causal graph is known.
Prostate cancer datasets. In this case, we are a interested in deploying a machine learning model for prostate cancer but have access to only labeled data in the UK Biobank dataset, which has approximately 10,000 patients. We would like to deploy our models in the United States, where we have access to many samples of patient features, but no labeled outcome. For this target domain, we use the SEER dataset, which has over 100,000 samples. Our objective is to predict the patient mortality, given the patient features and treatment provided.
Age
Choosent
Treadmment
Caroer
PSA
Figure 11: Interventional DAG for Prostate dataset.
To be able to evaluate the methods on predicting counterfactual outcomes on the target domain (and thus compute the PEHE), we create a semi-synthetic dataset where the outcomes are simulated according to the discovered causal graph. Thus, we build the semi-synthetic outcomes for the prostate cancer dataset similarly to the IHDP dataset. Let represent the patient covariates and let be the covariates affecting the outcome. We build the outcome for the treated patients f(x, 1) and for the untreated patients fæ, 0) as follows: f(x, 0) exp(3(v + })) + e and f(x, 1) = BB + where 3 consists of random regression coefficients uniformly sampled from [0.1, 0.2, 0.3, 0.4] and ~ (O 0.1), 7 (V are noise terms.
6 The data for the TWINS dataset can be found https://dat linked bi rth- innntn datta--v stalsstaistiia----e html
22

---

Under review as a conference paper at ICLR 2021
For the prostate cancer datasets, we also perform experiment where do not semi-synthetic data (to generate the counterfactual outcomes), but use only the factual outcomes the SEER dataset to evaluate method. We train 30 models with identical hyperparameters and semi-synthetic experiments. We repeat this for all of our benchmark ITE methods. Table 9 shows that ICMS improves in terms of test emrrr all methods and ITE models.
done in our synthetic
Computational settings. All experiments were performed on an Ubuntu 18.04 system with 12 CPUs and 64 GB of RAM.
COVID-19 EXPERIMENTAL DETAILS
J.1 DATASET
We obtained de-identified COVID-19 Hospitalization in England Surveillance System (CHESS) data from Public Health England (PHE) for the period from 8 February (data collection start) to 14th April 2020, which contains 7,714 hospital admissions, including 3,092 ICU admissions from 94 NHS trusts across England. The data set features comprehensive information on patients' general health condition, COVID-19 specific risk factors (e.g., comorbidities), basic demographic information (age, sex, etc.), and tracks the entire patient treatment journey: hospitalization time, ICU admission, what treatment (e.g., ventilation) they received, and their outcome by April 20th, 2020 (609 deaths and 384 discharges). We split the data set into source dataset containing 2,552 patients from urban areas (mostly Greater London area) and target dataset of the remaining 5,162 rural patients.
J.2 ABOUT THE CHESS DATA SET
COVID-19 Hospitalizations in England Surveillance System (CHESS) is surveillance scheme for monitoring hospitalized COVID-19 patients. The scheme has been created in response to the rapidly evolving COVID-19 outbreak and has been developed by Public Health England (PHE). The scheme has been designed to monitor and estimate the impact of COVID-19 on the population timely fashion, to identify those who most risk and evaluate the effectiveness of countermeasures.
The CHESS data therefore captures information to fulfill the following objectives:
To monitor and estimate the impact of COVID-19 infection on the population, including estimating the proportion and rates of COVID-19 cases requiring hospitalisation and/or ICU/HDU admission
2. To describe the epidemiology of COVID-19 infection associated with hospital/ICU admis- sion in terms of age, sex and underlying risk factors, and outcomes
3. To monitor pressures on acute health services
4. To inform transmission dynamic models to forecast healthcare burden and severity estimates
J.3 COVID- 19 PATIENT STATISTICS ACROSS GEOGRAPHICAL LOCATIONS
Figure 12 shows the histogram of age distribution for urban and rural patients. It is clear from the plot that the rural population is older, and therefore at higher risk of COVID-19. Table 10 presents statisties about the prevalence of preexisting medical conditions, the treatments received, and the final outcomes for patients in urban and rural areas. We can see that the rural patients tend to have more preexisting conditions such as chronic heart disease and hypertension. The higher prevalence's of comorbid conditions complicates the treatment for this population.
J.4 DATA SIMULATION AND ADDITIONAL RESULTS USING ICMS
In the CHESS dataset, only observe the factual patient outcomes. However, to be able to evaluate the selected ITE models on how well they estimate the treatment effects, we need to have access to both the factual and counterfactual outcomes. Thus, we have built semi-synthetic version of
23

---

Under review as a conference paper at ICLR 2021
0.025
0.015
density
Area
Rural
Urban
0.010
0.005
0.000
25
50
75
100
Age
Figure 12: Age distribution for urban and rural patients. The median age of rural patients five years older than the urban ones.
Improdng moSel aidd for GANTZ
Improwing model odection for CTR
Improvno mode ralicien for TAR
CARPY
DHANCUN
(a) GANITE
(b) CFR
(c) TAR
invwiing modellsdeddn la SIE
Irpunig nodd bokckai fo CMCP
trorovngmoot scceion NSDP
(d) SITE
(c) CMGP
(f) NSGP
Figure 13: Performance model selection methods terms on additional number of patients with improved outcomes compared to selecting models based on the factual error on the source domain for ITE models.
24

---

Under review as a conference paper at ICLR 2021
Table 10: Comparison of key features of urban and rural COVID-19 patients in the data set.
Urban Percentage
Rural Percentage
Count
Count
Sex at Birth
65%
1446
62%
3388
Chonic Respiratory
4%
81
6%
310
Obesity
121
225
Chronic Heart
4%
80
8%
444
Hypertension
13%
285
15%
798
Asthma
4%
92
6%
326
Diabetes
9%
197
11%
589
Chronic Renal
2%
45
3%
175
Noninvasive Ventilation
7%
160
6%
342
Invasive Ventilation
21%
456
16%
879
Death
18%
402
19%
1014
Discharge
12%
276
21%
1164
the dataset, with potential outcomes simulated according to the causal graph discovered for the COVID-19 patients Figure 3.
Let a represent the patient covariates and let u be the covariates affecting the outcome in the DAG represented in Figure Let far 1) the outcome for the patients that have received the ventilator (treatment) and let fz, 0) be the outcome for the patients that have not received the ventilator. The outcomes are simulated as follows: f(x, 0) 30++1 and fax 1) = exp(3u)-1 -11)))-+,,, 1+, where 9 consists of random regression coefficients uniformly sampled from [0.1, 0.2, 0.3, 0.4) and (N 0.1), 1) ~ (0, 0.1) are noise terms. We consider that the patient survives if f(x, t) O, where t {0, 13 indicates the treatment received.
Our training observational dataset consists of the patient features æ, ventilator assignment (treatment) t for the COVID-19 patients in the urban and the synthetic outcome generated using f(z, t). For evaluation, we use the set-up described in Section 5.2 for assigning ventilators to patients in the rural area based their estimated treatment effects. In Figure 13, indicate the additional number of patients with improved outcomes by using ICMS on top of existing UDA methods when selecting ITE models with different settings of the hyperparameters.
25


Reviews:
[{"review_id": "yWm2kx4fb0f", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: the paper attacks the problem of model selection for individual treatement effect (ITE) models when the domain of learning and prediction differ. Proposal is to use causal consistency as an additional \"regularizer\" in existing domain adaptation (DA) model selection methods. The \"regularizer\" would be scoring to which extent replacing factual outcomes by their counterfactual predictions would preserve conditional independence relations (induced by the causal graph) in the prediction domain. Experiments on a variety of datasets are conducted to show the added performance induced by using the \"regularizer\".\n\nGood points:\n- novel and very creative idea\n- excellent writing that introduces concepts as they are needed + illustrative figures (Fig. 1 and 2)\n- rigorous exposition of assumptions - quite important in this kind of problems\n- interesting, illustrative use-case on covid\n\nQuestions:\n- Is the dataset on covid openly accessible ? \n- It seems that all base models are deep i.e. involving a non-convex optimization problem that makes the model prone to initial solution. Have you taken the \"choice\" of the random seed into account in the same manner as the hyper-parameters to select the model ?\n\nPoints currently limiting the relevance of the paper:\n- [impact] model selection is here studied in isolation to other related techniques, notably causal feature selection (e.g. https://arxiv.org/abs/1911.07147) - it seems to me that it thus limits notably the generality of the conclusions as I don't see why practitioners would use one technique without the other. It is a big question mark for me and I'd like to read the point of view of authors on that point.\n- [relevance of baselines] In related works authors mention two methods that they place as \"state of the art\" for ITE model selection: Causal Assurance and Influence Functions. Even if they are not designed specifically for the DA use case why not include them in the experiments ? It is often the case in practice that methods not designed for a specific case are unexpectedly robust to more specific setting. In particular I think it would be most telling to compare an augmented/\"regularized\" method (e.g. DEV+ICMS) vs generic SOTA methods (CA or IF). Disclaimer: I don't know enough of these methods to evaluate how easy it would be to perform such a comparison.\n- [metrics] The standard PEHE metric seems to have been replaced by PEHE-10. I can't seem to find the definition of this metric in the paper ? why not use the standard ?\n- [reproducibility] I can't seem to find the hyper-parameter grid that was used for the different base models and among which the competing methods would choose the best candidate - even in supplementary ? To allow other researchers to independently reproduce your results this is just mandatory. \n- [reproducibility2] I can't find details on your strategy for seeding the different models. How many random seeds ? are they considered as hyper-params or part of an inner optimization loop ? \n- [insights on results] The provided result in Table 1 indicates that adding ICMS to DEV or IWCV uniformly improves model selection. It seems in a sense \"too good to be true\". Can you provide intuition on why it is so ? Could it be possible to check that using ICMS alone is not already a very strong baseline - or is it in combination only that it works ?\n\nPoints that could improve the paper (and my score):\n- provide a self-contained formula of NLL directly when introducing it in Sec. 4\n- give full hyper-param grid for all models in supplementary\n- define PEHE-10\n- clarify setting for random seeds\n- add an experiment with the Causal Assurance and Influence Functions baselines\n- add an experiment comparing best model selected by proposal (e.g. DEV ICMS) vs best model found by causal feature selection (e.g. Rojas-Carulla 2018 or Magliacane 2018)\n\nOverall, it seems to me the work deserves a more rigorous evaluation part as the underlying idea is very interesting and seems to be a very strong addition to the causal toolbox for ITE modeling.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IGAdDDepUSO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################\n\nSummary:\n\nThis paper proposes a novel interventional causal model selection (ICMS) score to select individualized treatment effects (ITE) models under the unsupervised domain adaption (UDA) setting. The problem is fundamentally challenging as counterfactual outcomes cannot be observed. The authors make an assumption that the underlying causal structure across domains remains unchanged when adapting the ITE models from one domain to another. The authors propose Theorem 1 that the conditional independence relationships in the interventional DAG are equal to that in the interventional distribution for the target domain, followed by augmenting the target domain data with the model's prediction of the potential outcomes. Then the model that generates the best augmented target data in the sense that matches best with the interventional DAG is selected. To access this fitness, authors use the negative log-likelihood of the interventional DAG given the augmented data on the target domain. Finally, results with both synthetic data and real-world COVID-19 dataset show improvement for all ITE models.\n\n\n\n##########################\n\nPros:\n \n1. This paper tackles an important and challenging task: adapting ITE models unsupervisedly with unobserved counterfactual outcomes.\n2. This paper is overall theoretically sound. The core assumption of causal structure invariance makes sense to me and the experimental results show significant improvement for all SOTA ITE models used.\n3. This paper is well organized and easy to follow.\n\nBased on these, I tend to vote for acceptance.\n\n\n##########################\n\nCons: \n\n1. In Eq (9) and Fig. 2, is the validation risk $v_r$ computed using the source dataset? This seems unjustified given that the risk computed in the source domain might not be a good approximation of that computed in the target domain.\n2. In Eq. (9) and the paragraph ``Assessing causal graph ﬁtness'', $c_r$ is computed by the negative log-likelihood of G given the augmented dataset Dhat_{tgt}. This fitness makes sense to me: the model that generates the augmented dataset in target domain that best align with $G_{\\bar{T}}$ should be selected. However, what is the relationship with Eq. (7-8)? It does not seem straightforward to see how Eq (9) and the NCI term are linked up.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Es1vbvkT1u-", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe present paper proposes a novel approach for model selection for individual treatment effect (ITE) estimation in the unsupervised domain adaptation (UDA) setting. The motivation for this approach, called interventional causal model selection (ICMS), is to exploit causal invariance to choose a model that has both good predictive power and fits the a priori belief for causal relationships.\nMore precisely, the authors prove a necessary condition for optimality of an ITE model and combine this condition with the classical target risk minimization from UDA model selection.\nThis necessary condition states preservation of all conditional independencies of the causal DAG by the interventional distribution of the target domain.\n\nRecommendation:\nClear accept. The framework proposed in this work constitutes an interesting new approach to model selection exploiting ideas from causal invariance and also allows to adapt existing ITE estimation methods to tackle covariate shift problems.\n\nStrong points:\n - This framework is theoretically motivated, using the structural causal model framework, and allows to improve various methods by allowing for model selection in UDA setting, as illustrated on a wide range of examples (simulations and real data).\n - The flexibility also extends to the quality of the data or prior knowledge, as this framework allows to incorporate as much expert structure knowledge as possible, or to specify its importance in the model selection via the hyperparameter $\\lambda$.\n\nWeak points:\n - The role of the causal DAG is key to this approach, it is however difficult to assess from the experiments and discussion, how much the proposed framework relies on the correctness of the used DAG, or, put differently, how sensitive this framework is to mis-specifications of the DAG.\n - It would be interesting to see the code for the presented examples and also to test it on other applications. Will it be made public at some point?\n - The presented work is considerable and it many experiments are provided. However I think that the experiments section is too long compared to the main section (Section 4). For instance, (parts of) the appendix A could be moved into the main part while shortening the Section 5.2.\n\nQuestions/Issues:\n - Sec. 5.2.: Reference for FGES is missing in the main part of the article (it's in the appendix but should also be in cited in Section 5.2)\n - Appendix B (additional related work) should be moved to the main part (Section 2).\n - Sec. 5.1: What is the reason for only shifting one ancestor of $Y$ in $G$? And can it also be, at the same time, an ancestor of the treatment, i.e., a confounder in the observational data (I think yes, since the intervention on the treatment variable cuts of incoming arrows, but has this been tested in practice as well?)?\n - Sec. 5.1: How do the authors choose the 30 candidate models for each model architecture? Is the variability across the different models similar for all comparable for all compared architecture?\n\nMinor comments (that did not impact the score):\n - p. 3: unconfoundness $>>$ unconfoundedness\n - p. 6: i.e $>>$ missing comma (i.e.,)\n - p. 6: return $>>$ returned\n - p. 7: For completeness, add reference for domain shift between rural and urban populations.\n - p. 7: hospital admission $>>$ hospital admissions", "extra_scores": {}, "extra_text": {}}, {"review_id": "1XEDVHDy4zb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a model selection metric for ITE models under the unsupervised domain adaptation setting. By modeling the data from a source domain, the metric is able to select optimal ITE models that perform well on a target domain, where only unlabeled data available. Experiments are conducted on both synthetic and real-world datasets.  \n\n+ The problem is significant in machine learning and the idea is incrementally innovative: causal inference under UDA setting. \n+ The idea flow is logical. The paper is very well-written.\n\nQuestions:\n1. How to evaluate the generated causal graph? The causal structure learned from the source domain is used to estimate the causal risk on the target domain. How to evaluate the goodness of a learned causal structure? Especially in the real-world dataset, the causal graph could be much more complex than the synthetic setting. How to evaluate the dependencies among each pair of nodes? \n\n2. How to evaluate the parameter lambda? In equation (8), a parameter lambda is used for controlling the weight of NCI and set to 1. Additional experiments should be conducted to evaluate the performance under different values of lambda. \n\n3. Some notations are not clear.  \n- In equation (7), the definition of NCI is not clear. A graphic illustration of NCI results in the causal graph should be provided. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the problem of identification of causal effects under the unsupervised domain adaptation setting. The authors assume the invariance of the causal structure and use it to regularize the predictor of causal effects. The method is interesting and looks effective, although this assumption may not hold always true (e.g., in some domains, some causal influences may disappear, leading to extra conditional independence relations). Hope the authors will update the paper to address the concerns raised by the reviewers, especially to conduct a sensitivity analysis of the framework to misspecification of the causal structure and make the motivation for the used evaluation metrics clear, and also provide a more thorough review of related work.
Decision: Reject