Zero-shot causal learning
Authors: Hamed Nilforoshan
Michael Moor
Yusuf H Roohani
Yining Chen
Anja Šurina
Michihiro Yasunaga
Sara Oblak
Jure Leskovec
Venue: NeurIPS 2023 spotlight (2023)
Keywords: causal inference
CATE
CATE estimation
causal machine learning
causal ML
heterogenous treatment effects
causality
potential outcomes
treatment effect
TL;DR: A personalized causal effect prediction framework that can generalize to novel treatments that did not exist during training (e.g. a newly invented drug that has never been prescribed before)

Abstract: Predicting how different interventions will causally affect a specific individual is important in a variety of domains such as personalized medicine, public policy, and online marketing. There are a large number of methods to predict the effect of an existing intervention based on historical data from individuals who received it. 
However, in many settings it is important to predict the effects of novel interventions (e.g., a newly invented drug), which these methods do not address.
Here, we consider zero-shot causal learning: predicting the personalized effects of a novel intervention. We propose CaML, a causal meta-learning framework which formulates the personalized prediction of each intervention's effect as a task. CaML trains a single meta-model across thousands of tasks, each constructed by sampling an intervention, its recipients, and its nonrecipients. By leveraging both intervention information (e.g., a drug's attributes) and individual features (e.g., a patient's history), CaML is able to predict the personalized effects of novel interventions that do not exist at the time of training. Experimental results on real world datasets in large-scale medical claims and cell-line perturbations demonstrate the effectiveness of our approach. Most strikingly, CaML's zero-shot predictions outperform even strong baselines trained directly on data from the test interventions.

Zero-shot causal learning
Hamed Nilforoshan"!
Michael Moor
Yusuf Roohani
Yining Chen'
Anja Surina
Michihiro Yasunaga'
Sara Oblak Jure Leskovec'
DDeaartment of Computer Science, Stanford University
Department of Biomedical Data Science, Stanford University
School of Computer and Communication Sciences, EPFL
*Department of Computer Science, University of Ljubljana
Correspondence to: hamedn @ @cs. stassfooord.dduu mdmoor@ @t..eeecsssss...ctt... jure@cs.. wt..sssssordd.c.
Abstract
Predicting how different interventions will causally affect specific individual is important in variety of domains such personalized medicine, public policy, and online marketing. There are a large number of methods to predict the effect of an existing intervention based historical data from individuals who received it. However, in many settings is important to predict the effects of novel inter- ventions (e.g., newly invented drug), which these methods do not address. Here, we consider zero-shot causal learning: predicting the personalized effects of a novel intervention. We propose CaML, causal meta-learning framework which formulates the personalized prediction of each intervention's 's effect as task. CaML trains single meta-model across thousands of tasks, each constructed by sampling an intervention, its recipients, and its nonrecipients. By leveraging both interven- tion information (e.g., a drug's attributes) and individual features (e.g.. patient's s history), CaML is able to predict the personalized effects of novel interventions that do not exist at the time of training. Experimental results on real world datasets large-scale medical claims and cell-line perturbations demonstrate the effectiveness of our approach. Most strikingly, CaML's zero-shot predictions outperform even strong baselines trained directly on data from the test interventions.
1 Introduction
Personalized predictions about how an intervention will causally affect a specific individual are important across many high impact applications the physical, life, and social sciences. For instance, consider a doctor deciding whether or not to prescribe drug to patient. Depending on the patient, the same drug could either (a) cure the disease, (b) have no effect, or (c) elicit life- life-threatening adverse reaction. Predicting which effect the drug will have for each patient could revolutionize healthcare by enabling personalized treatments for each patient.
The causal inference literature formalizes this problem as conditional average treatment effects (CATE) estimation, in which the goal is predict the effect of an intervention, conditioned patient characteristics (X). When natural experiment data is available, consisting of individuals who already did and did receive an intervention, variety of CATE estimators exist accomplish this
task (1 [6 23 30 34 44 55 36 [70]. These methods can then predict the effect of an existing
intervention (W) on a new individual
(X').
However, in many real-world applications natural experiment data is entirely unavailable, and yet CATE estimation is critical. For instance, when new drugs are discovered, or new government policies
Equal contribution. Code is available at: httpp:://github.com//mmat http:///o.cmoe//aaaaaaaaaaaaaaaa
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

are passed, it is important to know the effect of these novel interventions on individuals and subgroups in advance, i.e., before anybody treated. There is thus need for methods that can predict the effect of novel intervention (W") on a new individual (X') in a zero-shot fashion, i.e., without relying on any historical data from individuals who received the intervention.
Generalizing to novel interventions is especially challenging because it requires generalizing across two dimensions simultancously: to new interventions and new individuals. This entails efficiently "aligning" newly observed interventions to the ones previously observed in the training data.
Present work. Here, we first formulate the zero-shot CATE estimation problem. We then propose CaML (Causal Meta-learning leta-learning), a general framework for training a single meta-mode to estimate CATE across many interventions, including novel interventions that did not exist at the time of model training (Figure D. Our key insight is to frame CATE estimation for each intervention as a separate meta-learning task. For each task observed during training, we sample retrospective natural experiment consisting of both (a) individuals who did receive the intervention, and (b) individuals who did not receive the intervention. This natural experiment data is used to estimate the effect of the intervention for each individual (using any off-the-shelf CATE estimator), which serves as the training target for the task.
In order to achieve zero-shot generalization to new interventions, we include information (W) about the intervention (e.g.. drug's ss attributes), in the task. We then train single meta-mode! which fuses intervention information with individual-level features (X) to predict the intervention's 's effect. Our approach allows to predict the causal effect of novel interventions, i.e., interventions without sample-level training data, such as a newly discovered drug (Figure T.. We refer to this capability as zero-shot causal learning.
In our experiments, we evaluate our method on two real-world catasets-breakkee convention with the CATE methods literature which typically relies on synthetic and semi-synthetic datasets. Our experiments show that CaML is both scalable and effective, including the application to a large-scale medical dataset featuring tens of millions of patients. Most strikingly, CaML's zero-shot performance exceeds even strong baselines that were trained directly on data from the test interventions. We further discover that CaML is capable of zero-shot generalization even under challenging conditions: when trained only on single interventions, at inference time it can accurately predict the effect of combinations of novel interventions. Finally, we explain these findings, by proving zero-shot generalization bound.
2 Related work
We discuss recent work which is most closely related to zero-shot causal learning, and provide an extended discussion of other related work in Appendix B Most CATE estimators do not address novel interventions, requiring that all considered interventions be observed during training. A notable exception is recent methods which estimate CATE for an intervention using structured information about its attributes (25] 35). In principle, these methods can also be used for zero- shot predictions. These methods estimate CATE directly from the raw triplets (W, X, Y), without considering natural experiments. by tailoring specific existing CATE estimators (the S-learner 44 and Robinson decomposition (55). respectively) to structured treatments. The main drawback of these approaches is that they are inflexible, i.c., they are restricted to using a single estimator and are unable to take advantage of the recent advances in the broader CATE estimation literature (c.g.. recently developed binary treatment estimators [16] 21 40). This is limitation because any single CATE estimator can be unstable across different settings 13). Notably, the estimators which these methods build have already been shown to result in high bias in many domains [44] 37 10, 16). Likewise,
we find that these methods struggle with zero-shot predictions (Section 6). CaML's key difference
from prior work is that we construct separate task for each training intervention by synthesizing
natural experiments. This allows us to (a) flexibly wrap any existing CATE estimator to obtain labels for each task, and thus take advantage of the most recent CATE estimation methods and (b) leverage meta-learning, which requires lask-structured data. Consequently, CaML is able to achieve strong zero-shot performance (Section 6)
3 Background: single-intervention CATE estimation
Each task in the CaML. framework consists of estimating conditional average treatment effects (CATEs) for a single binary treatment. In this section, we first provide background on CATE
2

---

Training
Testing
Potential Outcomes
(fooee rellace '
2
4
5
[novel intervention)
Y( f observed
Y
unobsarvad
A
2
P O
A
6
unobserved
Y1O) unobserved
Y(F) Y(O)
T always unobserved
Figure 1: Overview of the zero-shot causal learning problem. Each individual has features (X), an intervention with features (W), and an outcome (Y). Lightning bolts (4) represent interventions (e.g. drugs). The personalized effect of an intervention (r) is always unobserved. The goal is to predict the T for a novel intervention (W") and individual (X') that did not exist during training.
estimation under this simple case of single treatment (W) and outcome (Y), and subsequently generalize it to our zero-shot setting. Under a single intervention and outcome, we consider n independent observations , Pn
drawn from distribution P. For unit i I,..., P. XW, Yi) ~ P collects: a binary or continuous outcome of interest Y; E y C IR instance features (i.e.. pre-treatment covariates) X, E X C Rd, and treatment- -asieenmtnssinmment indicator W; E {0. 1}. We use the Neyman-Rubin potential outcomes framework [33] in which Y((1), Y:(0) reflect the outcome of interest either under treatment (W; = 1), or under control (W; = 0), respectively. In our running medical example, (1(1) is the health status if exposed to the drug, and (0))) is the health status if not exposed to the drug. Notably, the fundamental problem of causal inference is that we only observe one of the two potential outcomes, Y, W; (1) + (1 W.) Y2()) (e.g., either health status with or without drug exposure can be observed for a specific individual, depending on whether they are prescribed the drug). However, it is possible to make personalized decisions by estimating treatment effects that are tailored to the attributes of individuals (based on features X). Thus, focus estimating T TTz), known the conditional average treatment effect (CATE):
CATE r(x) Ep ((1) - (()) X =
A variety of methods have been developed to estimate T(z) from observational data [16]. These rely on standard assumptions of unconfoundedness, consistency, and overlap (52). Unconfoundedness: there are unobserved confounders, i.e. Y,(0), (1 X,, Consistency: Y, Y.(W)), i.e. treatment assignment determines whether Y1(1) or (,))) is observed. Overlap: Treatment assignment nondeterministic, such that for all in support of X: 0 < P(W; = X; x) < 1.
Zero-shot causal learning
In many real- settings (e.g. drugs, online A/B tests) novel interventions are frequently introduced, for which no natural experiment data are available. These settings require zero-shot CATE estimates. The zero-shot CATE estimation problem extends the prior section, except the intervention variable W; is no longer binary, but rather contains rich information about the intervention: W, E W R (e.g.. drug's 's chemistry), where W,- 0 corresponds sample that did not receive any intervention. Thus, each intervention value w has its own CATE function that we seek to estimate:
CATE (z) Ep (w) - (0) | X =x].
(2)
During training, we observe independent observations PI...., Pn drawn from a distribution P. Each P. = (W, X,, Y,) ~ P. Let Waeen be set of all interventions observed during training. The zero-shot CATE estimation task consists of estimating CATE for novel intervention that was never observed during training:
(Zero-shot CATE estimation). Given (W,, X,, Yi)..., .., (W,, X,,Y, drawn from P. containing intervention information, individ- ual features, and outcomes... estimate (x) for novel intervention w' & W.een.
Problem
training observations
This problem formulation extends in a straightforward manner to combinations of interventions, by allowing a single intervention W, to consist of set of intervention vectors. CaMI. supports combinations of interventions, we elaborate on in Section 4.1
3

---

1. Sample natural experiment
2. Estimate pssudb-cuteomes (?)
3. Predict from individual and intervention information
X
A
2
&!
61R
A
W
W
!
C
O +
Predicted t
Y
+
Figure 2: Visual illustration of the CaML. (causal meta-learning) framework. (1) We sample a task (i.e., intervention) and natural experiment from the training data consisting of individuals who either received the intervention (W=(f)), or did not (W=()). Each individual has features (X) and an outcome (Y), and the intervention also has information (W) (c.g.. drug's attributes). (2) For each individual we estimate the effect of the intervention on the outcome (pseudo-outcomes F). (3) We predict an individual's pseudo-outcome outcomess using a model that fuses X and W. CaML trained by repeating this procedure across many tasks and corresponding natural experiments.
CaML overview. We propose novel framework for estimating CATE across multiple interventions, even including ones that were never encountered during training. Our framework consists of three key components (Figure 2). First, we formulate CATE estimation as a meta-learning problem in which each task corresponds the CATE estimation for a unique intervention. task dataset for a given intervention is constructed by sampling natural experiment of all individuals who received the intervention, and a sample individuals who did not. Tasks are augmented with intervention information (W). Synthesizing these natural experiments allows to compute noisy CATE label using any off-the-shell estimator (f is referred to as pseudo-outcome, by the causal inference literature (16). Finally, we train single meta-model to predict these labels using individual-level (X) and intervention-lers (W) information, such that it is able to generalize to novel tasks, i.e., estimating CATE for novel interventions.
The CaML framework incorporates three important design considerations: (I) Single meta-model. In domains such as electronic health records and online marketing, we observe that large-scale datasets contain thousands of interventions with rich feature information (W). Instead of training separate model for each intervention, CaMI. trains single meta-model that can estimate CATE across all interventions. This approach lets us leverage shared structure across tasks and generalize to novel interventions that were not present during training. (2) Pseudo-outcomes. Instead of directly modeling the response surfaces E| (w) X ø] and E ( X = z). we use pseudo-outcomes for each intervention to train model. This approach is informed by recent studies indicating bias in estimating CATE from direct predictions of observed outcomes Ioo 44 CaML outperforms strong baselines that meta- meta-learn Y(w) and (0) directly, as demonstrated in experiments (see Tables
2 and 3 rows S-learner and T-learner with meta-learming). (3) Discrete tasks from contimuous interventions. CaML takes advantage of the extensive literature CATE estimation for single, binary interventions. By creating a natural experiment for each intervention, CaML. taps into this literature and benefits from the high performance of recently developed nonparametric CATE estimators [16 55 44).
CaML identifies CATE for novel interventions under the assumptions that: (1) for each observed intervention w, Tu (x)) is identifiable under the binary treatment assumptions (unconfoundednesss, consistency, and overlap) in Section 3 This allows for valid training labels for each task. (2)
(x) = T(w, x), i.e., a global function T(w, x) unifies all intervention-specifi CATE functions, (3)
T(w, 2) is continuous in This allows the model to smoothly extrapolate the treatment effect to new interventions that are close to observed interventions in the intervention space. Lastly, (4) W follows continuous distribution.
4.1 Meta-dataset
We formulate CATE estimation meta-learning problem. For this, each task refers to CATE estimation for a distinct intervention. Interventions as well as tasks in our meta-dataset are jointly indexed by j E N with j < K, such that we can refer to the j-th intervention information with w())

---

We then construct a veta-eataset D in the following way:
D = { (D treated U D()
w )
}} with
control:
(3)
treated {X,, Yt) W. ) and D connrol [(X, W; 0)).
(4)
D6) denotes the natural experiment dataset for task j, composed of a treated group (instances which received the intervention, i.c. W; = w()) and control group (instances which did not receive any intervention, i.c. W = 0). Each sample represents an individual, for which the quantities (X2, Y;) are collected as introduced in Section 3 In practice, we down-sample both groups (i.e. to million samples for the treated and control groups) in our large-scale experiments.
We augment each task dataset D with intervention information, w() E IRS for zero-shot general- ization to new interventions [35 18 87 39]. The form of w'S) varies with the problem domain for text interventions, it could be a language model's text embedding 77
84 [58, while biomedical treatments can be represented as nodes in a knowledge graph 8 49 Additionally, domain-
features, like treatment categories from an ontology, may be included in w To handle com- binations of interventions (e.g., pairs of drugs), we aggregate the w for each intervention using an order-invariant pooling operation (we used the sum operator), and sample separate natural experiment for individuals who received the full combination.
4.2 Estimating pseudo-outcomes
We next estimate the training targets for each task (i.c. intervention) in the meta-dataset. The training target (+()) is an unbiased, but noisy, estimate of CATE. More formally, for each task j (which points to the natural experiment dataset for intervention w(i)),, we estimate =(), where (p |X =
z] T. (z). Thus, 7(0) denotes the target for the i-th sample in the j-th task (indexing will be omitted when clear from context). We refer to these targets as pseudo-outcomes, following prior literature [16). For prior work pseudo-outcomes, refer to Appendix B In Appendix E we demonstrate why these pseudo-outcomes provide an unbiased training objective. For a detailed explanation the necessity of using pseudo-outcone instead of directly modeling Y Yww) and (
please see [44 16 10].
CaML is agnostic to the specific choice of pseudo-outcome estimator. Thus, we assume a function v(DG)) which takes as input a task dataset D E D and returns a vector containing the pseudo- outcomes 5 for each sample in the task. We extend each task dataset D with the gseudo-outcome such that sample holds the elements (X,, Y., it). Our key insight that by collecting these aseudo-outcomec across multiple tasks, and predicting them using a combination of intervention and individual information (W, X) we can develop a CATE estimator which generalizes to novel interventions. In practice, we use the RA-learner [17 and treat pseudo-outcome estimation as data pre- pre-processing step (Appendix C..p.
4.3 Meta-model training
Given m target outcomes Y1. .... Ym (e.g., different drug side effects), our goal is then to learn model o Re Rdd + IRm that for parameters minimizes
f" argmin E)-U(D) Ew,x.=<Do) [V)))
(5)
where U(D) denotes the discrete uniform distribution over the tasks of the meta-dataset D, and where L(J) refers to a standard loss function between the pseudo-outcomes and the model output, i... I(f)= = (+ f (w, xz))² To assess whether the model generalizes to novel tasks, we partition our meta-dataset by task, into oveellellg subsets D - Dirain U Dval U D During training, vo is optimized on training tasks Dirin- We validate and test this model on Deat and Diest which are thus unseen during training tasks. While the CaML framework is agnostic to a specific training strategy, we based our approach (Algorithm T on the Reptile meta-learning algorithm (53) which we find performs better compared to straightforward empirical risk minimization (c.f. Section 6). For this, the objective is slightly modified to
0" argmin E)~U(D) [T. (A (Vo))].
(6)
0
5

---

Algorithm The CaML algorithm
Require: meta-dataset D, meta-model Vo with initialized parameters 0, hyperparameter k.
function ADAPT(Data D, Pseudo-outcomes f, Intervention information Model Va. of Steps V < Create copy of w
for iteration 2........44
j SAMPLETASK()
D D w0) + QUERYTASKDATA()
for 1=s=1122 1, ........h do
70, ESTUANTEREEDDOUUCOOESS
(
Draw batch of size b from D.
Compute loss L. by feeding instances through model, conditioned on task:
<- ADAPT((DEM) moay D #0), w()), No. k)
g - 0 - (Reptile gradient)
C. = Σ - (F: (ys,,,);;) Update parameters of Vy: 0 +- 0 -VV. L.
0 -- 0 8g (Gradient step for meta-model v/
end for return a
end for function
where Ab: F represents the operator that updates a model F using data sampled from the dataset T) for k gradient steps. This operator is defined in more detail as the ADAPT routine in Algorithm|] [ Note that depending on the choice of CATE estimator, this routine iterates only over treated samples of task dataset D (as in our experiments), or over all samples, including untreated ones.
4.4 CaML architecture
To parameterize Vo. we propose a simple but effective model architecture (see Section 6):
Vo (w, x) = MLP, ([x; with T MLP2(xr) and w MLPs(w).
where [::-] denotes concatenation. Equation 7 shows that the intervention information and individual features æ are encoded separately into dense vectors w and ž, respectively, Our MLPs consist of layers of the form g (z) = ReLU(Linear(z)).
5 Theoretical analysis
We now consider zero-shot causal learning from theoretical perspective. Under simplified assump- tions, we bound the prediction error in the zero-shot setting.
We formulate the setting as a supervised learning problem with noisy labels (pseudo-outcomes) where we learn a smooth function f = (ww x) + т among family F. We focus on E R, and assume [0, 1] without loss of generality, since can normalize interventions with m samples each, i.c. first i.i.d. draws from Pw: w(1), ..... (()) and then for each (i), m i.idd draws from Px: 21 ".,,..".,!!! (t)
this range. The training dataset has
The main theorem quantifies the rate that combining information across different interventions helps with zero-shot performance. We prove a finite-sample generalization bound for ERM variant of CaML. The ERM is special analysis. The advantage of Reptile over ERM orthogonal and we refer the readers to the original discussion [54]. We assume the estimated pscudo-outcomes 7 during training satisfy 7 where & is an independent zero-mean noise with |4| < almost surely for some e 2 0,
of ADAPT with k 1 that is more conducive to rigorous
L(f) - fef
= min
min
(1) (1))))))1)))) =10)33
j=1
The test error is L(f) = (f((,) r) Let min, L(f). We bound the excess loss L(f) L(f*). Our key assumption is that interventions with similar features W have similar effects in expectation. We assume that all functions in our family are smooth with respect to W, i.e., of E F.Ex.xx [|af//w||] 8?.

---

Theorem 1. Under our assumptions, with probability 1
I(f) $ L(F) + 8(1 +11+0 +. Rmm (F) 8/ (1 +x) Rmm (1g(1/)) 2 2 3n
+
n
(32CB² + 2(1 + c)? m)) log (1/6)
(1 (
where Rum is a novel notion of zero-shot Rademacher complexity defined in equation ( C' Poincaré constant that only depends on the distribution of W. For large n,m, the leading terms are the function complexity Rzo (F), and an O( O(vI/n) term with a numerator that scales with and (1 + e)² This validates our intuition that when the intervention information W more informative of the true treatment effects (smaller 3), and when the estimation of T in the training dataset is more accurate, the performance is better novel interventions. Please refer Section A for the full proof. Compared standard generalization bound which usually has a v I/n term, our main technical innovation involves bounding the variance by the smoothness of the function class plus Poincaré-tyy type inequalities. When is much smaller than we achieve a tighter bound.
Intervention type
Intervention in- formation (W)
Dataset
Samples
Features (X)
Outcome (Y)
Patient history (binned counts of medical codes)
Drug embedding (knowledge graph)
Drug intake (pre-
Claims
Patients
Pancytopenia onset
scription)
Molecular
Cancer cell encyclope- dia
Expression of land- mark genes (DEG)
Perturbagen (small molecule)
LINCS
Cell lines
embeddings
(RDKit)
Table I: High-level overview of our two experimental settings. Details in Appendix
Experiments
We explore to what extent zero-shot generalization is practical when predicting the effects of interven- tions. We thus design two novel evaluation settings using real-world data domains where zero-shot CATE estimation will be highly impactful:
(1) Health Insurance Claims: predicting the effect of a drug on a patient, and (2) LINCS: predicting the effect of perturbation on a cell. We use new datasets because existing causal inference benchmarks [31 73 focus on single intervention. By contrast, zero-shot causal learning must be conceptualized a multi-intervention setting.
Zero-shot Evaluation. Each task corresponds to estimating CATE for a single intervention, across many individual samples (e.g. patients). We split all tasks into meta-rraining/mett-vvlliee vaaliiaaaiiii,on and a hold-out meta-testing set for evaluating zero-shot predictions (Table 2 unseen drugs for Claims and Table 3 unseen molecular perturbations in LINCS). For the Claims dataset, also consider the challenging setting of combinations of unseen drugs (Table
Each meta-validation and meta-testing task contains natural experiment of many samples (e.g., patients) who received the unseen intervention, and many control samples who did not receive the intervention. The same patient (Claims) or cell-line (LINCS) appear in multiple tasks (if they received different interventions at different times). Thus, to ensure fair zero-shot evaluation, we exclude all samples who have received meta-testing intervention from meta-val/metaa-rriie Similarly, we exclude all meta-validation patients from meta-train. Details on holdout selection are provided in Appendix C.2
Table gives an overview of both benchmarks. In the Claims dataset, we compare zero-shot predictions with strong single-intervention baselines which cannot generalize to unseen interventions. To do so, we further split each task in meta- validation and meta-testing into a train/test (50/50) split of samples. These baselines are trained on task's 's train split, and all methods the test split the meta- meta-testing tasks. On the LINCS dataset, as each task consists of < 100 cells, single-intervention baselines performed weakly and are excluded from analysis.
evaluated on
Baselines. We compare the zero-shot performance of CaML. to two distinct categories of baselines. (1) Trained directly test interventions. These are strong CATE estimators from prior work and
7

---

can only be trained on a single intervention. Thus, we train a single model on each meta- meta-testing task's train split, and evaluate performance on its test split. This category includes T-learner [4]
X-learner 44 RA-learner [16], R-learnr 55 DragonNet 72), TARNet 70), and FleXTENet 17
(2) Zero-shot baselines trained across all meta-training tasks and are able to incorporate inter- vention information (W). These methods are thus, in principle, capable of generalizing to unseen interventions. We use GraphITE (25) and Structured Intervention Networks (SIN) [35] We also in- troduce two strong baselines which learn to directly estimate Y (w) and (0) by meta-learning across S-learner and T-learner with meta-learning. These extend the S-learner and T-learner from prior work [44] to incorporate intervention informa- tion (W) in their predictions. We elaborate on implementation details of baselines in Appendix C.7 For details on hyperparameter search and fair comparison, see Appendix C.T
all training interventions, without using
Ablations. In our first ablation experiment (w/o meta-learning), we trained the CaML. model without meta-learning, instead using the standard empirical risk minimization (ERM) technique [78]. Our second ablation (w/o RA-learner) assesses the sensitivity of CaML's performance to different pseudo- outcome estimation strategies. For further details on how these ablation studies were implemented, see Appendix C.3
We discuss the key findings from these ablations in Section 6.3
6.1 Setting 1: Personalized drug side effect prediction from large-scale medical claims
Our first setting (Claims) is to predict the increased likelihood of a life-threatening side effect caused by a drug prescription. We leverage a large-scale insurance claims dataset of over 3.5 billion claims across 30.6 million patients in the United State Each datestamped insurance claim contains set of diagnoses (ICD-10 codes), drug prescriptions (DrugBank ID), procedures (ICD-10 codes), and laboratory results (LOINC codes). Laboratory results were categorized by whether the result was high, low, normal, abnormal (for non-continuous labs), or unknown.
Interventions are administration of one drug (n 745), two drugs (n = 22,883) prescribed in combination. Time of intervention corresponds to the first day of exposure. Intervention information (W) was generated from pre-trained drug embeddings from large-scale biomedical knowledge graph 8 (Appendix C). We compute drug combination embeddings as the sum of the embeddings of the constituent drugs. We focus on the binary outcome (Y) of the occurrence of the side effect pancytopenia within 90 days of intervention exposure. Pancytopemia is deficiency across all three blood cell lines (red blood cells, white blood cells, and platelets). Pancytopenia is life-threatening with a 10-20% mortality rate (38 43), and is a rare side effect of many common medications (42) (e.g. arthritis and cancer drugs), which in turn require intensive monitoring of the blood work. Following prior work (24). patient medical history features (X) were constructed by time-binned counts of each unique medical code (diagnosis, procedure, lab result, drug prescription) at seven different time scales before the drug was prescribed, resulting in total of 143,940 features. For more details, refer to Appendix C.I
Metrics We rely on best practices for evaluating CATE estimators in observational data, as established by recent work [86] [1] which recommend to assess treatment rules by comparing subgroups across different quantiles of estimated CATE. We follow the high vs. others RATE (rank-weighted average treatment effect) approach from Yadlowsky et. al [86). which computes the difference in average treatment effect (ATE) of the top percent of individuals (ranked by predicted CATE), versus all individuals (for more details, see Appendix C.I). For instance, RATE @ 0.99 is the difference between the top 1% of the samples (by estimated CATE) vs. the average treatment effect (ATE) across all samples, which we would expect be high if the CATE estimator is accurate. Note that estimates of RATE can be negative if model predictions are inversely associated with CATE. We elaborate on the RATE computation in Appendix C.
The real-world use case of our model is preventing drug prescription for a small subset of high-risk individuals. Thus, more specifically, for each task j. intervention wy in the meta-dataset, and meta- model Vo, we compute RATE E @ u for each u in [0.999, 0.998, 0.995, 0.99] across individuals who received the intervention. We use narrow range for u because pancytopenia is a very rare event occurring in less than 0.3% of the patients in our dataset. Hence, in real-world deployment scenario, it is necessary to isolate the small subset of high-risk patients from the vast majority of patients for whom there is no risk of pancytopenia onset.
Insurance company undisclosed per data agreement.
8

---

RATE @
(1)
Rocall
@
(t)
Precision @hu
(1)
0.999
.908
0.995
0.99
0.999
0.988
0.995
0.99
0.989
0.996
0.996
0.99
Randomn
0.00
0.00
0.00
0.00
0.00
0.00
0.01
0.00
0.00
0.00
0.00
0.00
T-leamer
0.32
0.26
0.16
0.10
0.12
0.18
0.26
0.31
0.36
0.29
0.18
0.11
X-leamer
0.04
0.02
0.08
0.07
0.06
0.05
R. learnnr
0.19
0.17
0.12
0.08
0.06
0.1
0.19
0.26
0.24
0.21
0.15
0.11
Trained
RA-learner
0.47
0.37
0.23
0.14
0.17
0.26
0.38
0.45
0.54
0.42
0.26
0.16
on test
DragonNet
0.09
0.07
0.05
0.04
0.03
0.05
0.08
0.11
0.15
0,12
0.08
0.06
intervention
TARNet
0.15
0.12
0.07
0.05
0.05
0.08
0.12
0.14
0.18
0.15
0.09
0.06
Fle
0.10
0.09
0.06
0.04
0.04
0.06
0.11
0.16
0.15
0.13
0.09
0.06
BBdt undertined)
GraphITE
0.05
0.07
0.09
0.10
0,14
SIN
0.00
0.00
0,00
0.00
0.00
0.00
0.01
0.02
0.01
0.01
0.01
0.01
S. leammer w/ meta leurning
0.21
0.16
0.09
0.05
0.08
0.11
0.15
0.16
0.25
0.18
0.1
0.06
T-leamer w/ mota-lcarning
0.40
0.31
0.18
0.11
0.15
0.22
0.32
0.38
0.45
0,35
0.21
0.13
Zero-shot
CaML w/o meta- learning
0.39
0.31
0.18
0.11
0.15
0.22
0.32
0.39
0.45
0.35
0.22
0.14
CaML- RA-learner
0.45
0.36
0.22
0.14
0.16
0.24
0.34
0.41
0.48
0.38
0.26
0.16
CaML (ours)
0.23
0.18
0.38
0.45
0.54
0.43
0.16
(Best in bold)
Table 2: Performance results for the Claims dataset (predicting the effect of drug exposure on pancytopenia onset from patient medical history). Key findings are (1) CaML outperforms all zero- shot baselines (RATE is 18-27% higher than T-Learner w/ meta-learning, the strongest zero-shot baseline) (2) CaML performs stronger (up to 8x higher RATE values) than 6 of the baselines which are trained directly on the test interventions, and performs comparably to the strongest baseline trained directly on the test interventions (RA-learner). Mean reported across all runs; standard deviations included in (Appendix Table 4.. Analogous trends hold for generalization to pairs of unseen drugs (Appendix Table
Additionally, because our meta-testing dataset consists of individuals treated with drugs known to cause pancytopenia, observational metries of recall and precision are also rough prexy for successful CATE estimation (and highly correlated to RATE, Table 2). Thus, as secondary metrics, we also compute Recoll @ and Precision @ u for the same set of thresholds as RATE, where positive label is defined as occurrence of pancytopenia after intervention.
6.2 Setting 2: Cellular gene expression response due to perturbation
Our second setting (LINCS) is to predict how a cell's gene expression (Y) will respond to intervention from perturbagen (small molecule compound such drug). This is critical problem accurately predicting intervention response will accelerate drug-discovery. We use data for 10,325 different perturbagens from the LINCS Program (74). Each perturbagen corresponds to different small molecule. Molecular embeddings were generated using the RDKit featurizer [46] and used as intervention information (W). Outcomes (Y) of interest are post-intervention gene expression across the top-50 and top-20 differentially expressed landmark genes (DEGs) in the LINCS dataset. We did not look at all 978 genes since most do not show sigmificant variation upon perturbation. We use 19,221 features (X) from the Cancer Cell Line Encyclopedia (CCLE) [22 to characterize each cell-line (n = 99), each of which correspond unperturbed gene expression measured in a different lab environment using different experimental assay. For more details, see Appendix C.
Metrics. A key advantage of experiments cells is that at evaluation time we can observe both (0) and Y (1) for the same cell line X, through multiple experiments on clones of the same cell- line in controlled lab conditions. In the LINCS dataset, (() is also measured for all cells which received an intervention. Thus, we can directly compute the precision in estimating heterogencous effects (PEHE) on all treated cells in our meta-testing dataset, an established measure for CATE estimation performance analogous to mean equa-equared error 30 (see Appendix C.I
6.3 Key findings
CaML's zero-shot predictions outperform baselines with direct access to the target intervention. In the medical claims setting, single intervention baselines [Tables[] dark grey rows) are the highest performing baselines as we train them directly on the meta-test intervention. Still, CaML outper- forms 6 out of of these baselines (up to 8 3 higher RATE) and achieves comparable performance to the strongest of these baselines, the RA-learner. Furthermore, CaML. strongly outperforms alternative zero-shot CATE estimators (RATE is 18-27% higher than T-Learner w/ meta-learning, the strongest zero-shot baseline). In the LINCS data, multi-intervention learners are strongest as

---

PFHE 50 DEGs (1)
PEHE 20 DEGS (J)
Mcan.
3.78
4.11
GraphITE
3.58 0.023
3.82 1: 0.011
SIN
3.78.1 0.001
4.06 1: 0.001
S-leamer w/ meta-learning
3.63 1 0.004
3.90 1 0.004
T-lamee w/ meta-lcarning
3.61 0.007
3.85 = 0.006
CaML w/o meta-learning
3.57 1: 0.006
3.79 1 0.004
CaML w/o RA-leamner
4.28 1. 0.517
4.60 1 0.413
CaML (ours)
3.56 0.001
3.78 = 0.005
Table 3: Performance results for the LINCS dataset (predicting the effect of an unseen perturbation on the gene expression of unseen cell-line). CaML outperforms all baselines. Improvement is largest for the 20 most differentially expressed genes, where most signal is expected.
there are only small number of instances (cell lines) per intervention CaML outperforms both and multiiintervention learners by drawing from both of their strengths-it allows us use strong CATE estimation methods (i.e. the RA-leamer) which previously were restricted to single interventions, while sharing information across multiple interventions.
CaML learns generalize from single interventions combinations of unseen interventions (drug pairs). We evaluate CaML's performance in the challenging setting of predicting the personalized effects of combinations of two drugs which are both unseen during training, while only training on interventions consisting of single drugs. CaML. achieves strong performance results (see Appendix Table 5) surpassing the best baseline trained on the test tasks, and outperforms all zero-shot baselines, across all 12 metrics.
Understanding CaML's performance results. Our ablation studies explain that CaML's performance gains are due to (1) our meta-learning formulation and algorithm (in contrast to the w/o meta-learning row, in which ERM is used to train the model), and (2) the flexible CATE estimation strategy, allowing to take advantage of recently developed CATE estimators previously restricted to single interventions (in contrast to the w/o RA-learner row, in which alternative oseudo-outcome estimator is used). Lastly, (3) comparison to existing binary intervention CATE estimators trained separately on each meta-testing intervention (Table 2 grey rows) shows that gain from learning from thousands interventions. See Appendix
C.3 for details on ablations.
7 Conclusion
We introduce a novel approach to predict the effects of novel interventions. CaML consistently outperforms state-of-the-art baselines, by unlocking zero-shot capability for many recently developed CATE estimation methods which were previously restricted to studying single interventions in isolation. While study is limited retrospective data, we plan to prospectively validate findings. Future work includes designing new model architectures and CATE estimators which learn well under the CaML framework, developing well as more generally exploring novel learning strategies that enable zero-shot causal learning.
metrics to evaluate zero-shot CATE estimators, as
Societal impacts. In high-stakes decision- decision-making inaccurate predictions can lead to severe conse- quences. important not to overly rely on model predictions and proactively involve domain experts, such as doctors, in the decision-making process. Additionally, it is crucial to ensure that underserved communities are not disadvantaged by errors in treatment effect estimates due to un- derrepresentation in the training data. Important avenues for achieving equitable CATE estimation in future work include -orientd approaches (i.c., evaluating model errors for underserved demographics), and outcome-oriented methods (i.e., gauging model impacts on demographic utility)
12 57 69 2 Furthermore, the deployment of CATE models could raise privacy concerns. These models typically require access to individual patient data to estimate personalized treatment effects accurately. Ensuring the privacy and security of this sensitive information is crucial to avoid potential data breaches or unauthorized access, which could harm patients and erode public trust in healthcare systems.
'Single-takk baselines excluded from Table all performed similar or worse than mean baseline due to low task sample size.
10

---

Acknowledgements
We are deeply grateful to Stefan Wager for his invaluable insights and extensive contributions to our discussions. We thank Emma Pierson, Kexin Huang, Kaidi Cao, Yanay Rosen, Johann Gaebler, Maria Brbić, Kefang Dong, June Vuong for helpful conversations. H.N supported by Stanford Knight-Hennessy Scholarship and the National Science Foundation Graduate Research Fellowship under Grant No. DGE-1656518 18. M.M. was supported by DARPA N66001 1924033 (MCS), NIH NINDS R61 NS11865. GSK, Wu Tsai Neurosciences Institute. A.S and S.O were supported by the American Slovenian Education Foundation (ASEF) fellowship. M.Y was supported by the Microsoft Research PhD fellowship. Y.R was supported by funding from GlaxoSmithKline LLC. Y.C. was supported by Stanford Graduate Fellowship and NSF IIS 2045685. We also gratefully acknowledge the support of Stanford HAI for Google Cloud Credits, DARPA under Nos. HR001 12190039 (TAMI), N66001 1924033 (MCS); ARO under Nos. W991NNF16-D 1---16-1-0342 (MURT), W91 INF-16-1- 0171 (DURIP); NSF under Nos. OAC-183559 (CINES), OAC-1934577 (HDR), CCF- 191-9998999 19940 (Expeditions), NIH under No. 3U54HG010426-045 (HuBMAP), Stanford Data Science Initiative. Wu Tsai Neurosciences Institute, Amazon, Docomo, GSK, Hitachi, Intel, JPMorgan Chase, Juniper Networks, KDDI, NEC, and Toshiba.

---

References
[1] Ahmed M Alaa and Mihaela Van Der Schaar. Bayesian inference of individualized treatment effects using multi-task gaussian processes. Advances neural information processing systems, 30, 2017.
[2] Tim Althoff, Hamed Nilforoshan, Jenna Hua, and Jure Leskovec. Large-scale diet tracking data reveal disparate associations between food environment and diet. Nature communications, 13(1):267, 2022.
[3] Susan Athey and Guido Imbens. Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences, 1 733327))7333-7360 2016.
[4] Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a synaptic learning rule. Citeseer, 1990.
[5] Ioana Bica, Ahmed M Alaa, Craig Lambert, and Mihaela Van Der Schaar. From real-world patient data to individualized treatment effects using machine learning: current and future methods to address underlying challenges. Clinical Pharmacology Therapeutics, 109(1):87- 100, 2021.
[6] Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford university press, 2013.
[7] Olivier Bousquet. A bennett concentration inequality and its application to suprema of empirical processes. Comptes Rendus Mathematique, 334(6):495-500 2002.
[8] Payal Chandak, Kexin Huang, and Marinka Zitnik. Building knowledge graph to enable precision medicine. bioRxiv, 2022.
[9] Hong-Bin Bin Chen, Sinho Chewi, and Jonathan Niles-Weed. Dimension-free log-sobolev incquali- ties for mixture distributions. Journal of Functional Analysis, 281(11): 109223,, 2021.
[10] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters, 2018.
[11] Victor Chernozhukov, Mert Demirer, Esther Duflo, and Ivan Fernandez- Val. Generic machine learning inference on heterogeneous treatment effects in randomized experiments, with an application to immunization in india. Technical report, National Bureau of Economic Research, 2018.
[12] Sam Corbett-Davies, J Gaebler, Hamed Nilforoshan, Ravi Shroff, and Sharad Goel. The measure and mismeasure of fairness. Mach. Learn Res, 2023.
[13] Richard K Crump, V Joseph Hotz, Guido W Imbens, and Oscar A Mitnik. Nonparametric tests for treatment effect heterogeneity. The Review of Economics and Statistics, 90(3):389-405, 2008.
[14] Alicia Curth, David Svensson, Jim Weatherall, and Mihaela van der Schaar. Really doing great estimating cate? critical look at ml benchmarking practices in treatment effect estimation. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
(15) Alicia Curth and Mihaela van der Schaar. Doing great at estimating cate? on the ne- glected assumptions benchmark comparisons of treatment effect estimators. arXiv preprint arXiv:2107. /3346, 2021.
[16] Alicia Curth and Mihacla van der Schaar. Nonparametric estimation of heterogeneous treat- ment effects: From theory to learning algorithms. In International Conference on Artificial Intelligence and Statistics, pages 1810-1818, PMLR, 2021.
[17] Alicia Curth and Mihaela der Schaar. On inductive biases for heterogeneous treatment effect estimation. Advances in Neural Information Processing Systems, 34:1 15833-1888, 2021.
[18] Gerald DeJong and Raymond Mooney. Explanation-baan based learning: An alternative view. Machine learning. 1986.
[19] Qiaonan Duan, Corey Flynn, Mario Niepel, Mare Hafner, Jeremy L Mublich, Nicolas Fernandez, Andrew D Rouillard, Christopher M Tan, Edward Y Chen, Todd R Golub, et al. Lines canvas browser: interactive web app to query, browse and interrogate lines 11000 gene expression signatures. Nucleic acids research,
2014.
12

---

[20] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostee meta-learning for fast adap- tation of deep networks. In International conference on machine learning, pages 1135-115. PMLR, 2017.
[21] Dennis Frauen and Stefan Feuerriegel. Estimating individual treatment effects under unobserved confounding using binary instruments. arXiv preprint arXiv:2208. 08544, 2022.
[22] Mahmoud Ghandi, Franklin W Huang, Judit Jané- Valbuena, Gregory V Kryukov, Christopher C
Lo, E Robert McDonald, 3rd, Jordi Barretina, Ellen T Gelfand, Craig M Bielski, Haoxin Li, Kevin Hu, Alexander Y Andreev- Drabheev,, Jaegil Kim, Julian M Hess, Brian J Haas, François Aguet, Barbara A Weir, Michael V Rothberg, Brenton R Paolella, Michael S Lawrence, Rehan Akbani, Yiling Lu, Hong L Tiv, Prafulla C Gokhale, Antoine de Weck, Ali Amin Mansour, Coyin Oh, Juliann Shih, Kevin Hadi, Yanay Rosen, Jonathan Bistline, Kavitha Venkatesan, Anupama Reddy, Dmitriy Sonkin, Manway Liu, Joseph Lehar, Joshua M Korn, Dale A Porter, Michael D Jones, Javad Golji, Giordano Caponigro, Jordan E Taylor, Caitlin M Dunning, Amanda I. Creech, Allison C. Warren, James M McFarland, Mahdi Zamanighomi, Audrey Kauffmann, Nicolas Stransky, Marcin Imielinski. Yosef E Maruvka, Andrew D Cherniack, Aviad Tsherniak, Francisca Vazquez, Jacob D Jaffe. Andrew A Lane, David M Weinstock, Cory M Johannessen, Michael P Morrissey, Frank Stegmeier, Robert Schlegel, William C Hahn, Gad Getz, Gordon B Mills, Jesse S Boehm, Todd R Golub, Levi A Garraway, and William R Sellers. Next-gene cext-generation characterization of the cancer cell line encyclopedia. Nature, 569(7757):503-500 May 2019.
[23] Donald P Green and Holger L Kern. Modeling heterogeneous treatment effects in survey experiments with bayesian additive regression trees. Public opinion quarterly, 76(3):491-511, 2012.
[24] Lin Lawrence Guo, Ethan Steinberg, Scott Lanyon Fleming, Jose Posada, Joshua Lemmon, Stephen R Pfohl, Nigam Shah, Jason Fries, and Lillian Sung. Ehr foundation models improve robustness in the presence of temporal distribution shift. medRxiv, 2022.
[25] Shonosuke Harada and Hisashi Kashima. Graphite: Estimating individual effects of graph- structured treatments. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 659-668, 2021.
[26] Negar Hassanpour and Russell Greiner. Counterfactual regression with importance sampling weights. In IJCAI, pages 5880-5887, 2019.
[27] Negar Hassanpour and Russell Greiner. Learning disentangled representations for counterfactual regression. In International Conference on Learning Representations, 2019.
[28] Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining. inference, and prediction, volume 2. Springer, 2009.
[29] Leon Hetzel, Simon Böhm, Niki Kilbertus, Stephan Günnemann, Mohammad Lotfollahi, and Fabian Theis. Predicting single-cell perturbation responses for unseen drugs. arXiv preprint ar0iv:2204. 13545, 2022.
[30] Jennifer L Hill. Bayesian nonparametric modeling for causal inference. Journal of Computa- tional and Graphical Staristics, 20(1): 2011.
[31] Jennifer L Hill, Jeanne Brooks-Gure Gunn, and Jane Waldfogel. Sustained effects of high participation in early intervention for wew-iitth-weight premature infants. Developmental psychology, 39(4):730, 2003.
[32] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9):5149-5161 2021.
[33] Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015.
[34] Fredrik Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual inference. In International conference on machine learning. pages 3020-3029, PMLR, 2016.
[35] Jean Kaddour, Yuchen Zhu, Qi Liu, Matt J Kusner, and Ricardo Silva. Causal effect inference for structured treatments. Adyances in Neural Information Processing Systems, 14:24841-24855
2021.
13

---

[36] Edward H Kennedy, Optimal doubly robust estimation of heterogeneous causal effects, arXiv preprint arXiv:2004.14497 2020.
[37] Edward H Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects (2020). URL https://arxiv. org/abs, 2020.
[38] Jitender Mohan Khunger, S Arulselvi, Uma Sharma, Sumil Ranga, and VH Talib. Pancytopenia- a clinico haematological study of 200 cases. Indian journal of pathology & microbiology.
45(3):375-379, 2002.
[39] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, al. Wilds: A benchmark of ti-the-wild distribution shifts. In International Conference on Machine Learning (ICML), 2021.
[40] Andrei V Konstantinov, Stanislav R Kirpichenko, and Lev V Utkin. Heterogencous treatment effect with trained kemels of the nadaraya-wand watson regression. arXiv preprint arXiv:2207.091199 2022.
[41] Kostantinos. Gaussian mixtures and their applications to signal processing. Advanced signal processing handbook: theory and implementation for radar, sonar, and medical imaging real time systems, pages 3-1, 2000.
[42] Michael Kuhn, Ivica Letunic. Lars Juhl Jensen, and Peer Bork. The sider database of drugs and side effects. Nucleic acids research, (4DDI:D1075-D1077) 2016.
[43] R Kumar, SP Kalra, H Kumar, AC Anand, and H Madan. Pancytopenia-a six year study. The Journal of the Association of Physicians of India, 49:1078-1081. 2001.
[44] Sören R Künzel, Jasject S Sekhon, Peter Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the national academy of sciences, 1 116 (0)):4156-44666 2019.
[45] Nikolay Kuznetsov and Alexander Nazarov. Sharp constants in the poincaré, steklov and related inequalities (a survey). Mathematika, 61(2):328- 344, 2015.
[46] Greg Landrum et al. Rdkit: Open-source cheminformatics. 2006.
[47] Michel Ledoux. Concentration of measure and logarithmic sobolev inequalities. In Seminaire de probabilites XXXIII, pages 120-216. Springer, 1999.
[48] Hongzhu Li, Xiangrui Gao, and Yafeng Deng. Stargraph: A coarse-to-fine representation method for large-scale knowledge graph, 2022.
[49] Michelle M Li, Kexin Huang, and Marinka Zitnik. Graph representation learning in biomedicine and healthcare. Nature Biomedical Engineering, pages 1-17, 2022.
[50] Jing Ma, Ruocheng Guo, Aidong Zhang, and Jundong Li. Multi-cause effect estimation with disentangled confounder representation. In Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, 2021.
[51] Ron Meir and Tong Zhang. Generalization error bounds for bayesian mixture algorithms. Journal of Machine Learning Research, 4(Oct):839-88)) 860, 2003.
[52] Stephen L Morgan and Christopher Winship. Counterfactuals and causal inference. Cambridge University Press, 2015.
[53] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: 18880000999 2018
[54] Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint
10XVv:100.00299, 2(3):4, 2018.
[55] Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299-311 2021.
[56] Frank Nielsen and Richard Nock. On the chi square and higher-order chi distances for approxi- mating f-divergences. IEEE Signal Processing Letters, 21-1): 2013.
[57] Hamed Nilforoshan, Johann D Gaebler, Ravi Shroff, and Sharad Goel. Causal conceptions of fairness and their consequences. In International Conference on Machine Learning, pages 16848-16887. PMLR, 2022.
14

---

[58] Hamed Nilforoshan and Eugene Wu. Leveraging quality prediction models for automatic writing feedback. In Twelfth International AAAI Conference on Web and Social Media, 2018.
[59] Lawrence E Payne and Hans F Weinberger. An optimal poincaré inequality for convex domains. Archive for Rational Mechanics and Analysis, 5(1):286-229) 1960.
[60] Henri Poincaré. Sur les équations aux dérivées partielles de la physique mathématique. American Journal of Mathematics, pages 211-294, 1890.
[61] Zhaozhi Qian, Alicia Curth, and Mihaela van der Schaar. Estimating multi-cause treatment effects via single-cause perturbation. Advances in Neural Information Processing Systems, 34:23754-23767, 2021.
[62] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint 1oXiv:190900955: 2019.
[63] Bernardino Romera-Paredes and Philip Torr. An embarrassingly simple approach to zero-shot learning. In International conference on machine learning, pages 2152-2161. PMLR, 2015.
[64] Yusuf Roohani, Kexin Huang. and Jure Leskovec. Gears: Predicting transcriptional outcomes of novel multi-gene perturbations. bioRxiv, 2022.
[65] Shiv Kumar Saini, Sunny Dhamnani, Akil Arif Ibrahim, and Prithviraj Chavan. Multiple treatment effect estimation using deep generative model with task embedding. In The World Wide Web Conference, pages 1601-1611, 2019.
[66] Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016.
[67] André Schlichting. Poincaré and log-sobolev inequalities for mixtures. Entropy, 21(1):89, 2019.
[68] Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
[69] Laleh Seyyed-Kalantari,
Kalantari, Haoran Zhang, Matthew BA McDermott, Irene Y Chen, and Marzyeh Ghassemi. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature medicine, 27(12)2116-22182 2021.
[70] Uri Shalit. Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In International Conference on Machine Learning, pages 3076-3085. PMLR, 2017.
[71] Ankit Sharma, Garima Gupta, Ranjitha Prasad, Arna Chatterjee, Lovekesh Vig, and Gautam Shroff. Metaci: Meta-learning for causal inference in a heterogeneous population. arXiv preprint 1912:: 033960, 2019.
[72] Claudia Shi, David Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effects. Advances in neural information processing systems, 32, 2019.
[73] Yishai Shimoni, Chen Yanover, Ehud Karavani, and Yaara Goldschmnidt. Benchmarking frame- work for performance-evallutioon of causal inference analysis. arXiv preprint 1rXiv:1802.05U 2018.
[74] Aravind Subramanian, Rajiv Narayan, Steven M Corsello, David D Peck, Ted E Natoli, Xi- aodong Lu, Joshua Gould, John F Davis, Andrew A Tubelli, Jacob K Asiedu, David I. Lahr, Jodi E Hirschman, Zihan Liu, Mclanie Donahue, Bina Julian, Mariya Khan, David Wadden, Ian C Smith, Daniel Lam, Arthur Liberzon, Courtney Toder, Mukta Bagul, Marek Orzechowski, Oana M Enache, Federica Piccioni, Sarah A Johnson, Nicholas Lyons, Alice H Berger, Alykhan F Shamji, Angela N Brooks, Anita Vrcie, Corey Flynn. Jacqueline Rosains, David Y Takeda, Roger Hu, Desiree Davison, Justin Lamb, Kristin Ardlie, Larson Hogstrom, Peyton Greenside, Nathanael S Gray. Paul A Clemons, Serena Silver, Xiaoyun Wu, Wen-Ning Zhao, Willis Read-Button, Xiaohua Wu, Stephen Haggarty, Lucienne Ronco, Jesse Boehm, Stuart I. Schreiber, John G Doench, Joshua A Bittker, David E Root, Bang Wong, and Todd R Golub. A next generation connectivity map: I,1000 platform and the first 1,000,000 profiles. Cell, 17166::137-1452..lll November 2017.
15

---

[75] Nicholas P Tatonetti, Patrick P Ye, Roxana Daneshjou, and Russ B Altman. Data-driven prediction of drug effects and interactions. Science translational medicine, 4(125):125ra31- 125ra31, 2012.
[76] Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
[77] Nilesh Tripuraneni, Dhruv Madeka, Dean Foster, Dominique Perrault-Joncas, and Michael I Jordan. Meta-an ancta-snalysis of randomized experiments with applications to heavy-tailed response data. arXiv preprint arXiv:2112.07602 2021.
[78] Vladimir Vapnik. Principles of risk minimization for learning theory. Advances in neural information processing systems, 4, 1991.
[79] Victor Veitch, Dhanya Sridhar, and David Blei. Adapting text embeddings for causal inference.
In Conference on Uncertainty in Artificial Intelligence, pages 919-928. PMLR, 2020.
[80] Stefan Wager. Stats 361: Causal inference, 2020.
[81] Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, (3(522):1228-1221 2018.
[82] Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. A survey of zero-shot learning: Settings, methods, and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-37, 2019.
[83] Yixin Wang and David M Blei. The blessings of multiple causes. Journal of the American Statistical Association, 114(528) 2019.
[84] Galen Weld, Peter West, Maria Glenski, David Arbour, Ryan A Rossi, and Tim Althoff. Adjusting for confounders with text: Challenges and an empirical evaluation framework for causal inference. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pages 1109-1120, 2022.
[85] Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4582-4591, 2017.
[86] Steve Yadlowsky, Scott Fleming, Nigam Shah, Emma Brunskill, and Stefan Wager. Evaluating treatment prioritization rules via ranghweighted average treatment effects. arXiv preprint arXiv:2111.07966 2021.
[87] Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. QA-
GNN: Reasoning with language models and knowledge graphs for question answering. In North American Chapter of the Association for Computational Linguistics (NAACL), 2021.
[88] Shing-Tune -ung Yau. Isoperimetric constants and the first eigenvalue of a compact riemannian manifold. In Annales Scientifiques de l'École Normale Supérieure, volume pages 487-507, 1975.
[89] Jinsung Yoon, James Jordon, and Mihaela Van Der Schaar. Ganite: Estimation of individualized treatment effects using generative adversarial nets. In International Conference on Learning Representations, 2018.
[90] Long Yu, Zhicong Luo, Huanyong Liu, Deng Lin, Hongzhu Li, and Yafeng Deng. Triplere: Knowledge graph embeddings via tripled relation vectors. arXiv preprint 22020:220. 2022.
[91] Yao Zhang, Alexis Bellot, and Mihaela Schaar. Learning overlapping representations for the estimation of individualized treatment effects. In International Conference on Artificial Intelligence and Statistics, pages 1005-1014. PMLR, 2020.
[92] Marinka Zitnik, Monica Agrawal, and Jure Leskovec. Modeling polypharmacy side effects with graph convolutional networks. Bioinformatics, 34(13):445-ii466 2018.
16

---

A Zero-shot Rademacher complexity and Proof of Theorem
A.1 Problem setup and assumptions
Let w E W S R denote an intervention and x: E X S Rd denote an individual that received Assume the outcome to predict is scalar y E [0, 1]. The hypothesis class is F {f (w, a:) v)- The dataset has n interventions with m independent units which received each intervention, i.. first n i.i.d. draws from Pw and then m i.idd draws from Px for each w(t), During training we have access to noisy estimate y y where is independent noise with LEg 0 and |<| almost surely. We are tested directly on y.
The ERM is
f min L(f) min
(f(w)) [z()) - 11
==
The test error is
L(F) = Ew, (f(w, 11)
(8)
and let f = miny L(f).
We are interested bounding the excess error L(f) L(f").
Our key assumption is that interventions with similar attributes (w) have similar effects in expectation. More concretely, we assume that all hypotheses in our family are smooth with respect to w: Assumption 2.
[5 dw
27
Vʃ F, Eawa
13
Furthermore, we assume that Pw satisfies a Poincaré-type inequality: Assumption 3. For some constant C' that only depends on Pw, for any smooth function F,
Varw(F(w)) < CE [Iv))
For example, Pw can be any of the following distributions:
Multivariate Gaussian: w E R ~ (N,, Σ) for some vector E R and positive semi- definite matrix Σ E Rexe,
E H has independent coordinates; each coordinate has the symmetric exponential distribution 1/2e-Itt for t E R.
Pw is mixture over base distributions satisfying Poincaré inequalities, and their pair-wise chi-squared distances bounded.
Pw is mixture of isotropic Gaussians in R
Pw is the uniform distribution over W C R which is open, connected, and bounded with
Lipschitz boundary.
We note that isotropic Gaussian approximate any smooth densities in R are universal), showing that Assumption is fairly general. We define a novel notion of function complexity specialized the zero-shot setting. Intuitively, it measure how well can fit random labels, which is first drawing n interventions and m recipients for each intervention. For examples of concrete upper bound zero-shot Rademacher complexity see section A.4
41 (since RBF kernels
Rym (F)
E.
2,0 sup
() (
(9)
1=
where of are independently randomly drawn from 1-1,
17

---

A.2 Formal theorem statement
Theorem 4. Under Assumptions
with probability 1 6,
(1 + e) Rmm (F) log(1/0)
L(f) L(f)) 8(1 + em (F) + +8/
(32CB² + 2(1+c)² m
log (1/6)
2 2og (1/8)
+(1+
3n
A.3 Proof of the main theorem
We define the population loss on the noisy label L(f) Emg.z. (f(w, x) - -)) Due to independence
of E, E......e(( x) - 1 E)² ( xr) y)² + E L(F) EK for any f. so
I(f) - [( = I(f) - [.))). We shall focus on bounding the latter.
We first need lemma that bounds the supremum of empirical process indexed by bounded function class.
Lemma 5 (Theorem 2.3 of (7). Assume that X identically distributed according P. G is a countable set of functions from to R and, and all E G are P-measurable, square-integrable and we denote Z sup [(=) Suppose ²² 2 supgeg Var(g(X;)) almost surely, the for all 2 O, we have
and satisfy E[g] O. Suppose supgeg ll S
Pr 7 2 EZ + / /2l(no2 +2RZ) + 3
Sett
We apply
Lemma
with
Xj
(wlf), py...., It Fi ........ Hm), g(X,)
1 (f(((u =(-) )2 i(f)) σ² sup (Var( m E, (f(((()) xx (6)) 11))
m
(10)) Since f - 5 [1,, [-1,1]] with probability 1-
sup ((f) - I(n) s -(f)- + 11 2 no² + 2n 2np [(f)
( -
log 1
Multiplying both sides by 1/n, and using Va + b s Va + Vb.
Esup, [[() -i() l08(1/0)
2o log (1/8)
log (1/5)
sup [ iin) s Sup iU) - I(5) +21
n
3n
(10)
The next lemma bounds the variance ²² in equation 10). Lemma 6.
> (f(w))) ((1) - 2(ا0)33 s 4(1 <) + (1+6)2
Vf E F, Var m0 m
4m
==1
Proof of Lemma 6
Using law of total variance, if we write
g(wli) I..... H..... m E( 20))) -/))3
then
Varlg) Var (g v. [| + E, [Var slgllllwww., x,,,xxx0 x. () w|] (11)
To bound the first term of equation TT., we use Poincaré-type inequalities in Assumption For each of the example distributions, we show that they indeed satisfy Assumption 3
18

---

Lemma 7. Each of the example distributions in Assumption 3 satisfies a Poincare-tyee type inequality.
Proof.
When Pw is the uniform distribution over W E R which is open, connected, and bounded with Lipschitz boundary, we use Poincaré Poincaré-Wirtinger inequality [60] on the smooth function Elg w|: For some constant C' that only depends on Pw.
Var [B
w|] I CI
w]]3]]
(12)
C is the Poincaré constant for the domain W L2 norm. can be bounded by 1/Ar where a1 is the first eigenvalue of the negative Laplacian of the manifold W [88]. Many previous works study the optimal Poincaré constants for various domains [45]. For example, when w is uniform over W which is a bounded, convex, Lipschitz domain with diameter d, S d/m [59].
We can apply probabilisticc Poincaré inequalities over non-Lebesgut measure Pw:
When (H, Σ), the Gaussian Poincaré inequality (see e.g. Theorem 3.20 of [6] and using change variables),
Var[F(w)] S (. F(w)).
We apply this with F(w))= = Elg w). Since B T Au] = E(Tr(0 Ac)] = B[Tr(AvuT))
Tr(AE[ev"] T) & || ||
V Var Egg w|] | | Eig w]l
which satisfies equation 12 with C |||||
When E IR has independent coordinates wi....; We and each coordinate has the sym- metric exponential distribution 1/2e-14 for L E R. we first bound single dimension using T,2.
Lemma 4.1 of 47,, which says for any function
Var(k(w;)) I 4E [K' (w:))
which, combined with the Efro-Stein inequality (Theorem 3.1 of 6)),
Var(F(w)) = E
Var(F Wi...., W1-1, Wi+1..... Wp),
yields:
Var(F(w)) S 4E (1))
which satisfies equation
12 with C =
Lastly, consider the where Pw is mixture over base distributions satisfying Poincaré inequalities. We first consider the case where the pair-wise chi-squared distances are bounded. Next, show that mixture of isotropic Gaussians satisfies Poincaré inequality without further condition on pair-wise chi-squared distances.
When (Pi } hese is family of distributions, each satisfying Poincaré inequality with con- stant O9, and Pw is any mixture over {PW JoEQ with density μ. let KP(μ) = ess,, sup, C?,, which is an upper bound on the base Poincaré constants almost surely, and K" (H) =
E Eg ~ (1 + (P w W p)1/,, which is an upper bound on the pairwise x2 diveerggee Using Theorem 1 of (9) such that C < K (p) + (K)(u)) where p" is the dual exponent of satisfying 1/p 1.
we get that Pw satisfies Poincaré inequality with constant C
As an example, when base distributions are from the same exponential family and the natural parameter space is affine, such as mixture of Poisson or Multinomial distribu- tions, the pair-wise chi-squared distances are bounded (under some additional condi- tions) and hence the mixture satisfies Poincaré inequality. More formally, let pe(x)
19

---

exp (T(x)T F(0) + k(z)) where & θ is the natural parameter space and A(O) is the log partition function. Lemma [56] shows that
(pe, l|pe)) e -1) (2)-)))))))))
which is bounded as long as 202 - 0₁ E e. This is satisfied for mixture 1-D Pois- distributions which be written p(w[A) - exp (w log À - A) with natural parameter space I, and mixture of e-dimensional Multinomial distributions p(w](r
exp (w. log (T// (1-Ez m m:))) + log (1 E-= n.) with natural parameter space
R= When applied to Gaussian family the natural parameters are
2 My 0g 1-=(vec) Dec (- )
Since the covariance has to be positive definite matrices, 20, - 02y may not be a set of valid natural parameter. We deal with this in the next case.
When {Pw gee is mixture of isotropic Gaussians, each with mean My R and variance Σn = Te each satisfying Poincaré inequality with constant C (in the single- Gaussian case above we know that C I og)) Pw also satisifes Poincaré inequality. We prove this via induction. The key lemma is below:
Lemma 8 (Corollary of (67)). Suppose measure po is absolutely continuous with respect to measure pt, and Po. Pi satisfy Poincaré inequality with constants Co. C1 respectively, then for all [0, and = O, mixture measure p opn Bp1 satisfies Poincaré inequality with with C S max {Co, C1 + ax1)) where X1 = dp1 dpo 1.
We sort the components in the order non non-decreasin o and add in each component one by one. For each new component i = 2,..., 12), we apply the above lemma with po being mixture of P1 :..... Pür and p1 being the new component Ww- We only need to prove that ... is bounded at every step. Suppose Pa = Ej= ayPW with E/= 1, Pi. and PW
(2x) exp { 1-2(w- (4;) T (w - p3)). Therefore
dpo dp1
[ po(w)² dw (1w)
+1
dpo
1-1 n3 == oz
11w-
6- Ej== E
20y0y' azog.
|ww-;l
I py 11
cxp
exp
203
20
dw
(2m)a/2
| ||
exp
20
The convergence condition of the above integral is 2a? 2 20 for all j < which is satisfied when of 2 of.
Next we observe that
Elg w| V. [ (f(w x) p)) ddddd / (f(w,
(5) Of
0f (f(w,x) -
On p(x, D 21E
-w)
Jz.9
Ow
x.y
Since 1f(wxx) 5il 5 1 almost surely, E
e1 ttu
< B2.
2
([
0s
Ex Egg w]]l 4E
v)
< <4 E)
Therefore
Var
E[g
w| <
CE
[|. 11
I 4411 =
20

---

To bound the second term of equation variables.
we use concentration of mean of m i..d random
Conditioned on w each of the loss (f(w)), x. 2(0)) 110))² are i.d. and bounded in [0, (1 + )
Hence each variable has variance upper bound ((1 c) 0)2 (1 e)1/4 4 and the mean has variance upper bound (1 + e)¹ /4m.
Therefore Varlg] - 4(1 c) + +11+))/Amm.
Proof of Theorem
I(f) - T(5)) S sup fer
[T()) - F(DI
Esup, ((f) - [(f) log(1/6)
(32(1 + c) + 2(1+e)*
log (1/6)
2 log (1/0)
< 2E -())
+
n
n
3n
(13)
by equation 10 and Lemma
We now show that E sup, [ī( - L(f) 2(1 + R) (F). This is similar to the argument for classical Rademacher complexity
nm E(f(w() 2(10 21 E (f xx()) 2/
E @ sup f
-20) (f(w (00010,,220000 - Tboiisg
Es,s sup
1171
(lw()) o6 - (f(s (w (1'))),,,(0)) 1// 1/0119g1
Es, sup
Es sup Σ (o( x(() - - +
1
(0 16) - 10
< Es.. sup
-2 (C).
where the first inequality uses Jensen's 's inequality and convexity of sup.
Now we prove the equivalent of Talagrand's contraction lemma to show that Ru (2) S 2Rnm (F).
Note that the squared loss is 2(1 + e)-Lipschitzz since a(
01 2 - ūl s 2(1 + e). We use the
following lemma prove this:
Lemma 9 (Lemma 5 of [11). Suppose (di). (1i)), i= 1,..., N are two sets of fanctions on O such that for each an 0,6 0' E O, (0) (1) ) S (0) (0)) Then for all functions c: O R
a..(0)) }
c }
E
sup
c())
$ E, sup c(0) + Σ
For any set of w, we apply Lemma 9 with θ = F, f, N nm, diy (f) = (f (f((()) ,.,(0)
g))) (f) = 2(1 (0(1+))fo ())()) _,,0)),,,,,, _..5)) and c(0) = 0. Since (f - 7) (f' 7) < 2(1+c))
fl.. so the condition for Lemma 9 hold. We take expectation over w. x and divide both sides by nm to get
E nm
sup
n m
of (f(w(0,,(()
2(1 + c)
E.
sup
1
m
f( (())
num
1=1
j=1
which means Rum (L) < 2(1 + <) Rm)(F). Substituting this into inequality
13 finishes the proof.
21

---

A.4 Zero-shot Rademacher complexity bound for the linear hypothesis class
Consider the linear classifier F {[1o] wp x 1|w 1|2 S B, l-lwilla S C). Suppose ||w|| S and || S 1.
RN
(F) =
1
E wa sup
(w), Σ Mo of + (W2, Eofesoo)
nm
19
1
Bg w
Eotwwill ll2 + BrBo,
Σ
of o.
nm
1j
15
<
1
Bi
|101i112 + / m
Σ | (1) 11
mm
5
= (Bi + B2) /mm. nm..
We observe that the bound is the same as the standard Rademacher complexity for nm indepen- dent samples, which is interesting. The relationship between standard and zero-shot Rademacher complexity for other function classes is an important future direction.
B Extended Related Work
Our approach to zero-shot prediction of intervention effects is related recent advances in heteroge- nous treatment effect (HTE) estimation, zero-shot learning, and meta-learning.
B.1 Heterogenous treatment effect (HTE) estimation
Conditional average treatment effect (CATE) estimation. A number of approaches have been developed to predict the effect of an existing intervention on an individual or subgroup. based on historical data from individuals who received it. This problem is often referred to the literature as heterogeneous treatment effect (HTE) estimation 28 13,, to denote that the goal is to detect heterogencities in how individuals respond to an intervention. more specific instance of HTE estimation, which we focus on here, is conditional average treatment effect (CATE) estimation [8] 44, in which the goal is to predict the effect of a treatment conditioned on an individual's 's features. variety of methods and specific models have been developed to achieve this goal [28 B4 23 30, 81]
70, I 88 26 91
[4
36
13 3] and we refer to Bica et al. and Curth et al. for detailed
review of these methods 5 16 These methods estimate CATE for an existing intervention, based
on historical data from individuals who received and those that did not.
While these approaches have a number of useful applications, they do not address CATE for novel interventions which did not exist during training (zero-shot). Our primary contribution is a meta- learning framework to leverage these existing CATE estimators for zero-shot predictions. In the CaML framework (Figure ) each task corresponds to predicting CATE for single intervention. We synthesize task by sampling natural experiment for each intervention, and then use any existing CATE estimator to generate noisy target label for our the task (Step 2: estimate pseudo-outcomes)) We rely on pseoudo-outcome estimates training labels because prior work has shown that training on observed outcomes directly leads to biased CATE estimates [10 44 36). a result which we find holds true in our experiments as well (see T-leamer and S-learner w/ meta-learning in Tables 3).
Pseudo-outcome estimators. Prior work has developed a variety of methods to estimate CATE pseudo-outcomes, which are noisy but unbiased estimates of CATE, such as the X-leamer [44). R-learner (55). DR-learner [36], and RA-learner [16]. Moreover, the outputs of any other CATE estimation method, such as methods which directly estimate CATE via an end-to-end neural net- work 34 770 [22 are an equally valid choice of pseudo-outcome, The literature on pseudo-outcome estimation is growing continuously as new estimators are being developed [2] 40). Typically, these estimators are specific to a single binary intervention, for which set of nuisance models are trained and used to compute the pseoudo-outcomes. As such, applying meta-learning algorithms to these pseudo-outcomes requires synthesizing a natural experiment for each intervention, which corresponds to single task in the CaML framework.
22

---

Multi-cause estimators. Our methods to address zero-shot CATE estimation for combinations of interventions are distinct from multi-cause estimators for combinations of binary or categorical Recent work has shown that these methods can predict the effects of new combinations of interventions [50], when every intervention in the combination has been observed at some point during. However, these methods do not estimate CATE for novel interventions which did not exist during training. By contrast, CaML estimates CATE for zero-shot intervention combinations in which none of the interventions in the combo was ever observed during training (Appendix Table C
interventions [3] 61 65
B.2 Zero-shot learning
Zero-shot learning (ZSL) has traditionally aimed reason over new concepts and classes
g5
which did not exist during training time. While ZSL has primarily focused on natural language processing and computer vision [82], recent interest has been sparked in generalizing over novel
interventions (zero-shot) in the biomedical domain [64 29] in which data can be cheaply collected
for hundreds or thousands of possible interventions (92 75 19]. However, ceneral-purposs. zero-
shot causal methods have been largely unexplored. Notable exceptions include GranlTTE (25] and SIN (25), which each extend specific CATE estimation (55] 44 method to incorporate intervention
information (W). However, these approaches have significant drawbacks, which we discuss in Section
B.3 Meta-learning
Meta-leamiin, or learning learn, aims train models which can quickly adapt to new settings and tasks. The key idea is to enable a model to gain experience over multiple learning episodes in which episodes typically correspond distinct tasks to accelerate learning in subsequent learning episodes [32]. The meta-learning literature is rich and spans multiple decades [76] 688 66 4 with
recent interest focused on model-agnostic methods train deep learning models to quickly adapt to
A common focus in the meta-learning literature is few-shot learning, in which a model must adapt to new task given a small support set of labeled examples. By contrast, we focus on the zero-shot setting, in which no such support set exists. However, we hypothesize that the typical meta-learning problem formulation and training algorithms may also improve zero-shot performance. Thus, CaML's problem formulation and algorithm inspiration from the meta-learning literature, particularly the Reptile algorithm
new tasks 20
62 54
(54) and its application to other tasks in causal inference (7l. Our experimental results show that this meta-learning formulation improves CaML's performance, compared to a standard multi-task learning strategy.
C Experimental details
C.1 Experimental setup
Here, we provide more details about the experimental setup for each investigated setting. This serves to complement the high-level overview given in Table [ Experiments were using Google Cloud Services. Deep learning-based methods (i.e., CaML and its ablations, S-learner w/ meta-learning, T-leaner w/ meta-learning, SIN, GraphITE, FlexTENET TARNet, and DragonNet) were nl-highmem-66 64 machines with 4x NVIDIA T4 GPU devices. The remaining baselines (RA-leamer, R-leamer, X-learner, and T-leamer)) were run n1 al-highhem-666 machines featuring 64 CPUs.
Fair comparison. We perform hyper- hyper-parameter optimization with random search for all models, with the meta-testing dataset predetermined and held out. To avoid "hyperparameter hacking", hyperparameters ranges are consistent between methods wherever possible, and were chosen using defaults similar to prior work [35 25). Choice of final model hyype--praameten was determined using performance metrics (specific to each dataset) computed on the meta-validation dataset, using the best Syyer--prramttes over 48 runs (6 servers x 4 NVIDIA T4 GPUs per server x 2 runs per GPU) ) (Appendix C.4). All table results are computed as the mean across 8 runs of the final model with distinct random seeds.
23

---

C.1.1 Claims dataset
Interventions (W): We consider drug prescriptions consisting of either one drug. or two drugs prescribed in combination. We observed 745 unique single drugs, and 22,883 unique drug pairs, excluding interventions which occurred less than 500 times. Time of intervention corresponds to the first day of exposure. To obtain intervention information, we generated pre-trained drug embeddings from large-scale biomedical knowledge graph [8] (see Appendix C.5 Drugs correspond to nodes in the knowledge graph, which are linked to other nodes (e.g. genes, based on the protein target of the drug). Drug combination embeddings are the sum of the embeddings for their constituent drugs.
Control group. A challenge in such causal analyses of clinical settings is defining a control group. We randomly sample 5% (1.52M patients) to use as controls, with 40/20/40 split betweem meta- train/mmta-val/meen val/meta-testt When sampling a natural experiment for a given intervention, we select all patients from this control group that did not receive such intervention. An additional challenge is defining time of intervention for the control group. It is not possible to naively sample a random date, because there are large quiet periods in the claims dataset in which no data is logged. We thus sample a date in which the control patient received a random drug, and thus our measure of CATE estimates the increase in side effect likelihood from the drug(s) W, compared to another drug intervention chosen at random.
Outcome (Y): We focus on the side effect pancytopenia: deficiency across all three blood cell lines (red blood cells, white blood cells, and platelets). Pancytopenia is life-threatening, with a 10- 20% mortality rate (38 43), and is rare side effect of many common medications [42] (e.g. arthritis and cancer drugs), which in turn require intensive monitoring of the blood work. Our outcome is defined as the (binary) occurrence of pancytopenia within 90 days of intervention exposure.
Features (X): Following prior work 24 patient medical history features were constructed by time-binned counts of each unique medical code (diagnosis, procedure, lab result, drug prescription) before the drug was prescribed. In total, 443,940 features were generated from the following time bins: 0-24 hours, 24-48 hours, 2-7 days, 8-30 days, and 31-90 days, 91-365 days, and 365+ days prior. All individuals in the dataset provided by the insurance company had at least 50 unique days of claims data.
Metrics: We rely on best practices for evaluating CATE estimators as established established by recent work 186 which recommend to assess treatment rules by comparing subgroups across different quantiles of estimated CATE. We follow the high vs. others RATE (rank-weighted average treatment effect) approach from Yadlowsky et. al [86). which computes the difference in average treatment effect (ATE) of the top u percent of individuals (ranked by predicted CATE), versus all individuals:
RATE H RATE@u=E B(Y)) -()) Fs((())) 2 1- u -E E
(14)
where S(.) is priority score which ranks samples lowest to highest predicted CATE, and (- is the cumulative distribution function (CDF) of S(X;). ). For instance, RATE @ 0.99 would be the difference between the top 1 1% of the samples (by estimated CATE) vs. the average treatment effect (ATE) across all samples, which we would expect to be high if the CATE estimator is accurate. The real-world use case of our model would be preventing drug prescription a small subset of high-risk individuals. Thus, more specifically, for each task j. intervention wi in the meta-dataset, and meta-model Va (our priority score 5()). we compute R @ for each u in [0.999. 0.998, 0.995, 0.99] across individuals who received the intervention.
We now summarize how to estimate RATE performance metrics for a single intervention (task). As RATE performance is calculated separately per-intervention we are concerned with a single intervention, we use the simplified notation (i.e. Y1(1) instead of (w()) from Section 3 Due to the fundamental problem of causal inference (we can only observe (,()) or Y:(1) for a given sample), the true RATE, as defined above, cannot be directly observed.
We follow the method outlined in Section 2.2 and 2.4 of Yadlowsky et. al. 86 in which we compute Tr a (noisy but unbiased) estimate for CATE which is in turn used to estimate RATE:
r X R r(()) = E (M(1) Y,(0) X.).
(15)
24

---

Our data is observational, and as such we can estimate F; using a direct non non- parametric estimator 80
T. W.(Y; - (0(())-m(X)) (x, 0)) + (1- (X),m 1) Y.) m(r, w) = E (T)( X; = x]]
(16)
(17)
where m(r, w) a model that predicts the outcome. Here m w) represent nonparametric estimates of m(x, w), respectively, which we obtain by fitting cross-fitting model to the intervention natural experiment over 5-folds. We use random forest models for ri (i w., as they perform well (achieving 2 0.90 ROC AUC across all meta-testing tasks for predicting outcomes) and are robust choice of hyperparameters.
RATE can then be estimated via sample-averaging estimator. Specifically, we compute the difference between the average value of I; for those in the top u percent of individuals (based on our meta- model's 's predictions), compared to the average I across all individuals. For further discussion on estimating RATE, we refer readers to [6). Note that estimates of RATE are unbounded: RATE can be less than 0 (due to predictions inversely relating to CATE).
Finally, because our meta-testing dataset consists of individuals treated with drugs known in the medical literature to cause pancytopenia (identified by filtering drugs using the side effect database SIDER 421,, observational metries of recall and precision are also rough proxy for successful CATE estimation. Thus, as secondary metrics, we also compute Recall @ u and Precision @ u for the same set of thresholds as RATE, where a positive label is defined as occurrence of pancytopenia after intervention. We find that these metrics are highly correlated to RATE in our performance results.
Training& & Evaluation: For each method, we ran a hyperparameter search with 48 random config- urations (48 due to running 8 jobs in parallel on servers each) that were drawn uniformly from pre-defined hyperparameter search space (see Appendix C.4). Methods that can be trained on multiple tasks to then be applied to tasks unseen during training (i.e., CaML and its ablations, S-learner w/ meta-learning, T-learner w/ meta-learning, SIN, GraphITE) were trained for 24 hours (per run) on the meta-training tasks. Model selection was performed on the meta-validation tasks by maximizing the mean RATE@ @ 0..00.90 0,.98 across meta-validatio used to fit repetition runs across different random seeds. Each repetition model was then tested on the meta-testing tasks, where for all metrics averages across the testing tasks are reported. To make the setting of multi-task models comparable with single-task models that were trained on meta-testing tasks (requiring a train and test split of each meta-testing task), the evaluation of all models was computed on the test split of the meta-testing tasks, respectively. Single-task baselines (FleXTENET, TARNet, and DragonNet, RA-learner, R-learne, X-learner, and T-learner) were given access to the meta-testing tasks during training. Specifically, model selection was performed on the meta-validation tasks, while the best hyperparameter configuration was used to train 8 repetition models (using 8 random seeds) on the train split each meta-testing task. For the final evaluation, each single-task model that was fit on meta-testing task i was tested on the test split of the same meta-testing task i, and the average metrics were reported across meta- meta-testing tasks.
tasks. Then, the best hyperparameter configuration was
C.1.2 LINCS
Interventions (W): Interventions in the LINCS dataset consist of a single perturbagen (small molecule). For intervention information, we used the molecular embeddings for each perturbagen using the RDKit featurizer The same cell line-perturbagen combinations are tested with different perturbagen dosages and times of exposure. [46].To To maintain consistency in experimental conditions while also ensuring that the dataset is sufficiently large for training a model, we filter for most frequently occurring dosage and time of exposure in the dataset, which are 10pM and 24 hours, respectively. We use data from 10,322 different perturbagens.
Control group. For each perturbagen (at given timepoint and dose), use cell lines which did not receive that intervention as the control group.
Outcomes (Y): We measure gene expression across the top-50 and top-20 landmark differentially expressed genes (DEGs) in the LINCS dataset. Accurately predicting in gene expression these DEGs is most crucial to the drug discovery process.
25

---

Features (X): We use 19,221 features from the Cancer Cell Line Encyclopedia (CCLE) [2 to describe each cell-line, based on historical gene expression values in different lab environment. Our dataset consisted of 99 unique cell lines (after filtering for cell-lines with CCLE features).
cells is that at evaluation time we can observe both Metrics: A key advantage of experiments
(0) and Y for the same cell line X, through multiple experiments on clones of the same cell- line in controlled lab conditions. In the LINCS dataset, (0) is also measured for all cells which received an intervention. Thus, we can directly compute the Precision Estimation of Heterogenous Effects (PEHE) on all treated cells in our meta-testing dataset. PEHE is a standard metric for CATE estimation performance (30), analagous to mean squared enror (MSE).
1 PEHE
Σ TT R:) N
(18)
Training& & Evaluation: For each method, we ran a hyperparameter search with 48 random con- figurations (48 due to running 8 jobs in parallel on 6 servers each) that were drawn uniformly from a pre-defined hyperparameter search space (see Appendix C.4). Methods that can be trained on multiple tasks to then be applied to tasks unseen during training (i.e., CaMI. and its ablations, S-learner w/ meta-learning T-learner w/ meta-learning, SIN) were trained for 12 hours (per run) on the meta-training tasks. Model selection was performed on the meta-validation tasks by minimizing the overall PEHE for the Top-20 most differentially expressed genes (DEGs) across meta-validation tasks. Then, the best hyperparameter configuration was used to fit 8 repetition runs across 8 different random seeds. Each repetition model was then tested on the meta-testing tasks, where for all metrics averages across the testing tasks are reported.
Data augmentation: We augment each batch of data during training to also include treated samples that have their pseudo-outcome labels to 0, and their W set to the zero vector.
C.2 Selecting holdout interventions for meta-validation and meta-testing
C.2.1 Claims.
In the 30.4 million patient insurance claims dataset, each intervention task in meta-train/metta val/meta-testing corresponds to a natural experiment of multiple patients, with some interventions (e.g. commonly prescribed drugs) having millions of associated patients who were prescribed the drug. One challenge is that in this setting, there is overlap in subjects between the natural experiments sampled by CaML, which can lead to data leakage between training and testing. For instance, if patient received Drug (in meta-test) and Drug 2 (meta-train), they would appear in both natural experiments, resulting in data leakage.
We take a conservative approach and exclude all patients who have ever received meta-testing drug in their lifespan from the natural experiments for meta-val/meeta-rrain Similarly, we exclude all patients who received a vela-valio drug from meta- meta-training.
This approach means we must take great care in selecting meta-testing drugs. Specifically, we must trade off between selecting drugs that are important (covering enough patients) while not diminishing the training dataset size. For instance selecting commonly prescribed (e.g. aspirin) for meta-testing would deplete our meta-training dataset by over 50% of patients. Thus we only selected aetza-tes//meta-vvlidatti
drugs which were prescribed to between 1,000,000 and 100K patients our dataset, after filtering for only drugs which known to cause Pancytopenia [42) (using the SIDER database). From this subset of drugs, we randomly selected 10 meta-testing drugs and 2 meta velldaaiio drugs, resulting in a total and 685K patients respectively.
wewestiin/mmtt-validaattt pool of 4.1 million patients
To evaluate on unseen pairs of drugs on the same hold-out test dataset, we additionally created a second pairs testing dataset from the 5 most frequently occurring combinations from the meta- testing dataset. This allowed us to train single model on the same meta-train split and evaluate on both single drug and drug pair interventions without occurrence of data leakage. Designing a larger evaluation of pairs was not possible because while pairs of drugs are commonly prescribed as intervention, each particular pair of drugs is rare event, and accurately evaluating CATE estimation performance (for a rare outcome such as Pancytopenia) requires amassing natural experiment with at least several thousand patients who received the same intervention.
26

---

C.2.2 LINCS.
The goal in selecting holdout interventions for the meta- validation and meta-testing sets was to ensure that they consisted of both cell lines and tasks (small molecules) that had not been seen previously at the time of training (i.e. zero-shot on cell lines and tasks).
Using a random data splitting approach would result in large portions (up to 50%) of the data being unused to comply with the zero-shot requirements on cell lines and tasks. One approach to tackle this was reserve only those tasks in the held-out sets which had been tested This preserved the maximum amount of data but resulted in an average of just 1 cell line per task in the meta-testing and meta- meta-validation sets, which would not be fair the non-zero shot baselines.
the fewest cell lines.
To address these issues, we designed a new data split procedure that exploits the structure of how tasks and cell lines are paired. To do so, We first clustered tasks by the cell lines they are tested on. We then identified a set of 600 drugs that had all been tested on a shared set of roughly 20 cell lines. We divided the cell lines and tasks within this set into the meta-validation and meta-testing set, while enforcing zero-shot constraints on both. This resulted in roughly 10 cell lines per intervention in both the velidaaaiittionn and meta-testing sets, while still maintaining reasonably large size of 11 distinct cell lines and 300 distinct tasks in both sets. All remaining tasks and cell lines were reserved for the training set. (See Table 8
C.3 Understanding CaML's performance
Our comparison to CATE estimators which are restricted to single interventions (Grey, Table 2 shows that a key reason for CaML's strong performance the ability to joinly learn across from many intervention datasets, in order to generalize to unseen intervention.
Additionally, in both the Claims and LINCS settings, we conduct two key ablation studies to further understand the underlying reason for CaML's strong performance results.
In our first ablation experiment (w/o meta- meta-learning), meta-lcarning, instead using the standard empirical risk minimization (ERM) technique (78). This can be seen as a specific implementation of the CaML algorithm (refer to Algorithm I when i: 1 [54] The results of this experiment showed a varying degree of performance deterioration across our primary tests. In the Claims settings, 15%-22% (refer to Table D. while in the LINCS settings, the PEHE performance metric decreased by approximately 0.01 (see Table These results indicate that the absence of meta-learning affects the model's performance, although the impact varies depending on the specific setting. An important detail to consider is that the Claims data experiments dealt with substantially larger datasets, each comprising hundreds of thousands of patients per intervention. This extensive scale of data potentially amplifies the benefits of using meta-learning in the CaML model for the Claims dataset. The larger dataset enables the model to adapt to a given task over larger set of iterations without reusing the same data, thereby enhancing the efficacy of meta-learning.
trained the CaML model without employing
observed decrease in the RATE performance metric by
Our second ablation (w/o RA-learner) assesses the sensitivity of CaML's performance to different pseudo-outeome estimation strategies. A key aspect of CaML is flexibility in choice of any pseudo- outcome estimator to infer CATE, in contrast to prior work which uses specific CATE estimation We find that CaML. performance benefits strongly from flexibility of pseudo- outcome estimator choice. We assess this by using an alternative pseudo-outcome estimator. Firstly, we find that this ablation results in much noisier model training. For instance, the standard deviation in RATE across the 8 random seeds increases by 20x when using the alternative pscudo-outcome estimator in the claims setting. Moreover, the alternative pseudo-outcome estimator typyically worsens performance. decreasing RATE by up to 6% in the Claims setting, , and increasing PEHE by 20%-211% % in the LINCS setting (Table We note that this ablation performs slightly better at the 0.99 threshold, which may be a result of the high variance in this ablation. Specific choice of alternative pseudo-outcome estimator for this ablation varies by setting. We use the R-learner (55) for Claims as it also achieves strong single task performance (Table 2 grey) on Claims data. However, R-lerrner restricted to single-dimensional outcomes, and thus for LINCS (in which outcomes are 50 and 20 dimensional), we use the PW-learne instead [16].
strategies [25 35
27

---

C.4 Hyperparameter space
C.4.1 Claims dataset hyperparameter space
We list the hyperparameter search spaces for the medical claims dataset in the following tables. Table 9 represents the search space for CaML. The SIN baseline consists of two stages, Stage and Stage For the Stage 1 model, we searched the identical hyperparameter search space as for (Table For Stage 2, we used the hyperparameters displayed in Table TO] The search space for the Traphi'T baseline is displayed in Table For the S-learner and T-learner w/ meta-learning baselines, we use the same hyperparameter space as for CaML (Table 9 with the only major difference that the these baselines predicts the outcome instead of For all deep learning-bass
methods, we employed batch size of 8, 9,192, except for GraphITE, where we were restricted to using batch size of 512 due to larger memory requirements. Single-task neural network baselines (ElexT TARNet, and DragonNett) are shown in Tables 13, and [44 respectively. For the remaining baselines, i.e., the model-agnostic CATE estimators, the (shared) hyperparameter search space is shown in Table [5] Finally, applied LI regularization to the encoder layer of the customizable neural network models (that were not reused as external packages), i.e., SIN learner, GraphITE, T-learnr w/ meta-learning, and S-Iearner w/ meta-leaminn, and CaML..
C.4.2 LINCS hyperparameter space
We list the hyperparameter search spaces for LINCS in the following tables. CaMLis shown in Table [6] SIN Stage 1 used the same search space CaML (Table [6] The search space of SIN Stage 2 is shown in Table 17] S learner and T-learner w/ meta-learning used the same search space as CaML. The search space of GraphITE is shown in Table 18 All methods that were applied to LINCS used a batch size of 20.
C.5 More details on intervention information
Here we give more details about the intervention information used for the medical claims dataset. In order to perform zero-shot generalization, we acquired information about a specific intervention through the use of pretrained embeddings. We generated these embeddings on the Precision Medicine Knowledge Graph [8] that contains drug nodes as well as 9 other node types. We extracted embeddings for 7957 drugs from the knowledge graph.
To extract rich neighborhood information from the knowledge graph we used Stargraph [49 which is coarse-to fo-nie representation learning algorithm. StarGraph generates subgraph for each node by sampling from its neighbor nodes (all nodes in the one-hop neighborhood) and anchor nodes (a preselected subset of nodes appearing in the multihop neighborhood). In our case the anchor nodes were the 2% of graph nodes with the highest degree. For the scoring function we used the augmented version of TripleRE (90) presented in the StarGraph article
(49
We performed a hyperparameter optimization to compare different models and determine the one we used to calculate our final embeddings (see Table C.5 The hyperparameter search was random with the objective of minimizing the loss function used in training on held out data. The search range for each of the parameters is displayed in C.5 Since certain parameters did not seem influence the final score as much we decided to use them as constants and focus on optimizing the hyperparameters set 20 and 0.1 in the augmented
in the table. Therefore the number of sampled anchors TripleRE function, the values matching those seen in Stargraph [48]
Our final embeddings were 256-dimensional, the learning rate was 2e-4, the drop-ratio was 5e-3. We used the self-adversariall negative sampling loss with 7 = and we sampled 4 neighbor nodes for each subgraph.
To additionally evaluate the quality of the embeddings we assigned classes drug combinations and then scored them using multiple clustering metrics. We were interested to see if embeddings of drug combinations used for similar purposes would be embedded closer together than other drug combinations. For the class label of single drugs we used the first level of the Anatomical Therapeutic Chemical (ATC) code, which represents one the 14 anatomical or pharmacological groups. Since certain medications have more than one ATC code, we took the mode of all labels for specific drug. For multiple drugs we combined all distinct first level values and took the mode of them as the label. We used the Silhouette metric, Calinski Harabasz index and Davies Bouldin index as
28

---

well as the average classification accuracy over 10 runs of training a random forest classifier on random sample of 80% of the dataset and evaluating on the remaining 20%. Out of all tested embeddings the hyperparameter optimized StarGraph embeddings performed best (exceeding 93% in the classification accuracy metric).
C.6 Pseudo-outcome estimation
In our experiments, we estimate pseudo-outcomes 7 for given intervention w using the RA- learner [16]
f = W(Y - -x)(X) + (1 (x) -X)
(19)
where Pw is an estimate of (.. (X) = Ep X = x, W =
Furthermore, in both settings we only estimate CATE for treated individuals. We focus on treated individuals in the Claims setting because we care about the risk of an adverse event for prescribing a sick patients drugs that may cure their sickness, not the adverse event risk of prescribing healthy patients drugs (which is of less clinical interest). the LINCS setting, we focus on treated cells as for these cell-lines Y is also measured from cloned cell-line under similar laboratory conditions, which allows us to directly estimate CATE prediction performance using the PEHE metrie. As we focus on treated samples, the RA-learner can be simplified to F = Y - ( (X). We estimate (X) using a random forest model in the Claims setting, whereas in the LINCS setting we use the point estimate from the untreated control cell line's gene expression.
C.7 Baselines
Here we provide more details on the baselines used in our experiments.
Trained on test task: These baselines leverage CATE estimators which can only be trained on a single task (typically these are the strongest baselines, when there a large enough dataset for single task). Thus, we train single model for each meta-testing task on its train split, and evaluate performance on its test split. We use a number of strong baselines for CATE estimation developed by prior work including both model-agnostic and end-to-end deep learning approaches: T-learnrr. Specifically, we use the model-agnostic CATE estimators: (44), X-learner [44, RA-learner [16], R-learner (55).
We additionally use the end-to-end deep learning estimators DragonNet [72), TARNet [70]. and
FlexTENet [17. using implementations from [7]. For model-agnostic CATE estimators, we use
random forest models following prior work [14 81
Zero-shot. These baselines use CATE estimators which incorporate intervention information (W) and are capable of multi-task learning. We train these baselines on all meta-training tasks. These baselines have no access to the meta-testing tasks during training. We found in preliminary experiments that in some cases, baseline models trained with vanilla ERM would not even converge. To allow for fair comparison
baselines, we allow for all zero-shot baselines to be trained using Reptile (by training using the same optimization strategy as Algorithm I while allowing for training with ERM by including
in the hyperparameter search space).
Firstly, we use GraphITE (25) and Structured Intervention Networks 335. These are, to the best of our knowledge, the only methods from prior work which are (in principle) capable of zero-shot generalization. We use existing implementations provided by the authors 33
Additionally, implement two strong baselines which estimate CATE by modeling Y(w) and (O).. rather than via pseudo-outcomes These are variants of the S-lemrer and T-learer 44 with meta-learning which use the intervention information as input, rather than one-hot encoded vectors of the different interventions-suchh that they also have zero-shot capability. Specifically, we train MI.s using the same architecture CaML. estimate the response function from observed outcomes:
p(x, w) = Ep Y
X
= x.
W
w
(20)
and estimate CATE by
H. (x) (0 w) f0
(21)
Where w denotes the corresponding intervention information w for an intervention, and 0 denotes a null intervention vector. In the LINCS setting, we represent 0 as vector of zeros, whereas in the
29

---

Claims setting we represent as the mean embedding of all drugs (as the estimand is the increase in adverse event likelihood compared randomly chosen drug). The difference between the T-learner and the S-learner is that the T-learner estimates two models, one for control units and one for treated units, By contrast, the S-learner estimates shared model across all units.
30

---

D Additional Experiments
In general, limited training examples, biases in the training data, will degrade model perfor- mance-ane and the CaMI. algorithm is no exception this regard. For instance, there are too few examples of prior interventions (e.g., only handful of training drugs), then zero-shot generalization may become more challenging. Therefore, it is important to study the robustness of CaML's per- formance to limitations in the training dataset. To this end, we conduct additional experiments in which we downsample the number of training interventions. As expected, we find that: (1) zero-shot capabilities improve the set of unique training interventions increases in size and (2) achieve strong performance on smaller datasets (e.g., runs with 60% and 80%, the interventions achieve similar performance).
still
31

---

Effect of Decreasing Training Intervention Set Size (Claims RATE @ 0.998)
0.34
0.32
0.30
RATE 0.998
0.28
0.26
0.24
0.22
0.01
0.05
0.1
0.2
0.4
0.6
0.8
1.0
Train Set Fraction of Interventions Used
Effect of Decreasing Training Intervention Set Size (LINCS PEHE 20 DEGS)
3.90
PEHE Error
3.85
3.80
3.75
0.2
0.4
0.6
0.8
1.0
Train Set Fraction of Interventions Used
Figure 3: Measuring the robustness of CaML to limitations in the training intervention data. We downsample the number of training interventions and measure CaML's performance, Overall, we find that CaML's zero-shot capabilities improve as the set of unique training interventions increases in size. Nevertheless, CaML still achieves strong performance 60% and 80%, of the interventions achieve similar performance). Results are analogous for other metrics on both datasets. Top: Performance on the Claims dataset at predicting the effect on novel drug on the likelihood of Pancytopenia onset (RATE 0.998). Bottom: Performance on LINCS dataset at predicting the gene expression of the Top 20 and Top 50 most differentially expressed genes (DEGs).
smaller datasets (e.g., runs with
32

---

E Unbiasedness of CATE estimates
Unbiasedness of Pseudo-outcome labels
We show for an example pseudo-outcome label, the RA-learner [16]. that the estimated pseudo- outcome labels are unbiased estimates of the true CATE, i.e.:
E [ x] = T E (1) Y())X =x]
The pseudo-outcome for the RA-learner defined as 7; Y; - (X;) for treated units (W; = 1), and F, (xx Y; for control units (W, 0).
Here, (X)) مr (X) denote unbiased and correctly specified nuisance models for the outcomes Y Y())
and Y (1) respectively. In other words, E [() (=( = IE (0) x] and E'' [Āz (x)]
Ma (z) (1) z²]-
We consider the treated and control units separately. For treated units (W; = 1), we have:
5 = Y - (X,).
Hence, their expectation, conditioned on covariates X can be written as: | = x] = E - Ao (X) = x] = E [ = x|| E (X) x| E X a] -B[(() x] = ><()).
which by applying the consistency assumption (paper Line 98) for treated units is equivelant to:
E Y = z] E (0) = z] = E (1) (()) X = x]| = T(x).
Analogously, we can make the same argument for control units (W = 0). Here, the pseudo computed as:
ने = ix,,) Y,
Hence, we have
E = (x) Y| = x
- x] x),
which under consistency (for control units) equivalent to:
[ x]
E (Y = z) = - x=
= x] = T(x).
Unbiasedness of Model Loss
We consider parametrized CATE estimators so: R Rd + R that take as input intervention information w E RS (e.g.. drug's 's attributes) and individual features E Rd (e.g., patient medical history) return a scalar for the estimated CATE (e.g., the effect of the drug on patient health).
We denote the loss function L with regard to a CATE estimator V and a target as:
L(V,, y) = ( (w, .x) -)²
y)²
As in Theorem I we assume pseudo-outcomes targets 7 during training satisfy i = T + E where 7 is the true (unobserved) CATE and & is an independent zero-mean noise.
Lemma 10. Given two different CATE estimators Lo,, Loo parameterized by and 02:
[ F) < E (I(Vos: #)
E [[(Vor ) S E [((oo.:) r))
R
Proof. We follow a similar argument as Tripuraneni et al. [77].
E I( +) = E ( 1 (w, 2) - ] = E ( i (w, x)) - T + r =ト) })]
E
Vo (w,x) -r)
+ (T 7)² + 2 ( (w,x) - 7) (T F)
( (w, 22) -
27
+ E [T - m)²] + 2 [ @ (w, xx) - -7) (r -+]
E

---

Now we subtract the loss for the two models parameterized by by 01 and 0g:
E L +) - E [ 1 02 F)
=
(I)
E Co (w,xx) - } 27
+ E [ [
2E [ (uo, (w, - - -) (T F)
Vo (w,x) - +) 27
+ E [(= -) m/
+ 21 ( Vog (w, - 7) (r - +)
E
E
6 (w, -r)
+ 2E L (w, x)) - 7) (T +)
02 (u, -r) 21
2 { (oo (w,x) - r) (r +)
E
Expanding out the righthand terms give us:
E L (w, - (
+ 2E Woo (w,x) . -wo (w, - 72 +
E
( 02 (w, x²) -T)
2E /o (w,x) . U/ (w.r) - 72 +
E
1 0: (w, xx) - rT
+ 21E
Lo. (w.x) - Me, (w, x)) - - -E [ E f7 7
27
E
W 02 (w, xx) -T)
21 W 02 (w.xx) - L (w, x)) -
+ EE [r2] BB
L (W, xx) - -) 27
E
2E L (w, T Lo. (w,x) -
1 02 (W, x)) - 7) 27
E
22E Joo (w, x)) -T -Woo (w,z) -7]
Lo (w, -r) -B E (50, (Bo (w, x) -77
21E [ (wo, (w,z)) (r - -+)) (uos (w, (T - -F))]
E
E
(e (w, x²) - 27 E
-7)
(0 (w, x))
(0,,)))-77 -7) 27 + 2E (wo, (w, - Uos (u,, (T - =
E
Vo. (w, xr) -r) 27
E
Vog (w, xr) -7)
2E
[ (wo) (w., - Vi (1c, xr)
-
E
1 01 (wr, -r)
E
V (w,, xr) - 7)
+ 2E Vo, (w..x) -io0 (w. xx)
E [-
Vo, (w, x²) ) 27
2 ( (w, x) -7) )
E
E
+ Vog (w,x) /W (w,x)
E E1-E]
E [ L( [( - E [[
from shich-due- to equality with Equation I the claim follows.
2

---

0.111--0000
0.051
0.11+-C0.00
0.16+ 0.000
0.05
0.06 0.004
0.06 00000
0.0410.003
0.01
0.06++0.014
0.13
0.14
0.16
0.16 0..003
This table extends Table (2 with
0.99
0.001
0.
0.00
0.181-c0.00
0.06=<00.00
0.15+-20.00
122==00.00)
0.07-0.023
0.09:00007
0.08=0.010
0.07.0.000
0.01
0.10+0.023
021
0.22000008
0.26=0.078
0.2610.000 10.004
0.996
0.26
Precision @M
0.00 -0000x0000
0.9 0291 -0000
0.07=<00.00
0.24 0.24+-50.00 0,21
042=<00.0)
0.1011.0036
0.1520.012
0.11
0.14100015
0.01
0.18
0.35
0.35
0.41
0.43 10.008
0.998
Table 4: Performance results for the Claims dataset (predicting pancytopenia onset from drug exposure using patient medica history. standard deviations.
0.004:-00.00
0.361--00000
0.09:
0,54:0.001
0.1210..45
0.18:00013
0.12400018
0
0.01
0,25
0.45:00113
0.45:00100
0.51
0.54 154010112
0.909
(000--0000)
0.311
0.124
0,261-20.00
0.45+<0.00
0. 0.101:0027
0. 14410.111
0.1740.017
0.1010.000
0.0210.002
0.16
0.38100114
0.39
0.4123
0.45 0.45 10.000
0.99
0,004
0.995
0.01 +-00000000
0.36 1.36--000
0.08
190--00001
0.8=<0000 0.001
0.071:0.000
0. 0.12000008
0.12+00011
0.0910000
0.01
0.15+
0.3240013
0.321:0014
0.35#:0.16
0.3810.006
10,00
(t)
0.38
Recall @u
1000cc00000
0.18L <0.00
0,04 <0.001
0.101 -0000000000
0.26 c0.001
0,041:00012
0,0810.0006
0.07+0.009
0.001::001
0.11
0,2240000S
0,224:0007
0.24+:00019
0.27 17.00.005
0.998
0.00 00.00
0.121-c0000
0.02+-c0.00
0.06-1--00000
0.17+-00.00
0.02=00008
0.05-0.003
0.01=0.066
0.07-00000
0.00=0001
0
0.15=0006
0.15=0000
0.16=0.020
0.18 0.00
0.999
<0.00
0.10 10000000000
<0.=<00.001
D.0- -0,000
014==00.01
0.04:0.011
0.05=00000
0.01=000066
0:0310.000 002
0.00
0 0.05 +0.011
0.11
00.050
0.14=0.041
0.13 10.002
0.99
0.00
0.001 200000.000 000
0.26 0221--0002 -0.00000 0.161-0.000
0,044-00000
0.14--0.000
0.234<0.001
100019
0.07110006
0.06+0.008
100000000
0.0011.001
0.09
0,1840007
0.18.10008
0.22+:0.067
0.23 10.003
0.996
(1)
RATE @h
0.00
0.05 <0.001
0.19 119--11000 0,17+-00000
0.30000.00
0.07-00.000
0.12-0011
0.09=0.016
0.12100113
0.0010.001
0.16+00028
0.31
0.31200006
0. 0.36=00006
0.38 L0.007
.998
0.00=<c0.00
0.32 (022-0000
1.06=c0.00
0.4=00.001
0.09:000337
0.12000111
0.10000015
19100024
0.00
0.21+0.022
0.40200112
0.39:00012
0.45=0.058
0.48 10.010
0.999
S-lemeer w/ meta meta- learning
T-karner w/ meta- leamning
CaML- wio meta- metu-learming learning
CaML- wio RA-leamer
CaML (ouns)
Random
E-lamne
X-leamer
R leamer
RA-leamer
DragonNet
TARNet
FletTENet
GraphITE
SIN
3

---

0.00-1 00001-0000000
0.044 0.000
0.01
0.01
0.11++000001
0.00
0.02
0.04
0.05=0.032
0.05
0.04
0.01 10.005
0.15=0.008
0.17=0.010
0.17=0.013
Table 5: Performance results for the medical claims dataset, in which the task is predict the effect of a pair drugs the drug on pancy sanecytopenic occurrence. Mcan and standard deviation between runs is reported. Single-task methods were trained on the meta meta-testing testing tasks (best model underlined). Methods that were capable of training across multiple tasks were trained on meta-tre setta-taainin tasks and applied to previously unseen meta-tesiin testing tasks (best model in bold).CMMI outperforms the strongest baseline that had access to testing tasks on 12 out of 12 metrics, and outperforms all zero-shot baselines. lines. Notably: due to the small sample size for natural experiments with combinations of drugs, the RATE estimation process is very noisy which is reflected in high variability of the measured RATE, Here, the secondary metrics (Recall and Precision) that are not affected, additionally assert the dominance of CaML. over all baselines lines.
0.012: -01=00000000
-000000000
0.00= -0000000000
0.00 <0.000
0.18- 18==<0000
0.00::0.004
0.03:000013
0.04
0.09=:0.05
0.09
0.07
0.01 10.099
0.23=20.17
0.26=00014
0.25
Pracision (1)
0.06
000
=- 001
0.0001
0,00==0.000
0.00 00.001
0.28+<0-A +8=+<00001
0,00-10.010
0,04:00:032
0:03100000
0.18 123
0.18 123
0.1611057
0,01
0.41:00057
0.36+00033
0.40:0004
0.008
1011--200000
0.01
0.08
0.28
0.01 2000014
CO 001
0.00 00.001
0.00==-00000 00.001
0.30==-0.001
0.00 10:0000000
0.05 046
0.02:10.02 027
0. 0.29:10. 180
0. 0.29:10.8 180
0.27:10.09)
0.00.1..00 007
0.49:50.02 070
0.4420000 099
0.51
079
0.00
0.011:-00.00
0. 1.13::c0000
0.01 0.01=<00000 <0.001 0.02+=-0.001
0.041=-0.001
0.37+-20.001
0.01 0.0110009
0.061-0002
O. 0.121:0037
0. 0.15::0.000
0.151
0.13100040
0,03100009
0.45220.019
0.50=:0.028
0.51--0027
0.01 2 <0000
0.00 0.00=<<0.00 <0.001
0,30- M <0.001
0.00
0.0413
0.08200012
0.13=0.044
0.13:0.044
0.11=0.045
0.0210.015
0.38==0.25
0.42+0.024
0.43=0.024
0.11=<0000
Recall @ (1)
0.998
0.0%2 -0.00
0.074-c0.000
0.00 -0.000
0.00 -0.000000000
1119--0000
0.00 1.00003
0,02 102-10015
000310018
0. 0.10=0.000
0.10=:0070
0. 0.10::004
0.00 0.00
10,00
0,28-:0.026
0.28--0.100
0.001-00016
0.0.-00000
0.05+ 1055=<00.001
0.00==c0.00
0.00: 1.00=1-0.000
c0.001
0.10 100+-20.00
0.0011 10.1--00000
0.01 00.011
0.01= 009
0.09=0.055
0.09
0.08:0.024
0.0010.005
0.19:0.019
0.18=00025
0.20=00.015
0.001 1000<<00001
-040000000
0.00+
0.00 c0.001
0110+ c0.001
0.00 1.0003
002 10002
003
00000000
00000000
00300011
100000
011340008
011550100
0.15+10013
0.00
0:00 -000000000
0.05 +00=<00000
0.00 0000=000000
0.011
-11==<20.00
0.00 100:000004
0.02-00:13
0.04::0:122
0.08 008=00055
0.08 000-00052
0.06-00022
0.001000000
0.31-00017
0:24:10014
0:33-10222
(T)
RATTE
10000-c0.001
0.07++-0.00)
-0.01==00.00
-0.01 (.01==c0.00) -0.000
1.26+-c0.00)
0.00
0.03-00000
0,02=0019
0,16=0.118 118
0,16 118
0,1520084
0.00
0,38=0.57
0.33=0.044
0.37=0.044
0.00-2
000011-0.001
0.0011 -0.00
0.10:1-0000 <0.001
0.00+=<0.000
-0.01==00.00
0.28+ -00.000000
.0.01 0..11
000002
0.04 100004
0.02:10.024
0,2 0.27=10.13 173
0.27 173
0:25-10.088
0.00:10.008
+45-+0.070
0.40=+0.100 101
0.47=+0.084
S-leumer w/ metu- Jeaming
T-ecaner w/ meta- Reaanng
CaML- wlo learning
CsML wll RA- Aclacre
Randon
Teleamer
Seleamer
Relearner
RAA teamer
DragonNet
TARNet
Flex TENet
GraphITE
CaML (ours)
SIN

---

Split # of Patients
Allopurinol
Test
815,921
Pregabalin
Test
636,995
Mirtazapine
Test 623,980
Indomethacin
Test
560,380
Colchicine
Test
370,397
Hydralazine
Test 363,070
Hydroxychloroqqunee
Test 324,750
Methotrexate
Test 323,387
Memantine
Test
306,832
Fentanyl
Test 261,000
Etodolac
Val 438,854
Azathioprine
Val 100,000
Table 6: Held-out test and validation drugs for single-drug meta-testing and meta-validation
datasets for our Claims evaluation Table Drugs are unseen (excluded) during training. All drugs known to cause pancytopenia 42
Split # of Patients
Allopurinol Hydralazine
Test 7,859
Methotrexate Hydroxychloroquine
Test 25,716
Pregabalin + Fentanyl
Test 5,424
Indomethacin Colchicine
Test 42,846
Mirtazapine + Memantine
Test 10,215
Table Held-out test pairs of drugs for our meta-testing and meta-validation datasets Appendix
Table 5 Both drugs are unseen (excluded) during training. All drugs are known to cause pancytope-
Split Meta-training 9717
Perturbagens
Cell-Lines
Mean #Cell Lines/Task
77
5.79
Meta-validation Meta-testing
304
11
9.99
301
10.77
Table 8: Composition of the meta-training meta-validation and meta-testing sets for the LINCS dataset. No cell lines or drugs (tasks) were shared across any of the splits.
Hyperparameter
Search range
Num. of layers
[2, 4, 6} {128, 256}
Dim. of hidden layers
Dropout
{0, 0.1}
Learning rate
x 10 1 10 10-3 3 x 10 x 10-4]
Meta learning rate
{1}
Weight decay
(5 10-3)
Reptile k
{1, 10,
LI regularization coefficient
{0, x 10-7,5 10-
Table 9: Hyperparameter search space for CaML (our proposed method) on the medical claims dataset.
5

---

Hyperparameter
Search range
Num. of como layers
{2, (2,4,6)
Num. of covariate layers
{2,
Num. of propensity layers
[2,4,6]
Num. of treatment layers
{2,4.6]
Dim. hidden como layers
{128, 256)
Dim. hidden covariate layers
{128, 256)
Dim. of hidden treatment layers
(128, 2563
Dim. hidden propensity layers
{ [16, 32, 64, 128]
Dropout
{0,0.1}
Learning rate
{3 x 10 10-3,
x 10 3- 3 x 10-4, x 10-4)
Meta learning rate
{1}
Sin Weight decay
{0,5 x 10-3)
Pro Weight decay
{0,5 x 10-3)
GNN Weight decay
{0,5 x 10-3}
Reptile
{1, 10, 50}
L1 regularization coefficient
{0, 10 x 10-3
Table 10: Hyperparameter search space for SIN on the medical claims dataset. The SIN model
consists of two stages, Stage and Stage 2. For the Stage 1 model searched the identical hyperparameter search space as for CaML (Table 9). For Stage 2, we used the hyperparameters shown in this table.
Hyperparameter
Search range
Num. of covariate layers
{2, 4, 6]
Num. of treatment layers
[2, 6]
Dim. of hidden treatment layers
{128, 256)
Dim. of hidden covariate layers
{128, 256)
Dropout
{0,0.1]
Independence regularization coefficient
[0, 0.,0, 0.1,
Learning rate
{3 x 10-3 1 x 10-3,3 10 1 10-1
Meta learning rate
{1}
Weight decay
{5 x 10
Reptile k
10, 50)
regularization coefficient
(0, 1 x 10-7,5 10-7
Table 11: Hyperparameter search space for GraphITE on the medical claims dataset.
Hyperparameter
Search range
Num. of out layers
{1, 4}
Num. of r layers
[2,4, 6}
Num. units p out
{32, 64, 128, 256)
Num. units out
{32, 64, 128, 256)
Num. units s Num. units pr
{32, 64, 128, 256)
{32, 64, 128, 256)
Weight decay
(5 10-3)
Orthogonal penalty
{0, 1 x 10-5 5, 1 10-3.0.13
Private out
(True, False }
Learning rate
{3 x 10-3 3, 1 10 10-3,3 x 10 x 10-4]
Table 12: Hyperparameter search space for FlexTENet on the medical claims dataset.

---

Hyperparameter
Search range
Num. of out layers
[1, 4]
Num. of layers
{2, [2,4,6]
Num. units out
{128, 256)
Weight decay
{5 10-3)
Penalty disc
{0, 1 10-3)
Learning rate
{3 x 10 1 x 10-3,3 10-4,1 10-4)
Table 13: Hyperparameter search space for TARNet on the medical claims dataset.
Hyperparameter
Search range
Num. of out layers
[1,2,4]
Num. of r layers Num. units r
{2,4,6]
[128, 256]
Num. units out
{128, 256]
Weight decay
{5 x 10-3}
Learning rate
{3 10 10-3,03000 x 10 10-4, 1 10-4)
Table 14: Hyperparameter search space for DragonNet on the medical claims dataset.
Hyperparameter
Search range
Num. of estimators
[50, 250]
Max depth
[10, 50]
Min sample split
[2, 8]
Criterion regress
(squared error, absolute error)
Criterion binary
{gini, entropy]
Max features
(sqrt, log2, auto)
Table 15: Hyperparameter search space for model-agnostic CATE estimators, i.c., R-learner, X-
learner, RA-learner, and T-learner on the medical claims dataset.
Hyperparameter
Search range
Num. of layers
{2, 4, 6} {512, 1024]
Dim. of hidden layers
Dropout
(..0.13
Learning rate
{3 10 1 x 10 1,03,,3
10-4, 10-4) {0.1, 0.5,
3,3
Meta learning rate
Weight decay
(0.1)
Reptile
[1.2.3}
L.1 regularization coefficient
{0,
10 10-7)
Table 16: Hyperparameter search space for CaML (our proposed method) on the LINCS dataset.

---

Hyperparameter
Search range
Num. of como layers
{2, 4, 6]
Num. of covariates layers
{2, 4]
Num. of propensity layers
[2, 4,6
Num. of treatment layers
{2, 22]
Dim. output
{128, 256)
Dim. of hidden treatment layers
{ [128,
Dim. hidden covariate layers
128, 256)
Dim. of hidden como layers
{ [128, 256}
Dim. of hidden propensity layers
{ {16, 32, 64, 128}
Model dim.
{512, 1024]
Dropout
{0,0.1}
Learning rate
{3 10
x 10 10-00.30 10 10
10 10
Meta learning rate
{0.1, 0.,
Sin weight decay
{0.0, 0.005)
Pro weight decay
(0.0, 0.005)
GNN weight decay
{0.0. 0.005)
Weight decay
{0.1}
Reptile
[1,2,3}
LI regularization coefficient
{0. 1 x 10 7,5 x 10
Table 17: Hyperparameter search space for the SIN baseline on the LINCS dataset.
Hyperparameter
Search range
Num. of covariate layers
{2,4, 6)
Num. of treatment layers
(2, 4,
Num. of layers
{2, 6]
Dim. of hidden covariate layers
{128, 256}
Independence regularization coefficient
{0, 0.010 0.1, 1.0)
Dropout
(0,0.1}
Model dim.
{512, 1024]
Learning rate
(3 x 10-3
x 10 11003,33 3 10-4,1 10-1)
Meta learning rate
{0.1, 0.9.
Weight decay
{0.1)
Reptile k
[1,2,3}
LI regularization coefficient
{0, 1 x 10-7,5x 55 10-7}
Table 18: Hyperparameter search space for the GraphITE baseline on the LINCS dataset.
Hyperparameter
Search range
Dropout
[le-4,le-1]
Learning rate
[le-5,1e-3]
Weight decay
[le-5,1e-2]
Adversarial temperature
[1,10]
Gamma
[0,30]
Num. of sampled neighbors
0-10
Dim. of hidden layers
( 64, 256, 512)
Table 19: The hyperparameter optimization search ranges used in the selection of the optimal model for the generation of knowledge graph node embeddings that would serve intervention information for the medical claims dataset.


Reviews:
[{"review_id": "a6tXj46SVP", "reviewer": "Reviewer_uxCL", "summary": "In this paper, the authors propose a new Meta Learning algorithm called CaML - to predict the personalized effects of a novel intervention. The idea is neat, and its applicability seems reasonable given the empirical results.", "questions": "- Are there any other guarantees you can make on the out of sample error without a theorem like Lemma 1 above?\n- What covariates can be used?", "limitations": "The authors address some limitations - I have outlined others in the \"Weaknesses\" section above", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper's algorithm seems to perform well for novel settings.\n- The PAC-like theorem is a nice addition\n\nBased on the response I have upgraded my score to Accept", "weaknesses": "- I'd like to see a theorem like the one in https://arxiv.org/pdf/2112.07602.pdf (Lemma 1 or Lemma 2), which would validate unbiasedness in different settings (assuming that pre-treatment covariates are used)\n- A more complex set of theorems (which requires extensions that are more complex) would be ones where the distribution of the actual meta-datasets are drifting. I think even then strong results can be obtained, which would allow the use of CaML even in settings where the novel interventions are drawn from different distributions. These might not be feasible for this paper but maybe worth addressing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> (...) The idea is neat, and its applicability seems reasonable given the empirical results (...) The paper's algorithm seems to perform well for novel settings (...) The PAC-like theorem is a nice addition\n\nWe thank reviewer uxCL for their positive feedback on our method, the strength of our empirical results, and our zero-shot generalization bound. We prove the additional theorem requested by the reviewer below, which validates the unbiasedness of our training objective. \n\n\nIf the reviewer has any further concerns, please let us know! If not, we would appreciate it if the reviewer would be willing to consider raising their score.\n\n> 1. I'd like to see a theorem (...) which would validate unbiasedness in different settings (assuming that pre-treatment covariates are used)\n\nWe thank the reviewer for this insightful comment. We have updated our paper to prove a similar lemma as Lemma 1 from Tripuraneni et al. [1], which we agree strengthens our arguments. We have also cited Tripuraneni et al. as our proof follows a similar structure.\n\nFor each task (corresponding to a single intervention) we obtain pseudo-outcome labels (denoted as $\\tilde{\\tau}$), which are unbiased but noisy estimates of CATE (denoted as $\\tau$). Specifically, we have $\\mathbb{E}[\\tilde{\\tau} | X=x] = \\tau(x)$ for a sample with pre-treatment covariates $x$. An extensive literature develops methods to obtain these pseudo-outcomes [2, 3, 4, 5]. CaML flexibly wraps any of these existing methods to obtain labels for each task.\n\nWe now prove that by leveraging these unbiased (yet noisy) pseudo-outcome labels, we can formulate an unbiased (yet noisy) optimization objective to train and evaluate an estimator for the true CATE. In other words, if estimator A outperforms estimator B on the noisy pseudo-outcome loss (in expectation), then it also holds that estimator A will outperform estimator B at predicting the true CATE (in expectation).\n\nMore formally, we consider parametrized CATE estimators $\\Psi_{\\theta} \\colon \\mathbb{R}^e \\times \\mathbb{R}^d \\to \\mathbb{R}$ that take as input intervention information $w \\in \\mathbb{R}^e$ (e.g., a drug's attributes) and individual features $x \\in \\mathbb{R}^d$ (e.g., patient medical history) to return a scalar for the estimated CATE (e.g., the effect of the drug on patient health).\n\nWe denote the loss function $L$ with regard to a CATE estimator $\\Psi$ and a target $y$ as\n\n$L(\\Psi,y) = \\left(\\Psi(w, x) - y\\right)^2$\n\nAs in Theorem 1 in our paper, we assume pseudo-outcome targets $\\tilde{\\tau}$ during training satisfy $\\tilde{\\tau}=\\tau+\\xi$ where $\\tau$ is the true (unobserved) CATE and $\\xi$ is an independent zero-mean noise.\n\n**Lemma 10**: Given two different CATE estimators $\\hat{\\Psi_{\\theta_{1}}}$, $\\hat{\\Psi_{\\theta_{2}}}$, parameterized by $\\theta_{1}$ and $\\theta_{2}$:\n\n$\\mathbb{E}[L(\\hat{\\Psi_{\\theta_{1}}}, \\tilde{\\tau})] \\leq \\mathbb{E}[L(\\hat{\\Psi_{\\theta_{2}}}, \\tilde{\\tau})] \\implies\n\\mathbb{E}[L(\\hat{\\Psi_{\\theta_{1}}}, \\tau)] \\leq \\mathbb{E}[L(\\hat{\\Psi_{\\theta_{2}}}, \\tau)]$\n\nThe proof is provided in the PDF shared with the AC on OpenReview (as external links in reviewer responses are disallowed by NeurIPS).\n\n> Are there any other guarantees you can make on the out of sample error without a theorem like Lemma 1 above? (...)  A more complex set of theorems (...) would be ones where the distribution of the actual meta-datasets are drifting. (...) These might not be feasible for this paper but maybe worth addressing.  (...)  \n\nWe thank the reviewer for highlighting these important questions for theoretical inquiry. As we are the first paper to extensively explore whether zero-shot CATE estimation is possible in real-world settings, we focused primarily on its empirical aspects; we develop the CaML framework and demonstrate its strong performance at generalizing to novel interventions across two real-world datasets.\n\nWe recognize the exciting theoretical questions our work raises, and in future research we will indeed delve deeper into addressing these exact questions. To study the scenario where testing interventions are drawn from a different distribution from the training interventions, we need additional assumptions on the nature of the domain shift to provide non-trivial OOD generalization guarantees. In the simplest case where we have linear models, if the test intervention is in the span of training interventions, we can prove that CATE for the test intervention is learnable. Alternatively, in the case where the test intervention distribution has sufficient overlap with training distributions, we can use generalization bounds like the ones in Ben-David et al. [6] or Zhang et al. [7] to bound the test error using training error plus a term bounding the divergence between two distributions. Thank you for your valuable feedback, which will shape our considerations for subsequent extensions of our paper, which we are actively investigating,\n\n> 3. What covariates can be used.   \n\nConsistent with the CATE estimation literature, covariates must be pre-treatment. For example, in our Claims evaluation we use patient medical history. We will clarify this in our revised paper. \n\n**References**  \n\n[1] Tripuraneni, et al., 2021. Meta-Analysis of Randomized Experiments with Applications to Heavy-Tailed Response Data.  \n[2] Künzel, et al., 2019. Metalearners for estimating heterogeneous treatment effects using machine learning.    \n[3] Kennedy, et al., 2020. Towards optimal doubly robust estimation of heterogeneous causal effects.  \n[4] Wager, et al., 2018. Estimation and inference of heterogeneous treatment effects using random forests.   \n[5] Curth, et al., 2021. Nonparametric estimation of heterogeneous treatment effects: From theory to learning algorithms.  \n[6] Ben-David, et al., 2010. A theory of learning from different domains.   \n[7] Zhang, et al., 2019. Bridging theory and algorithm for domain adaptation. "}, {"review_id": "ON5DxNwejJ", "reviewer": "Reviewer_qej5", "summary": "The paper presents a novel framework for zero-shot causal learning, which is the problem of predicting the personalized effects of a novel intervention that has no historical data. The framework, called CaML (Causal Meta-learning), formulates the prediction of each intervention's effect as a meta-learning task, and trains a single meta-model across thousands of tasks, each constructed by sampling an intervention. The paper demonstrates the effectiveness of CaML on two real-world datasets in large-scale medical claims and cell-line perturbations, and shows that CaML can outperform existing methods that are either trained on the test interventions or capable of zero-shot generalization. The paper also provides a theoretical analysis of CaML's zero-shot generalization bound. The paper contributes to the field of causal inference by unlocking zero-shot capability for many recently developed CATE estimation methods that were previously restricted to single interventions.\n", "questions": "Above weaknesses part", "limitations": "Above weaknesses part", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper presents an original approach to addressing the challenging problem of zero-shot causal learning, which has important applications in various domains including personalized medicine, public policy, and online marketing. The paper's framework is comprehensive and rigorous, and its effectiveness is demonstrated through experiments on two real-world datasets in large-scale medical claims and cell-line perturbations. Additionally, the paper unlocks the zero-shot capability for many recently developed CATE estimation methods, which were previously restricted to single interventions, and provides a promising direction for future research in causal inference.", "weaknesses": "While the paper provides a thorough evaluation of its framework using metrics such as RATE and PEHE, future research should explore additional evaluation methods and metrics that can provide a more comprehensive understanding of the performance of zero-shot causal learning methods. Additionally, the paper's section on societal impacts could benefit from a more detailed discussion of the potential risks and limitations of the proposed approach. For instance, the risk of underrepresentation of underserved communities in the training data could lead to biased results, which could ultimately have negative societal impacts. Therefore, future research should address these limitations and consider approaches to mitigate potential risks associated with the proposed framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> (...) The paper presents an original approach to addressing the challenging problem of zero-shot causal learning, which has important applications in various domains including personalized medicine, public policy, and online marketing. The paper's framework is comprehensive and rigorous, and its effectiveness is demonstrated through experiments on two real-world datasets in large-scale medical claims and cell-line perturbations. Additionally, the paper unlocks the zero-shot capability for many recently developed CATE estimation methods, which were previously restricted to single interventions, and provides a promising direction for future research in causal inference.  \n\nWe thank the reviewer for their overwhelmingly positive feedback on the CaML framework, our experimental analyses, theoretical results, and the broader impacts of zero-shot CATE estimation.\n\n> 1. While the paper provides a thorough evaluation of its framework using metrics such as RATE and PEHE, future research should explore additional evaluation methods and metrics that can provide a more comprehensive understanding of the performance of zero-shot causal learning methods.\n\nWe agree that an important direction for future work is to explore additional evaluation metrics. In this paper, we focus on PEHE and RATE as they are prominent metrics of CATE error (PEHE being prominent in the seminal literature [1, 2, 3], and RATE in more recent work [4, 5, 6]). We have extended our discussion to stress the importance of developing and exploring further metrics for CATE estimation, including measures of calibration and bias (a point we further address below in our discussion of the risks of CATE models). \n\n**References** \n\n[1] Hill, Jennifer L. \"Bayesian nonparametric modeling for causal inference.\" Journal of Computational and Graphical Statistics 20.1 (2011): 217-240.  \n[2] Shalit, Uri, Fredrik D. Johansson, and David Sontag. \"Estimating individual treatment effect: generalization bounds and algorithms.\" International conference on machine learning. PMLR, 2017.  \n[3] Künzel, Sören R., et al. \"Metalearners for estimating heterogeneous treatment effects using machine learning.\" Proceedings of the national academy of sciences 116.10 (2019): 4156-4165.  \n[4] Offer-Westort, Molly, Leah R. Rosenzweig, and Susan Athey. \"Battling the CoronavirusInfodemic'Among Social Media Users in Africa.\" arXiv preprint arXiv:2212.13638 (2022).  \n[5] Metz-Peeters, Maike. The Effects of Mandatory Speed Limits on Crash Frequency-A Causal Machine Learning Approach. No. 982. Ruhr Economic Papers, 2023.  \n[6] Xu, Yizhe, et al. \"Principled estimation and evaluation of treatment effect heterogeneity: A case study application to dabigatran for patients with atrial fibrillation.\" Journal of Biomedical Informatics (2023): 104420.  \n\n> 2. Additionally, the paper's section on societal impacts could benefit from a more detailed discussion of the potential risks and limitations of the proposed approach. For instance, the risk of underrepresentation of underserved communities in the training data could lead to biased results, which could ultimately have negative societal impacts. Therefore, future research should address these limitations and consider approaches to mitigate potential risks associated with the proposed framework.\n\nIndeed, it is of utmost importance to quantify, understand, and mitigate the effects of personalized causal models on underserved communities—and more broadly, to understand and address the risks of deploying CATE estimators in practice. \n\nRegarding equitable and fair CATE estimation, two promising directions are process-oriented and outcome-oriented approaches [1]. A process-oriented approach would quantify model error (e.g. calibration, false positive rates) for underserved groups to understand whether they may be disparately impacted by the deployment of CATE models [2, 3]. An outcome-oriented approach would quantify the expected impact of the deployment of such models on utility of each demographic subgroup [4, 5], and tailor resulting decision policies to increase the utility of the underserved population. Both are important areas which we will investigate in our follow-up work.  \n\nRegarding the broader risks of deploying CATE models, there is a potential for overreliance on CATE models, leading to decreased human oversight and decision-making. While CATE estimators can provide valuable insights into treatment effects, they should not replace clinical judgment and expertise. Relying solely on model predictions without considering individual patient circumstances and medical expertise could lead to inappropriate or ineffective treatments. Additionally, the deployment of CATE models could raise privacy concerns. These models typically require access to individual patient data to estimate personalized treatment effects accurately. Ensuring the privacy and security of this sensitive information is crucial to avoid potential data breaches or unauthorized access, which could harm patients and erode public trust in healthcare systems.\n\nIn our camera-ready, we will include a significantly extended discussion on these key points. \n\n**References** \n\n[1] Corbett-Davies, Sam, et al. \"The measure and mismeasure of fairness.\" arXiv preprint arXiv:1808.00023 (2023).      \n[2] Obermeyer, Ziad, et al. \"Dissecting racial bias in an algorithm used to manage the health of populations.\" Science 366.6464 (2019): 447-453.  \n[3] Seyyed-Kalantari, Laleh, et al. \"Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations.\" Nature medicine 27.12 (2021): 2176-2182.  \n[4] Nilforoshan, Hamed, et al. \"Causal conceptions of fairness and their consequences.\" International Conference on Machine Learning. PMLR, 2022.  \n[5] Liu, Lydia T., et al. \"Delayed impact of fair machine learning.\" International Conference on Machine Learning. PMLR, 2018.  "}, {"review_id": "mfZwt1nik0", "reviewer": "Reviewer_o8uy", "summary": "The paper proposes a zero short cate estimation algorithm. Building natural experiments sub datasets and estimating pseudo outcomes the paper attempts at estimating the cate of unseen interventions", "questions": "- if the authors could address the latter point in the weakness section ", "limitations": "- adequate discussion of limitations and impact ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- interesting premise \n- interesting and novel method\n- adequate review of related works\n- good experimentation and discussion \n", "weaknesses": "- unclear why the pseudo outcome target \\tau is unbiased \n- unclear how the algorithm would fair if the training data was biased in the prevalence of specific interventions . For example if the available data only contains a small subset of possible interventions it is possible that the generalisation of the model is not adequate to span the entire set of possible interventions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The paper proposes a zero short cate estimation algorithm. Building natural experiments sub datasets and estimating pseudo outcomes the paper attempts at estimating the cate of unseen interventions\n> - interesting premise\n> - interesting and novel method\n> - adequate review of related works\n> - good experimentation and discussion\n\nWe thank the reviewer for their overwhelmingly positive feedback on our problem formulation, methodology, experimental evaluation, and discussion. \n\n> 1. unclear why the pseudo outcome target $\\tau$ is unbiased\n\nThank you for pointing out that this could be more clear. As we note in Line 170, here unbiased means that the expected value of the pseudo outcome target $\\tilde{\\tau}$ is equal to $\\tau$, i.e., $E[\\tilde{\\tau} |X= x] = \\tau(x)$. \n\nWe will revise our paper to more explicitly refer to the prior work which develops these pseudo outcome targets and analyzes their theoretical properties, under standard causal inference assumptions of unconfoundedness, consistency, and overlap (Lines 97-100) [1, 2, 3, 4]. While each pseudo-outcome estimation method is unique, performance varies by problem setting, and hence CaML flexibly wraps any pseudo-outcome label. \n\nPrior work shows that pseudo-outcomes are unbiased, when they are constructed for a given sample using correctly specified nuisance models trained on an independent split of the data (e.g., via cross-fitting or treated/control split). To further clarify this point, we revise our paper to include an illustrative example for the RA-learner [1] and show that its pseudo-outcome labels are unbiased CATE estimates, i.e.:\n\\begin{align*}\n\\mathbb{E}\\left[\\tilde{\\tau} | X = x\\right] = \\tau(x) = \\mathbb{E}\\left[Y(1) - Y(0)|X=x\\right]\n\\end{align*}\n\nThe pseudo-outcome $\\tilde{\\tau}$ for the RA-learner is defined as $\n    \\tilde{\\tau}_i = Y_i - \\hat{\\mu}_0(X_i)$\nfor treated units ($W_i=1$), and \n$\n    \\tilde{\\tau}_i = \\hat{\\mu}_1(X_i) - Y_i\n$\nfor control units ($W_i=0$). Here, $\\hat{\\mu}_0(X), \\hat{\\mu}_1(X)$ denote unbiased and correctly specified nuisance models for the outcomes $Y(0)$ and $Y(1)$ respectively. In other words, $\\mathbb{E}\\left[\\hat{\\mu}_0(x)\\right] = \\mu_0(x) = \\mathbb{E}\\left[Y(0) | X = x\\right]$ and $\\mathbb{E}\\left[\\hat{\\mu}_1(x)\\right] = \\mu_1(x) = \\mathbb{E}\\left[Y(1) | X = x\\right]$.\n\nWe consider the treated and control units separately. For treated units ($W_i=1$), we have: \n\\begin{align*}\n    \\tilde{\\tau}_i = Y_i - \\hat{\\mu}_0(X_i).\n\\end{align*}\n\nHence, their expectation, conditioned on covariates $X$ can be written as:\n\n\\begin{align*}\n    \\mathbb{E}\\left[\\tilde{\\tau} | X=x\\right] = \\mathbb{E}\\left[Y - \\hat{\\mu}_0(X) | X=x \\right] = \\mathbb{E}\\left[Y|X=x\\right] - \\mathbb{E}\\left[\\hat{\\mu}_0(X) |X=x\\right]\n\\end{align*}\n\\begin{align*}\n = \\mathbb{E}\\left[Y|X=x\\right] - \\mathbb{E}\\left[Y(0)|X=x\\right] = \\tau(x),\n\\end{align*}\n\nwhich by applying the consistency assumption (paper Line 98) for treated units is equivelant to: \n\n\\begin{align*}\n     \\mathbb{E}\\left[Y(1)|X=x\\right] - \\mathbb{E}\\left[Y(0)|X=x\\right] = \\mathbb{E}\\left[Y(1) - Y(0)|X=x\\right] = \\tau(x).\n\\end{align*}\n\nAnalogously, we can make the same argument for control units ($W_i=0$). Here, the pseudo-outcome is computed as: \n\\begin{align*}\n    \\tilde{\\tau}_i = \\hat{\\mu}_1(X_i) - Y_i.\n\\end{align*}\n\nHence, we have \n\\begin{align*} \\mathbb{E}\\left[\\tilde{\\tau} | X=x\\right] = \\mathbb{E}\\left[\\hat{\\mu}_1(X) - Y | X=x \\right] = \\mathbb{E}\\left[\\hat{\\mu}_1(X)|X=x \\right]- \\mathbb{E}\\left[Y|X=x\\right],\n\\end{align*}\n\nwhich under consistency (for control units) is equivalent to:\n\n\\begin{align*}\n     \\mathbb{E}\\left[Y(1)|X=x\\right] - \\mathbb{E}\\left[Y(0)|X=x\\right] = \\mathbb{E}\\left[Y(1) - Y(0)|X=x\\right] = \\tau(x).\n\\end{align*}\n\n**References**\n\n[1] Curth, Alicia, and Mihaela van der Schaar. \"Nonparametric estimation of heterogeneous treatment effects: From theory to learning algorithms.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2021.  \n[2] Kennedy, Edward H. \"Towards optimal doubly robust estimation of heterogeneous causal effects.\" arXiv preprint arXiv:2004.14497 (2020).  \n[3] Künzel, Sören R., et al. \"Metalearners for estimating heterogeneous treatment effects using machine learning.\" Proceedings of the national academy of sciences 116.10 (2019): 4156-4165.  \n[4] Nie, Xinkun, and Stefan Wager. \"Quasi-oracle estimation of heterogeneous treatment effects.\" Biometrika 108.2 (2021): 299-319.  \n\n> 2. unclear how the algorithm would fair if the training data was biased in the prevalence of specific interventions . For example if the available data only contains a small subset of possible interventions...\n\nWe agree that this point should be further clarified, and will revise our paper to do so.\n\nIn general, limited training examples, or biases in the training data, will degrade model performance—and the CaML algorithm is no exception in this regard. For instance, if there are too few examples of prior interventions (e.g. only a handful of training drugs), then zero-shot generalization may become more challenging. Therefore, we agree that it is important to study the robustness of CaML’s performance to limitations in the training dataset.\n\nTo clarify this point, we have implemented additional experiments in which we downsample the number of training interventions. In the **PDF attached to our global response**, we present two result figures for our cell-line perturbations evaluation. Experiments for the (considerably larger) medical claims dataset are still running and will be included in the camera-ready. \n\nAs expected, we find that: (1) zero-shot capabilities improve as the set of unique training interventions increases in size and (2) we can still achieve strong performance on smaller datasets (e.g. runs with 60% and 80%, of the interventions achieve similar performance). We will include these new experiments in our revised paper. \n\n"}, {"review_id": "WCinLSB5I4", "reviewer": "Reviewer_fcYE", "summary": "In this paper, authors have proposed causal meta-learning framework that trains a single model across several intervations and estimate CATE for novel interventions at inference time which were not present at the training time. They frame CATE estimation for each intervention as a separate meta-learning task. The proposed framework is evaulated on two real world large-scale datasets as medical claims and cell-line perturbations, which show the effectiveness of the proposed framework.", "questions": "- What is the ratio of treated and untreated while preparing meta-datasets?\n- CaML comes in different variations, like CaML - w/o RA-learner, etc -- if you could expalin all those at one place with other baselines then that would be easy to follow.\n- Why very narrow range is considered for u in [0.999,0.998,0.995,0.99]?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- A new meta-learning framework, called CaML, is proposed to estimate treatment effect for novel interventions by considering each intervention as a separate task. This is simple but interesting formulation.\n- CaML is evaluated on two real world large-scale datasets as medical claims and cell-line perturbations, which show the effectiveness of the proposed framework\n- Theoretical results are provided to bound the prediction error (I have not verified these results).\n- Code is provided with the paper (I have not checked it).", "weaknesses": "- The idea seemed simple and interesting but I found it hard to follow, especially the results. \n\nWhy are the treatment effect estimates first calculated and used as kind of lables, and why not direclty learn the model? The selected CATE learner (RA-Learner) can create biased estimates.\n\nThere are typos which also make some sentences hand to interpret. Please revise accordingly.\n\nIn standard ML, use of validation dataset makes sense but what does validation tasks/interventions mean?\n\nWhat do you mean by 'Given m outcomes of interest'? Does that mean the interventions have multiple outcomes discrete/continous?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback on our problem formulation, meta-learning framework, real-world experimental evaluation, and theoretical results. \nWe also appreciate the clarifying questions below, which have helped us to revise the manuscript to be more clear. If the reviewer has any further concerns, please let us know! If not, we would appreciate it if the reviewer would be willing to consider raising their score.\n\n> 1. The idea seemed simple and interesting but I found it hard to follow, especially the results.\n\nWe thank the reviewer for this feedback and thoroughly revise the presentation for our camera-ready to clarify each of the points below. \n\n> 2. Why are the treatment effect estimates first calculated and used as kind of lables, and why not direclty learn the model? The selected CATE learner (RA-Learner) can create biased estimates.\n\n**TL;DR:** It is not possible to directly model CATE because it is unobserved. While there is an alternative to the pseudo-outcomes approach (i.e. modeling potential outcomes and then subtracting them), it has been widely shown to suffer from estimation bias (Künzel et al. 2019; Nie and Wager 2021). We discuss these points in Lines 90-93, Lines 133-138. We further empirically show via two baselines (S-learner and T-learners w/meta-learning; Tables 2 and 3) that this approach significantly underperforms compared to CaML.\n\n In our revision, we will clarify and elaborate on the following points:   \n- CATE estimation by modeling potential outcomes, and why it underperforms compared to the pseudo-outcome approach (due to regularization bias)\n- Why pseudo-outcome targets are unbiased, using an illustrative example of the RA-learner (see response to Reviewer o8uy’s point 1)\n- That CaML flexibly wraps *any* pseudo-outcome estimator and is not in any way restricted to the RA-learner\n\n*We have prepared a more detailed response for the question above, but cannot post it until the discussion period due to OpenReview character limitations.*\n\n> 3. There are typos which also make some sentences hand to interpret. Please revise accordingly.\n\nThank you for pointing this out, and we apologize for the typos present in the initial submission. We have further thoroughly proofread the paper and made the necessary corrections to ensure clarity. \n\n> 4. What does validation tasks/interventions mean?\n\nIn the CaML meta-learning framework, each task corresponds to CATE estimation for a single intervention (e.g., a single drug). Validation tasks contain interventions which were excluded from training, and allow us to validate our model’s ability to perform zero-shot CATE estimation on novel interventions. A validation task contains a natural experiment of many samples (e.g. patients) who received the unseen intervention, and many control samples who did not receive the intervention. All samples included in the validation task are excluded from the training. \n\nIn our revised paper, we will elaborate on this point to make it more clear in the zero-shot evaluation subsection (Lines 236-244).\n\n> 5. What do you mean by 'Given m outcomes of interest'? \n\nIn many settings, we may be interested in how an intervention will affect multiple outcomes. For example, in our LINCS evaluation, a single perturbation can affect many different genes. Our CATE estimation task thus corresponds to predicting the effect of a perturbation on the top 50 most differentially expressed genes $(Y_{1}, \\dots ,Y_{50})$. \n\nWe will include this concrete example on Line 182 to ensure that this point is clear.\n \n> 6. What is the ratio of treated and untreated while preparing meta-datasets?\n\nWe use all available data, and hence this varies by the specific intervention and evaluation dataset. For instance, in our medical claims data provided to us by an insurance company, the number of patients that received each of the the held-out test drugs ranges from 261K to 815K patients, and the held-out control group is a sample of 1 million patients. Hence the fraction of treated patients for meta-testing tasks ranges from 20.7% to 44.9% depending on the test intervention. This class imbalance is an important reason for why we chose to use the RA-learner, as it is robust to class imbalance, as long as there is sufficient data in both classes to estimate the necessary nuisance models. \n\nWe will include a supplementary table containing the treated vs. control ratios for all the tasks in our two real-world datasets.\n\n> 7. CaML comes in different variations, like CaML - w/o RA-learner, etc -- if you could explain all those at one place with other baselines then that would be easy to follow.\n\nWhile we had previously described these details in our supplements (Lines 968-1000), we will now present them in one place of the main paper.\n\n> 8. Why very narrow range is considered for $u$ in [0.999,0.998,0.995,0.99]?\n\nOur medical claims evaluation is tailored to the real-world deployment of CaML, in which we want to predict the effect of a drug on the likelihood of a patient developing the life-threatening side effect pancytopenia. Pancytopenia is a very rare event occurring in less than 0.3% of the patients in our dataset. Hence we choose this narrow range of u to isolate the small subset of high-risk patients from the vast majority of patients for whom there is no risk of pancytopenia onset. Notably, many approved drugs show life-threatening side effects (like pancytopenia) only very rarely. Hence, when considering a deployment scenario, it generally makes sense to consider narrow thresholds in order to focus on the small subset of patients at risk of a rare but serious side effect. In other settings, lower RATE thresholds may become more interesting: e.g., when considering treatments for which the majority of recipients have some treatment effect, such as surge pricing in online marketing.\n\nWe elaborate on this rationale for choice of $u$ in our revision. \n\n"}]

Author rebuttal:
We thank the reviewers for their valuable feedback, recommendations for improvements, and generally positive reviews of our work. Overall, it appears that the reviewers found the problem setting of zero-shot CATE estimation important, well-motivated, and applicable (Reviewers qej5, o8uy, uxCL), noted the novelty of our algorithm and its strong empirical performance in our experiments (Reviewers fcYE, o8uy, qej5, uxCL), and appreciated our theoretical contribution of a finite-sample bound on the zero-shot error (Reviewers uxCL, qej5, fcYE). 


The reviewers raise several questions, which we appreciate and answer in our individual responses. To address Reviewer uxCL's request for additional theoretical guarantees, we prove a new lemma (see **note below** for proof) which validates that by wrapping unbiased pseudo-outcome labels from prior work, we can formulate an unbiased optimization objective to train and evaluate an estimator for the true CATE. We were additionally inspired by Reviewer o8uy's question concerning zero-shot generalization under limited training data, and conducted new experiments which show how CaML’s performance varies when the number of training interventions is limited (see **PDF attached** to this response). 

These insightful comments and suggestions have helped us to enhance the clarity and strength of our manuscript. Thank you!

***

**Note**: Per NeurIPS OpenReview policy, we can't include text (besides figure captions) in the attached PDF. To adhere to this, we've uploaded the lemma proof separately and shared an anonymous link with the AC, following NeurIPS instructions for code requests (copied below). Please inform us during the discussion period if there are any issues in accessing the proof, and if an alternative sharing method is required.

> If you were asked by the reviewers to provide code, please send an anonymized link to the AC in a separate comment (make sure the code itself and all related files and file names are also completely anonymized).


Meta-review: This paper proposes a meta-learning framework to estimate causal effects for novel treatments (zero-shot). The method treats each intervention+data-specific task to predict pseudo outcomes or treatment effects, and a meta-learner that samples an intervention to predict the pseudo outcomes. Theoretical results in the form of a generalization bound on zero-shot learner are presented. Experiments demonstrate utility in two tasks.

Guaranteeing good zero-shot performance for causal effect estimation is a highly challenging task. Positivity assumptions are required for any causal effect estimation. Therefore in my reading of the paper some subtle inherent smoothness assumptions might have been made that are a necessity to allow such extrapolation to novel interventions. It may be that the meta-learning process learns to estimate this structure easily compared to other vanilla methods. However, a nuanced discussion of this does not appear in the manuscript. The authors demonstrate the utility of their method in for predicting drug side effects and gene expression tasks. It is unclear how much the method generalizes beyond these tasks. I encourage the authors to include a discussion on this in the final version of the paper. 

All reviewers have acknowledged the utility and novelty of the contribution. Authors have successfully addressed questions around potential deterioration when fewer training interventions are available that may jeopardize generalization. Authors provide additional empirical evaluation to test this deterioration in their experiments.

Considering the utility of the empirical contributions, I recommend an accept.
Decision: Accept (spotlight)