SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To additionally evaluate the quality of the embeddings we assigned classes to drug combinations and then scored them using multiple clustering metrics. We were interested to see if embeddings of drug combinations used for similar purposes would be embedded closer together than other drug combinations. For the class label of single drugs we used the first level of the Anatomical Therapeutic Chemical (ATC) code, which represents one of the 14 anatomical or pharmacological groups. Since certain medications have more than one ATC code, we took the mode of all labels for a specific drug. For multiple drugs we combined all distinct first level values and took the mode of them as the label. We used the Silhouette metric, Calinski Harabasz index and Davies Bouldin index as
well as the average classification accuracy over 10 runs of training a random forest classifier on a random sample of  $80\%$  of the dataset and evaluating on the remaining  $20\%$ . Out of all tested embeddings the hyperparameter optimized StarGraph embeddings performed best (exceeding  $93\%$  in the classification accuracy metric).
C.6 Pseudo-outcome estimation
In our experiments, we estimate pseudo-outcomes  $\tilde{\tau}$  for a given intervention  $w$  using the RA-learner [16]:
$$
\tilde {\tau} = W \left(Y - \hat {\mu} _ {0} (X)\right) + (1 - W) \left(\hat {\mu} _ {1} (X) - Y\right) \tag {19}
$$
where  $\hat{\mu}_w$  is an estimate of  $\mu_w(X) = \mathbb{E}_{\mathcal{P}}\left[Y \mid X = x, W = w\right]$ .
Furthermore, in both settings we only estimate CATE for treated individuals. We focus on treated individuals in the Claims setting because we care about the risk of an adverse event for prescribing a sick patients drugs that may cure their sickness, not the adverse event risk of prescribing healthy patients drugs (which is of less clinical interest). In the LINCS setting, we focus on treated cells as for these cell-lines  $Y(0)$  is also measured from a cloned cell-line under similar laboratory conditions, which allows us to directly estimate CATE prediction performance using the PEHE metric. As we focus on treated samples, the RA-learner can be simplified to  $\tilde{\tau} = Y - \hat{\mu}_0(X)$ . We estimate  $\hat{\mu}_0(X)$  using a random forest model in the Claims setting, whereas in the LINCS setting we use the point estimate from the untreated control cell line's gene expression.
C.7 Baselines
Here we provide more details on the baselines used in our experiments.
Trained on test task: These baselines leverage CATE estimators which can only be trained on a single task (typically these are the strongest baselines, when there is a large enough dataset for a single task). Thus, we train a single model for each meta-testing task on its train split, and evaluate performance on its test split. We use a number of strong baselines for CATE estimation developed by prior work including both model-agnostic and end-to-end deep learning approaches: T-learner. Specifically, we use the model-agnostic CATE estimators: [44], X-learner [44], RA-learner [16], R-learner [55]. We additionally use the end-to-end deep learning estimators DragonNet [72], TARNet [70], and FlexTENet [17], using implementations from [17]. For model-agnostic CATE estimators, we use random forest models following prior work [14, 81].
Zero-shot. These baselines use CATE estimators which incorporate intervention information  $(W)$  and are capable of multi-task learning. We train these baselines on all meta-training tasks. These baselines have no access to the meta-testing tasks during training. We found in preliminary experiments that in some cases, baseline models trained with vanilla ERM would not even converge. To allow for fair comparison to baselines, we allow for all zero-shot baselines to be trained using Reptile (by training using the same optimization strategy as Algorithm 1, while allowing for training with ERM by including  $k = 1$  in the hyperparameter search space).
Firstly, we use GraphITE [25] and Structured Intervention Networks [35]. These are, to the best of our knowledge, the only methods from prior work which are (in principle) capable of zero-shot generalization. We use existing implementations provided by the authors [35].
Additionally, we implement two strong baselines which estimate CATE by modeling  $Y(w)$  and  $Y(0)$ , rather than via pseudo-outcomes. These are variants of the S-learner and T-learner [44] with meta-learning, which use the intervention information as input, rather than one-hot encoded vectors of the different interventions—such that they also have zero-shot capability. Specifically, we train MLPs using the same architecture as CaML to estimate the response function from observed outcomes:
$$
\mu (x, w) = \mathbb {E} _ {\mathcal {P}} \left[ Y \mid X = x, W = w \right] \tag {20}
$$
and estimate CATE by
$$
\hat {\tau} _ {w} (x) = \hat {\mu} (x, w) - \hat {\mu} (x, \mathbf {0}) \tag {21}
$$
Where  $w$  denotes the corresponding intervention information  $w$  for an intervention, and  $\mathbf{0}$  denotes a null intervention vector. In the LINCS setting, we represent  $\mathbf{0}$  as a vector of zeros, whereas in the
Claims setting we represent  $\mathbf{0}$  as the mean embedding of all drugs (as the estimand is the increase in adverse event likelihood compared to a randomly chosen drug). The difference between the T-learner and the S-learner is that the T-learner estimates two models, one for control units and one for treated units. By contrast, the S-learner estimates a shared model across all units.
In general, limited training examples, or biases in the training data, will degrade model performance—and the CaML algorithm is no exception in this regard. For instance, if there are too few examples of prior interventions (e.g., only a handful of training drugs), then zero-shot generalization may become more challenging. Therefore, it is important to study the robustness of CaML's performance to limitations in the training dataset. To this end, we conduct additional experiments in which we downsample the number of training interventions. As expected, we find that: (1) zero-shot capabilities improve as the set of unique training interventions increases in size and (2) we can still achieve strong performance on smaller datasets (e.g., runs with  $60\%$  and  $80\%$ , of the interventions achieve similar performance).
Effect of Decreasing Training Intervention Set Size (Claims - RATE @ 0.998)
Effect of Decreasing Training Intervention Set Size (LINCS - PEHE 20 DEGS)
Figure 3: Measuring the robustness of CaML to limitations in the training intervention data. We downsample the number of training interventions and measure CaML's performance. Overall, we find that CaML's zero-shot capabilities improve as the set of unique training interventions increases in size. Nevertheless, CaML still achieves strong performance on smaller datasets (e.g., runs with  $60\%$  and  $80\%$ , of the interventions achieve similar performance). Results are analogous for other metrics on both datasets. Top: Performance on the Claims dataset at predicting the effect on a novel drug on the likelihood of Pancytopenia onset (RATE @ 0.998). Bottom: Performance on the LINCS dataset at predicting the gene expression of the Top 20 and Top 50 most differentially expressed genes (DEGs).
E Unbiasedness of CATE estimates
Unbiasedness of Pseudo-outcome labels
We show for an example pseudo-outcome label, the RA-learner [16], that the estimated pseudo-outcome labels are unbiased estimates of the true CATE, i.e.:
$$
\mathbb {E} \left[ \tilde {\tau} | X = x \right] = \tau (x) = \mathbb {E} \left[ Y (1) - Y (0) | X = x \right]
$$
The pseudo-outcome  $\tilde{\tau}$  for the RA-learner is defined as  $\tilde{\tau}_i = Y_i - \hat{\mu}_0(X_i)$  for treated units  $(W_{i} = 1)$ , and  $\tilde{\tau}_i = \hat{\mu}_1(X_i) - Y_i$  for control units  $(W_{i} = 0)$ .
Here,  $\hat{\mu}_0(X),\hat{\mu}_1(X)$  denote unbiased and correctly specified nuisance models for the outcomes  $Y(0)$  and  $Y(1)$  respectively. In other words,  $\mathbb{E}[\hat{\mu}_0(x)] = \mu_0(x) = \mathbb{E}[Y(0)|X = x]$  and  $\mathbb{E}[\hat{\mu}_1(x)] = \mu_1(x) = \mathbb{E}[Y(1)|X = x]$ .
We consider the treated and control units separately. For treated units  $(W_{i} = 1)$ , we have:
$$
\tilde {\tau} _ {i} = Y _ {i} - \hat {\mu} _ {0} (X _ {i}).
$$
Hence, their expectation, conditioned on covariates  $X$  can be written as:
$$
\mathbb {E} \left[ \tilde {\tau} | X = x \right] = \mathbb {E} \left[ Y - \hat {\mu} _ {0} (X) | X = x \right] = \mathbb {E} \left[ Y | X = x \right] - \mathbb {E} \left[ \hat {\mu} _ {0} (X) | X = x \right] = \mathbb {E} \left[ Y | X = x \right] - \mathbb {E} \left[ Y (0) | X = x \right] = \tau (x),
$$
which by applying the consistency assumption (paper Line 98) for treated units is equivalent to:
$$
\mathbb {E} \left[ Y (1) | X = x \right] - \mathbb {E} \left[ Y (0) | X = x \right] = \mathbb {E} \left[ Y (1) - Y (0) | X = x \right] = \tau (x).
$$
Analogously, we can make the same argument for control units  $(W = 0)$ . Here, the pseudo-outcome is computed as:
$$
\tilde {\tau} _ {i} = \hat {\mu} _ {1} (X _ {i}) - Y _ {i}.
$$
$$
\mathbb {E} \left[ \tilde {\tau} | X = x \right] = \mathbb {E} \left[ \hat {\mu} _ {1} (X) - Y | X = x \right] = \mathbb {E} \left[ \hat {\mu} _ {1} (X) | X = x \right] - \mathbb {E} \left[ Y | X = x \right],
$$
which under consistency (for control units) is equivalent to:
Unbiasedness of Model Loss
We consider parametrized CATE estimators  $\Psi_{\theta}\colon \mathbb{R}^{e}\times \mathbb{R}^{d}\to \mathbb{R}$  that take as input intervention information  $w\in \mathbb{R}^e$  (e.g., a drug's attributes) and individual features  $x\in \mathbb{R}^d$  (e.g., patient medical history) to return a scalar for the estimated CATE (e.g., the effect of the drug on patient health).
We denote the loss function  $L$  with regard to a CATE estimator  $\Psi$  and a target  $y$  as:
$$
L (\Psi , y) = \left(\Psi (w, x) - y\right) ^ {2}
$$
As in Theorem 1, we assume pseudo-outcomes targets  $\tilde{\tau}$  during training satisfy  $\tilde{\tau} = \tau +\xi$  where  $\tau$  is the true (unobserved) CATE and  $\xi$  is an independent zero-mean noise.
Lemma 10. Given two different CATE estimators  $\hat{\Psi}_{\theta_1}, \hat{\Psi}_{\theta_2}$ , parameterized by  $\theta_1$  and  $\theta_2$ :
$$
\mathbb {E} \left[ L (\hat {\Psi} _ {\theta_ {1}}, \tilde {\tau}) \right] \leq \mathbb {E} \left[ L (\hat {\Psi} _ {\theta_ {2}}, \tilde {\tau}) \right] \Rightarrow \mathbb {E} \left[ L (\hat {\Psi} _ {\theta_ {1}}, \tau) \right] \leq \mathbb {E} \left[ L (\hat {\Psi} _ {\theta_ {2}}, \tau) \right]
$$
Proof. We follow a similar argument as Tripuraneni et al. [77].
$$
\begin{array}{l} \mathbb {E} \left[ L (\hat {\Psi} _ {\theta}, \tilde {\tau}) \right] = \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta} (w, x) - \tilde {\tau}\right) ^ {2} \right] = \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta} (w, x) - \tau + \tau - \tilde {\tau}\right) ^ {2} \right] \\ = \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta} (w, x) - \tau\right) ^ {2} + (\tau - \tilde {\tau}) ^ {2} + 2 \left(\hat {\Psi} _ {\theta} (w, x) - \tau\right) (\tau - \tilde {\tau}) \right] \\ = \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta} (w, x) - \tau\right) ^ {2} \right] + \mathbb {E} \left[ (\tau - \tilde {\tau}) ^ {2} \right] + 2 \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta} (w, x) - \tau\right) (\tau - \tilde {\tau}) \right]. \\ \end{array}
$$
Now we subtract the loss for the two models parameterized by  $\theta_{1}$  and  $\theta_{2}$ :
$$
\begin{array}{l} \mathbb {E} \left[ L \left(\hat {\Psi} _ {\theta_ {1}}, \tilde {\tau}\right) \right] - \mathbb {E} \left[ L \left(\hat {\Psi} _ {\theta_ {2}}, \tilde {\tau}\right) \right] = \tag {1} \\ \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {1}} (w, x) - \tau\right) ^ {2} \right] + \mathbb {E} \left[ (\tau - \tilde {\tau}) ^ {2} \right] + 2 \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {1}} (w, x) - \tau\right) (\tau - \tilde {\tau}) \right] - \\ \left(\mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {2}} (w, x) - \tau\right) ^ {2} \right] + \mathbb {E} \left[ (\tau - \tilde {\tau}) ^ {2} \right] + 2 \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {2}} (w, x) - \tau\right) (\tau - \tilde {\tau}) \right]\right) = \\ \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {1}} (w, x) - \tau\right) ^ {2} \right] + 2 \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {1}} (w, x) - \tau\right) (\tau - \tilde {\tau}) \right] - \\ \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {2}} (w, x) - \tau\right) ^ {2} \right] - 2 \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {2}} (w, x) - \tau\right) (\tau - \tilde {\tau}) \right] = \\ \end{array}
$$
Expanding out the righthand terms give us:
$$
\mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {1}} (w, x) - \tau\right) ^ {2} \right] + 2 \mathbb {E} \left[ \hat {\Psi} _ {\theta_ {1}} (w, x) \cdot \tau - \hat {\Psi} _ {\theta_ {1}} (w, x) \cdot \tilde {\tau} - \tau^ {2} + \tau \cdot \tilde {\tau} \right] -
$$
$$
\mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {2}} (w, x) - \tau\right) ^ {2} \right] - 2 \mathbb {E} \left[ \hat {\Psi} _ {\theta_ {2}} (w, x) \cdot \tau - \hat {\Psi} _ {\theta_ {2}} (w, x) \cdot \tilde {\tau} - \tau^ {2} + \tau \cdot \tilde {\tau} \right] =
$$
$$
\mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {1}} (w, x) - \tau\right) ^ {2} \right] + 2 \mathbb {E} \left[ \hat {\Psi} _ {\theta_ {1}} (w, x) \cdot \tau - \hat {\Psi} _ {\theta_ {1}} (w, x) \cdot \tilde {\tau} \right] - \underbrace {\mathbb {E} [ \tau^ {2} ]} + \underbrace {\mathbb {E} [ \tau \cdot \tilde {\tau} ]} -
$$
$$
\mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {2}} (w, x) - \tau\right) ^ {2} \right] - 2 \mathbb {E} \left[ \hat {\Psi} _ {\theta_ {2}} (w, x) \cdot \tau - \hat {\Psi} _ {\theta_ {2}} (w, x) \cdot \tilde {\tau} \right] + \mathbb {E} \left[ \tau^ {2} \right] - \mathbb {E} [ \tau \cdot \tilde {\tau} ] =
$$
$$
\mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {1}} (w, x) - \tau\right) ^ {2} \right] + 2 \mathbb {E} \left[ \hat {\Psi} _ {\theta_ {1}} (w, x) \cdot \tau - \hat {\Psi} _ {\theta_ {1}} (w, x) \cdot \tilde {\tau} \right] -
$$
$$
\mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {2}} (w, x) - \tau\right) ^ {2} \right] - 2 \mathbb {E} \left[ \hat {\Psi} _ {\theta_ {2}} (w, x) \cdot \tau - \hat {\Psi} _ {\theta_ {2}} (w, x) \cdot \tilde {\tau} \right] =
$$
$$
\mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {1}} (w, x) - \tau\right) ^ {2} \right] - \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {2}} (w, x) - \tau\right) ^ {2} \right] + 2 \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {1}} (w, x) \cdot (\tau - \tilde {\tau})\right) - \left(\hat {\Psi} _ {\theta_ {2}} (w, x) \cdot (\tau - \tilde {\tau})\right) \right] =
$$
$$
\mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {1}} (w, x) - \tau\right) ^ {2} \right] - \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {2}} (w, x) - \tau\right) ^ {2} \right] + 2 \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {1}} (w, x) - \hat {\Psi} _ {\theta_ {2}} (w, x)\right) \cdot \underbrace {(\tau - \tilde {\tau})} _ {- \xi} \right] =
$$
$$
\mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {1}} (w, x) - \tau\right) ^ {2} \right] - \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {2}} (w, x) - \tau\right) ^ {2} \right] + 2 \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {1}} (w, x) - \hat {\Psi} _ {\theta_ {2}} (w, x)\right) \cdot - \xi \right] =
$$
$$
\mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {1}} (w, x) - \tau\right) ^ {2} \right] - \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {2}} (w, x) - \tau\right) ^ {2} \right] + 2 \mathbb {E} \left[ \hat {\Psi} _ {\theta_ {1}} (w, x) - \hat {\Psi} _ {\theta_ {2}} (w, x) \right] \mathbb {E} [ - \xi ] =
$$
$$
\mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {1}} (w, x) - \tau\right) ^ {2} \right] - \mathbb {E} \left[ \left(\hat {\Psi} _ {\theta_ {2}} (w, x) - \tau\right) ^ {2} \right] + 2 \mathbb {E} \left[ \hat {\Psi} _ {\theta_ {1}} (w, x) - \hat {\Psi} _ {\theta_ {2}} (w, x) \right] \underbrace {\mathbb {E} [ - \xi ]} _ {0} =
$$
$$
\mathbb {E} \left[ L (\hat {\Psi} _ {\theta_ {1}}, \tau) \right] - \mathbb {E} \left[ L (\hat {\Psi} _ {\theta_ {2}}, \tau) \right],
$$
from which—due to equality with Equation 1—the claim follows.
RATE @u(↑)	Recall @u(↑)	Precision @u(↑)
0.999	.998	0.995	0.99	0.999	0.998	0.995	0.99	0.999	0.998	0.995	0.99
Random	0.00±<0.001	0.00±<0.001	0.00±<0.001	0.00±<0.001	0.00±<0.001	0.00±<0.001	0.01±<0.001	0.00±<0.001	0.00±<0.001	0.00±<0.001	0.00±<0.001	0.00±<0.001
T-learner	0.32±<0.001	0.26±<0.001	0.16±<0.001	0.10±<0.001	0.12±<0.001	0.18±<0.001	0.26±<0.001	0.31±<0.001	0.36±<0.001	0.29±<0.001	0.18±<0.001	0.11±<0.001
X-learner	0.06±<0.001	0.05±<0.001	0.04±<0.001	0.03±<0.001	0.02±<0.001	0.04±<0.001	0.08±<0.001	0.12±<0.001	0.09±<0.001	0.07±<0.001	0.06±<0.001	0.05±<0.001
R-learner	0.19±<0.001	0.17±<0.001	0.12±<0.001	0.08±<0.001	0.06±<0.001	0.10±<0.001	0.19±<0.001	0.26±<0.001	0.24±<0.001	0.21±<0.001	0.15±<0.001	0.11±<0.001
RA-learner	0.47±0.001	0.37±<0.001	0.23±<0.001	0.14±<0.001	0.17±<0.001	0.26±<0.001	0.38±<0.001	0.45±<0.001	0.54±0.001	0.42±<0.001	0.26±<0.001	0.16±<0.001
DragonNet	0.09±0.037	0.07±0.030	0.05±0.019	0.04±0.013	0.02±0.008	0.04±0.012	0.07±0.020	0.10±0.027	0.12±0.045	0.10±0.036	0.07±0.023	0.05±0.015
TARNet	0.15±0.011	0.12±0.011	0.07±0.006	0.05±0.004	0.05±0.003	0.08±0.006	0.12±0.008	0.14±0.011	0.18±0.013	0.15±0.012	0.09±0.007	0.06±0.004
FlexTENet	0.10±0.015	0.09±0.016	0.06±0.008	0.04±0.006	0.04±0.006	0.07±0.009	0.12±0.011	0.17±0.017	0.12±0.018	0.11±0.019	0.08±0.010	0.06±0.007
GraphITE	0.19±0.024	0.12±0.013	0.05±0.004	0.03±0.002	0.07±0.009	0.08±0.010	0.09±0.008	0.10±0.008	0.23±0.027	0.14±0.015	0.07±0.005	0.04±0.003
SIN	0.00±0.002	0.00±0.001	0.00±0.001	0.00±0.001	0.00±0.001	0.00±0.001	0.01±0.001	0.02±0.002	0.01±0.002	0.01±0.001	0.01±0.001	0.01±0.001
S-learner w/ meta-learning	0.21±0.032	0.16±0.028	0.09±0.020	0.05±0.012	0.08±0.013	0.11±0.022	0.15±0.035	0.16±0.038	0.25±0.034	0.18±0.031	0.10±0.023	0.06±0.014
T-learner w/ meta-learning	0.40±0.012	0.31±0.010	0.18±0.007	0.11±0.004	0.15±0.006	0.22±0.008	0.32±0.013	0.38±0.014	0.45±0.013	0.35±0.011	0.21±0.008	0.13±0.004
CaML - w/o meta-learning	0.39±0.012	0.31±0.006	0.18±0.008	0.11±0.006	0.15±0.005	0.22±0.007	0.32±0.014	0.39±0.021	0.45±0.010	0.35±0.006	0.22±0.008	0.14±0.006
CaML - w/o RA-learner	0.45±0.058	0.36±0.066	0.22±0.067	0.14±0.041	0.16±0.020	0.24±0.019	0.35±0.016	0.41±0.023	0.51±0.076	0.41±0.082	0.26±0.078	0.16±0.048
CaML (ours)	0.48±0.010	0.38±0.007	0.23±0.003	0.13±0.002	0.18±0.004	0.27±0.005	0.38±0.006	0.45±0.010	0.54±0.012	0.43±0.008	0.26±0.004	0.16±0.003
Table 4: Performance results for the Claims dataset (predicting pancytopenia onset from drug exposure using patient medical history. This table extends Table 2 with standard deviations.
Split	# of Patients
Allopurinol	Test	815,921
Pregabalin	Test	636,995
Mirtazapine	Test	623,980
Indomethacin	Test	560,380
Colchicine	Test	370,397
Hydralazine	Test	363,070
Hydroxychloroquine	Test	324,750
Methotrexate	Test	323,387
Memantine	Test	306,832
Fentanyl	Test	261,000
Etodolac	Val	438,854
Azathioprine	Val	100,000
Table 6: Held-out test and validation drugs for our single-drug meta-testing and meta-validation datasets for our Claims evaluation in Table 2. Drugs are unseen (excluded) during training. All drugs are known to cause pancytopenia [42]
Split	# of Patients
Allopurinol + Hydralazine	Test	7,859
Methotrexate + Hydroxychloroquine	Test	25,716
Pregabalin + Fentanyl	Test	5,424
Indomethacin + Colchicine	Test	42,846
Mirtazapine + Memantine	Test	10,215
Table 7: Held-out test pairs of drugs for our meta-testing and meta-validation datasets in Appendix Table 5. Both drugs are unseen (excluded) during training. All drugs are known to cause pancytopenia [42]
Split	# Perturbagens	# Cell-Lines	Mean #Cell Lines/Task
Meta-training	9717	77	5.79
Meta-validation	304	11	9.99
Meta-testing	301	11	10.77
Table 8: Composition of the meta-training, meta-validation and meta-testing sets for the LINCS dataset. No cell lines or drugs (tasks) were shared across any of the splits.
Hyperparameter	Search range
Num. of layers	{2,4,6}
Dim. of hidden layers	{128,256}
Dropout	{0,0.1}
Learning rate	{3 × 10-3,1 × 10-3,3 × 10-4,1 × 10-4}
Meta learning rate	{1}
Weight decay	{5 × 10-3}
Reptile k	{1,10,50}
L1 regularization coefficient	{0,1 × 10-7,5 × 10-7}
Table 9: Hyperparameter search space for CaML (our proposed method) on the medical claims dataset.
Hyperparameter	Search range
Num. of como layers	{2,4,6}
Num. of covariate layers	{2,4,6}
Num. of propensity layers	{2,4,6}
Num. of treatment layers	{2,4,6}
Dim. of hidden como layers	{128,256}
Dim. of hidden covariate layers	{128,256}
Dim. of hidden treatment layers	{128,256}
Dim. of hidden propensity layers	{16,32,64,128}
Dropout	{0,0.1}
Learning rate	{3×10-3,1×10-3,3×10-4,1×10-4}
Meta learning rate	{1}
Sin Weight decay	{0,5×10-3}
Pro Weight decay	{0,5×10-3}
GNN Weight decay	{0,5×10-3}
Reptile k	{1,10,50}
L1 regularization coefficient	{0,1×10-7,5×10-7}
Table 10: Hyperparameter search space for SIN on the medical claims dataset. The SIN model consists of two stages, Stage 1 and Stage 2. For the Stage 1 model we searched the identical hyperparameter search space as for CaML (Table 9). For Stage 2, we used the hyperparameters shown in this table.
Hyperparameter	Search range
Num. of covariate layers	{2,4,6}
Num. of treatment layers	{2,4,6}
Dim. of hidden treatment layers	{128,256}
Dim. of hidden covariate layers	{128,256}
Dropout	{0,0.1}
Independence regularization coefficient	{0,0.01,0.1,1.0}
Learning rate	{3 × 10-3,1 × 10-3,3 × 10-4,1 × 10-4}
Meta learning rate	{1}
Weight decay	{5 × 10-3}
Reptile k	{1,10,50}
L1 regularization coefficient	{0,1 × 10-7,5 × 10-7}
Table 11: Hyperparameter search space for GraphITE on the medical claims dataset.
Hyperparameter	Search range
Num. of out layers	{1,2,4}
Num. of r layers	{2,4,6}
Num. units p out	{32,64,128,256}
Num. units s out	{32,64,128,256}
Num. units s r	{32,64,128,256}
Num. units p r	{32,64,128,256}
Weight decay	{5 × 10-3}
Orthogonal penalty	{0,1 × 10-5,1 × 10-3,0.1}
Private out	{True, False}
Learning rate	{3 × 10-3,1 × 10-3,3 × 10-4,1 × 10-4}
Table 12: Hyperparameter search space for FlexTENet on the medical claims dataset.
Hyperparameter	Search range
Num. of out layers	{1,2,4}
Num. of r layers	{2,4,6}
Num. units out	{128,256}
Weight decay	{5 × 10-3}
Penalty disc	{0,1 × 10-3}
Learning rate	{3 × 10-3,1 × 10-3,3 × 10-4,1 × 10-4}
Table 13: Hyperparameter search space for TARNet on the medical claims dataset.
Hyperparameter	Search range
Num. of out layers	{1,2,4}
Num. of r layers	{2,4,6}
Num. units r	{128,256}
Num. units out	{128,256}
Weight decay	{5 × 10-3}
Learning rate	{3 × 10-3,1 × 10-3,3 × 10-4,1 × 10-4}
Table 14: Hyperparameter search space for DragonNet on the medical claims dataset.
Hyperparameter	Search range
Num. of estimators	[50, 250]
Max depth	[10, 50]
Min sample split	[2, 8]
Criterion regress	{squared error, absolute error}
Criterion binary	{gini, entropy}
Max features	{sqrt, log2, auto}
Table 15: Hyperparameter search space for model-agnostic CATE estimators, i.e., R-learner, X-learner, RA-learner, and T-learner on the medical claims dataset.
Hyperparameter	Search range
Num. of layers	{2,4,6}
Dim. of hidden layers	{512,1024}
Dropout	{0,0.1}
Learning rate	{3 × 10-3,1 × 10-3,3 × 10-4,1 × 10-4}
Meta learning rate	{0.1,0.5,0.9}
Weight decay	{0.1}
Reptile k	{1,2,3}
L1 regularization coefficient	{0,1 × 10-7,5 × 10-7}
Table 16: Hyperparameter search space for CaML (our proposed method) on the LINCS dataset.
Hyperparameter	Search range
Num. of como layers	{2,4,6}
Num. of covariates layers	{2,4,6}
Num. of propensity layers	{2,4,6}
Num. of treatment layers	{2,4,6}
Dim. output	{128,256}
Dim. of hidden treatment layers	{128,256}
Dim. of hidden covariate layers	{128,256}
Dim. of hidden como layers	{128,256}
Dim. of hidden propensity layers	{16,32,64,128}
Model dim.	{512,1024}
Dropout	{0,0.1}
Learning rate	{3×10-3,1×10-3,3×10-4,1×10-4}
Meta learning rate	{0.1,0.5,0.9}
Sin weight decay	{0.0,0.005}
Pro weight decay	{0.0,0.005}
GNN weight decay	{0.0,0.005}
Weight decay	{0.1}
Reptile k	{1,2,3}
L1 regularization coefficient	{0,1×10-7,5×10-7}
Table 17: Hyperparameter search space for the SIN baseline on the LINCS dataset.