SlowGuess/NeurIPS_2023_OCR huggingface dataset:
obtained by fitting the corresponding candidate models, which can be parametric, semiparametric, or nonparametric machine learning models. Let  $\widehat{m}_{a,k}(X_i) = \sum_{l=1}^{L} \widehat{\Omega}_l \widehat{m}_{a,k}^l(X_i)$  be the predictions with ensemble weights  $\widehat{\Omega}_l$  of candidate outcome models under treatment  $a$  in site  $k$ . We derive the ensemble weights  $\widehat{\Omega}_l$  based on the cumulative predictive risk of candidate models. In particular, we denote the data corresponding to treated and control units as  $D_{k,1}$  and  $D_{k,0}$  respectively, and the sample sizes of  $D_{k,1}$  and  $D_{k,0}$  are denoted as  $n_{k,1}$  and  $n_{k,0}$ . Consider the treated samples first; we randomly partition  $D_{k,1}$  into a training set  $D_{k,1}^{\mathrm{train}}$  of units  $i \in \{1, \dots, n_{k,1}^{\mathrm{train}}\}$  and a validation set  $D_{k,1}^{\mathrm{val}}$  of units  $i \in \{n_{k,1}^{\mathrm{train}} + 1, \dots, n_{k,1}\}$ . Then, each candidate outcome model is fit on  $D_{k,1}^{\mathrm{train}}$  to obtain  $\widehat{m}_{a,n_{k,1}^{\mathrm{train}}}^l$  for  $l \in \mathcal{L}$ .
If the outcome is binary, the ensemble weights are determined by the fitted models' predictive risks evaluated on  $D_{k,1}^{\mathrm{val}}$  according to the Bernoulli likelihood as shown in (7). If the outcome is continuous, the ensemble weights are alternatively determined by the mean squared errors of the fitted models on  $D_{k,1}^{\mathrm{val}}$  [15, 4]. Specifically,
$$
\widehat {\Omega} _ {l} = \left(n _ {k, 1} - n _ {k, 1} ^ {\text {t r a i n}}\right) ^ {- 1} \sum_ {i = n _ {k, 1} ^ {\text {t r a i n}} + 1} ^ {n _ {k, 1}} \widehat {\Omega} _ {l, i} \text {a n d}
$$
$$
\widehat {\Omega} _ {l, i} = \frac {\exp \left[ - \kappa \sum_ {q = n _ {k , 1} ^ {\text {t r a i n}}} ^ {i - 1} + 1 \left\{Y _ {q} - \widehat {m} _ {n _ {k , 1} ^ {\text {t r a i n}}} ^ {l} \left(X _ {q}\right) \right\} ^ {2} \right]}{\sum_ {l ^ {\prime} = 1} ^ {L} \exp \left[ - \kappa \sum_ {q = n _ {k , 1} ^ {\text {t r a i n}}} ^ {i - 1} + 1 \left\{Y _ {q} - \widehat {m} _ {n _ {k , 1} ^ {\text {t r a i n}}} ^ {l ^ {\prime}} \left(X _ {q}\right) \right\} ^ {2} \right]} \quad \text {f o r} n _ {k, 1} ^ {\text {t r a i n}} + 2 \leq i \leq n _ {k, 1}, \tag {20}
$$
where  $\widehat{\Omega}_{l,n_{k,1}^{\mathrm{train}} + 1} = 1 / L$  and the ensemble predictions for the conditional outcomes under treatment is  $\hat{m}_{1,k}(X_i) = \sum_{l = 1}^{L}\widehat{\Omega}_l\hat{m}_{1,k}^l (X_i)$ . The above procedure is then repeated in the control group  $D_{k,0}$  and  $a = 0$  to obtain the ensemble predictions for the conditional outcomes under control.
The tuning parameter  $\kappa$  in (20) can be selected via cross-validation and [4] showed that the performance of model mixture estimators is generally robust across different choices of  $\kappa$ ; they recommended choosing  $\kappa = \max(1, \lfloor \log(L) \rfloor)$ .
C Practical considerations
C.1 Assessing the identification assumptions
In general, our proposed causal identification assumptions in Section 2 are similar to those in existing work in the domain of causal generalizability and transportability [2, 1, 16]. Further, the estimation assumptions that we make for consistency require only one of  $J + K$  models to be correctly specified, which is weaker than the other work, requiring one of the outcome and treatment models to be correctly specified [2, 1].
More specifically, some of the assumptions we proposed can be verified empirically. For instance, Assumptions (A3), (A5), and (A7) can be verified by computing the treatment probabilities for specific patient subgroups of interest. Diagnostic plots can be valuable tools for detecting potential positivity violations in practice. For a more comprehensive discussion and practical guidance, one can refer to [1] and [7]. Furthermore, in specific real-world scenarios, these assumptions can also be justified with domain knowledge. For example, in our case study with the CMS dataset, Assumptions (A3) and (A5) hold because each state has some hospitals performing PCI treatment and there are no baseline covariates that a contraindications for the treatment. Assumption (A7) is also plausible since none of the states deny admission to AMI patients on the basis of any of the baseline covariates.
The Assumptions (A2) and (A6) are not testable, so it is reasonable to explore how the conclusion changes with different degrees of assumption violations. Various sensitivity analysis methods can be applied, for example, [13] and [3]. Discerning the plausibility of assumptions can also be facilitated using directed acyclic graphs; the graphical identification algorithms for assessing transportability can be useful [14]. In the important practical scenario that treatment is randomly assigned (e.g., a multi-site randomized trial), then consistency, exchangeability, and positivity of treatment assignment will hold by design.
When identification assumptions do not hold, we cannot identify the TATE from the observed data. However, we can still derive bounds under relaxed identification requirements. These bounds are useful because they provide a range of plausible values for the TATE. In practice, we suggest conducting sensitivity analyses. For example, we can relax identification assumption (A6) as follows:
$$
\left| E \left\{Y _ {i} (a) \mid V _ {i} = v, R _ {i} = T \right\} - E \left\{Y _ {i} (a) \mid V _ {i} = v, R _ {i} = k \right\} \right| \leq \delta_ {k} \text {a l m o s t s u r e l y f o r a l l} a. \tag {21}
$$
The specific value for  $\delta_{k}$  can be determined with domain knowledge. Investigators can also learn the value of  $\delta_{k}$  that changes results substantially (e.g., flips the sign of the treatment effect).
C.2 The target population can be defined generally
In this work, without loss of generality, we assume that our target population is a population of a specific site, namely the target site population. However, our target population can be generally specified to correspond to different goals. For example, a target population can be some covariate profiles of the patients admitted to the hospital or any covariate profiles of patients admitted to hospitals within a geographic region, the definition we delved into during our case study. As one reviewer suggested, we can even define our target population at the level of an individual's covariate profile, with all other individuals being regarded as source individuals. Within this framework, our method can potentially be extended to estimate individual treatment effects. However, we identify some challenges in extending our method to the individual treatment effect estimation. First, obtaining a reliable estimate of the target individual effect to obtain an initial "anchor estimate" is not a simple task given that we perform estimation and inference for only one unit. Another challenge is the privacy concern. If we operate on an individual level, we must be more careful about possible privacy leakage and membership inference attacks.
C.3 Detecting the model heterogeneity
In practice, it can be challenging to assess whether the same outcome and treatment models should be adopted across sites. Our primary recommendation is to consult with investigators at each site and draw upon their domain-specific knowledge. For example, in the context of hospital quality measurement, a shared model may not be suitable, as certain hospitals might excel in treating specific diagnoses compared to others. Previous research has also indicated that regression models can diverge across sites, even within the same population [10].
In cases where domain-specific expertise is unavailable, we advocate using the model-mixing algorithm by [11] as a useful diagnosis of model heterogeneity across sites. Specifically, we apply the algorithm in each site with a set of candidate outcome and treatment models and examine the associated mixing weight of each model. If substantial variations in mixing weights exist across sites (i.e. certain sites favor one model while others favor a different one), model heterogeneity is likely to occur across sites. In such a case, we recommend the use of the MR- $L_{1}$  approach over the AIPW- $L_{1}$  approach.
In the case study, we consider two outcome models and two treatment models. We specify the first model with the patient's demographic variables and the second model with the patient's demographic and comorbidity variables. We extract the model-mixing weights for the two outcome and treatment models and present them in Figure 5. We observe substantial variations in the assigned mixing weights across different states, especially in terms of the treatment model. For instance, in the State of Alabama (AL), the first model receives a mixing weight of 0.24, while the second model is assigned a weight of 0.76. In contrast, in the State of Nebraska (NE), the second model is favored with a mixing weight of 0.84, whereas the first model is assigned a weight of 0.16. These findings suggest that simply assuming the same treatment model across states may not be suitable.
C.4 Stabilize the density ratio estimation
In practice, some sites can have extreme covariate shifts against the target site, meaning that their patient population can be very different from the target patient population. Under this scenario, correctly specifying the density ratio model can be challenging. To increase the flexibility of the density ratio models, we allow for different basis functions to capture potential nonlinearities. Furthermore, in the implementation, we suggest stabilizing density ratio estimates by trimming extreme values [12].
To further protect against potential density ratio model misspecification, we introduce the adaptive ensemble method, described in Section 4, so that the source site estimates will be down-weighted if they are extremely biased or the variance is too large due to the extreme density ratio weights. Finally, as we have shown in Theorem 2, if we can estimate the  $\tau$  function well enough, our site-specific estimator can still be consistent, even when the density ratio models are completely misspecified.
(a)
Figure 5: Model-mixing weights for the two outcome models and two treatment models across states.
We consider five different estimators: (i) an augmented inverse probability weighted (AIPW) estimator using data from the target site only (Target), (ii) an AIPW estimator that weights each site proportionally to its sample size (SS), (iii) an AIPW estimator that weights each site inverse-proportionally to its variance (IVW), (iv) an AIPW estimator that weights each site with the  $L_{1}$  weights defined in (9) (AIPW-  $L_{1}$ ), and (v) a multiply robust estimator with the  $L_{1}$  weights defined in (9) (MR-  $L_{1}$ ).
We consider a total of five sites and fix the first site as the target site with a relatively small sample size of 300. The source sites have larger sample sizes of  $\{500, 500, 1000, 1000\}$ . We model heterogeneity in the covariate distributions across sites with skewed normal distributions and varying levels of skewness in each site,  $X_{kp} \sim \mathcal{SN}\left(x; \Xi_{kp}, \Omega_{kp}^2, \gamma_{kp}\right)$ , where  $k \in \{1, \dots, 5\}$  indexes each site and  $p \in \{1, \dots, 4\}$  indexes the covariates. Specifically, for any given site  $k \in \mathcal{K}$ , we set the location parameters  $\Xi_{kp} = 0$  and the scale parameter  $\Omega_{kp} = 1$ . Moreover, in the case of the target site, we specifically assign the skewness parameter  $\gamma_{kp}$  a value of zero, denoting a symmetrical distribution. However, for source site  $k \in \mathcal{S}$ , we adopt different values for  $\gamma_{kp}$  based on the sample size. If the sample size is equal to 500, we assign  $\gamma_{kp}$  the value of  $(1/2)^p$ , reflecting a positively skewed distribution. On the other hand, if the sample size is 1000, we assign  $\gamma_{kp}$  the value of  $-(1/2)^p$ , indicating a negatively skewed distribution. Following [8], we generate covariates  $Z_{kp}$  as nonlinear transformation of  $X_{kp}$  such that  $Z_{k1} = \exp(X_{k1}/2)$ ,  $Z_{k2} = X_{k2}/\{1 + \exp(X_{k1})\} + 10$ ,  $Z_{k3} = (X_{k1}X_{k3}/25 + 0.6)^3$  and  $Z_{k4} = (X_{k2} + X_{k4} + 20)^2$ .
To obtain the site-specific TATE estimates, we adopt the model-mixing procedure in [11], partitioning the data within each site into two equally sized training and validation datasets five times. The model-mixing weights for outcome models and treatment models are determined by computing 20 and 7, respectively. We follow the recommendation in [11], setting the tuning parameter  $\kappa$  in 20 as  $\kappa = \max(1, \lfloor \log(L) \rfloor)$ .
To compute the optimal ensemble weights, we solve the adaptive LASSO problem 9. The tuning parameter  $\lambda$  is selected through cross-validation using a grid of values  $\{0,10^{-3},10^{-2},0.1,0.5,1,2,5,10\}$ . To perform cross-validation, the simulated datasets in each
site are split into two equally sized training and validation datasets. To account for cases where source sites have extreme covariate shifts against the target site, we stabilize density ratio estimates by trimming extreme values such that all estimated density ratios are within a range from 0.01 to 100.
D.1 No covariate mismatch
The main text presented simulation results in Table 1 based on 500 simulations, assuming there is covariate mismatch. Here, we investigate the performance of the estimators assuming no covariate mismatch, i.e.  $V_{k} = X_{k} = \{X_{k1},X_{k2},X_{k3},X_{k4}\}$  for target and source sites. We generate potential outcomes as
$$
Y _ {k} (a) = 2 1 0 + X _ {k} \beta_ {x} + Z _ {k} \beta_ {z} + \varepsilon_ {k} \tag {22}
$$
where  $\beta_{x} = \beta_{z} = (27.4, 13.7, 13.7, 13.7)$ . For units in the target site, we generate outcomes with  $X_{k}$  only by setting  $\beta_{z} = 0$ ; for units in the source sites, either  $X_{k}$  or  $Z_{k}$  is used to generate outcomes. If  $\beta_{x} \neq 0$ , then  $\beta_{z} = 0$  and vice versa. Similarly, the treatment is generated as
$$
A _ {k} \sim \operatorname {B e r n o u l l i} \left(\pi_ {k}\right) \quad \pi_ {k} = \operatorname {e x p i t} \left(X _ {k} \alpha_ {x} + Z _ {k} \alpha_ {z}\right) \tag {23}
$$
where  $\alpha_{x} = \alpha_{z} = (-1,0.5, - 0.25, - 0.1)$ . For units in the target site, we generate treatments with  $X_{k}$  only by setting  $\alpha_{z} = 0$ ; for units in the source sites, either  $X_{k}$  or  $Z_{k}$  is used to generate treatments. If  $\alpha_{x}\neq 0$ , then  $\alpha_{z} = 0$  and vice versa. With this data generation scheme, the true ATE is  $\Delta_T = 0$ .
We compare the performance of the five estimators under the following settings ( $C$  denotes the proportion of source sites that correctly specify the outcome and treatment models):
Setting 1 ( $C = 0$ ): outcomes and treatments in source sites are generated with  $Z_{k}$  by setting  $\beta_{x} = 0$  and  $\beta_{z} = (27.4, 13.7, 13.7, 13.7)$ . However, all source sites misspecify both models with  $X_{k}$ . The target site correctly specifies both models with  $X_{k}$ .
Setting 2 ( $C = 1/2$ ): outcomes and treatments are generated with  $X_{k}$  in Sites 2 and Site 4 by setting  $\beta_{x} = (27.4, 13.7, 13.7, 13.7)$  and  $\beta_{z} = 0$ . In Sites 3 and 5, outcomes and treatments are generated with  $Z_{k}$  by setting  $\beta_{x} = 0$  and  $\beta_{z} = (27.4, 13.7, 13.7, 13.7)$ . Thus, the outcome and treatment models are misspecified in Sites 3 and 5 with  $Z_{k}$ , and only half of the source sites correctly specify the models.
Setting 3 ( $C = 1$ ): outcomes and treatments in all source sites are generated with  $X_{k}$  by setting  $\beta_{x} = (27.4, 13.7, 13.7, 13.7)$  and  $\beta_{z} = 0$ , so all source sites correctly specify outcome and treatment models with  $X_{k}$ .
Table 2: Mean absolute error (MAE), root mean squared error (RMSE), coverage (Cov.), and length (Len.) of  $95\%$  CIs based on 500 simulated data sets in three (mis)specification settings.
Target	SS	IVW	AIPW-L1	MR-L1
C=0
MAE	0.109	1.933	0.177	0.110	0.050
RMSE	0.141	1.987	0.219	0.144	0.061
Cov.	0.950	0.998	0.826	0.936	0.960
Len.	0.551	7.035	0.567	0.547	0.234
C=1/2
MAE	0.109	1.111	0.107	0.109	0.050
RMSE	0.141	1.189	0.139	0.140	0.062
Cov.	0.950	1.000	0.942	0.950	0.962
Len.	0.551	6.010	0.540	0.547	0.242
C=1
MAE	0.109	0.036	0.035	0.050	0.049
RMSE	0.141	0.045	0.044	0.064	0.063
Cov.	0.950	0.968	0.956	0.958	0.960
Len.	0.551	0.195	0.191	0.260	0.253
The results in Table 2 indicate that the MR- $L_{1}$  estimator has lower RMSE than the Target estimator when some source sites have correctly specified models ( $C = 1/2$  and  $C = 1$ ). Relative to the MR- $L_{1}$  estimator, the SS and IVW estimators demonstrate larger biases and RMSE, and lower coverage when some source sites have misspecified models ( $C = 0$  and  $C = 1/2$ ). The MR- $L_{1}$  estimator
shows reduced biases and RMSE compared to the AIPW- $L_{1}$  estimator, while maintaining similar coverage; this improvement can be attributed to the inclusion of an additional model that closely resembles the true model. When all source sites correctly specify working models ( $C = 1$ ), the IVW estimator performs optimally with the shortest confidence interval as expected.
To provide further insights into the site-specific estimates, we present simulation results in Figure 6. The results show that in cases where some sites fail to specify their working models properly, AIPW estimators exhibit significant bias, while the multiply robust estimators can accurately recover the true TATE. This can be attributed to the fact that the additional candidate model closely approximates the true underlying models. These findings underscore the enhanced safeguard against model misspecification provided by the multiply robust estimators.
(a) No Site Correctly Specify Models  $(C = 0)$
(b) Site 2 and Site 4 Correctly Specify Models  $(C = 1 / 2)$
(c) All Sites Correctly Specify Models  $(C = 1)$
Figure 6: Results of site-specific estimation based on 500 simulated data sets in three (mis)specification settings.
D.2 Non-comparable source site sample sizes
To study if the proposed method is robust in the case where the number of samples in different sites is not comparable, we generated a new experiment building on the one in Section D.1. Specifically, the target site has a sample size of 300, and the two source sites have sample sizes of 300 (Source Site 2) and 3000 (Source Site 3). We conducted 300 iterations of the experiment due to the computing resource constraints. The results in Table 3 indicate that even for non-comparable target and source site sample sizes, the proposed MR- $L_{1}$  still demonstrates superior performance than other methods.
Table 3: Mean absolute error (MAE), root mean squared error (RMSE), coverage (Cov.), and length (Len.) of  $95\%$  CIs based on 300 simulated data sets in three specification settings.
Target	SS	IVW	AIPW-L1	MR-L1
MAE	0.111	1.938	0.177	0.110	0.070
RMSE	0.143	1.985	0.226	0.142	0.086
Cov.	0.953	1.000	0.880	0.957	0.973
Len.	0.555	15.140	0.642	0.554	0.378
(a) Setting 1: Only the target site (size of 300) correctly specifies both models
Target	SS	IVW	AIPW-L1	MR-L1
MAE	0.111	1.821	0.155	0.111	0.073
RMSE	0.143	1.872	0.200	0.143	0.092
Cov.	0.953	1.000	0.907	0.953	0.947
Len.	0.555	16.100	0.627	0.554	0.387
(b) Setting 2: Target site (size of 300) and Source Site 2 (size of 300) correctly specify both models
Target	SS	IVW	AIPW-L1	MR-L1
MAE	0.111	0.241	0.051	0.109	0.072
RMSE	0.143	0.292	0.065	0.140	0.088
Cov.	0.953	1.000	0.990	0.953	0.973
Len.	0.555	1.716	0.374	0.545	0.382
(c) Setting 3: Target site (size of 300) and Source Site 3 (size of 3000) correctly specify both models
D.3 Alternative data generation process
We designed a data generation process based on Section D.1 such that there is extra information in the source sites which is not helpful for specifying models. We conducted two experiments which enabled us to study the efficiency loss due to specifying more models. Specifically, for the first experiment, we generate the source and target site data according to 14 and 15. For units in the target and source sites, we generate outcomes with  $X_{k}$  only by setting  $\beta_{z} = 0$  and  $\beta_{x} = (27.4, 13.7, 13.7, 13.7)$ . Similarly, for units in the target and source sites, we generate treatments with  $X_{k}$  only by setting  $\alpha_{z} = 0$  and  $\alpha_{x} = (-1, 0.5, -0.25, -0.1)$ . The true ATE is still set to  $\Delta_{T} = 0$ . To examine the estimator's performance under covariate mismatch setting, the target site correctly specifies outcome and treatment models with  $X_{k}$ , while all source sites specify outcome and treatment models with  $X_{k}$  and  $(Z_{k1}, Z_{k2})$ ;  $(Z_{k1}, Z_{k2})$  are two additional dimensions that are not helpful for specifying models. The results in Table 4 indicate that including additional covariates does not lead to a degradation in the performance of the MR-  $L_{1}$  estimator. The MR-  $L_{1}$  estimator demonstrates comparable levels of bias, RMSE, confidence interval coverage, and interval length as observed with the AIPW-  $L_{1}$  estimator. As expected, the MR-  $L_{1}$  estimator outperforms the Target estimator by achieving lower RMSE values, and the IVW estimator attains optimal performance with the shortest confidence interval since all source sites correctly specify the working models.
Table 4: Mean absolute error (MAE), root mean squared error (RMSE), coverage (Cov.), and length (Len.) of  $95\%$  CIs based on 500 simulated data sets in covariate mismatch setting.
Target	SS	IVW	AIPW-L1	MR-L1
MAE	0.109	0.036	0.035	0.050	0.050
RMSE	0.141	0.045	0.044	0.064	0.063
Cov.	0.950	0.968	0.956	0.958	0.954
Len.	0.551	0.195	0.191	0.260	0.251
D.4 Reproducibility and computing resources
All experiments in this study were performed using the statistical programming language R (version 4.2.2). The package sn (version 2.1.0) was employed for generating covariates that follow a skewed-normal distribution within each site. To solve the density ratio estimating equations, we utilized the rootSolve package (version 1.8.2.3). For the estimation of adaptive ensemble weights based on penalized regression of site-specific influence functions, we employed the glmnet package (version 4.1-4).
To enhance computational efficiency, parallel computing packages were employed. Specifically, the packages foreach (version 1.5.2) and doParallel were employed to facilitate the replication of experiments. For the purpose of model-mixing in multiply robust estimation, we employed the parallel package (version 4.2.2). The replication of experiments was carried out using ten CPU cores, while the implementation of the model-mixing algorithm utilized five CPU cores.
If Assumptions (A1) - (A3) hold, the mean counterfactual outcomes in the target population can be identified using data from the target site;
$$
\begin{array}{l} \mu_ {a, T} = E \left\{Y _ {i} (a) \mid R _ {i} = T \right\} \\ = E \left[ E \left\{Y _ {i} (a) \mid V _ {i} = v, R _ {i} = T \right\} \mid R _ {i} = T \right] \\ = E \left[ E \left\{Y _ {i} (a) \mid V _ {i} = v, A _ {i} = a, R _ {i} = T \right\} \mid R _ {i} = T \right] \\ = E \left[ E \left\{Y _ {i} \mid V _ {i} = v, A _ {i} = a, R _ {i} = T \right\} \mid R _ {i} = T \right]. \tag {24} \\ \end{array}
$$
The second line follows the law of total expectation; the third line follows Assumption (A2); the last line follows Assumption (A1).
If Assumptions (A1), (A4) - (A7) hold, the mean counterfactual outcomes in the target population can be identified using data from source sites;