SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We note that  $s$ -values can be used to create sensitivity plots. Under various distribution shifts, the parameter can attain a range of values. More concretely, for different choices of  $E$  and an upper bound on the distribution shift  $c \in \mathbb{R}$  we define upper and lower bounds for parameter values as follows:
$$
\theta_ {\text {u p p e r - b o u n d}} = \sup  \theta (P) \text {s u c h t h a t}
$$
$$
\begin{array}{l} P \in \mathcal {P}: P (\cdot | E = e) = P _ {0} (\cdot | E = e) \text {f o r a l l} e \in \mathcal {E} \text {a n d} D _ {K L} (P \| P _ {0}) \leq c \\ 1 = \inf  \theta (P) \text {s u c h t h a t} \end{array} \tag {9}
$$
$$
P \in \mathcal {P}: P (\cdot | E = e) = P _ {0} (\cdot | E = e) \text {f o r a l l} e \in \mathcal {E} \text {a n d} D _ {K L} (P \| P _ {0}) \leq c.
$$
In experiments, we plot estimated versions of these upper and lower bounds across  $c$  for different choices of the variable  $E$ . Note that if  $\theta(P_0) > 0$ , then  $s(\theta, P_0)$  is the minimum  $c$  for which  $\theta_{\mathrm{lower - bound}} = 0$ .
5.1 National supported work demonstration data (NSW)
Here, we analyze the stability of the average treatment effect estimator in the presence of covariate shift using the NSW dataset [29], which consists of  $n = 722$  participants randomly assigned to a treatment or control group (variable  $A$ ) in an employment program field experiment conducted between January 1976 and July 1977. Covariates  $X$  include 'age', 'education', 'black', 'hispanic', 'married', 'nodegree', and 're75', where 're75' denotes pre-intervention earnings in 1975. The outcome variable is 're78', corresponding to post-intervention earnings in 1978. We apply augmented inverse probability weighting (AIPW) using causal forests [50] to estimate the average treatment effect, resulting in an estimate of 820 with a standard deviation of 492. We evaluate the performance of the two-stage transfer learning procedure presented in Section 4.
Distributional stability of average treatment effect. We investigate the stability of the average treatment effect estimator under distributional changes. Specifically, we examine how  $\mathbb{E}_P[\tau(X)]$  changes when there is a shift in the underlying distribution  $P$  of each predictor separately. We measure the distributional stability of  $\mathbb{E}_P[\tau(X)]$  while keeping the conditional distribution of the other variables given the predictor constant. The study uses the NSW dataset [29], where the outcome variable is 're78', and the covariates  $X$  include 'age', 'education', 'black', 'hispanic', 'married', 'nodegree', and 're75', where 're75' denotes pre-intervention earnings in 1975. We estimate the average treatment effect using augmented inverse probability weighting (AIPW) implemented with causal forests [50], resulting in an estimate of 820 with a standard deviation of 492. Our findings show that the  $s$ -values of average treatment effect conditional on 'age', 'education', 'black', 'hispanic', and 're75' are non-zero. We also note that the average treatment effect is unstable with respect to changes in the marginal distribution of 'age', 'education', and 're75' indicating that the average treatment effect can change its sign with a shift in the marginal distribution of these covariates ( $s_X > 0.85$ ). We present the directional  $s$ -values in Table 1.
Table 1: S-values for NSW data set
Feature	Age	Education	Black	Hispanic	Married	Nodegree	Re75
Directional s-value	0.97	0.91	0.52	0.54	0	0	0.96
Parameter transfer. In this section, we evaluate the two-stage transfer learning procedure described in Section 4 using the DJW subset of the original Lalonde data extracted by Dehejia and Wahba [14] to generate training and test datasets with different distributions. The remaining samples are referred to as DJWC. The DJW subset includes 185 treated and 260 control observations and has additional information on pre-interventional earnings in 1974. We present the pre-intervention characteristics of the two subsets in Appendix J and find that they differ in distribution along several variables, which are statistically significant. This split into training and test datasets allows us to evaluate the transfer learning method in a setting with strong covariate shift. We estimate the average treatment effect separately in the two subsets using causal forest [50]. The estimate on the DJW subset was 1636.7 with a standard deviation of 668.8, while on DJWC, it was -847.5 with a standard deviation of 657.2.
Next, we obtain our training set by adding some proportion  $\alpha$  of randomly chosen samples from the DJW subset to the DJWC subset, where  $\alpha$  takes values in the set 0.05, 0.1, 0.2, 0.3, and use
the remaining samples as the test set. We use the procedure described in Section 4 to obtain a projection of the training distribution that closely approximates the test distribution. We use two transfer methods: full transfer, where we use all the covariates for the transfer, and partial transfer, where we use only the subset of covariates with which the ATE is most unstable, namely 'age', 'education', and 're75'. We display our results in Figure 3, where we find that both transfer methods lead to lower ATE estimation error than the naive procedure. Further, we observe that there is not much gain with the full transfer method that uses all the covariates over the partial transfer method.
Figure 2: The plot shows the estimated minimum and maximum value of the average treatment effect for NSW data achievable when allowing for distribution shift in some covariate (cf. equation (9)).
Figure 3: Parameter transfer on the NSW data set. The transfer procedure described in Section 4 compared to a naive procedure that uses only the training distribution, and a full transfer procedure that uses data on all covariates from the new distribution. The green, red and blue bars represent performance of transfer learning with partial, full new data, and naive method respectively. Error bars show the range of error over 20 repetitions.
5.2 Wine Quality data set
We evaluate the effectiveness of our method using the wine quality dataset from the UCI Machine Learning Repository [13, 18]. The dataset includes subgroups of red and white wines, each with 11 chemical properties used as predictors. The response is a continuous quality assessment measured on a scale of 0 to 10. The dataset consists of 1599 red wines and 4898 white wines. We use all red wines as the training set and randomly select a proportion  $\alpha$  from the white wines, where  $\alpha \in \{0.01, 0.05, 0.1\}$ . The remaining observations are used for testing. We include a small proportion of white wines in the red wine training set to ensure that the shifted distribution is absolutely continuous with respect to the training distribution. This step is necessary to avoid making transfer learning very challenging when datasets deviate from this assumption.
Distributional stability of regression coefficients. We use directional  $s$ -values to assess the distributional stability of ordinary least-squares regression coefficients. We focus our analysis on the predictors "pH" and "density," although similar results can be obtained for other variables. Figure 4 shows estimates of the minimum and maximum achievable value of a regression coefficient given
a specific shift in the marginal distribution of a covariate (as defined in equation (9)). We observe that the coefficient of "pH" is unstable with respect to shifts in "fixed.acidity," "chlorides," "pH," "sulphates," and "alcohol" ( $s_X > 0.85$ ). The coefficient of "density" is unstable with respect to shifts in "volatile.acidity," "total.sulfur.dioxide," "sulphates," and "alcohol" ( $s_X > 0.85$ ). We present the directional  $s$ -values in Tables 2 and 3.
Parameter transfer. We employ Jin and Rothenhausler [28]'s transfer procedure to estimate the parameter under shifted distributions. This transfer procedure combines a re-weighting step with a bias-correction step for semi-parametrically efficient transfer. We compare three different estimators. The first one uses only the training distribution, the second transfers the parameter using the covariates found to be unstable in the previous step, and the third employs all covariates for transfer (full transfer). Figure 5 displays the estimation error of the parameter under the projected and training distributions. Both transfer learning methods have smaller errors than the naive estimator that uses only the training distribution. However, the full transfer method that employs all covariates does not yield much improvement over partial transfer. For  $\alpha = 0.01$ , there is not much enhancement in the coefficient of "density," which might be due to a partial violation of the assumption that the test distribution is absolutely continuous with respect to the training distribution.
Table 2: Directional S-values for wine quality data set (parameter "pH")
f.a = fixed.acidity, v.a = volatile.acidity, c.a = citric.acid, f.so2 = free.sulfur.dioxide, t.so2 = total.sulfure.dioxide, cl = chlorides, so4 = sulphates
Feature	f.a	v.a	c.a	r.s	cl	f.so2	t.so2	density pH	so4	alcohol
s-value	0.86	0.81	0.65	0.83	0.94	0.55	0.8	0.83	0.97	0.97	0.88
Table 3: Directional S-values for wine quality data set (parameter "density")
f.a = fixed.acidity, v.a = volatile.acidity, c.a = citric.acid, f.so2 = free.sulfur.dioxide, t.so2 = total.sulfure.dioxide, cl = chlorides, so4 = sulphates
Feature	f.a	v.a	c.a	r.s	cl	f.so2	t.so2	density pH	so4	alcohol
s-value	0.80	0.94	0.83	0.81	0.78	0.81	0.93	0.84	0.79	0.9	0.98
The generalizability and replicability of statistical findings are crucial in scientific research. However, classical statistical measures only account for uncertainty due to sampling and not other sources of variation, such as distributional shift. Since distributions are expected to vary between settings and locations, it is essential to understand how statistical parameters are affected by such shifts to assess the stability of a finding.
In this work, we propose stability measures to quantify the impact of distributional shifts on statistical parameters at an overall and variable-specific level, respectively, enabling a more detailed evaluation of instability. We expect that stability measures will be used in tandem with transfer learning procedures. Initially, the stability of a conclusion can be assessed with respect to various shifts, and then, once the sensitivities are determined, the data scientist can collect data from target distributions for the most sensitive covariates and update the model accordingly. As an example, researchers are often interested in a causal effect estimate for a new location. For some covariates such as age and education, partial data is available via surveys such as American National Election Study (ANES) or Cooperative Election Study (CES). Additional partial data can be cheaply obtained via Amazon Mechanical Turk. However, some covariates are hard to collect, since they require running a study in the new location. Our numerical results show that the proposed approach can help prioritize data collection.
The proposed approach has several limitations. First, we consider worst-case shifts, which can be somewhat pessimistic for situations where the distribution is expected to change due to random
Shift in marginal distribution of a given covariate with respect to KL divergence
(a)
Figure 5: This figure shows the effectiveness of a two-stage transfer procedure for the wine quality data set. The green, red and blue bars represent performance of transfer learning with partial, full new data, and naive method respectively. Error bars show the range of error over 20 repetitions. Transfer learning outperforms the naive method in almost all cases.
Figure 4: The plot shows the estimated minimum and maximum value of the regression coefficient for win $^e$  quality data set achievable under a distribution shift in one covariate (as defined in equation (9)).
(b)
perturbations in background characteristics. Modelling distribution shift as random is an attractive alternative [41, 26]. Secondly, the Kullback-Leibler divergence only allows for certain types of distribution shift. The shifted distribution might have a different support than the training distribution. In this case, it might be more appropriate to model the changes with the total variation distance or the Wasserstein distance. Thirdly, in this paper we focus on distribution shift in the covariates  $X$ . There is evidence that distributions shift not only in the covariates  $X$ , but also in  $Y|X$  [27, 31]. Developing new tools to better understand distributional shifts in  $Y|X$  is an exciting research direction.
[44] Shimodaira, H. (2000). Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90(2):227-244.
[45] Sinha, A., Namkoong, H., and Duchi, J. (2018). Certifying some distributional robustness with principled adversarial training. In Proceedings of the Sixth International Conference on Learning Representations.
[46] Splawa-Neyman, J., Dabrowska, D. M., and Speed, T. P. (1990). On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9. Statistical Science, 5(4):465 - 472.
[47] Subbaswamy, A., Adams, R., and Saria, S. (2021). Evaluating model robustness to dataset shift. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics.
[48] van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press.
[49] van der Vaart, A. W. (2002). The statistical work of Lucien Le Cam. The Annals of Statistics, 30(3):631-682.
[50] Wager, S. and Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):1228-1242.
[51] Wen, J., Yu, C.-N., and Greiner, R. (2014). Robust learning under uncertain test distributions: Relating covariate shift to model misspecification. In Proceedings of the 31st International Conference on Machine Learning, pages 631-639.
[52] Yu, B. and Kumbier, K. (2020). Veridical data science. Proceedings of the National Academy of Sciences, 117(8):3920-3929.
[53] Zhao, Q., Small, D. S., and Bhattacharya, B. B. (2019). Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrap. Journal of the Royal Statistical Society Series B: Statistical Methodology, 81(4):735-761.