The s-value: evaluating stability with respect to distributional shifts
Authors: Suyash Gupta
Dominik Rothenhaeusler
Venue: NeurIPS 2023 poster (2023)
Keywords: Distributional Stability
Distributional Robustness
Distributional Shifts
Generalizability
TL;DR: A new measure to evaluate distributional (in)-stability

Abstract: Common statistical measures of uncertainty such as $p$-values and confidence intervals quantify the uncertainty due to sampling, that is, the uncertainty due to not observing the full population. However, sampling is not the only source of uncertainty. In practice, distributions change between locations and across time. This makes it difficult to gather knowledge that transfers across data sets. We propose a measure of instability that quantifies the distributional instability of a statistical parameter with respect to Kullback-Leibler divergence, that is, the sensitivity of the parameter under general distributional perturbations within a Kullback-Leibler divergence ball. In addition, we quantify the instability of parameters with respect to directional or variable-specific shifts. Measuring instability with respect to directional shifts can be used to detect under which kind of distribution shifts a statistical conclusion might be reversed. We discuss how such knowledge can inform data collection for transfer learning of statistical parameters under shifted distributions. We evaluate the performance of the proposed measure on real data and show that it can elucidate the distributional instability of a parameter with respect to certain shifts and can be used to improve estimation accuracy under shifted distributions.

The s-value: evaluating stability with respect to distributional shifts
Suyash Gupta
Dominik Rothenhäusler Department of Statistics Stanford University Stanford, CA 94305 stominik@stanfortt edu
Department of Statistics
Stanford University Stanford, CA 94305
suyash028@gmail. com
Abstract
Common statistical measures of uncertainty such p-values and confidence inter- vals quantify the uncertainty due to sampling, that is, the uncertainty due to not observing the full population. However, sampling is not the only source of uncer- tainty. practice, distributions change between locations and across time. This makes it difficult to gather knowledge that transfers across data sets. We propose a measure of instability that quantifies the distributional instability of a statistical parameter with respect to Kullback-Leible Leibler divergence, that is, the sensitivity of the parameter under general distributional perturbations within Kullback-Leiih divergence ball. In addition, quantify the instability of parameters with respect to directional or variable-specific shifts. Measuring instability with respect to directional shifts can be used detect under which kind of distribution shifts a statistical conclusion might be reversed. We discuss how such knowledge can inform data collection for transfer learning of statistical parameters under shifted distributions. We evaluate the performance of the proposed measure on real data and show that can elucidate the distributional instability of a parameter with respect to certain shifts and can be used to improve estimation accuracy under shifted distributions.
Introduction
Test data sets collected in different locations at different time points often drawn from different distributions, due to changing circumstances, changes in unmeasured confounders, time shifts in distribution, distributional shifts in covariates [44] [9 [6, 20). This makes it difficult to gather knowledge that transfers across data sets. Statistical estimands such as a regression coefficient or the average treatment effect (ATE) may vary the underlying distribution changes and hence, statistical findings (such as that the treatment effect is positive) may not replicate across data sets (22].
quantifies the stability of an estimate with respect to unobserved confounding. Roughly speaking, this line of work sees stability analysis part of uncertainty quantification. Inspired by this line of work, we aim to bring a similar type of stability analysis a wider range of statistical procedures.
In causal inference, the rapidly growing field of sensitivity analysis 122 40 16 53
In this paper, we propose a measure of instability, called the s-value, to investigate the stability of given statistical parameter with respect shift in the underlying distribution (Figure [ The s-value quantifies the minimum shift in distribution required to tilt the parameter to given value, using Kullback-Leite Leibler divergence. We also investigate the stability of parameters with respect to directional or variable-specinn shifts. The proposed measure can be used as an exploratory tool to identify the kind of distribution shift that could reverse statistical conclusion. We further discuss
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Training distribution, Po
Shifted distribution, P
Parameter, ((Pb) > 0
Parameter, ((P) <
Figure Distribution shift can change the parameter of interest.
how s-values can be used obtain improved estimates of statistical parameters under shifted distribution with limited information about the new distribution.
1.1 Our contribution
We propose measure of instability that quantifies the sensitivity of one-dimensional statistical parameter to changes in the underlying probability distribution. We focus on shifts in distributions that are absolutely continuous with respect the training distribution. Let Po E be the training distribution on the measure space (2. A). where P is the set of probability measures, Z is a random the dinedimensional statistical parameter of interest. We are
element of Z, and 8 P + interested in the minimum amount of shift in distribution that changes the sign of the parameter. To
this end, we define the stability value (s-value) for as
sup exp DDI Po) s.t. ((P) = 0, PEP where DEL is the Kullback-Leibler divergence between P and Po given by
s0, Po)
(1)
/
dP
DKL(P Pb) (20)=/
dP.
dPo
We provide some more discussion on the thought process that led the proposed definition in Appendix, Section A Note that the s-value lies in [0, 1J. with values close to 1 indicating that small shift in distribution may alter the findings, and hence, the finding not distributionally stable. S-values close to 0 indicate that the sign of the parameter is stable under distributional changes. In Section 3 we discuss estimation of s-values for parameters that are linear in the distribution P.
The proposed procedure can be generalized to parameters defined via risk minimization, including parameters generalized linear models (see Appendix, Section D
Considering overall distributional shift does not give information about what kind of distribution shifts the parameter is sensitive Hence, we also quantify the instability of parameters with respect to shifts in the distribution of certain exogenous or endogenous variables F,, assuming that the conditional distribution of the remaining variables given constant. Let be a random variable taking values in the space E. We define the directional or variable-specific s-value as
sup sg PEP:P(|E=e)= E==)EE=ee for eeE
exp - (KL I| Po) s.t. 0(P) = 0.
(2)
where P denotes the set of probability distributions over joint random variable (Z, E). If practitioner discovers that parameter sensitive with respect to changes in the distribution of certain variable E, this knowledge can be used to update the parameter estimate. We discuss how our method can be used to prioritize data collection about the new distribution in Section 4 and use the same to re-estimate the parameter under shifted distribution. The proposed procedure shows promise for the task of prioritizing data collection from the new distribution the experiments.
2 Related work
Quantifying the uncertainty of statistical estimators is a crucial objective in statistics, typically accomplished using classical statistical measures such as p-values and confidence intervals to quantify sampling uncertainty. However, these methods typically rely on strong, potentially unjustified

---

assumptions about fixed underlying distributions, which may lead to false discoveries. To improve reliability and reproducibility in statistical estimation, Yu and Kumbier [52] propose the predictability. computability, and stability (PCS) framework. While they investigate the stability of data results under data and method perturbations, we foeus specifically on evaluating the stability of statistical parameters under distributional shifts.
Model misspecification can result in distributional instability. Buja et al. [6] highlight fundamental issues with model misspecification or non-linearity in linear models. They propose reinterpreting population slopes as statistical functionals of data generating distributions and develop diagnostic tests for detecting model deviations. We introduce measures to illustrate coefficient instability under various distributional shifts for parametric and semi- parameetei estimators.
introduce a novel framework to analyze the stability of decision policies and prediction models under distribution shifts. Central their approach is the notion of stability, characterized as the minimal alteration in the underlying environment required to push system's performance beyond a specified threshold. In contrast, we focus on understanding stability of parameters with respect to shift in distribution.
In machine learning, there is much work on computing which data features contribute to prediction, e.g. using Shapley values [3] [1]. contrast, we are interested in how distributional changes in features (or covariates) lead to parameter changes.
Sensitivity analyses in the causal inference literature aim to investigate the stability of causal estimates with respect to unmeasured confounding [12 40) 16 53 [1]. Our proposal can be seen version of sensitivity analysis for general estimands where we evaluate both the stability of an estimand with respect to the overall shift in distribution and the stability with respect to directional distribution
shifts.
Classical robust statistics 24 addresses robustness against contaminations and outliers using mea- sures like leverage scores and influence functions to construct estimators that are not unduly influenced by such outliers. Influence functions play an important role in this work, since it corresponds to the functional derivative of a parameter with respect to the distribution. Different from classical robust statistics, in our case the perturbation is not a contamination but corresponds to an actual change in the underlying population. Rather than robustifying estimators, our aim is to equip practitioners with tools to detect sources of instability and facilitate the transfer of estimators across different settings.
There has been a resurgence of research addressing the challenges posed by distributional shifts. This research has mostly focused on building distributionally robust estimators where more weight is given the outliers by considering worst-case distributional shifts neighborhood of the training distribution 199 non- non-linear statistical parameters under both overall and variable-specific distributional shifts.
47). In contrast, we quantify the stability of potentially
There is exciting empirical work by Devaux and Egami [1] who build a library of reference stability values based on survey data sets. They recommend thresholds for s-values based on empirical investigations of how data sets change between settings. This work allows for the contextualization of distributional stability values.
Closely related to our work are empirical likelihoods 335 In the empirical likelihood framework, small overall distributional tilts are used to construct p-values and confidence intervals for given parameter. In our work, we use distributional shifts of various strengths to evaluate the (directional) stability of an estimand with respect to distribution shifts.
3 S-value of the mean
In this section, we discuss estimation of the s-value of the mean of -ie-dimenioonal real-valued random variable followed by some examples. Estimation of s-values for more general settings, including parameters defined via risk minimization, is discussed in Appendix, Section D We first focus on the special case of mean estimation as allows us to develop procedures that will be helpful in more general settings.
3

---

3.1 Estimation of the s-value
Consider mne-iimensionl real-valued random variable Po. where Pg E We recall from that the s-value for the mean (p(Po) Er, [Z]) is defined as
s(p, Po) = sup exp( (epp(-Dx((PIIB))) s.t. Ep[2] =0
In words, we are interested in finding the distribution closest to our training distribution Po under which the mean of the random variable is 0. At first sight, s-values might seem difficult to esti- mate since the supremum in equation is taken over the crinite-dimensional space of probability distributions P. However, it turns out that the s-value of the mean can be obtained by solving one-dimensional convex optimization problem.
Theorem 1 (Theorem 5.2, Donsker and Varadhan 17]) Let Z Po be real-valued random variable with mean p (P) = P. [7] and finite moment generating function on R. Then, we have
s(1,, Po) = inf inf [e
Further. if the infimum in 44 is attained at some X R then the infinuum in is attained some probability distribution Q given by
x²=
dQ(z)= Epg l dPo(z) all E R.
We note that Mz(X) = E[e^2] is the moment generating function of Z. Since, Mz(0) = 1, we have s(1s, Pa) [0, 1]. In practice, we only have access to finitely many realizations of the data generating distribution. Let P,, be the empirical distribution of Z itd. Po for i [n]. obtain estimator of the s-value via the plugin estimator
5(,, P.) inf Ep, [e^2] inf
AZ;
(5)
Using classical results for Meestimators (see Chapter of van der Vaart 488 we show that 8(H, Pa) is consistent and asymptotically normal in Appendix C.I
Directional s-values. The previous form of distributional stability might be very conservative. In practice, we do not expect all aspects of distribution to change from setting to setting, To allow for more finengrained evaluation of stability, we also consider directional shifts, which only change certain aspects of the distribution. In the following, we will make this more precise.
Let Po be the joint distribution of the multivariate random variable (Z, F)) where Z takes values in 2 C R and takes values in & C IRP for some positive integer p. E may be an exogenous or endogenous variable. We consider directional shift, i.e. situation where the marginal distribution of E may change while keeping the conditional distribution of Z given E' constant. To be more precise, we seek estimate
(0 Po)
sup PEP:P(|E=<)= Pot (==e) for all egg
exp(-Drr(PP(1B6) s.t. O(P) 0.
(6)
We next show that 8 is a solution to one-dimensional convex optimization problem. The proof of the following result can be found in Appendix [.2]
Theorem 2. Let Po be the joint distribution function of the random variable (Z, E) taking values in Z x & with μ Ep. (2) and finite moment generating finction on R. Then,
[e SE Po) = inf Ep, [BA A.Bel12 12 El].
Further, if the infimum is attained at some X e R then the infimum in 2 is attained at some probability distribution Q given by
Ap 7 F
e)(2,e) Ep.e * [
dPo(z, e) for all (z,e) Z 8.

---

This result allows us to estimate the directional s-value. Let fn(() be estimator of E|Z E]. Then, can define a plug-in estimator by setting
$ inf AL)(E.))
(8)
We prove the consistency of (1, P..) in Appendix, Section C.I In the Appendix, also discuss how to form a de-biased estimator the directional s-value that is asymptotically normal.
Examples
Example 1 (Distribution with positive support). If Z is a random variable that has positive support with probability 1, then s(u, Po) = O, which reflects the fact that for any distribution shift within the KL-divergence ball, will always have positive mean.
Example (Gaussian distribution). I ~ N(, o²), then s(1, Po) =
Thus, in the Gaussian case the stability measure is monotonous transformation of the signal- to-noise ratio. High signal-to-noise ratio yields lower values of s indicating stronger distributional stability.
us now develop some intuition for directional shifts. First, we derive conditions under which the directional stability is zero, that is, conditions under which sn(U, Po) 0.
Example 3 (Directional stability). Ler Ep, [Z > 0. Then,
F (u, Po) inf E (AR [Z] 2-
lim 1211))) 0.
Example 4 (Average treatment effect). Here we consider estimating the causal effect of treatment via the potential outcome framework (46 42V. We have binary treatment random variable A {0. 1}, potential outcomes Y (1) and (0) corresponding to the potential outcome under treatment and control respectively and some covariates X. Under the consistency assumption, we observe (1) (0). One can write the average treatment
if A = and (0) if A 0. i.e AY(1)
effect (ATE) as
T Ex-Px 1EY (0) X| ExxPx (xxxxx((()())) (X)],
where M(a)( (a) X). Hence, we only consider shifts in marginal distribution of covariates X keeping the conditional distribution of other variables given the covariates fixed, we obtain s -x-vuuues as above with Z = (X))()) Poo (X). In practice pany (X) and pro) (X) are often unknown. We can use plug-in in estimators (X) and (X))(X) to form the estimator
n e Aoo
6x(T, Po) = -inf
Consistency of this estimator can be shown with the same technique as Lemma C.3 Appendix C.1
In statistical analysis, risk minimization is often used to define various parameters, including regres- sion coefficients and general M-estimators. However, unlike parameters that can be represented as the mean of a random variable, these parameters lack simple representation as they not linear in the underlying probability distribution. This makes the optimization problem involved in obtaining valuee non-convex. To address this issue, present methods for obtaining s-values for such parameters in Section of the Appendix.
Parameter transfer using s-values
we introduced s-values that measure the distributional stability statistical parameters with respect to various shifts. In this section, we discuss how we can use s-values to guide further data collection. The above problem re-estimating parameters under shifted distribution is related to the transfer learning literature that overlaps with various fields including robust machine learning,
In Section 3
5

---

causal inference, and conformal inference transfer learning.
36 51 3 Here, we discuss how s-values can guide
a parameter is unstable with respect to a shift in marginal distribution of certain covariates, then knowledge about those covariates be used to transfer parameters across distributions. As an example, assume that we have collected some data on a job program in New York. We now want to estimate how efficient this job program would be in Boston. We have not run this job program in Boston yet, so we do not know all covariates of the participants. However, we can find that the efficiency of the job program is likely unstable with respect to changes in the demographics of job seekers in Boston. How can we use this knowledge to estimate the efficiency of the job program in Boston, based on limited data about the population in Boston? In the following, we will discuss this problem in a formal framework. We discuss another important case. Researchers are often interested causal effect estimate for new location. For some covariates such as age and education, partial data is available via surveys such as American National Election Study (ANES) or Cooperative Election Study (CES). Additional partial data can be cheaply obtained via Amazon Mechanical Turk. However, some covariates are hard to collect, since they require running study in the new location. Our numerical results show that the proposed approach can help prioritize data collection. This may drastically reduce the cost compared to running full-scale replication studies.
Assume that we want to estimate a parameter (Paain) for Pahnt + Po. but we only have observa- tions from Po- In addition, we may be able to collect some information about Pahin for example, observations of a subset of variables X5 E Rd For example, one may know the age distribution of job seekers in Boston. Intuitively, we' d like to re-weight Po so that the distribution of age matches the distribution of age in Boston. However, there may be infinitely many choices of weights. These different choices of weights will correspond to different values of A. Thus, in practice, it is crucial to use a form of regularization when finding a re-weighted distribution (Pproj).
We can define P Paroj as the solution of the following optimization problem:
Pproj = min
pr min DKL (Pz(PP)))) such that P' = = (X =),
where DEL is the Kullback-Ieiblee divergence. The objective is similar to Hainmueller 23 where the author proposes entropy balancing to achieve covariate balance hetween treated and control sets for estimating the average treatment effect. This objective can be solved explicitly, leading to covariate shift setting [36). More specifically, a short calculation shows that if the minimum finite, then
dP dPrroj xs) = dPo(z(as Pa
Since data collection can be costly, one would like to prioritize collecting data that is relevant for the transfer learning task. If sx; (0 c, P) = 0 for all c & ((P). then the parameter is constant under shifts in the marginal distribution of Xs. On the other hand, xxx P) z 1, then small changes in the distribution of Xs might induce large change in the parameter 0(-). These heuristics motivate the following approach:
1. Find variables Xs with respect to which the parameter of interest is most sensitive to as determined by directional s-values. Collect observations of X under the shifted distribution.
2. Estimate 0( Pproj).
There are several existing approaches to deal with part 2. In particular, it is possible to estimate 0( PPoi proj for a large range of estimands 0(+), with asymptotically normal and efficient estimators 38 32 28 We can leverage these existing estimators in workflow that we have statistical guarantees for all steps in this pipeline.
We will investigate the empirical performance of this two-stage approach in Section S
5
Experiments
In this section, we consider real-world data to illustrate the effectiveness of the proposed methods in elucidating the distributional instability of various statistical procedures. addition, we evaluate the two-stage transfer learning procedure described in Section

---

We note that s-values can be used to create sensitivity plots. Under various distribution shifts, the parameter can attain a range of values. More concretely, for different choices of E and an upper bound on the distribution shift E IR we define upper and lower bounds for parameter values as follows:
0 Dupper-bound = sup suup((P) ((P) such that
PE : P = e) = Po e) for all E & and Dxz(PIIPo) S c
(9)
(owor-bound = inf ((P) such that
PE P: P(-E =) = Po e) for all e & and Dkz(PIIPo) Sc.
In experiments, we plot estimated versions of these upper and lower bounds across e for different choices of the variable E. Note that if (PP)) > 0, then s(0, Po) is the minimum for which 0lower-bound = 0.
5.1 National supported work demonstration data (NSW)
Here, we analyze the stability of the average treatment effect estimator in the presence of covariate shift using the NSW dataset [29]. which consists of 722 participants randomly assigned to a treatment or control group (variable A) in an employment program field experiment conducted between January 1976 and July 1977. Covariates include "age'. 'education', 'black', 'hispanic', 'married', 'nodegree'. and 're75', where 're75' denotes pre-intervention carnings in 1975. The outcome variable is 're78', corresponding to post-intervention earnings in 1978. We apply augmented inverse probability weighting (AIPW) using causal forests [50] to estimate the average treatment effect, resulting in an estimate of 820 with a standard deviation of 492. We evaluate the performance of the two-stage transfer learning procedure presented in Section 4
Distributional stability of average treatment effect. We investigate the stability of the average treatment effect estimator under distributional changes. Specifically, we examine how [X]] changes when there is a shift in the underlying distribution P' of each predictor separately. We measure the distributional stability of Ep(r(X) while keeping the conditional distribution of the other variables given the predictor constant. The study uses the NSW dataset [29). where the outcome variable is 're78', and the covariates X include 'age', 'education', 'black', "hispanic', 'married', 'nodegree', and 're75', where 're75' denotes pre-intervention earnings in 1975. We estimate the average treatment effect using augmented inverse probability weighting (AIPW) implemented with causal forests (50), resulting in an estimate of 820 with standard deviation of 492. Our findings show that the s- s-values of average treatment effect conditional on 'age', 'education', 'black', 'hispanic', and 're75' are non-zero. We also note that the average treatment effect is unstable with respect to changes in the marginal distribution of "age', 'education', and 're75' indicating that the average treatment effect can change its sign with a shift in the marginal distribution of these covariates (8x > 0.85). We present the directional s-values in Table 1
Table 1: S-values for NSW data set
Feature
Age
Education
Black
Hispanic
Married
Nodegree
Re75
Directional s-value
0.97
0.91
0.52
0.54
0
0.96
Parameter transfer. In this section, we evaluate the two-stage transfer learning procedure described in Section Jusing the DJW subset of the original Lalonde data extracted by Dehejia and Wahba [14] to generate training and test datasets with different distributions. The remaining samples are referred to as DJWC. The DJW subset includes 185 treated and 260 control observations and has additional information on pre-interventional carnings in 1974. We present the pre-intervention characteristics of the two subsets in Appendix and find that they differ in distribution along several variables, which are statistically significant. This split into training and test datasets allows us to evaluate the transfer learning method in a setting with strong covariate shift. We estimate the average treatment effect separately in the two subsets using causal forest (50]. The estimate on the DJW subset was 1636.7 with standard deviation of 668.8, while on DJWC,
-847.5 with a standard deviation of 657.2.
Next, we obtain our training set by adding some proportion a of randomly chosen samples from the DJW subset to the DJWC subset, where takes values in the set 0.05, 0.1, 0.2, 0.3, and use
7

---

the remaining samples as the test set. We use the procedure described in Section 4 to obtain projection of the training distribution that closely approximates the distribution. We use two transfer methods: full transfer, where we use all the covariates for the transfer, and partial transfer, where we use only the subset of covariates with which the ATE is most unstable, namely 'age', 'education', and 're75'. We display our results in Figure where we find that both transfer methods lead to lower ATE estimation error than the naive procedure. Further, we observe that there is not much gain with the full transfer method that uses all the covariates over the partial transfer method.
2000
1600
Treatment effece effect
1000
500
Age
0
Education
Black
-500
Hispanic
Married
Nodegres
Ra75
Shift in marginal distribution of given covariate with respect to KL divergence
Figure The plot shows the estimated minimum and maximum value of the average treatment effect for NSW data achievable when allowing for distribution shift in some covariate (cf. equation (9).
5000
(T(P) - teP
3000
trmeler Partel manafer Traniigg
2000
1000
0.05
0.2
0.5
0.7
alpha
Figure 3: Parameter transfer on the NSW data set. The transfer procedure described in Section compared to a naive procedure that uses only the training distribution, and a full transfer procedure that uses data on all covariates from the new distribution. The green, red and blue bars represent performance of transfer learning with partial, full new data, and naive method respectively. Error bars show the range of error over repetitions.
5.2 Wine Quality data set
We evaluate the effectiveness of our method using the wine quality dataset from the UCI Machine Learning Repository [13 18). The dataset includes subgroups of red and white wines, each with 11 chemical properties used as predictors. The response is continuous quality assessment measured on scale of 0 to 10. The dataset consists of 1599 red wines and 4898 white wines. We use all red wines as the training set and randomly select a proportion a from the white wines, where E {0.01, 0.05, 0. }. used for testing. We include small proportion of white wines in the
The remaining observations red wine training set to ensure that the shifted distribution is absolutely continuous with respect to the training distribution. This step is necessary avoid making transfer learning very challenging when datasets deviate from this assumption.
Distributional stability regression coefficients. We use directional s-values to assess distributional stability of ordinary least-squares regression coefficients. We focus our analysis on the predictors "pH" and "density," although similar results can be obtained for other variables. Figure 4 shows estimates of the minimum and maximum achievable value of a regression coefficient given

---

specific shift in the marginal distribution of a covariate (as defined in equation (9). We observe that the coefficient of "pH" is unstable with respect shifts in acidit.acidity. "chlorides," " "pH," "sulphates." and "alcohol" (sx >> 0.85). The coefficient of "density" is unstable with respect to shifts in acidity sogaal.ulfur.dioxidd directional s-values in Tables 2 [nd
"sulphates," " and "alcohol" (sx x > 0.85). We present the
transfer procedure estimate the parameter under shifted distributions. This transfer procedure combines re- re-weighting step with a birs-corre step for semi parametriiaalllyy efficient transfer. We compare three different estimators. The first one uses only the training distribution, the second transfers the parameter using the covariates found to be unstable the previous step, and the third employs all covariates for transfer (full transfer). Figure 5 displays the estimation error of the parameter under the projected and training distributions. Both transfer learning methods have smaller errors than the naive estimator that uses only the training distribution. However, the full transfer method that employs all covariates does not yield much improvement over partial transfer. For a 0.01, there not much enhancement in the coefficient of "density," " which might be due to a partial violation of the assumption that the test distribution is absolutely continuous with respect to the training distribution.
Parameter transfer. We employ Jin and Rothenhäusler 28
Table 2: Directional S-values for wine quality data set (parameter "pH")
f.a = Exxd_acidity, v.a = volatile aclitile.acidity, c.a citric.acid, f.so2 = free.sulfur dioxide, t.so2 = lotal.sulfure. cl chlorides, so4 sulphates
Feature f.a v.a
c.a
cl
f.so2
t.so2
density pH
so4
alcohol
s-value
0.86
0.81
0.65
0.83
0.94
0.55
0.8
0.83
0.97
0.97
0.88
Table 3: Directional S-values for wine quality data set (parameter "density")
f.a = fcieitacidity, v.a = mual.sulfure.dioxie cl = chlorides, so4 sulphates
volatile.acidity, c.a citric.acid, f.so2 = free.sulfur. dioxide, t.so2 =
Feature
f.a V.a
c.a
cl
f.so2
t.so2
density
pH
so4
alcohol
s-value
0.80
0.94
0.83
0.81
0.78
0.81
0.93
0.84
0.79
0.9
0.98
Discussion
The generalizability and replicability of statistical findings are crucial in scientific research. However, classical statistical measures only account for uncertainty due to sampling and not other sources of variation, such distributional shift. Since distributions expected to vary between settings and locations, is essential to understand how statistical parameters are affected by such shifts to assess the stability of finding.
In this work, we propose stability measures to quantify the impact of distributional shifts on statistical parameters at an overall and variable-specific level, respectively, enabling more detailed evaluation of instability. We expect that stability measures will be used in tandem with transfer learning procedures. Initially, the stability of a conclusion can be assessed with respect to various shifts, and then, once the sensitivities are determined, the data scientist can collect data from target distributions for the most sensitive covariates and update the model accordingly. As example, researchers are often interested in a causal effect estimate for a new location. For some covariates such as age and education, partial data is available via surveys such American National Election Study (ANES) or Cooperative Election Study (CES). Additional partial data can be cheaply obtained via Amazon Mechanical Turk. However, some covariates are hard to collect, since they require running study in the new location. Our numerical results show that the proposed approach can help prioritize data collection.
The proposed approach has several limitations. First, we consider worst-case shifts, which can be somewhat pessimistic for situations where the distribution is expected to change due to random

---

fixed
volatie acidily
citric acid
sugar
chlorides
roe.suufar dioxide
densily
pH
sulphates
Shift marginal distribution a given covariate with respect to KL. divergence
alcohol
100
50
fixed acidity
volatie. acidity
density
-50
citric acid
residual..sgaa sugar
-100
chiorides
sulfur dioxide
total sulfur. dioxxde
-000
density
pH
0.0
0.2
0.4
0.6
0.8
1.0
sulphates
alcohol
Shift in marginal distribution of given covariate with respect to KI. divergence
Figure 4: The plot shows the estimated minimum and maximum value of the regression coefficient for win" quality data achievable under distribution shift in one covariate (as defined in equation
2.0
400
3an Pes
deensit
1.5
300
Variable vonster Parsal sonstor
Variable
1.0
200
tonster Parsal sanstor
Lemsity (P)
0.5
100
0.0
0.01
0.05
0.1
0.2
0.3
0.5
0.7
0.01
0.05
0.1
0.2
0.3
0.5
0.7
alpha
alpha
This figure shows the effectiveness of two-stage transfer procedure for the wine quality data set. The green, red and blue bars represent performance of transfer learning with partial, full new data, and naive method respectively. Error bars show the range of error over 20 repetitions. Transfer learning outperforms the naive method almost all cases.
Figure 5:
perturbations in background characteristics. Modelling distribution shift random an attractive alternative [4 26] Secondly, the Kullback-Leible divergence only allows for certain types of distribution shift. The shifted distribution might have different support than the training distribution. In this case, it might be more appropriate to model the changes with the total variation distance or the Wasserstein distance. Thirdly, in this paper we focus on distribution shift in the covariates X. There is evidence that distributions shift not only in the covariates X, but also in Y|X [27 3I. Developing new tools better understand distributional shifts in Y X is exciting research direction.
10

---

References
[1] Andersen, and Gill, R. (1982). Cox's 's regression model for counting processes: large sample study. The Annals of Statistics, 10(4): 1000-1200
[2]
Anscombe, F. (1973). Graphs in statistical analysis. The American Statistician, 27(1):17-21.
[3] Barber, R. F., Candès, E. J., Ramdas, A., and Tibshirani, R. (2019). Conformal prediction under covariate shift. arXiv:1904.0601 [stat.ME].
[4] Basu, S., Sussman, J. B., and Hayward, R. A. (2017). Detecting heterogeneous treatment effects guide personalized blood pressure treatment: modeling study of randomized clinical trials. Annals of Internal Medicine, 166(5):354-36..
[5] Bertsekas, D. (1999). Nonlinear Programming. Athena Scientific.
[6] Buja, A., Berk, R., Brown, L., George, E., Pitkin, E., Traskin, M., Zhao, L., and Zhang, K. (2019). Models as approximations Consequences illustrated with linear regression. arXiv: /404. 1578 [stat.ME].
[7] Cauchois, M., Gupta, S., Ali, A., and Duchi, J. (2020). Robust validation: Confident predictions even when distributions shift. VrXiv:2008.04267 [stat.ML].
[8] Cauchois, M., Gupta, S., Ali, A., and Duchi, J. (2022). Predictive inference with weak supervision. arXiv:2201.085 08311 [s].
[9] Cauchois, M., Gupta, S., and Duchi, J. (2021). Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction. Journal of Machine Learning Research, 22(81)):144
[10] Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1-C6..
[11] Cinelli, C. and Hazlett, C. (2020). Making sense of sensitivity: Extending omitted variable bias. Journal of the Royal Statistical Society Series B- 4-Statistical Methodology, 82(1):39-67.
[12] Cornfield, J., Haenszel, W., Hammond, E. C., Lilienfeld, A. M., Shimkin, M. B., and Wynder, E. L. (1959). Smoking and lung cancer: Recent evidence and a discussion of some questions. Journal of the National Cancer Institute, 22(1):173-201
[13] Cortez, P., Cerdeira, A., Almeida, F., Matos, T., and Reis, J. (2009). Modeling wine preferences by data mining from physicochemical properties. In Decision Support Systems, 47(4):547-553
[14] Dehejia, R. H. and Wahba, S. (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of American Statistical Association, 94(448):1053- 1062.
[15] Devaux, M. and Egami, N. (2022). Quantifying robustness to external validity bias. Available at
[16] Ding, P. and VanderWeele, J. (2016). Sensitivity analyses without assumptions. Epidemiology. 27(3):66--377.
[17] Donsker, M. and Varadhan, S. (1976). Asymptotic evaluation of certain markov process expectations for large time, iii. Communications on Pure and Applied Mathematics, 29:389-461
[18] Dua, D. and Graff, C. (2019). Uci machine learning repository.
[19] Duchi. C. and Namkoong, H. (2018). Learning models with uniform performance via distributionally robust optimization. arXiv:1810.08750 [stat.ML].
[20] Esfahani, P. M. and Kuhn, D. (2018). Data-driven distributionally robust optimization using the wasserstein metric: Performance guarantees and tractable reformulations. Mathematical Programming, Series A, 115-166)

---

[21] Ghorbani, A. and Zou, J. (2019). Data shapley: Equitable valuation of data for machine learning. International conference on machine learning, pages 2242-2251. PMLR.
[22] Gijsberts, C. M., Groenewegen, A., Hoefer, I. E., and Bijkemans, M. J. (2015). Race/ethnic differences in the associations of the framingham risk factors with carotid imt and cardiovascular events. PLoS One, 10(7).
[23] Hainmueller, (2012). Entropy balancing for causal effects: multivariate reweighting method to produce balanced samples in observational studies. Political Analysis, 20(1):25-46.
[24] Huber, P. J. (1981). Robust Statistics. John Wiley and Sons, New York.
[25] Jeong, S. and Namkoong, H. (2020). Robust causal inference under covariate shift via worst-case subpopulation treatment effects. arXiv:2007.02411 [stat.ML].
[26] Jeong, Y. and Rothenhäusler, D. (2023). Calibrated inference: statistical inference that accounts for both sampling uncertainty and distributional uncertainty. arXiv preprint arXiv:2202./1888
[27] Jin, Y., Guo, K., and Rothenhäusler, D. (2023). Diagnosing the role of observable distribution shift in scientific replications. arXiv preprint arXiv:2309.01056
[28] Jin, Y. and Rothenhäusler, D. (2023). Tailored inference for finite populations: conditional validity and transfer across distributions. Biometrika, page asad022.
[29] Lalonde, R. (1986). Evaluating the econometric evaluations of training programs. American Economic Review, 76(2):1148 1178.
[30] Lange, K. (2013). Optimization. Springer.
[31] Liu, J., Wang, T., Cui, P., and Namkoong, H. (2023). On the need for a language describing distribution shifts: Illustrations tabular datasets. arXiv preprint arXiv:2307.0522
[32] Liu, M., Zhang, Y., and Cai, T. (2020). Doubly robust covariate shift regression with semi- nonparametric nuisance models. arXiv preprint arXiv:2010.02521.
[33] Lundberg, S. M. and Lee, S.-L. (2017). A unified approach to interpreting model predictions. Advances in neural information processing systems, 30.
[34] Namkoong, H., Ma, Y., and Glynn, W. (2022). Minimax optimal estimation of stability under distribution shift, arXiv:22 [st.
[35] Owen, A. B. (2001). Empirical likelihood. CRC press.
[36] Pan, S. J. and Yang. Q. (2010). A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering. 224109:1345-1339.
[37] Ramsey, (1969). Tests for specification errors in classical linear least squares regression analysis. Journal of the Royal Statistical Society, Series B, 31:350-371.
[38] Robins, J. M., Rotnitzky, A., and Zhao, L. P. (1994). Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association, 89(427):846880) 8666.
[39] Rockafellar, R. T. (1970). Convex Analysis. Princeton University Press.
[40] Rosenbaum, P. R. (1987). Sensitivity analysis for certain permutation inferences in matched observational studies. Biometrika, 74(1):13-26.
[41] Rothenhäusler, D. and Bühlmann, (2023+). Distributionally robust and generalizable infer- ence. Stutistical Science, to appear.
[42] Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Pyschology, 6685))688-701.
[43] Shafieeradeh-Ahadeh, S., Esfahani, M., and Kuhn, D. (2015). Distributionally robust logistic regression. In Advances in Neural Information Processing Systems 28, pages 1576-1584.
12

---

[44] Shimodaira, H. (2000). Improving predictive inference under covariate shift by weighting the logllikeihhod function. Journal of Statistical Planning and Inference. 90(2)227-244..
[45] Sinha, A., Namkoong, H., and Duchi, J. (2018), Certifying some distributional robustness with principled adversarial training. In Proceedings of the Sixth International Conference Learning Representations.
[46] Splawa-Neyman, J., Dabrowska, D. M., and Speed, T. P. (1990). On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9. Statistical Science, 5(4):465 - 472.
[47] Subbaswamy, A., Adams, R., and Saria, S. (2021). Evaluating model robustness to dataset shift. Proceedings of the 24th International Conference on Artificial Intelligence and Statistics.
[48] van der Vaart, W. (1998). Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press.
[49] der Vaart, A. W. (2002). The statistical work of Lucien Le Cam. The Annals of Statistics, 30-3).631-68..
[50] Wager, S. and Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 1228523)
[51] Wen, J., Yu, C.-N., and Greiner, R. (2014). Robust learning under uncertain test distributions: Relating covariate shift to model misspecification. In Proceedings of the 31st International Conference on Machine Learning. pages 631-639.
[52] Yu, B. and Kumbier, (2020). Veridical data science. Proceedings of the National Academy of Sciences, 117(8):3920-392)
[53] Zhao, Q., Small, D. S., and Bhattacharya, B. B. (2019). Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrap. Journal of the Royal Statistical Society Series B: Statistical Methodology, 8144)7735-76.
13


Reviews:
[{"review_id": "h0LGKMxq6f", "reviewer": "Reviewer_JAUw", "summary": "This paper proposes method to quantify instability of a statistical parameter with respect to pertubrations around the KL divergence ball. This has implications to detect where statistical conclusions no longer hold when there is a distribution shift. The authors show this metric across both overall and directional shifted. The paper provides a two-step transfer learning strategy over two datasets to demonstrate the effectiveness of using stability as a measure of where to collect extra data for transfer learning.", "questions": "There are some uncertainties about the experimental methodology. Why were these particular datasets chosen? Something with an average treatment effect seems like it would be useful to be seen in medical datasets for a particular intervention? Would be interested to see if there would be significant improvements over standard transfer learning with all covariates if this procedure was done with datasets with more unstable covariate variables. It would be helpful to see a broader set of experiments, or a more thorough analysis of which covariates are helpful in this case or not.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Tthe paper was well written, with its objectives and motivations clear.  Theorems and mathematical notation were easy to follow. The examples in Section 3.2 illustrated well how this metric is concretely used for different distributions. It is evident that  s-values can be useful in determining when to re-train models or re-estimate statistical queries, depending on the stability of a particular parameter. Detecting under which variables shifts occurs is an important open problem that the authors provide clear insight in, as well as a procedure to use s-values for improved transfer learning.", "weaknesses": "However, while an interesting and intuitive idea, it does not seem that it provides significant improvements over a transfer learning approach with all covariates. In Figures 3 and 5, transfer learning outperforms the naive approach, but this is an expected result. What is the reason for a partial transfer if a full transfer works better than or just as well? I would also be interested to see an experiment where a greater amount of data can be collected to improve the transfer learning (aka a higher $\\alpha$ value for the second experiment).\n\nIt would be also helpful to provide recommendations for practitioners for what it means when a parameter is stable (is there a general threshold of $s$ at which transfer learning works well over a parameter works well?) Is the $s_X > 0.85$ threshold as shown in the paper recommended? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful feedback and comments.\n\n> What is the reason for a partial transfer if a full transfer works better than or just as well?\n\nCost! Full transfer relies on having data on all covariates. Some of the data might be very expensive to collect. Partial data is often available in the form of surveys (e.g. ANES or CES) or can be collected via Amazon Mechanical Turk [1]. Our numerical results show that s-values help decide which covariates one should obtain for transfer learning. In our view, these are exciting results that may help reduce the cost of planning and running replication studies.\n\n> I would also be interested to see an experiment where a greater amount of data can be collected to improve the transfer learning (aka a higher alpha value for the second experiment)\n\nSure! We have updated the figures and included them in the pdf attached to \"Rebuttal Summary\". Our conclusions remain unchanged.\n\n> It would also be helpful to provide recommendations for practitioners for what it means when a parameter is stable (is there a general threshold of s at which transfer learning works well over a parameter works well?) Is the $s_X > .85$ threshold as shown in the paper recommended?\n\nFor the data sets we have worked with, the threshold .85 has worked well. However, in the end, this is an empirical question and depends on the subject of study. There is some exciting parallel work by Egami and Devaux [1] who build a library of reference values based on survey data sets. Their empirically validated thresholds are similar to ours.\n\n> Why were these particular datasets chosen?\n\nThe NSW data set (often referred to as the “LaLonde data”) is the most commonly studied data set in causal inference. The wine quality data set was chosen since it has appeared in recent work studying distribution shifts [2]. We have chosen these data sets to ensure high familiarity of the target audience with the data sets.\n\n> Would be interested to see if there would be significant improvements over standard transfer learning with all covariates if this procedure was done with datasets with more unstable covariate variables. It would be helpful to see a broader set of experiments or a more thorough analysis of which covariates are helpful in this case or not.\n\nThank you for the suggestion. We suspect this might be the case since our procedure ignores small shifts in unimportant variables, stabilizing the transfer learning procedure. We consider this an exciting direction for future research. For the current paper, we focus on the foundations of the method, which include theoretical guarantees and showing that it can perform as well as an oracle procedure in practice.\n\n### References\n[1] Devaux, M., & Egami, N. (2022). Quantifying robustness to external validity bias. Available at SSRN 4213753.\n\n[2] Podkopaev, A., & Ramdas, A. (2021). Distribution-free uncertainty quantification for classification under label shift. In Uncertainty in Artificial Intelligence (pp. 844-853). PMLR.\n\n\n"}, {"review_id": "T6VrFmo3Xn", "reviewer": "Reviewer_qNys", "summary": "This paper proposes a novel metric, called the s-value, for evaluating the stability of statistical parameters with respect to distributional shifts. This metric is based on a variational problem involving the KL divergence between the target distribution and the shifted distribution, which can be solved via an equivalent one-dimensional convex problem. The authors also introduce the notion of directional s-value that quantifies the instability of directional shifts. Moreover, consistency and asymptotic normality results are proven for the plug-in estimators of these s-values. Finally, the authors illustrate the interest of the s-values on some real datasets.", "questions": "1. Can you prove the consistency of $\\hat s_E(\\mu, P_n)$ under a weaker assumption on $\\hat f_n(E)$?\n2. How does the rate of convergence of $\\hat s_E(\\mu, P_n)$ depends on the one of $\\hat f_n(E)$?\n3. Can you provide empirical results supporting the practical usefulness of s-values? For example, is a statistical parameter with a large s-value really more sensitive to distribution shifts than one with a small s-value?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I found the proposal novel and interesting.\n2. Quantifying the stability of statistical findings under distributional shifts is an important question and this work takes a first step towards answering this question.\n3. The paper is well-organized and easy to follow.", "weaknesses": "1. The treatment for the consistency of $\\hat s_E(\\mu, P_n)$ seems to be weak. In particular,\n  (1) Estimating the conditional expectation $E[Z | E]$ is a challenging problem, especially when the dimension of $E$ is large as in Example 4. The uniform convergence assumption (Assumption 1 in Appendix C) seems to be too strong.\n  (2) The rate of convergence of $\\hat f_n(E)$ can be very slow when the dimension is large. It is of interest to know how the rate of convergence of $\\hat s_E(\\mu, P_n)$ depends on the one of $\\hat f_n(E)$.\n2. The practical usefulness of this metric is questionable. In the experiments the authors only computed a few s-values without giving empirical evidence supporting the reasonableness of these numbers. For example, is a statistical parameter with a large s-value really more sensitive to distribution shifts than one with a small s-value? It would be good to perform at least simulation studies to investigate this question.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful remarks.\n\n> Can you prove the consistency of $\\hat s_E (\\mu, P_n) $ under a weaker assumption on $\\hat f_n(E)$?\n\nYes, such a result can be obtained under $L_p$ convergence and a boundedness assumption, with small modifications to the current proof. We are happy to relax this in the final version of the manuscript if desired.\n\n> How does the rate of convergence of $\\hat s_E(\\mu,P_n)$ depends on the one of $\\hat f_n(E)$?\n\nIf $\\hat f_n$ has slow convergence (slower than $1/\\sqrt{n}$), then the naive estimation of $\\hat s_E$ also has a slow convergence rate. This can be improved, as explained below.\n\nIn Lemma C.4 in the Appendix, we derive a debiasing procedure that shows that even if the convergence rate of $\\hat f_n$ is slow (e.g. $n^{-1/4}$), one can still obtain a fast $1/\\sqrt{n}$ rate of convergence for $\\hat s_E$. The proof is based on de-biasing techniques from the semiparametric literature that have recently attracted considerable attention under the name of “double machine learning” [1].\n\n> Can you provide empirical results supporting the practical usefulness of s-values? For example, is a statistical parameter with a large s-value really more sensitive to distribution shift than one with a small value?\n\nChecking whether s-values are “reasonable” is best done with a small toy example where it is clear what the correct answer should be. In the Appendix, we give such a toy example (Figure 6 and Table 1) based on the famous “Anscombe quartet”. We realize that it might be helpful to move this example to the main paper for the final version.\n\nRegarding practical usefulness, the applied community has signaled interest in the proposed procedures. For example, Egami and Devaux [2] estimate the KL divergence between multiple surveys (such as ANES or CES) and use this to derive appropriate thresholds for s-values. Other ongoing applied work focuses on shifts in $Y|X$ “concept drift”.\n\n### References\n[1] Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters.\n\n[2] Devaux, M., & Egami, N. (2022). Quantifying robustness to external validity bias. Available at SSRN 4213753.\n"}, {"review_id": "YH3OfS4QbM", "reviewer": "Reviewer_oQ9C", "summary": "This work introduces the s-value, a novel metric to measure the stability of statistical parameters. It is defined as the exponential of minus the largest KL-divergence for which the statistical parameter is 0. The smaller the s-value the more stable the parameter is. When the statistical parameter is the mean of a random variable, the s-value is shown to have a simple expression. The paper also introduces the less conservative directional s-value, which constrains the type of distribution shift possible.", "questions": "How useful would s-values be for non-convex models? In the appendix, it is mentioned that a small s-value cannot be necessarily interpreted as a proof of stability in that case. What results would you expect to obtain if you were to run your parameter transfer experiment using a non-linear model?", "limitations": "I think limitations could be better addressed by e.g. adding a paragraph in the appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Many methods exist to find the worst case distribution shift within some radius as a way to obtain a robust estimator. This paper instead is finding the largest radius up to which a parameter is robust, and then use this value to derive a measure of stability. I find this approach novel and I can imagine this paper having some impact in its area. \n\nThe idea behind the paper and the derivations are sound, I also appreciate the exhaustive appendix.", "weaknesses": "I found the experiments hard to interpret, e.g. in figure 4 and 5, what is $\\beta$? Maybe you could improve the legend for those figures to make them easier to interpret. In figure 4, why are the scales so different ([-2,2] vs [-200,100]). It might have been more valuable to dive into a single example and describe it more in depth. Considering the main focus of this paper, I would also suggest showing the s-values for different parameters e.g. in a table. \n\n\nGiven the venue, I was expecting that the estimation of the s-values for parameters defined via risk minimization would be in the main paper. \n\n\nOverall, I appreciate the quality of this paper, but I would have liked to see more clearly how impactful the s-values can be in concrete cases. Achieving this might simply require improving the experiment section to address it to a broader audience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review. Also, we are glad to hear that you “appreciate the quality of this paper”.\n\n\n\n> I found the experiments hard to interpret, e.g. in Figure 4 and 5, what is beta? Maybe you could improve the legend for those figures to make them easier to interpret. In Figure 4, why are the scales so different. It might have been more valuable to dive into a single example and describe it in more depth. Considering the main focus on the paper, I would also suggest showing the s-values for different parameters, e.g. in a table.\n\nThank you for the thoughtful feedback. We will improve the descriptions for the final version. In Figure 4, the scales are very different since the plots correspond to different covariates. Following your suggestion, we will show the s-values in a table.\n\n### S-values for NSW data set\n\n|Feature| Age  | Education|  Black |   Hispanic | Married |Nodegree|Re75 |\n| -----------| ----------- | ----------- |----------- | ----------- | ----------- |----------- |----------- |\n|Directional s-value|  0.97 | 0.91|0.52 | 0.54|0 |0 |0.96 |\n\n\n### S-values for wine quality data set (parameter \"pH\")\n\n| Feature | fixed.acidity | volatile.acidity | citric.acid | residual.sugar | chlorides | free.sulfur.dioxide | total.sulfur.dioxide | density | pH  | sulphates | alcohol |\n| ------- | ------------- | ---------------- | ----------- | -------------- | --------- | ------------------- | ------------------- | ------- | --- | --------- | ------- |\n| Directional s-value | 0.86          | 0.81             | 0.65        | 0.83           | 0.94      | 0.55                | 0.8                 | 0.83    | 0.97 | 0.97      | 0.88    |\n\n### S-values for wine quality data set (parameter \"density\")\n\n| Feature              | fixed.acidity | volatile.acidity | citric.acid | residual.sugar | chlorides | free.sulfur.dioxide | total.sulfur.dioxide | density | pH   | sulphates | alcohol |\n| -------------------- | ------------- | ---------------- | ----------- | -------------- | --------- | ------------------- | ------------------- | ------- | ---- | --------- | ------- |\n|Directional  s-value              | 0.80          | 0.94             | 0.83        | 0.81           | 0.78      | 0.81                | 0.93                | 0.84    | 0.79 | 0.9       | 0.98    |\n\n> How useful would s-values be for non-convex models? In the appendix, it is mentioned that a small s-value cannot be necessarily interpreted as a proof of stability in that case. What results would you expect to obtain if you were to run your parameter transfer experiment using a non-linear model?\n\nTo summarize, we believe that the s-value is useful for cases that are most important for empirical applications, despite computational difficulties for non-convex models. Let’s discuss this in more detail.\n\nWe expect s-values to be accurate for small distributional changes since in that case the distribution change is well-approximated by a Taylor expansion. For large distributional changes, the s-value may exhibit high variance and may not be interpreted as proof of stability. However, in this case, the weights in transfer learning procedures will get extreme, leading to large confidence intervals. From a practical perspective, transfer learning in such settings is challenging if not impossible.\n\n> Given the venue, I was expecting that the estimation of s-values for parameters defined via risk minimization would be in the main paper.\n\nFrom a conceptual perspective, there is no new step from what Section 3 covers in the main paper. Our goal with the main paper is to lay out the ideas and concepts as clearly as possible, without losing “tempo”. We believe that this is a question of style, and hope that the reviewer appreciates our reasoning.\n\n> Overall, I appreciate the quality of this paper, but I would have liked to see more clearly how impactful the s-values can be in concrete cases. Achieving this might simply require improving the experiment section to address it to a broader audience.\n\nAs written above, we will update the description of the figures in the final version. \n\nRegarding “impact”: The NSW “LaLonde” data set [1] is one of the most widely used data sets in observational causal inference over the last few decades. On this data set, our two-stage procedure performs as well as a “potentially unobtainable” oracle procedure and improves more than 50% over a naive baseline. Thus, we believe that a large audience will be able to appreciate the results of our analysis. However, we would be happy to receive further input and address them in the final version.\n\n### References\n[1] Dehejia, R. H., & Wahba, S. (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association, 94(448), 1053-1062.\n"}, {"review_id": "IwcHkEQ6Ju", "reviewer": "Reviewer_jftc", "summary": "For a given data distribution P0 and family of distributions around it, mathcalP, the authors propose a measure of stability in estimating\na parameter theta(P) when there are distribution shifts within mathcalP.  The idea is that you have data P0, but might really be interested\nin estimating theta for some P' != P0 that you might get when deploying your model. That is, from data P0, know how bad you will\nbe at estimating theta(P'). Sensitivity to changes depend on the parameter of interest, and the idea here is to quantify how much of a shift it\nwould take (within P, from P0) to change the sign of theta(P0). The resulting quantity s(theta, P0), which should maybe be caled s(theta, P0, mathcalP) is called the s-value, which takes values in [0,1] for 1 meaning \"small shifts will change the sign of theta\" to 0 meaning \"the sign is stable\".\n\nThe authors not only study marginal shifts, but also those resulting when one knows that some given conditionals will be fixed. Finally, they describe a two stage procedure for determining if it would be useful to collect test set samples for certain subsets of features (those for which s value is most unstable).", "questions": "\n- The two-stage procedure involves testing the stability of various subsets of features and seeing if there are some for which it would help to collect samples from the test set. Since it would be usually not possible to do this kind of search in the real world, what would be the practical take-away for how to make use of this in non synthetic studies? Did I understand correctly that testing a given subset Xs requires collecting it from the test set? It's fine to study this ideal case in a research paper, but it seems like it necessitates more discussion.\n\n- NSW experiment: your initial estimator has a std dev of 492 which is large for an effect size of 820. How was the std dev estimated? Is this the expected ATE std dev using this method on this data? \n\n- There is a claim in the paper that \"full transfer\" doesn't provide much gain over \"partial transfer\". This probably needs some more explanation. Is the claim that  using Pproj to estimate theta is as good as using Ptest? This must only be true under some assumptions. Even if these are mentioned implicitly when describing the data, it would be helpful for the authors to restate what might cause this phenomenon. It's fine to report this phenomenon if it does hold on your data, but please contextualize.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- distribution shift is discussed in ML for predictive tasks but less in statistical inference / parameter estimation settings. It's great that the authors are bringing together the two areas, and providing tools for reasoning about inferences in the real world where we do expect shifts\n- Moreover, we don't always care just about absolute shifts, but instead qualitative things, like parameter sign changes, which the authors focus on\n- The authors discuss the conditional case where shift is only limited to some features, which helps us study more specific instances of the problem as well as gain statistical efficiency when we do have some knowledge", "weaknesses": "The weaknesses are not in the mathematical method nor experiments, but more so in the discussion and contextualization of this work.\n\n- I think the iterative two stage procedure for selecting features to collect test data for is a pretty cool use case, but there is some lacking discussion about when this would actually be feasible in the real world. Please further discuss when one should or should not be able to do this.\n\n- The authors somewhat quickly dismiss influence-function (IF) based estimation in the Related Work, but my strong feeling is that a more thorough discussion of the relationship to this field is necessary. IF based estimation is not just about robustness to outliers, but actually has a fairly deep connection to this work where influence functions show up in the functional derivative of a parameter to be estimated with respect to an underlying distribution. Moreover, just like the authors of this work consider the case of parameter sensitivity conditionally on certain factors of the joint distribution, the IF literature likewise considers projections that describe parameter sensitivity to changes in distributions when some factors are fixed. I admit that the IF literature is somewhat dense, but I believe it would be useful to state some relationships even at high level to the derivatives and Von-Mises (\"distributional taylor\") expansions in e.g. Kennedy's review here: https://arxiv.org/pdf/2203.06469.pdf\n\n---- just some writing comments below ---\n\n- I assume figure 2 is about the data NSW given the legend mentioning demographics, but the figure did not explicitly mention this. The figure caption should mention the name of the dataset to avoid confusion.\n\n- Figure 3 does not mention which data it is about in the caption, and one has to work backwards to the NSW experiment to find out which data the figure is about by searching for the text \"Figure 3\". Please mention the data in the figure caption.\n\n- when you say \"We employ Jin and Rothenhäusler [22]’s transfer procedure to estimate\", it would be helpful to be able to read what the method is from a re-statement in your paper rather than having to open the citation, since it's a detail that is part of your experiments.\n\n- in lines 59-61, \"We discuss how our ... same to re-estimate the parameter under shifted distribution\", it would be correct to change \"under shifted distribution\" to either \"under a shifted distribution\" or \"under shifted distributions\".\n\n- The remark in 63-70 could be moved to the end of work since it sort of interrupts the flow", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your careful and thorough review. We are glad that you find it “great that the authors are bringing together the two areas” [distribution shift in ML for predictive tasks and estimation problems in statistics]!\n\nThank you for your thoughtful comments on writing, we will address them in the final version of the manuscript.\n\n> I think the iterative two-stage procedure for selecting features to collect test data is a pretty cool use case, but there is some lacking discussion about when this would actually be feasible in the real world. Please further discuss when one should or should not be able to do this.\n\nIn the following, we will discuss an example from the social sciences since applied researchers from political science have signaled interest in our procedures.\n\nResearchers are often interested in a causal effect estimate for a new location. For some covariates such as age and education, partial data is available via surveys such as American National Election Study (ANES) or Cooperative Election Study (CES). Additional partial data can be cheaply obtained via Amazon Mechanical Turk. However, some covariates are hard to collect, since they require running a study in the new location. Our numerical results show that the proposed approach can help prioritize data collection. This may drastically reduce the cost compared to running full-scale replication studies.\n\nWe will add this discussion to the final manuscript.\n\n> The authors somewhat quickly dismiss influence-function (IF) based estimation in the Related Work, but my strong feeling is that a more thorough discussion of the relationship to this field is necessary. IF-based estimation is not just about robustness to outliers, but actually has a fairly deep connection to this work where influence functions show up in the functional derivative of a parameter to be estimated with respect to an underlying distribution.\n\nWe agree that functional derivatives are closely related to our work and apologize if the “related work” section gave this impression. In fact, the theory in the Appendix is based on functional derivatives of the parameter. We will rephrase the corresponding sentences in the final version.\n\n> The two-stage procedure involves testing the stability of various subsets of features and seeing if there are some for which it would help to collect samples from the test set. Since it would be usually not possible to do this kind of search in the real world, what would be the practical take-away for how to make use of this in non synthetic studies? Did I understand correctly that testing a given subset Xs require collecting it from the test set? It’s fine to study this ideal case in a research paper, but it seems like it necessitates more discussion.\n\nTo compute the s-value it is *not* necessary to have data from the test set. Searching over several subsets can be done without having additional data. Only for conducting the transfer procedure, one needs partial data from the test set. For a more concrete example discussing a use-case, see above.\n\n> NSW experiment: your initial estimator has a std dev of 492 which is large for an effect size of 820. How was the std dev estimated? Is this the expected ATE std dev using this method on this data?\n\nWe chose the NSW data set since it is the most widely used data set in observational causal inference. Standard deviations for classical procedures can be found in [1], and range between 500 and 1100. Our baseline procedure is based on SOTA cross-fitted doubly-robust machine learning and thus performs a bit better than classical approaches.\n\nWe estimated the standard deviation by the empirical standard deviation of the cross-fitted influence function, which is the standard implementation in the R-package GRF [2].\n\n> There is a claim in the paper that “full transfer” doesn’t provide much gain over “partial transfer”. This probably needs more explanation. Is the claim that using Pproj to estimate theta is as good as using Ptest? This must only be true under some assumptions. Even if these are mentioned implicitly when describing the data, it would be helpful for the authors to restate what might cause this phenomenon. It’s fine to report this phenomenon if it does hold on your data, but please contextualize.\n\nThanks for helping us clarify this. Our intuition is that in these cases most of the shift is in observables covariates $X$ (while $Y|X$ is roughly invariant). The s-value captures which types of $X$-shifts the parameter is sensitive to. Full transfer does not improve performance, since it additionally updates the distribution along directions that the parameter is not sensitive to.\n\n### References\n[1] Dehejia, R. H., & Wahba, S. (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association, 94(448), 1053-1062.\n\n[2] Athey, Susan, Julie Tibshirani, and Stefan Wager. \"Generalized random forests.\" (2019): 1148-1178.\n"}, {"review_id": "eH4vioPVNi", "reviewer": "Reviewer_msmy", "summary": "This paper proposes a measure to quantify the instability of a statistical parameter under distribution shifts, which calculates the minimal KL divergence to flip the sign of the estimated parameter. The authors demonstrate its usage in helping to collect target samples in transfer learning. The idea is clear and the theoretical analysis is solid, while the experiments seem a little inadequate.", "questions": "Please refer to weaknesses.", "limitations": "How could the proposed measure be used in neural networks or in large-scale problems?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea makes sense and the paper is easy to follow.\n2. The proposed measure is novel, and the theoretical analysis is solid.\n3. The proposed measure could help to collect data in transfer learning, and the two-stage method is simple but seems efficient.", "weaknesses": "Generally, I love this paper, but there are some drawbacks that stop me from giving a higher score:\n1. The experiments seem a litter inadequate. I view this paper as technical work, but there are almost no baselines to compare in experiments. There are some recent papers sharing similar ideas (efficiently collecting data in transfer learning), and I think the authors should compare or at least mention them. For example:\n* Data Shapley: Equitable Valuation of Data for Machine Learning. ICML 2019\n* Shapley values for feature selection: The good, the bad, and the axioms\n* Algorithms to estimate Shapley value feature attributions\n* Towards Efficient Data Valuation Based on the Shapley Value\nAlso, maybe the authors could compare with some feature selection methods in experiments.\n\n2. There are some typesetting problems in Figure 4 on page 9. Some contents are covered by figures.\n\n3. The authors measure the instability via the effects on estimation. I wonder whether it is a better way to directly measure the model performance since model performance has already taken into consideration of the estimated parameters. For example, this paper: \"Minimax Optimal Estimation of Stability Under Distribution Shift  Hongseok Namkoong, Yuanzhe Ma, Peter Glynn\". I hope that the authors could discuss these two perspectives more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We are excited to hear that you “love this paper”!\n\n> The experiments seem a litter inadequate. I view this paper as technical work, but there are almost no baselines to compare in experiments. There are some recent papers sharing similar ideas (efficiently collecting data in transfer learning), and I think the author should compare or at least mention them. For example: (...)\n\nThank you for these references, they help us clarify differences. We will include a short discussion in the final version. Here is our view:\n\nFor Shapley values, the goal is usually to evaluate the value of data (sets) or features in the context of prediction. In contrast, we are interested in parameter changes under distribution shift. As an example, if you want to predict whether someone has a headache you might want to take into account the situation an individual is in; but understanding whether the causal effect of taking an aspirin generalizes to a new situation is a different question.\n\nRegarding baselines: We are unaware of direct competitors to our approach. To the best of our knowledge, there exists no other method that measures the stability of a parameter with respect to various covariate shifts. Thus we compare it to an oracle procedure that gets to use additional data from the target distribution and conducts semi-parametrically efficient transfer using double-machine learning [1,2]. In our view, this is among the strongest oracles possible. Motivated by your question, we have tried feature selection based on the Lasso. As discussed in the rebuttal summary, feature selection based on the Lasso fails at our task of selecting the important variables for parameter transfer.\n\n> Also, maybe the authors could compare with some feature selection methods in experiments.\n\nThank you for the suggestion. Based on your suggestion, we have applied a popular feature selection method (Lasso) to the experiment. For the NSW data set, Lasso selects no covariates and for the wine quality data set it selects all covariates. We attribute this failure to the fact that the Lasso quantifies feature importance in prediction problems, instead of sensitivity in parameter estimation problems.\n\n> There are some typesetting problems in Figure 4 on page 9. Some contents are covered by figures.\n\nThank you. We will fix this in the final version.\n\n> The authors measure the instability via the effects on estimation. I wonder whether it is a better way to directly measure the model performance since model performance has already taken into consideration of the estimated parameters. For example, this paper: “Minimax Optimal Estimation of Stability Under Distribution Shift”\n\nThank you for the thoughtful comment. We love the mentioned paper, since it employs a similar philosophy, albeit for prediction problems instead of parameter estimation problems. Let’s consider the example of fitting a model for causal inference. One parameter might correspond to the treatment, while the other parameters might correspond to nuisance parameters (such as the effect of age, education, socioeconomic status,...). Model performance would now measure the stability of the model with respect to shifts (including changes in age and education) and would be misleading if someone is just wondering whether the treatment would still work in a new situation.\n\nIn short, evaluating model stability is useful for predictions under shifts, while parameter stability is useful for parameter estimation under shifts.\n\n### References\n[1] Jin, Ying, and Dominik Rothenhäusler. \"Tailored inference for finite populations: conditional validity and transfer across distributions.\" Biometrika (2023)\n\n[2] Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters.\n\n[3] Wager, S., & Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523), 1228-1242.\n"}, {"review_id": "seTR6LFslr", "reviewer": "Reviewer_ktSh", "summary": "This work defines a novel statistical measure of the “stability” of a parameter in a distribution with respect to changes in that distribution. From a high level, it is defined as the minimum KL distance to a perturbation that flips the sign of the parameter. A method is given to calculate the s-value for mean parameters and several experiments are performed to show its utility.", "questions": "Could you extend to more general cases by learning a flexible neural density approximator to P_0, and solving (3) by (constrained) variational inference?", "limitations": "Even though it’s addressed in the appendix, it seems non-practical to calculate the s-value for parameters other than the mean or functions of the mean.\n\nThe paper restricts discussion to distributions that are absolutely continuous with respect to the training distribution, although this is not much of a limitation as it covers most practical cases of interest.", "rating": 3, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The method extends to the directional case. I can see the application of this to average treatment effect to be useful in industry in online experimentation platforms. Strong arguments for the utility of s-value for doing robust transfer learning.", "weaknesses": "One weakness is that the calculation of the s-value uses a theorem that is only applicable to mean value parameters. The difficulty is that the definition uses an optimization problem over a set of functions rather than real-valued variables. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time in evaluating our manuscript. We hope that we can address your concerns with this rebuttal.\n\n> One weakness is that the calculation of the s-value uses a theorem that is only applicable to mean value parameters.\n\nPlease note that we have many results that cover more general cases (e.g., Example 6 in the Appendix covers generalized linear models). We provide an extensive theory (Appendix D, E, and F) to cover a variety of real-world cases.\n\n> The difficulty is that the definition uses an optimization problem over a set of functions rather than real-valued variables.\n\nIn statistical and machine learning problems, it is common to initially state a problem as an optimization over an infinite-dimensional space. For example, non-linear prediction can be stated as arg min_f E[(Y- f(X))^2]. In observational causal inference, under ignorability the parameter is a functional of an infinite-dimensional nuisance parameter. Thus, we believe that our approach is well within mainstream ML.\n\nFrom a practical side, our theoretical results allow us to re-cast the problem as finite-dimensional optimizations (sometimes even one-dimensional convex problems, see lines 127 and 146).\n\n> Could you extend to more general cases by learning a flexible neural density approximator to $P_0$, and solving (3) by (constrained) variational inference?\n\nIn general, such a direct approach is possible and might lead to interesting new procedures. However, our proposed procedure has two advantages that may not be straightforward to implement with the reviewer’s approach. First, neural density estimation does not consider the particular functional form of the estimand. As Donsker-Varadhan shows, taking into account the functional form can reduce the infinite-dimensional problem to a finite-dimensional one. Secondly, density estimation usually has extremely slow convergence rates and (in our experience) might lead to unreliable estimates unless penalized very carefully.\n\n> Even though it’s addressed in the appendix, it seems non-practical to calculate the s-value for parameters other than the mean or functions of the mean.\n\nWe respectfully disagree with this statement. From an optimization perspective, the question is not whether the parameter is a function of the mean; it is whether the parameter is a linear functional in the distribution space. Parameters of linear regression are non-convex functionals when viewed as functionals on the distribution space. Thus, the empirical example in Section 5.2 already captures this challenge. More general cases (such as empirical risk minimization) are treated in extensive detail in the appendix.\n"}, {"review_id": "XekC86Dnpl", "reviewer": "Reviewer_buuF", "summary": "The paper presents a metric (s-value) that quantifies the uncertainty of statistical estimators in terms of their distributional instability. In addition, the techniques proposed can quantify the effect of directional shifts and the authors also discuss how the s-value can be used to improve estimation accuracy under shifted distributions.", "questions": "The definition in (2) is unclear in page 2. The authors should mention there that the set \\mathcal{P} in (2) denotes joint distributions of Z and E, since in that page those distributions correspond to r.v. Z only.\n\nIn line 248 the authors mention \"Our findings show that the average treatment effect is unstable with respect to changes in the marginal distribution of ‘age’, ‘education’, and ‘re75’. We also find that s-values conditional on ‘age’, ‘education’, ‘black’, ‘hispanic’, and ‘re75’ are non-zero, indicating that the average treatment effect can change its sign with a shift in the marginal distribution of these covariates (sX > 0.85).\" Are those results shown in Figure 2? \n\nThe authors should improve the quality of figures that currently is rather poor.", "limitations": "The authors should improve the description of the limitations of the methods proposed as described in the \"weaknesses\" section. ", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The development of methods that quantify the effect of distributions shifts is very relevant since such shifts are very common in practice. The overall approach in the paper is of interest and the results in the paper provide a promising initial step towards novel measures of uncertainty that account for distributional shifts.\n", "weaknesses": "The interpretation of the s-value in (1) for scalar parameters is somehow clear since it measures the smallest divergence needed to change the sign of the parameter (assuming the parameter continuously depends on the distribution). However, the usefulness of its generalization to the multidimensional case in (11) needs further clarification. In the scalar case, the divergence needed to achieve a zero value reflects the divergence needed to have a change of sign. In the multidimensional case, it is not clear why measuring the divergence needed to achieve a zero vector valued \\eta quantifies the instability. \n\nThe experimental results are not correctly described. It is hard to grasp the main takeaways of such results and the relationship with the theoretical results provided. In general, the paper presentation is rather poor and multiple results appear in the appendices. It is clear that the paper can be significantly improved.\n\nThe contribution of the proposed two-stage approach for transfer learning is hard to quantify. In general, the authors should compare their methods with existing techniques so that the paper's contributions can be better assessed. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful suggestions and interest in our work. We hope that the additional evidence and our explanations will address your concerns.\n\n> The interpretation of the s-value in (1) for scalar parameters is somehow clear since it measures the smallest divergence needed to change the sign of the parameter (assuming the parameter continuously depends on the distribution). However, the usefulness of its generalization to the multidimensional case in (11) needs further clarification. In the scalar case, the divergence needed to achieve a zero value reflects the divergence needed to have a change of sign. In the multidimensional case, it is not clear why measuring the divergence needed to achieve a zero vector valued $\\eta$ quantifies the instability.\n\nWe agree that (11) does not have many direct applications. The s-value in (11) is just an intermediate result. In practice, the stability of scalar components of multidimensional parameters is of paramount importance (Section 5.1, Section 5.2, Section D.1). For example, in causal inference, we are often interested in the average treatment effect (a scalar component) in the presence of multiple nuisance parameters (confounders). Such an example is presented in Section 5.1.\n\n> The experimental results are not correctly described. It is hard to grasp the main takeaways of such results and the relationship with the theoretical results provided. In general, the paper presentation is rather poor and multiple results appear in the appendices. It is clear that the paper can be significantly improved.\n\nWe are not sure what is meant by “not correctly described”. In the final version, we will include tables of s-values (see response to reviewer oQ9C) and improve the quality of the figures. We believe that this will improve readability. If this does not address your concern, we would appreciate additional input.\n\nRegarding the Appendix: We have decided to explain the conceptual ideas in the main paper and move more technical results to the appendix. The goal is to confer the main ideas and numerical evidence, without losing “tempo”. We believe that this is a stylistic choice and hope that the reviewer agrees with our reasoning.\n\n> The contribution of the proposed two-stage approach for transfer learning is hard to quantify. In general, the authors should compare their methods with existing techniques so that the paper's contributions can be better assessed.”\n\nUnfortunately, for the s-value itself, there exists no direct competitor (see rebuttal summary above).\n\nBased on your feedback we have evaluated a popular feature selection method, the Lasso. For the wine quality data, it selects all covariates while for the NSW data, it selects no covariates. The reason for this failure is that the Lasso selects based on feature importance in a prediction problem instead of stability in an estimation problem.\n\nThe naive approach is to conduct semi-parametrically efficient transfer learning based on data on all covariates. This is an oracle procedure that provides an upper bound on other procedures. We show that our approach is competitive with this data-hungry oracle method while requiring much less data.\n\n> The definition in (2) is unclear in page 2. The authors should mention there that the set $\\mathcal{P}$ in (2) denotes joint distributions of Z and E, since in that page those distributions correspond to r.v. Z only.\n\nWe will address this and clarify that the set  $\\mathcal{P}$ in (2) denotes joint distributions of Z and E in the final version.\n\n> In line 248 the authors mention \"Our findings show that the average treatment effect is unstable with respect to changes in the marginal distribution of ‘age’, ‘education’, and ‘re75’. We also find that s-values conditional on ‘age’, ‘education’, ‘black’, ‘hispanic’, and ‘re75’ are non-zero, indicating that the average treatment effect can change its sign with a shift in the marginal distribution of these covariates ($s_X > 0.85$).\" Are those results shown in Figure 2?\n\nYes, the s-values can be obtained from Figure 2 by checking at which x-values the line cross zero, but we agree that it is more complicated than necessary. Following your feedback, we have decided to include a table with the s-values in the main paper (tables can be found in the response to reviewer oQ9C). Thank you for your input!\n\n> The authors should improve the quality of figures that currently is rather poor.\n\nWe realize that some characters are only partially visible in Figures 2, 4 and 5. We will address this in the final version. We will also fix some color issues. If you have any other feedback, please let us know. \n\n> The authors should improve the description of the limitations of the methods proposed as described in the \"weaknesses\" section.\n\nWe will expand the discussion of the weaknesses; in particular, we will describe more directly situations that might require other types of distributional stability values. For example, ongoing work covers shifts in $Y|X$. We believe there is room for a variety of new stability values that cover different types of distributional shifts. If you have additional feedback, please let us know."}]

Author rebuttal:
# Rebuttal Summary

We thank you for your constructive feedback. We appreciate that you find the approach “novel and interesting” (qNys), that you “appreciate the quality of this paper” (oQ9C), that the “theoretical analysis is solid” (msmy), that it is “a promising initial step towards novel measures of uncertainty” (buuF) and that you see “strong arguments for the utility of s-value for doing robust transfer learning” (ktSh).

In the following, we will address two points that the majority of the reviewers have brought up as weaknesses: [1] practicality and feasibility, and [2] empirical evaluation.

## Practicality and feasibility

Several reviewers have asked for better contextualization, in particular, whether partial test data is available in practice, and how “impactful” stability values can be in practice.

Recall that the proposed workflow is to (i) evaluate the instability of the parameter with respect to shifts in different covariates; and (ii) update the parameter using data from the covariates flagged in the first step. Our procedure focuses on the first step. The second step can be conducted using existing procedures. In the following, we will discuss an example from the social sciences since applied researchers from political science have signaled interest in our procedures.

Researchers are often interested in a causal effect estimate for a new location. For some covariates such as age and education, partial data is available via surveys such as American National Election Study (ANES) or Cooperative Election Study (CES). Additional partial data can be cheaply obtained via Amazon Mechanical Turk. However, some covariates are hard to collect, since they require running a study in the new location. Our numerical results show that the proposed approach can help prioritize data collection. This may drastically reduce the cost compared to running full-scale replication studies.

As a side note, there is exciting empirical work by Egami and Devaux [1] who build a library of reference stability values based on survey data sets. They recommend thresholds for s-values based on empirical investigations of how data sets change between settings.


## Empirical evaluation

Overall, the reviewers have (i) asked about additional baselines, (ii) asked for a larger range of alpha in plots, and (iii) identified typesetting problems in some of the figures.

(i) Baselines

To the best of our knowledge, there exists no other method that measures the stability of a parameter with respect to various covariate shifts. Thus, there is no direct competing method. This is why we have compared our procedures to an oracle procedure that has access to additional data.

As suggested by the reviewers, we have evaluated a baseline based on feature importance. For the wine quality data set, Lasso selects all features, and for the NSW data set, Lasso selects no covariates. Thus, for NSW the performance of Lasso is equal to “Training”, while for “wine quality”, the performance of Lasso is equal to “Full Transfer”.

Why does Lasso feature selection fail? It solves a different problem; it captures the *feature importance in a prediction problem*, whereas our stability value captures the *sensitivity in a parameter estimation problem*. 

(ii) A larger range of alpha

In the pdf attached to this summary, you find a plot corresponding to a larger range of alphas. Our conclusions remain unchanged.

(iii) Typesetting problems in some figures

The reviewers have criticized that some characters were only partially visible in Figures 2, 4 and 5. We will fix this in the final version of the manuscript. In addition, we will fix some color issues.


## References

[1] Devaux, M., & Egami, N. (2022). Quantifying robustness to external validity bias. Available at SSRN 4213753.




Meta-review: This paper proposes a novel metric, called the s-value, for evaluating the instability of statistical parameters with respect to distributional shifts. The paper contains many interesting and novel ideas and is well-written. There are some concerns regarding the computational challenges in evaluating this metric as well as as the quality of numerical experiments and the authors are encouraged to take into account the related suggestions by the reviewers in the revision of their paper.
Decision: Accept (poster)