Title: The s-value: evaluating stability with respect to distributional shifts
Venue: NeurIPS 2023
Accepted: yes
Keywords: Distributional Stability

Distributional Robustness

Distributional Shifts

Generalizability
Abstract: Common statistical measures of uncertainty such as $p$-values and confidence intervals quantify the uncertainty due to sampling, that is, the uncertainty due to not observing the full population. However, sampling is not the only source of uncertainty. In practice, distributions change between locations and across time. This makes it difficult to gather knowledge that transfers across data sets. We propose a measure of instability that quantifies the distributional instability of a statistical parameter with respect to Kullback-Leibler divergence, that is, the sensitivity of the parameter under general distributional perturbations within a Kullback-Leibler divergence ball. In addition, we quantify the instability of parameters with respect to directional or variable-specific shifts. Measuring instability with respect to directional shifts can be used to detect under which kind of distribution shifts a statistical conclusion might be reversed. We discuss how such knowledge can inform data collection for transfer learning of statistical parameters under shifted distributions. We evaluate the performance of the proposed measure on real data and show that it can elucidate the distributional instability of a parameter with respect to certain shifts and can be used to improve estimation accuracy under shifted distributions.

The s -value: evaluating stability with respect to distributional shifts

Abstract

Common statistical measures of uncertainty such as p -values and confidence intervals quantify the uncertainty due to sampling, that is, the uncertainty due to not observing the full population. However, sampling is not the only source of uncertainty. In practice, distributions change between locations and across time. This makes it difficult to gather knowledge that transfers across data sets. We propose a measure of instability that quantifies the distributional instability of a statistical parameter with respect to Kullback-Leibler divergence, that is, the sensitivity of the parameter under general distributional perturbations within a Kullback-Leibler divergence ball. In addition, we quantify the instability of parameters with respect to directional or variable-specific shifts. Measuring instability with respect to directional shifts can be used to detect under which kind of distribution shifts a statistical conclusion might be reversed. We discuss how such knowledge can inform data collection for transfer learning of statistical parameters under shifted distributions. We evaluate the performance of the proposed measure on real data and show that it can elucidate the distributional instability of a parameter with respect to certain shifts and can be used to improve estimation accuracy under shifted distributions.

1 Introduction

Test data sets collected in different locations or at different time points often are drawn from different distributions, due to changing circumstances, changes in unmeasured confounders, time shifts in distribution, or distributional shifts in covariates [44, 19, 16, 20]. This makes it difficult to gather knowledge that transfers across data sets. Statistical estimands such as a regression coefficient or the average treatment effect (ATE) may vary as the underlying distribution changes and hence, statistical findings (such as that the treatment effect is positive) may not replicate across data sets [4, 22].

In causal inference, the rapidly growing field of sensitivity analysis [12, 40, 16, 53, 11] quantifies the stability of an estimate with respect to unobserved confounding. Roughly speaking, this line of work sees stability analysis as part of uncertainty quantification. Inspired by this line of work, we aim to bring a similar type of stability analysis to a wider range of statistical procedures.

In this paper, we propose a measure of instability, called the s -value, to investigate the stability of a given statistical parameter with respect to a shift in the underlying distribution (Figure 1). The s -value quantifies the minimum shift in distribution required to tilt the parameter to a given value, using Kullback-Leibler divergence. We also investigate the stability of parameters with respect to directional or variable-specific shifts. The proposed measure can be used as an exploratory tool to identify the kind of distribution shift that could reverse a statistical conclusion. We further discuss

Training distribution,

P

0

Shifted distribution,

P

Parameter,

θ

(

P

0

)

>

0

Parameter,

θ

(

P

)

<

0

how s -values can be used to obtain improved estimates of statistical parameters under a shifted distribution with limited information about the new distribution.

1.1 Our contribution

We propose a measure of instability that quantifies the sensitivity of a one-dimensional statistical parameter to changes in the underlying probability distribution. We focus on shifts in distributions that are absolutely continuous with respect to the training distribution. Let P 0 ∈ P be the training distribution on the measure space ( Z , A ) , where P is the set of probability measures, Z is a random element of Z , and θ : P ↦→ R is the one-dimensional statistical parameter of interest. We are interested in the minimum amount of shift in distribution that changes the sign of the parameter. To this end, we define the stability value ( s -value) for θ as

where D KL is the Kullback-Leibler divergence between P and P 0 given by

We provide some more discussion on the thought process that led to the proposed definition in Appendix, Section A. Note that the s -value lies in [0 , 1] , with values close to 1 indicating that a small shift in distribution may alter the findings, and hence, the finding is not distributionally stable. S -values close to 0 indicate that the sign of the parameter is stable under distributional changes. In Section 3, we discuss estimation of s -values for parameters that are linear in the distribution P . The proposed procedure can be generalized to parameters defined via risk minimization, including parameters in generalized linear models (see Appendix, Section D).

Considering overall distributional shift does not give information about what kind of distribution shifts the parameter is sensitive to. Hence, we also quantify the instability of parameters with respect to shifts in the distribution of certain exogenous or endogenous variables E , assuming that the conditional distribution of the remaining variables given E is constant. Let E be a random variable taking values in the space E . We define the directional or variable-specific s -value as

where P denotes the set of probability distributions over joint random variable ( Z, E ) . If a practitioner discovers that a parameter is sensitive with respect to changes in the distribution of a certain variable E , this knowledge can be used to update the parameter estimate. We discuss how our method can be used to prioritize data collection about the new distribution in Section 4 and use the same to re-estimate the parameter under shifted distribution. The proposed procedure shows promise for the task of prioritizing data collection from the new distribution in the experiments.

2 Related work

Quantifying the uncertainty of statistical estimators is a crucial objective in statistics, typically accomplished using classical statistical measures such as p -values and confidence intervals to quantify sampling uncertainty. However, these methods typically rely on strong, potentially unjustified

assumptions about fixed underlying distributions, which may lead to false discoveries. To improve reliability and reproducibility in statistical estimation, Yu and Kumbier [52] propose the predictability, computability, and stability (PCS) framework. While they investigate the stability of data results under data and method perturbations, we focus specifically on evaluating the stability of statistical parameters under distributional shifts.

Model misspecification can result in distributional instability. Buja et al. [6] highlight fundamental issues with model misspecification or non-linearity in linear models. They propose reinterpreting population slopes as statistical functionals of data generating distributions and develop diagnostic tests for detecting model deviations. We introduce measures to illustrate coefficient instability under various distributional shifts for parametric and semi-parametric estimators.

[34] introduce a novel framework to analyze the stability of decision policies and prediction models under distribution shifts. Central to their approach is the notion of stability, characterized as the minimal alteration in the underlying environment required to push a system's performance beyond a specified threshold. In contrast, we focus on understanding stability of parameters with respect to shift in distribution.

In machine learning, there is much work on computing which data or features contribute to a prediction, e.g. using Shapley values [33, 21]. In contrast, we are interested in how distributional changes in features (or covariates) lead to parameter changes.

Sensitivity analyses in the causal inference literature aim to investigate the stability of causal estimates with respect to unmeasured confounding [12, 40, 16, 53, 11]. Our proposal can be seen as a version of sensitivity analysis for general estimands where we evaluate both the stability of an estimand with respect to the overall shift in distribution and the stability with respect to directional distribution shifts.

Classical robust statistics [24] addresses robustness against contaminations and outliers using measures like leverage scores and influence functions to construct estimators that are not unduly influenced by such outliers. Influence functions play an important role in this work, since it corresponds to the functional derivative of a parameter with respect to the distribution. Different from classical robust statistics, in our case the perturbation is not a contamination but corresponds to an actual change in the underlying population. Rather than robustifying estimators, our aim is to equip practitioners with tools to detect sources of instability and facilitate the transfer of estimators across different settings.

There has been a resurgence of research addressing the challenges posed by distributional shifts. This research has mostly focused on building distributionally robust estimators where more weight is given to the outliers by considering worst-case distributional shifts in a neighborhood of the training distribution [19, 45, 20, 43, 25, 7, 9, 8, 47]. In contrast, we quantify the stability of potentially non-linear statistical parameters under both overall and variable-specific distributional shifts.

There is exciting empirical work by Devaux and Egami [15] who build a library of reference stability values based on survey data sets. They recommend thresholds for s-values based on empirical investigations of how data sets change between settings. This work allows for the contextualization of distributional stability values.

Closely related to our work are empirical likelihoods [35]. In the empirical likelihood framework, small overall distributional tilts are used to construct p -values and confidence intervals for a given parameter. In our work, we use distributional shifts of various strengths to evaluate the (directional) stability of an estimand with respect to distribution shifts.

3 S-value of the mean

In this section, we discuss estimation of the s -value of the mean of a one-dimensional real-valued random variable followed by some examples. Estimation of s -values for more general settings, including parameters defined via risk minimization, is discussed in Appendix, Section D. We first focus on the special case of mean estimation as it allows us to develop procedures that will be helpful in more general settings.

3.1 Estimation of the s -value

Consider a one-dimensional real-valued random variable Z ∼ P 0 , where P 0 ∈ P . We recall from (1) that the s -value for the mean ( µ ( P 0 ) = E P 0 [ Z ]) is defined as

In words, we are interested in finding the distribution closest to our training distribution P 0 under which the mean of the random variable is 0 . At first sight, s -values might seem difficult to estimate since the supremum in equation 3 is taken over the infinite-dimensional space of probability distributions P . However, it turns out that the s -value of the mean can be obtained by solving a one-dimensional convex optimization problem.

Theorem 1 (Theorem 5.2, Donsker and Varadhan [17]) . Let Z ∼ P 0 be a real-valued random variable with mean µ ( P 0 ) = E P 0 [ Z ] and finite moment generating function on R . Then, we have

Further, if the infimum in (4) is attained at some λ ∗ ∈ R then the infimum in (3) is attained at some probability distribution Q given by

We note that M Z ( λ ) = E [ e λZ ] is the moment generating function of Z . Since, M Z (0) = 1 , we have s ( µ, P 0 ) ∈ [0 , 1] . In practice, we only have access to finitely many realizations of the data generating distribution. Let P n be the empirical distribution of Z i i.i.d. ∼ P 0 for i ∈ [ n ] , we obtain an estimator of the s -value via the plugin estimator

Using classical results for M -estimators (see Chapter 5 of van der Vaart [48]), we show that ˆ s ( µ, P n ) is consistent and asymptotically normal in Appendix C.1.

Directional s -values. The previous form of distributional stability might be very conservative. In practice, we do not expect all aspects of distribution to change from setting to setting. To allow for a more fine-grained evaluation of stability, we also consider directional shifts, which only change certain aspects of the distribution. In the following, we will make this more precise.

Let P 0 be the joint distribution of the multivariate random variable ( Z, E ) where Z takes values in Z ⊆ R and E takes values in E ⊆ R p for some positive integer p . E may be an exogenous or endogenous variable. We consider a directional shift, i.e. a situation where the marginal distribution of E may change while keeping the conditional distribution of Z given E constant. To be more precise, we seek to estimate

We next show that s E is a solution to a one-dimensional convex optimization problem. The proof of the following result can be found in Appendix I.2.

Theorem 2. Let P 0 be the joint distribution function of the random variable ( Z, E ) taking values in Z × E with µ = E P 0 [ Z ] and finite moment generating function on R . Then,

Further, if the infimum in (7) is attained at some λ ∗ ∈ R then the infimum in (2) is attained at some probability distribution Q given by

This result allows us to estimate the directional s -value. Let ˆ f n ( E ) be an estimator of E [ Z | E ] . Then, we can define a plug-in estimator by setting

We prove the consistency of ˆ s E ( µ, P n ) in Appendix, Section C.1. In the Appendix, we also discuss how to form a de-biased estimator of the directional s -value that is asymptotically normal.

3.2 Examples

Example 1 (Distribution with positive support) . If Z is a random variable that has positive support with probability 1, then s ( µ, P 0 ) = 0 , which reflects the fact that for any distribution shift within the KL-divergence ball, we will always have a positive mean.

Thus, in the Gaussian case the stability measure s is a monotonous transformation of the signalto-noise ratio. High signal-to-noise ratio yields lower values of s indicating stronger distributional stability.

Let us now develop some intuition for directional shifts. First, we derive conditions under which the directional stability is zero, that is, conditions under which s E ( µ, P 0 ) = 0 .

Example 3 (Directional stability) . Let E P 0 [ Z | E ] > 0 . Then,

Example 4 (Average treatment effect) . Here we consider estimating the causal effect of a treatment via the potential outcome framework [46, 42]. We have a binary treatment random variable A ∈ { 0 , 1 } , potential outcomes Y (1) and Y (0) corresponding to the potential outcome under treatment and control respectively and some covariates X . Under the consistency assumption, we observe Y (1) if A = 1 and Y (0) if A = 0 , i.e. Y = AY (1) + (1 -A ) Y (0) . One can write the average treatment effect (ATE) as

where µ ( a ) ( X ) = E [ Y ( a ) | X ] . Hence, if we only consider shifts in marginal distribution of covariates X keeping the conditional distribution of other variables given the covariates as fixed, we obtain s X -values as above with Z = µ (1) ( X ) -µ (0) ( X ) . In practice, µ (1) ( X ) and µ (0) ( X ) are often unknown. We can use plug-in estimators ˆ µ (1) ( X ) and ˆ µ (0) ( X ) to form the estimator

Consistency of this estimator can be shown with the same technique as Lemma C.3 in Appendix C.1.

In statistical analysis, risk minimization is often used to define various parameters, including regression coefficients and general M-estimators. However, unlike parameters that can be represented as the mean of a random variable, these parameters lack a simple representation as they are not linear in the underlying probability distribution. This makes the optimization problem involved in obtaining s -values non-convex. To address this issue, we present methods for obtaining s -values for such parameters in Section D of the Appendix.

4 Parameter transfer using s -values

In Section 3, we introduced s -values that measure the distributional stability of statistical parameters with respect to various shifts. In this section, we discuss how we can use s -values to guide further data collection. The above problem of re-estimating parameters under a shifted distribution is related to the transfer learning literature that overlaps with various fields including robust machine learning,

causal inference, and conformal inference [36, 51, 3]. Here, we discuss how s -values can guide transfer learning.

If a parameter is unstable with respect to a shift in marginal distribution of certain covariates, then knowledge about those covariates can be used to transfer parameters across distributions. As an example, assume that we have collected some data on a job program in New York. We now want to estimate how efficient this job program would be in Boston. We have not run this job program in Boston yet, so we do not know all covariates of the participants. However, we can find that the efficiency of the job program is likely unstable with respect to changes in the demographics of job seekers in Boston. How can we use this knowledge to estimate the efficiency of the job program in Boston, based on limited data about the population in Boston? In the following, we will discuss this problem in a formal framework. We discuss another important case. Researchers are often interested in a causal effect estimate for a new location. For some covariates such as age and education, partial data is available via surveys such as American National Election Study (ANES) or Cooperative Election Study (CES). Additional partial data can be cheaply obtained via Amazon Mechanical Turk. However, some covariates are hard to collect, since they require running a study in the new location. Our numerical results show that the proposed approach can help prioritize data collection. This may drastically reduce the cost compared to running full-scale replication studies.

̸

Assume that we want to estimate a parameter θ ( P shift ) for P shift = P 0 , but we only have observations from P 0 . In addition, we may be able to collect some information about P shift, for example, observations of a subset of variables X S ∈ R d . For example, one may know the age distribution of job seekers in Boston. Intuitively, we'd like to re-weight P 0 so that the distribution of age matches the distribution of age in Boston. However, there may be infinitely many choices of weights. These different choices of weights will correspond to different values of θ . Thus, in practice, it is crucial to use a form of regularization when finding a re-weighted distribution ( P proj ).

We can define P proj as the solution of the following optimization problem:

where D KL is the Kullback-Leibler divergence. The objective is similar to Hainmueller [23], where the author proposes entropy balancing to achieve covariate balance between treated and control sets for estimating the average treatment effect. This objective can be solved explicitly, leading to a covariate shift setting [36]. More specifically, a short calculation shows that if the minimum is finite, then

̸

Since data collection can be costly, one would like to prioritize collecting data that is relevant for the transfer learning task. If s X S ( θ -c, P ) = 0 for all c = θ ( P ) , then the parameter is constant under shifts in the marginal distribution of X S . On the other hand, if s X S ( θ, P ) ≈ 1 , then small changes in the distribution of X S might induce a large change in the parameter θ ( · ) . These heuristics motivate the following approach:

Find variables X S with respect to which the parameter of interest is most sensitive to as determined by directional s -values. Collect observations of X S under the shifted distribution.

Estimate θ ( P proj ) .

There are several existing approaches to deal with part 2. In particular, it is possible to estimate θ ( P proj ) for a large range of estimands θ ( · ) , with asymptotically normal and efficient estimators [38, 32, 28]. We can leverage these existing estimators in our workflow so that we have statistical guarantees for all steps in this pipeline.

We will investigate the empirical performance of this two-stage approach in Section 5.

5 Experiments

In this section, we consider real-world data to illustrate the effectiveness of the proposed methods in elucidating the distributional instability of various statistical procedures. In addition, we evaluate the two-stage transfer learning procedure described in Section 4.

We note that s -values can be used to create sensitivity plots. Under various distribution shifts, the parameter can attain a range of values. More concretely, for different choices of E and an upper bound on the distribution shift c ∈ R we define upper and lower bounds for parameter values as follows:

In experiments, we plot estimated versions of these upper and lower bounds across c for different choices of the variable E . Note that if θ ( P 0 ) > 0 , then s ( θ, P 0 ) is the minimum c for which θ lower-bound = 0 .

5.1 National supported work demonstration data (NSW)

Here, we analyze the stability of the average treatment effect estimator in the presence of covariate shift using the NSW dataset [29], which consists of n = 722 participants randomly assigned to a treatment or control group (variable A ) in an employment program field experiment conducted between January 1976 and July 1977. Covariates X include 'age', 'education', 'black', 'hispanic', 'married', 'nodegree', and 're75', where 're75' denotes pre-intervention earnings in 1975. The outcome variable is 're78', corresponding to post-intervention earnings in 1978. We apply augmented inverse probability weighting (AIPW) using causal forests [50] to estimate the average treatment effect, resulting in an estimate of 820 with a standard deviation of 492 . We evaluate the performance of the two-stage transfer learning procedure presented in Section 4.

Distributional stability of average treatment effect. We investigate the stability of the average treatment effect estimator under distributional changes. Specifically, we examine how E P [ τ ( X )] changes when there is a shift in the underlying distribution P of each predictor separately. We measure the distributional stability of E P [ τ ( X )] while keeping the conditional distribution of the other variables given the predictor constant. The study uses the NSW dataset [29], where the outcome variable is 're78', and the covariates X include 'age', 'education', 'black', 'hispanic', 'married', 'nodegree', and 're75', where 're75' denotes pre-intervention earnings in 1975. We estimate the average treatment effect using augmented inverse probability weighting (AIPW) implemented with causal forests [50], resulting in an estimate of 820 with a standard deviation of 492 . Our findings show that the s -values of average treatment effect conditional on 'age', 'education', 'black', 'hispanic', and 're75' are non-zero. We also note that the average treatment effect is unstable with respect to changes in the marginal distribution of 'age', 'education', and 're75' indicating that the average treatment effect can change its sign with a shift in the marginal distribution of these covariates ( s X > 0 . 85 ). We present the directional s -values in Table 1.

Parameter transfer. In this section, we evaluate the two-stage transfer learning procedure described in Section 4 using the DJW subset of the original Lalonde data extracted by Dehejia and Wahba [14] to generate training and test datasets with different distributions. The remaining samples are referred to as DJWC. The DJW subset includes 185 treated and 260 control observations and has additional information on pre-interventional earnings in 1974. We present the pre-intervention characteristics of the two subsets in Appendix J and find that they differ in distribution along several variables, which are statistically significant. This split into training and test datasets allows us to evaluate the transfer learning method in a setting with strong covariate shift. We estimate the average treatment effect separately in the two subsets using causal forest [50]. The estimate on the DJW subset was 1636.7 with a standard deviation of 668.8, while on DJWC, it was -847.5 with a standard deviation of 657.2.

Next, we obtain our training set by adding some proportion α of randomly chosen samples from the DJW subset to the DJWC subset, where α takes values in the set 0 . 05 , 0 . 1 , 0 . 2 , 0 . 3 , and use

the remaining samples as the test set. We use the procedure described in Section 4 to obtain a projection of the training distribution that closely approximates the test distribution. We use two transfer methods: full transfer, where we use all the covariates for the transfer, and partial transfer, where we use only the subset of covariates with which the ATE is most unstable, namely 'age', 'education', and 're75'. We display our results in Figure 3, where we find that both transfer methods lead to lower ATE estimation error than the naive procedure. Further, we observe that there is not much gain with the full transfer method that uses all the covariates over the partial transfer method.

Shift in marginal distribution of a given covariate with respect to KL divergence

500100015002000

Treatmenteffect

0

00S-

0.0

0.2

0.4

0.6

0.8

1.0

Age

Education

Black

Hispanic

Married

Nodegree

Re75

|

τ

(

P

.

)

-

τ

(

P

test

)

|

alpha

5000

4000

arror

3000

Variable

Full transfer

2000

日

自

Training

Partial transfer

1000

0

0.05

0.1

0.2

0.3

0.5

0.7

5.2 Wine Quality data set

We evaluate the effectiveness of our method using the wine quality dataset from the UCI Machine Learning Repository [13, 18]. The dataset includes subgroups of red and white wines, each with 11 chemical properties used as predictors. The response is a continuous quality assessment measured on a scale of 0 to 10 . The dataset consists of 1599 red wines and 4898 white wines. We use all red wines as the training set and randomly select a proportion α from the white wines, where α ∈ { 0 . 01 , 0 . 05 , 0 . 1 } . The remaining observations are used for testing. We include a small proportion of white wines in the red wine training set to ensure that the shifted distribution is absolutely continuous with respect to the training distribution. This step is necessary to avoid making transfer learning very challenging when datasets deviate from this assumption.

Distributional stability of regression coefficients. We use directional s -values to assess the distributional stability of ordinary least-squares regression coefficients. We focus our analysis on the predictors "pH" and "density," although similar results can be obtained for other variables. Figure 4 shows estimates of the minimum and maximum achievable value of a regression coefficient given

a specific shift in the marginal distribution of a covariate (as defined in equation (9)). We observe that the coefficient of "pH" is unstable with respect to shifts in "fixed.acidity," "chlorides," "pH," "sulphates," and "alcohol" ( s X > 0 . 85 ). The coefficient of "density" is unstable with respect to shifts in "volatile.acidity," "total.sulfur.dioxide," "sulphates," and "alcohol" ( s X > 0 . 85) . We present the directional s -values in Tables 2 and 3.

Parameter transfer. We employ Jin and Rothenhäusler [28]'s transfer procedure to estimate the parameter under shifted distributions. This transfer procedure combines a re-weighting step with a bias-correction step for semi-parametrically efficient transfer. We compare three different estimators. The first one uses only the training distribution, the second transfers the parameter using the covariates found to be unstable in the previous step, and the third employs all covariates for transfer (full transfer). Figure 5 displays the estimation error of the parameter under the projected and training distributions. Both transfer learning methods have smaller errors than the naive estimator that uses only the training distribution. However, the full transfer method that employs all covariates does not yield much improvement over partial transfer. For α = 0 . 01 , there is not much enhancement in the coefficient of "density," which might be due to a partial violation of the assumption that the test distribution is absolutely continuous with respect to the training distribution.

f.a = fixed.acidity, v.a = volatile.acidity, c.a = citric.acid, f.so2 = free.sulfur.dioxide, t.so2 = total.sulfure.dioxide, cl = chlorides, so4 = sulphates

f.a = fixed.acidity, v.a = volatile.acidity, c.a = citric.acid, f.so2 = free.sulfur.dioxide, t.so2 = total.sulfure.dioxide, cl = chlorides, so4 = sulphates

6 Discussion

The generalizability and replicability of statistical findings are crucial in scientific research. However, classical statistical measures only account for uncertainty due to sampling and not other sources of variation, such as distributional shift. Since distributions are expected to vary between settings and locations, it is essential to understand how statistical parameters are affected by such shifts to assess the stability of a finding.

In this work, we propose stability measures to quantify the impact of distributional shifts on statistical parameters at an overall and variable-specific level, respectively, enabling a more detailed evaluation of instability. We expect that stability measures will be used in tandem with transfer learning procedures. Initially, the stability of a conclusion can be assessed with respect to various shifts, and then, once the sensitivities are determined, the data scientist can collect data from target distributions for the most sensitive covariates and update the model accordingly. As an example, researchers are often interested in a causal effect estimate for a new location. For some covariates such as age and education, partial data is available via surveys such as American National Election Study (ANES) or Cooperative Election Study (CES). Additional partial data can be cheaply obtained via Amazon Mechanical Turk. However, some covariates are hard to collect, since they require running a study in the new location. Our numerical results show that the proposed approach can help prioritize data collection.

The proposed approach has several limitations. First, we consider worst-case shifts, which can be somewhat pessimistic for situations where the distribution is expected to change due to random

Shift in marginal distribution of a given covariate with respect to KL divergence

β

pH

2

2

0.0

0.2

0.4

0.6

0.8

1.0

fixed.acidity

volatile.acidity

citric.acid

residual.sugar

chlorides

free.sulfur.dioxide

total.sulfur.dioxide

density

pH

sulphates

alcohol

Shift in marginal distribution of a given covariate with respect to KL divergence

β

density

100

F

5

0

-100-50

0.0

0.2

0.4

0.6

0.8

1.0

fixed.acidity

volatile.acidity

citric.acid

residual.sugar

chlorides

free.sulfur.dioxide

total.sulfur.dioxide

density

pH

sulphates

alcohol

|

β

pH

(

P

.

)

-

β

pH

(

P

test

)

|

alpha

(a)

|

β

density

(

P

.

)

-

β

density

(

P

test

)

|

alpha

(b)

2.0

1.5

Variable

1.0

日

白

Training

Full transfer

Partialtransfer

0.010.050.10.20.30.50.7

Variable

200

日

白

Partial transfer

Training

Full transfer

0

0.010.050.10.20.30.50.7

perturbations in background characteristics. Modelling distribution shift as random is an attractive alternative [41, 26]. Secondly, the Kullback-Leibler divergence only allows for certain types of distribution shift. The shifted distribution might have a different support than the training distribution. In this case, it might be more appropriate to model the changes with the total variation distance or the Wasserstein distance. Thirdly, in this paper we focus on distribution shift in the covariates X . There is evidence that distributions shift not only in the covariates X , but also in Y | X [27, 31]. Developing new tools to better understand distributional shifts in Y | X is an exciting research direction.

References

Andersen, P. and Gill, R. (1982). Cox's regression model for counting processes: A large sample study. The Annals of Statistics , 10(4):1100-1120.

Anscombe, F. (1973). Graphs in statistical analysis. The American Statistician , 27(1):17-21.

Barber, R. F., Candès, E. J., Ramdas, A., and Tibshirani, R. J. (2019). Conformal prediction under covariate shift. arXiv:1904.06019 [stat.ME] .

Basu, S., Sussman, J. B., and Hayward, R. A. (2017). Detecting heterogeneous treatment effects to guide personalized blood pressure treatment: a modeling study of randomized clinical trials. Annals of Internal Medicine , 166(5):354-360.

Bertsekas, D. (1999). Nonlinear Programming . Athena Scientific.

Buja, A., Berk, R., Brown, L., George, E., Pitkin, E., Traskin, M., Zhao, L., and Zhang, K. (2019). Models as approximations i: Consequences illustrated with linear regression. arXiv:1404.1578 [stat.ME] .

Cauchois, M., Gupta, S., Ali, A., and Duchi, J. (2020). Robust validation: Confident predictions even when distributions shift. arXiv:2008.04267 [stat.ML] .

Cauchois, M., Gupta, S., Ali, A., and Duchi, J. (2022). Predictive inference with weak supervision. arXiv:2201.08315 [stat.ML] .

Cauchois, M., Gupta, S., and Duchi, J. (2021). Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction. Journal of Machine Learning Research , 22(81):1-42.

Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal , 21(1):C1-C68.

Cinelli, C. and Hazlett, C. (2020). Making sense of sensitivity: Extending omitted variable bias. Journal of the Royal Statistical Society Series B-Statistical Methodology , 82(1):39-67.

Cornfield, J., Haenszel, W., Hammond, E. C., Lilienfeld, A. M., Shimkin, M. B., and Wynder, E. L. (1959). Smoking and lung cancer: Recent evidence and a discussion of some questions. Journal of the National Cancer Institute , 22(1):173-203.

Cortez, P., Cerdeira, A., Almeida, F., Matos, T., and Reis, J. (2009). Modeling wine preferences by data mining from physicochemical properties. In Decision Support Systems , 47(4):547-553.

Dehejia, R. H. and Wahba, S. (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of American Statistical Association , 94(448):10531062.

Devaux, M. and Egami, N. (2022). Quantifying robustness to external validity bias. Available at SSRN:https://ssrn.com/abstract=4213753 .

Ding, P. and VanderWeele, T. J. (2016). Sensitivity analyses without assumptions. Epidemiology , 27(3):368-377.

Donsker, M. and Varadhan, S. (1976). Asymptotic evaluation of certain markov process expectations for large time, iii. Communications on Pure and Applied Mathematics , 29:389-461.

Dua, D. and Graff, C. (2019). Uci machine learning repository.

Duchi, J. C. and Namkoong, H. (2018). Learning models with uniform performance via distributionally robust optimization. arXiv:1810.08750 [stat.ML] .

Esfahani, P. M. and Kuhn, D. (2018). Data-driven distributionally robust optimization using the wasserstein metric: Performance guarantees and tractable reformulations. Mathematical Programming, Series A , 171(1-2):115-166.

Ghorbani, A. and Zou, J. (2019). Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning , pages 2242-2251. PMLR.

Gijsberts, C. M., Groenewegen, K. A., Hoefer, I. E., and Eijkemans, M. J. (2015). Race/ethnic differences in the associations of the framingham risk factors with carotid imt and cardiovascular events. PLoS One , 10(7).

Hainmueller, J. (2012). Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies. Political Analysis , 20(1):25-46.

Huber, P. J. (1981). Robust Statistics . John Wiley and Sons, New York.

Jeong, S. and Namkoong, H. (2020). Robust causal inference under covariate shift via worst-case subpopulation treatment effects. arXiv:2007.02411 [stat.ML] .

Jeong, Y. and Rothenhäusler, D. (2023). Calibrated inference: statistical inference that accounts for both sampling uncertainty and distributional uncertainty. arXiv preprint arXiv:2202.11886 .

Jin, Y., Guo, K., and Rothenhäusler, D. (2023). Diagnosing the role of observable distribution shift in scientific replications. arXiv preprint arXiv:2309.01056 .

Jin, Y. and Rothenhäusler, D. (2023). Tailored inference for finite populations: conditional validity and transfer across distributions. Biometrika , page asad022.

Lalonde, R. (1986). Evaluating the econometric evaluations of training programs. American Economic Review , 76(2):1148 - 1178.

Lange, K. (2013). Optimization . Springer.

Liu, J., Wang, T., Cui, P., and Namkoong, H. (2023). On the need for a language describing distribution shifts: Illustrations on tabular datasets. arXiv preprint arXiv:2307.05284 .

Liu, M., Zhang, Y., and Cai, T. (2020). Doubly robust covariate shift regression with seminonparametric nuisance models. arXiv preprint arXiv:2010.02521 .

Lundberg, S. M. and Lee, S.-I. (2017). A unified approach to interpreting model predictions. Advances in neural information processing systems , 30.

Namkoong, H., Ma, Y., and Glynn, P. W. (2022). Minimax optimal estimation of stability under distribution shift. arXiv:2212.06338 [stat.ML] .

Owen, A. B. (2001). Empirical likelihood . CRC press.

Pan, S. J. and Yang, Q. (2010). A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering , 22(10):1345-1359.

Ramsey, J. (1969). Tests for specification errors in classical linear least squares regression analysis. Journal of the Royal Statistical Society, Series B , 31:350-371.

Robins, J. M., Rotnitzky, A., and Zhao, L. P. (1994). Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association , 89(427):846-866.

Rockafellar, R. T. (1970). Convex Analysis . Princeton University Press.

Rosenbaum, P. R. (1987). Sensitivity analysis for certain permutation inferences in matched observational studies. Biometrika , 74(1):13-26.

Rothenhäusler, D. and Bühlmann, P. (2023+). Distributionally robust and generalizable inference. Statistical Science, to appear .

Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Pyschology , 66(5):688-701.

Shafieezadeh-Abadeh, S., Esfahani, P. M., and Kuhn, D. (2015). Distributionally robust logistic regression. In Advances in Neural Information Processing Systems 28 , pages 1576-1584.

Shimodaira, H. (2000). Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference , 90(2):227-244.

Sinha, A., Namkoong, H., and Duchi, J. (2018). Certifying some distributional robustness with principled adversarial training. In Proceedings of the Sixth International Conference on Learning Representations .

Splawa-Neyman, J., Dabrowska, D. M., and Speed, T. P. (1990). On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9. Statistical Science , 5(4):465 - 472.

Subbaswamy, A., Adams, R., and Saria, S. (2021). Evaluating model robustness to dataset shift. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics .

van der Vaart, A. W. (1998). Asymptotic Statistics . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press.

van der Vaart, A. W. (2002). The statistical work of Lucien Le Cam. The Annals of Statistics , 30(3):631-682.

Wager, S. and Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association , 113(523):1228-1242.

Wen, J., Yu, C.-N., and Greiner, R. (2014). Robust learning under uncertain test distributions: Relating covariate shift to model misspecification. In Proceedings of the 31st International Conference on Machine Learning , pages 631-639.

Yu, B. and Kumbier, K. (2020). Veridical data science. Proceedings of the National Academy of Sciences , 117(8):3920-3929.

Zhao, Q., Small, D. S., and Bhattacharya, B. B. (2019). Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrap. Journal of the Royal Statistical Society Series B: Statistical Methodology , 81(4):735-761.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a metric (s-value) that quantifies the uncertainty of statistical estimators in terms of their distributional instability. In addition, the techniques proposed can quantify the effect of directional shifts and the authors also discuss how the s-value can be used to improve estimation accuracy under shifted distributions.\n\nSTRENGTHS:\nThe development of methods that quantify the effect of distributions shifts is very relevant since such shifts are very common in practice. The overall approach in the paper is of interest and the results in the paper provide a promising initial step towards novel measures of uncertainty that account for distributional shifts.\n\nWEAKNESSES:\nThe interpretation of the s-value in (1) for scalar parameters is somehow clear since it measures the smallest divergence needed to change the sign of the parameter (assuming the parameter continuously depends on the distribution). However, the usefulness of its generalization to the multidimensional case in (11) needs further clarification. In the scalar case, the divergence needed to achieve a zero value reflects the divergence needed to have a change of sign. In the multidimensional case, it is not clear why measuring the divergence needed to achieve a zero vector valued \\eta quantifies the instability. \n\nThe experimental results are not correctly described. It is hard to grasp the main takeaways of such results and the relationship with the theoretical results provided. In general, the paper presentation is rather poor and multiple results appear in the appendices. It is clear that the paper can be significantly improved.\n\nThe contribution of the proposed two-stage approach for transfer learning is hard to quantify. In general, the authors should compare their methods with existing techniques so that the paper's contributions can be better assessed.\n\nQUESTIONS:\nThe definition in (2) is unclear in page 2. The authors should mention there that the set \\mathcal{P} in (2) denotes joint distributions of Z and E, since in that page those distributions correspond to r.v. Z only.\n\nIn line 248 the authors mention \"Our findings show that the average treatment effect is unstable with respect to changes in the marginal distribution of ‘age’, ‘education’, and ‘re75’. We also find that s-values conditional on ‘age’, ‘education’, ‘black’, ‘hispanic’, and ‘re75’ are non-zero, indicating that the average treatment effect can change its sign with a shift in the marginal distribution of these covariates (sX > 0.85).\" Are those results shown in Figure 2? \n\nThe authors should improve the quality of figures that currently is rather poor.\n\nLIMITATIONS:\nThe authors should improve the description of the limitations of the methods proposed as described in the \"weaknesses\" section.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-27"}

comments: SUMMARY:
This work defines a novel statistical measure of the “stability” of a parameter in a distribution with respect to changes in that distribution. From a high level, it is defined as the minimum KL distance to a perturbation that flips the sign of the parameter. A method is given to calculate the s-value for mean parameters and several experiments are performed to show its utility.

STRENGTHS:
The method extends to the directional case. I can see the application of this to average treatment effect to be useful in industry in online experimentation platforms. Strong arguments for the utility of s-value for doing robust transfer learning.

WEAKNESSES:
One weakness is that the calculation of the s-value uses a theorem that is only applicable to mean value parameters. The difficulty is that the definition uses an optimization problem over a set of functions rather than real-valued variables.

QUESTIONS:
Could you extend to more general cases by learning a flexible neural density approximator to P_0, and solving (3) by (constrained) variational inference?

LIMITATIONS:
Even though it’s addressed in the appendix, it seems non-practical to calculate the s-value for parameters other than the mean or functions of the mean.

The paper restricts discussion to distributions that are absolutely continuous with respect to the training distribution, although this is not much of a limitation as it covers most practical cases of interest.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a measure to quantify the instability of a statistical parameter under distribution shifts, which calculates the minimal KL divergence to flip the sign of the estimated parameter. The authors demonstrate its usage in helping to collect target samples in transfer learning. The idea is clear and the theoretical analysis is solid, while the experiments seem a little inadequate.\n\nSTRENGTHS:\n1. The idea makes sense and the paper is easy to follow.\n2. The proposed measure is novel, and the theoretical analysis is solid.\n3. The proposed measure could help to collect data in transfer learning, and the two-stage method is simple but seems efficient.\n\nWEAKNESSES:\nGenerally, I love this paper, but there are some drawbacks that stop me from giving a higher score:\n1. The experiments seem a litter inadequate. I view this paper as technical work, but there are almost no baselines to compare in experiments. There are some recent papers sharing similar ideas (efficiently collecting data in transfer learning), and I think the authors should compare or at least mention them. For example:\n* Data Shapley: Equitable Valuation of Data for Machine Learning. ICML 2019\n* Shapley values for feature selection: The good, the bad, and the axioms\n* Algorithms to estimate Shapley value feature attributions\n* Towards Efficient Data Valuation Based on the Shapley Value\nAlso, maybe the authors could compare with some feature selection methods in experiments.\n\n2. There are some typesetting problems in Figure 4 on page 9. Some contents are covered by figures.\n\n3. The authors measure the instability via the effects on estimation. I wonder whether it is a better way to directly measure the model performance since model performance has already taken into consideration of the estimated parameters. For example, this paper: \"Minimax Optimal Estimation of Stability Under Distribution Shift  Hongseok Namkoong, Yuanzhe Ma, Peter Glynn\". I hope that the authors could discuss these two perspectives more.\n\nQUESTIONS:\nPlease refer to weaknesses.\n\nLIMITATIONS:\nHow could the proposed measure be used in neural networks or in large-scale problems?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFor a given data distribution P0 and family of distributions around it, mathcalP, the authors propose a measure of stability in estimating\na parameter theta(P) when there are distribution shifts within mathcalP.  The idea is that you have data P0, but might really be interested\nin estimating theta for some P' != P0 that you might get when deploying your model. That is, from data P0, know how bad you will\nbe at estimating theta(P'). Sensitivity to changes depend on the parameter of interest, and the idea here is to quantify how much of a shift it\nwould take (within P, from P0) to change the sign of theta(P0). The resulting quantity s(theta, P0), which should maybe be caled s(theta, P0, mathcalP) is called the s-value, which takes values in [0,1] for 1 meaning \"small shifts will change the sign of theta\" to 0 meaning \"the sign is stable\".\n\nThe authors not only study marginal shifts, but also those resulting when one knows that some given conditionals will be fixed. Finally, they describe a two stage procedure for determining if it would be useful to collect test set samples for certain subsets of features (those for which s value is most unstable).\n\nSTRENGTHS:\n- distribution shift is discussed in ML for predictive tasks but less in statistical inference / parameter estimation settings. It's great that the authors are bringing together the two areas, and providing tools for reasoning about inferences in the real world where we do expect shifts\n- Moreover, we don't always care just about absolute shifts, but instead qualitative things, like parameter sign changes, which the authors focus on\n- The authors discuss the conditional case where shift is only limited to some features, which helps us study more specific instances of the problem as well as gain statistical efficiency when we do have some knowledge\n\nWEAKNESSES:\nThe weaknesses are not in the mathematical method nor experiments, but more so in the discussion and contextualization of this work.\n\n- I think the iterative two stage procedure for selecting features to collect test data for is a pretty cool use case, but there is some lacking discussion about when this would actually be feasible in the real world. Please further discuss when one should or should not be able to do this.\n\n- The authors somewhat quickly dismiss influence-function (IF) based estimation in the Related Work, but my strong feeling is that a more thorough discussion of the relationship to this field is necessary. IF based estimation is not just about robustness to outliers, but actually has a fairly deep connection to this work where influence functions show up in the functional derivative of a parameter to be estimated with respect to an underlying distribution. Moreover, just like the authors of this work consider the case of parameter sensitivity conditionally on certain factors of the joint distribution, the IF literature likewise considers projections that describe parameter sensitivity to changes in distributions when some factors are fixed. I admit that the IF literature is somewhat dense, but I believe it would be useful to state some relationships even at high level to the derivatives and Von-Mises (\"distributional taylor\") expansions in e.g. Kennedy's review here: https://arxiv.org/pdf/2203.06469.pdf\n\n---- just some writing comments below ---\n\n- I assume figure 2 is about the data NSW given the legend mentioning demographics, but the figure did not explicitly mention this. The figure caption should mention the name of the dataset to avoid confusion.\n\n- Figure 3 does not mention which data it is about in the caption, and one has to work backwards to the NSW experiment to find out which data the figure is about by searching for the text \"Figure 3\". Please mention the data in the figure caption.\n\n- when you say \"We employ Jin and Rothenhäusler [22]’s transfer procedure to estimate\", it would be helpful to be able to read what the method is from a re-statement in your paper rather than having to open the citation, since it's a detail that is part of your experiments.\n\n- in lines 59-61, \"We discuss how our ... same to re-estimate the parameter under shifted distribution\", it would be correct to change \"under shifted distribution\" to either \"under a shifted distribution\" or \"under shifted distributions\".\n\n- The remark in 63-70 could be moved to the end of work since it sort of interrupts the flow\n\nQUESTIONS:\n- The two-stage procedure involves testing the stability of various subsets of features and seeing if there are some for which it would help to collect samples from the test set. Since it would be usually not possible to do this kind of search in the real world, what would be the practical take-away for how to make use of this in non synthetic studies? Did I understand correctly that testing a given subset Xs requires collecting it from the test set? It's fine to study this ideal case in a research paper, but it seems like it necessitates more discussion.\n\n- NSW experiment: your initial estimator has a std dev of 492 which is large for an effect size of 820. How was the std dev estimated? Is this the expected ATE std dev using this method on this data? \n\n- There is a claim in the paper that \"full transfer\" doesn't provide much gain over \"partial transfer\". This probably needs some more explanation. Is the claim that  using Pproj to estimate theta is as good as using Ptest? This must only be true under some assumptions. Even if these are mentioned implicitly when describing the data, it would be helpful for the authors to restate what might cause this phenomenon. It's fine to report this phenomenon if it does hold on your data, but please contextualize.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces the s-value, a novel metric to measure the stability of statistical parameters. It is defined as the exponential of minus the largest KL-divergence for which the statistical parameter is 0. The smaller the s-value the more stable the parameter is. When the statistical parameter is the mean of a random variable, the s-value is shown to have a simple expression. The paper also introduces the less conservative directional s-value, which constrains the type of distribution shift possible.\n\nSTRENGTHS:\nMany methods exist to find the worst case distribution shift within some radius as a way to obtain a robust estimator. This paper instead is finding the largest radius up to which a parameter is robust, and then use this value to derive a measure of stability. I find this approach novel and I can imagine this paper having some impact in its area. \n\nThe idea behind the paper and the derivations are sound, I also appreciate the exhaustive appendix.\n\nWEAKNESSES:\nI found the experiments hard to interpret, e.g. in figure 4 and 5, what is $\\beta$? Maybe you could improve the legend for those figures to make them easier to interpret. In figure 4, why are the scales so different ([-2,2] vs [-200,100]). It might have been more valuable to dive into a single example and describe it more in depth. Considering the main focus of this paper, I would also suggest showing the s-values for different parameters e.g. in a table. \n\n\nGiven the venue, I was expecting that the estimation of the s-values for parameters defined via risk minimization would be in the main paper. \n\n\nOverall, I appreciate the quality of this paper, but I would have liked to see more clearly how impactful the s-values can be in concrete cases. Achieving this might simply require improving the experiment section to address it to a broader audience.\n\nQUESTIONS:\nHow useful would s-values be for non-convex models? In the appendix, it is mentioned that a small s-value cannot be necessarily interpreted as a proof of stability in that case. What results would you expect to obtain if you were to run your parameter transfer experiment using a non-linear model?\n\nLIMITATIONS:\nI think limitations could be better addressed by e.g. adding a paragraph in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel metric, called the s-value, for evaluating the stability of statistical parameters with respect to distributional shifts. This metric is based on a variational problem involving the KL divergence between the target distribution and the shifted distribution, which can be solved via an equivalent one-dimensional convex problem. The authors also introduce the notion of directional s-value that quantifies the instability of directional shifts. Moreover, consistency and asymptotic normality results are proven for the plug-in estimators of these s-values. Finally, the authors illustrate the interest of the s-values on some real datasets.\n\nSTRENGTHS:\n1. I found the proposal novel and interesting.\n2. Quantifying the stability of statistical findings under distributional shifts is an important question and this work takes a first step towards answering this question.\n3. The paper is well-organized and easy to follow.\n\nWEAKNESSES:\n1. The treatment for the consistency of $\\hat s_E(\\mu, P_n)$ seems to be weak. In particular,\n  (1) Estimating the conditional expectation $E[Z | E]$ is a challenging problem, especially when the dimension of $E$ is large as in Example 4. The uniform convergence assumption (Assumption 1 in Appendix C) seems to be too strong.\n  (2) The rate of convergence of $\\hat f_n(E)$ can be very slow when the dimension is large. It is of interest to know how the rate of convergence of $\\hat s_E(\\mu, P_n)$ depends on the one of $\\hat f_n(E)$.\n2. The practical usefulness of this metric is questionable. In the experiments the authors only computed a few s-values without giving empirical evidence supporting the reasonableness of these numbers. For example, is a statistical parameter with a large s-value really more sensitive to distribution shifts than one with a small s-value? It would be good to perform at least simulation studies to investigate this question.\n\nQUESTIONS:\n1. Can you prove the consistency of $\\hat s_E(\\mu, P_n)$ under a weaker assumption on $\\hat f_n(E)$?\n2. How does the rate of convergence of $\\hat s_E(\\mu, P_n)$ depends on the one of $\\hat f_n(E)$?\n3. Can you provide empirical results supporting the practical usefulness of s-values? For example, is a statistical parameter with a large s-value really more sensitive to distribution shifts than one with a small s-value?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes method to quantify instability of a statistical parameter with respect to pertubrations around the KL divergence ball. This has implications to detect where statistical conclusions no longer hold when there is a distribution shift. The authors show this metric across both overall and directional shifted. The paper provides a two-step transfer learning strategy over two datasets to demonstrate the effectiveness of using stability as a measure of where to collect extra data for transfer learning.\n\nSTRENGTHS:\nTthe paper was well written, with its objectives and motivations clear.  Theorems and mathematical notation were easy to follow. The examples in Section 3.2 illustrated well how this metric is concretely used for different distributions. It is evident that  s-values can be useful in determining when to re-train models or re-estimate statistical queries, depending on the stability of a particular parameter. Detecting under which variables shifts occurs is an important open problem that the authors provide clear insight in, as well as a procedure to use s-values for improved transfer learning.\n\nWEAKNESSES:\nHowever, while an interesting and intuitive idea, it does not seem that it provides significant improvements over a transfer learning approach with all covariates. In Figures 3 and 5, transfer learning outperforms the naive approach, but this is an expected result. What is the reason for a partial transfer if a full transfer works better than or just as well? I would also be interested to see an experiment where a greater amount of data can be collected to improve the transfer learning (aka a higher $\\alpha$ value for the second experiment).\n\nIt would be also helpful to provide recommendations for practitioners for what it means when a parameter is stable (is there a general threshold of $s$ at which transfer learning works well over a parameter works well?) Is the $s_X > 0.85$ threshold as shown in the paper recommended?\n\nQUESTIONS:\nThere are some uncertainties about the experimental methodology. Why were these particular datasets chosen? Something with an average treatment effect seems like it would be useful to be seen in medical datasets for a particular intervention? Would be interested to see if there would be significant improvements over standard transfer learning with all covariates if this procedure was done with datasets with more unstable covariate variables. It would be helpful to see a broader set of experiments, or a more thorough analysis of which covariates are helpful in this case or not.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-28"}