Estimating and Explaining Model Performance When Both Covariates and Labels Shift
Authors: Lingjiao Chen
Matei Zaharia
James Y. Zou
Venue: NeurIPS 2022 Accept (2022)
Keywords: ML models
data distribution shift
model deployment and monitoring

Abstract: Deployed machine learning (ML) models often encounter new user data that differs from their training data. Therefore, estimating how well a given model might perform on the new data is an important step toward reliable ML applications. This is very challenging, however, as the data distribution can change in flexible ways, and we may not have any labels on the new data, which is often the case in monitoring settings. In this paper, we propose a new distribution shift model, Sparse Joint Shift (SJS), which considers the joint shift of both labels and a few features. This unifies and generalizes several existing shift models including label shift and sparse covariate shift, where only marginal feature or label distribution shifts are considered. We describe mathematical conditions under which SJS is identifiable. We further propose SEES, an algorithmic framework to characterize the distribution shift under SJS and to estimate a model’s performance on new data without any labels. We conduct extensive experiments on several real-world datasets with various ML models. Across different datasets and distribution shifts, SEES achieves significant (up to an order of magnitude) shift estimation error improvements over existing approaches.

Estimating and Explaining Model Performance When Both Covariates and Labels Shift
Lingjiao Chen', Matei Zaharia' James Zoul Department of Computer Sciences, Department of Biomedical Data Science Stanford University
Abstract
Deployed machine learning (ML) models often encounter new user data that differs from their training data. Therefore, estimating how well a given model might perform on the new data is an important step toward reliable ML applications. This is very challenging. however, as the data distribution ways, and we may not have any labels on the new data, which often the case in monitoring settings. In this paper, we propose new distribution shift model, Sparse Joint Shift (SJS), which considers the joint shift of both labels and few features. This unifies and generalizes several existing shift models including label shift and sparse covariate shift, where only marginal feature or label distribution shifts are considered. We describe mathematical conditions under which SJS is identifiable. We further propose SEES, an algorithmic framework to characterize the distribution shift under SJS and to estimate model's performance data without any labels. We conduct extensive experiments on several real-world datasets with various ML models. Across different datasets and distribution shifts, SEES achieves significant (up to an order of magnitude) shift estimation error improvements over existing approaches.
change in flexible
new
Introduction
Deployed machine learning (ML) models often face new data different from their training data. For example, mismatch of deploymenttdeevellpmenn data in geographical locations (21), demographic features [16], and label balance [20] is widely observed and known to affect model performance. Thus, estimating and explaining how model's performance changes on the new data an important step toward reliable ML applications.
Estimating and explaining performance shift is challenging for several reasons, however. One major challenge is that the data distribution might shift flexible ways. Another obstacle is that we often do not have labels the new data, especially in MI monitoring applications. Without any assumption on the distribution shift, it's impossible to estimate how well the model would perform on the unlabeled new data. Previous work often assumes (i) label shift [22], where feature distributions conditional on the labels are fixed, or (ii) covariate shift (32), where label distributions conditional on features stay the same. However, label or covariate shift, and naively applying estimation methods designed for one shift may produce inaccurate assessments [26]. Moreover, labels and features may shift simultaneously in practice, invalidating these common assumptions.
often do not know whether the real data shift is limited to
To tackle the above challenges, here propose (SJS), to consider the joint shift of both labels and a few features. SJS assumes labels and a few features shift, but the remaining features' distribution conditional is fixed. This unifies and generalizes sparse covariate shift and label shift: both of them are SJS, but some SJS is not label sparse covariate shift (Figure Dp. We describe mathematical conditions
distribution shift model, Sparse Joint Shift
the shifted features and labels
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Labeled source data
Unlabeled target data
Sparse Joint Shift
age
loc
wage
chum
age
sex
wage
Squared 12 loss
30
M
CA
100K
Yes
25
F
CA
100K
10
SEES
Label Shift
45
F
WA
150K
No
57
M
WA
200K
BBSE
KLIEP
No Shift
Sparse Covariate Shift
Source accuracy: 92%
SEES
5 Number of samples (1k)
Accuracy shift: -6%
Shift reason: loc & chumn
(a) Sparse joint shift (SJS)
(b) Shift estimation and explanation under SJS
(c) Shift estimation error
Figure 1: Overview of sparse joint shift (SJS). (a) Both label shift and sparse covariate shift are SJS, but SJS contains additional shifts as well. (b) illustrates SEES, a framework for performance shift estimation and explanation under SJS. Given labeled source and unlabeled target data, SEES exploits the joint shift modeled by SJS to estimate the model performance change and explain which factors drive the shift. In this example, the goal is to predict churn. (c) SEES significantly reduces the shift estimation error over existing methods when both labels and covariates shift.
under which SJS is provably identifiable: if the non-shifted features are weakly correlated, then the marginal feature distribution uniquely determines the joint distribution under SJS. This makes it possible to quantify the shift and estimate model performance on new data without any labels.
When SJS occurs: for customer chum prediction. Two years later, churn rate rose in some states but dropped in others, while the distribution of other features given label (churn) and location (state) remained. This shift in customer distribution is a natural SJS and challenging to estimate without labels on the new data.
motivating example. Consider Alice, data scientist who built an ML model
Furthermore, we propose SEES, an algorithmic framework for performance shift estimation and explanation under SJS. SEES exploits correlation shifts between features and labels modeled by SJS to improve performance estimation accuracy over existing methods, such BBSE (22] for label shift and KLIEP 32 for covariate shift (Figure Dp. It uses the identified shifted features and labels as natural explanation of the performance shift. We present extensive empirical evaluation on real world datasets with both simulated SJS and real shifts. Our experiments validate the effectiveness of SEES: the estimation eπrоr of SEES is often order of magnitude smaller than that of previous state-of-the-aa methods. All together, our contributions are:
1. We formulate and study sparse joint shift (SJS), the joint shift of both labels and features. We show how it unifies and generalizes existing shift models and when it is identifiable with unlabeled data.
distribution shift model that considers
2. We propose SEES, general framework for performance shift estimation and explanation under SJS. We design efficient substantiations of SEES for different data types. We also release our "
3. We provide comprehensive empirical analysis of our methods. On real world datasets with natural distribution shifts, we found SEES leads up to 66% performance estimation error reduction over standard approaches.
Related Work. Label shift: Label shift has been observed and studied in various domains, such as epidemiologic (5), economics 233 and data mining [13.. Recently, there is an increasing interest in model evaluation and development under under label shift. For example, 22 study how to quantify label shift and evaluate an ML model's performance under label shift accordingly. [40) gives an algorithm for active learning under label shift. Online adaptation label shift (37] is also shown to be feasible. Label shift can be viewed as a subset of our proposed shift model SJS. Although simpler and easier to estimate, label shift does not capture the joint shift of labels and features. Thus, methods developed for label shift may not work well under SJS.

---

Covariate shift: Covariate shift [30 38 24 is perhaps the most widely adopted assumption in data
distribution shift. Since studied in the seminal work [30], various methods have been proposed to
estimate covariate shift, including KLIEP (3), KMM [5 and IWCV [31 Adaptation to covaraite
shift has been found useful in many applications, such as spam filtering 4 emotion recognition [T9]
and human activity detection
[88]. More recently, covariate shift adaptation is jointly optimized with
model robustness [29]. fairness (28) and conformal prediction [35].
Unsupervised model performance evaluation: Model performance evaluation without labels has received relatively limited attention. Domain-specific models' performance can be estimated via certain statistics, such as confidence score [17]. rotation prediction [9] and feature statistics of the datasets sampled from meta-datase [10] for image recognition. General model evaluation often relies on different assumptions and accessibility [14] 6
12 8 (36. 7 assumee covariate shift and requires users to provide an approximation (slice) of the shifted features, while [6] needs white-box access to the MI. models to train an ensemble as reference. [12] assumes the label distributions are known, while labels can be obtained, [36] proposes an active model evaluation approach. To our knowledge, this is the first paper that explicitly models the joint shift of both labels and a few features with provable identification guarantees. Moreover, we do not require access to side information such as model design or metadata.
needs feature independence structure given the labels. When a small number of
2 Preliminaries and Problem Statement
We start by giving the preliminaries and the problem of estimating and explaining performance shift.
Prediction tasks and ML models. In this paper consider the standard classification task: given a d-dimensional feature vector x E X C Rd from the feature space X, the goal is to predict its associated label y E y in the label space y. Let f(-) Rd + ) denote an ML model designed for such a task. For simplicity, we assume that y = {1, 2,
L}. Given the model's s prediction f(x) and its true label y. its performance is quantified by some loss function E(-, )) A popular choice is the standard 0-1 loss: l(a b) = La=b, which we focus on, but other losses are also applicable.
Joint distribution shift. The training and inference data for ML often come from two different distributions, referred to as source domain and target domain. Here, we consider the general case when the joint distribution vary across the source and target domains, and call this joint distribution shift. Formally, let P., Pt. X' x y + [0, 1] denote the source and target domains, and ps. be their probability density (or mass) functions. Then joint distribution shift means pa (I.
y) + pc
Problem statement. Suppose we are given labeled dataset D. A (<x ")"") from the source distribution P.,. an unlabeled dataset D, _ { from the target distribution P,,, and an MI. model f() predicting the associated label given any feature vector æ. Our goal is estimate how much performance changes from the source domaín to the target domain. More formally, we aim at estimating the performance shift A & E(z,y)=>, [(((())) y)] (a.y)) [((f(x), v)]- This is challenging we do not observe labels the target domain. In many applications, attributing the performance shift to certain features is also desired. Thus, we are also interested in identifying a set of features to explain the performance shift.
3 SJS: A Tractable Unification of Label Shift and Sparse Covariate Shift
At first glance, estimating the performance shift under joint distribution shift without observing labels from target domain seems hopeless: if the marginal feature distributions are identical for both domains, then observing the features alone should give 0 the estimated performance shift. However, the label distribution given any feature on the target domain is arbitrary, and thus the estimated shift can be arbitrarily bad. In other words, joint distribution shift is not identifiable with no target labels.
To mitigate nonidentifiability, it's necessary to make additional assumptions. The most popular assumptions in literature are label shift 22 and covariate shift (30] Label shift assumes that only label distribution may change, but the feature distribution given a label remains, i.e., Ps (2ly) = Pe (xly). On the other hand, covariate shift assumes that feature distribution can shift, but the label
3

---

distribution given the features is fixed, i.., (=(zz) - However, those assumptions disallow simultaneous changes of both features and labels. which often happen in real- woal-world data [2T 27 34 To enable joint feature and label estimation which is tractable, we introduce a subclass of joint distribution shift, Sparse Joint Shift (SIS), as follows.
Definition (Sparse Joint Shift (SJS)). Suppose for an integer m. S and an index set C [d] with size at most (i.e., [4| S m). PT( 31,3) (1 y). Then say the source and target pair (ps. Pt) is under m n-Sparse Joint Shift, m-SJS. Here, I - [d] I. We call I the shift index set.
Roughly speaking, SJS allows both labels and few features to shift, but assumes the remaining features' conditional distribution to stay the same. Section[givve one example when SJS occurs, and more examples and discussions can be found in the appendix. Next, we will study when this assumption allows tractable performance shift estimation. All proofs are left to the appendix.
3.1
When is sparse joint shift identifiable?
Recall that additional assumptions are needed because the general joint distribution shift is not identifiable. However, when = d. m-SJS simply becomes general joint distribution shift. Thus, it is worthy understanding when m-SJS resolves the identifiability issue. To do so, let us first formally introduce the notation of identifiability.
Definition 2 (Identifiable). Suppose the source-target tuple (Ps.Pt) is under m-SJS. (Ps. Pt) is identifiable if and only if for any alternative distribution Pa (a y). if Pa (x) = pi( and EJ C [d]. |T| I m, such that Pa (Ig.) y) pa ( V). then pa Pa y)
y).
The identifiability can be easily interpreted in words: If joint feature and label distribution matches the target feature distribution and satisfies the m-SJS requirement together with the source distribution, it has to be the target distribution. The following statement shows when (Pa, Pt) is identifiable.
Theorem 1. Suppose (ps. Pt) is under m- m-SJS. Assume for any set J [d] |J| S m and any fixed x E X, the probability density (or mass) functions (P. ; = IJ i are linearly independent. Then (ps, (P). is identifiable.
This statement sheds light on why uniquely identifying the target distribution without target label is feasible under sparse joint shift. Roughly speaking, m-SJS requires that given the shifted features and labels, the remaining features' distribution remains the same on both domains. If those remaining features are different enough (linear independence), they can uniquely determine the distribution of the shifted features and labels. We stress that the linear independence is necessary: if it does not hold, then for any mi we can always find some source-target pair (Pe,Pt) Pt) which is not identifiable. Linear independence implicitly requires sparsity: if m > d/2, then J n I can be empty and the linear independence does not hold. In other words, the sparsity is necessary for the shift to be identifiable.
3.2 How does SJS relate to label shift and covariate shift?
natural question is how does SIS relates to standard label shift and covariate shift. To answer this, let us first introduce label and sparse covariate shift formally.
Definition 3. The source and target (Ps.pe) is under Label Shift iff px(xly) pr (x) (yly), and under m-Sparse Covariate Shift iff ps (#1=, y/ = (z () for some index set with size m < d.
Now we are ready to answer the above question.
Theorem 2. If (pe. Pt) is under label shift, then it is also under O-SJS. If (pe, Pc) is under m-sparse covariate shift, then it is also under m-SJS. In addition, there exists (Par (Pa.Pt) under m-SIS such that it is under neither label shift or covariate shift.
There are several takeaways. First, label shift implies SJS without additional requirements. In fact. as certain distribution pairs are under SJS but not label shift, SJS is strictly more general than label shift. Second, SJS also includes sparse covariate shift. When m = SJS completely unifies both label shift and covariate shift, though it is not identifiable. Identifiable SJS, on the other hand, unifies label shift and sparse covariate shift. Finally, SJS also allows shifts not covered by label shift and covaraite shift: the correlation between label and (a set of) features can be shifted.

---

Labeled source samples D, {()*
Empirical Gap Calculator
D Gap
Sparsity-Aware Density Matcher
weight ui v)
Unlabeled target samples D; (w 1=1
Shifted Feature Selector
> Index set
Figure 2: How SEES works. Given labeled source and unlabeled target data, SEES uses sparsity- aware density matcher to learn a weight function w(E y). Next, an empirical gap calculator computes the performance gap by weighing the source samples with the learned v0( y). The shifted feature selector extracts the features on which the weight function depends heavily.
4 Shift Estimation and Explanation under Sparse Joint Shift
Now we present SEES, an algorithmic framework to estimate and explain the performance shift A when the source and target domain is under m-SJS. As shown in Figure 2 it consists of three components: a sparsity-aware density matcher, an empirical gap calculator, and a shifted feature selector. Given the labeled source samples and unlabeled target samples, we first adopt the sparsity- aware density matcher to obtain an estimated ratio of the target and source density functions, denoted by (zz y). Next, the empirical gap calculator responsible via appropriately reweighting source samples with ratio ai Iz y). Finally, the shifted feature selector picks a set of features as the explanation for the shift. We explain each component as follows.
estimate the performance shift A
4.1 Sparsity-aware density matcher
A key component of SEES is the sparsity-aware density matcher. Here we want to find some weight function (xx y) to induce an estimated target distribution P. (x, - w(x y) (z, y). Our goals include (i) small distance between the estimated target distribution and the true target distribution, (ii) m-SJS between the source and the estimated target distributions, and (iii) flexible parameterization of the weight function. To achieve those goals, we propose the following optimization framework
min w(z.:))EW
((pr(x)
L w( v) (2( and w(x, v) depends on at most m features of æ.
(4.1)
s.t. px(x)
Σ y=1
Here, D).,)) is some distance metric that measures the difference between two density functions. We minimize the distance between the induced feature density (x) and the target feature density Pz()). The minimization is not over joint label and feature distributions since target labels not available. The induced feature density function can be casily derived from source density function and the weight function, encoded in the first constraint. m-SJS is enforced by the second constraint: m-SJS means given m features and labels, the distributions of remaining features are fixed across source and the induced domain, which holds if and only if their density ratio w(x y) only depends on those m features. W represents the set of all feasible weight functions. Different parameterization can be easily realized by adopting different W. Assume access density functions pa y) and (e()), and weight function set W containing the true weight w² (æx y) 2 p=(2.y)
(=,.))) One can easily show the above optimization returns the true weight function w' (xr, v) for identifiable m-SJS.
One benefit of the above framework the flexibility of concrete instantiations. Different design choices, including the distance metric D(-..) and the weight parameterization space W, can fit differ- ent feature types, incorporate domain knowledge, and tradeoff different sample and computational complexity. We give two instantiations of the above optimization: SEES-c for continuous features, and SEES-d for discrete features.
SEES-c: SEES for continuous features. For continuous features, we use KL-divergence DæL(-.) (...) as the distance metric, and initialize the parameterization space W by linear combinations of K fixed basis functions, éi (z, y), d2 (). ... Ox (z, v). That is to say, W {w(x, v)(w(())))
5

---

Σ
ok.y 2 (( E ax.yos (z.y) okk 2 03. The last two con- straints ensure w(x, y) pp, y) is valid probability density. Those basis functions encode users' prior knowledge about the shift. A simple choice, for example, is linear functions (when 22: 2 0): setting K = d and (x y) = Zz. To model the dependence on different features, let e, denote all indexes lo such that (+).) depends on feature Z,, and introduce vector B R' such that B, v EnEr Eym1 B.n The feature dependence requirement in Problem 4.1 is equivalent to sparsity constraint |18||o I s. We can relax the O-norm by 1-norm and obtain one instantiation as
I log (y)a
K Σ
I.
max a1,1 (1,, JK..
Ex-P,
Σ y=l
pa
+
k.y
2
k=1
i=1
kEEE
y=1
K any (2, = 1, 1, 2 0
s.l. W
where > O controls the trade-off between sparsity and the KL distance. One benefit of this instantiation is computational efficiency: the constraint is linear in the optimization variables, and the objective is convex. Thus, the problem is convex and can be efficiently solved. The label distribution given feature p. (x) is unknown but can be approximated by the ML model f(-) trained on the source domain. Given finite samples, the expectations can be replaced by their empirical estimation.
SEES-d: SEES for discrete features. Another interesting instantiation exists for discrete features. With no prior knowledge, we parameterize W to include all possible m-SJS: specifically, W contains all tuple (J, wy (z., v)), where index set J C [d], |J] represents the shifted features, and weight function W (x y) only depends on z, and y-
Features only take finite values, so we can view the density (mass) functions as vectors with finite dimensions. Thus, we adopt the squared lg distance, i.e., D(z, 2') > (z; - =1)² measure distance. However, naively measuring & distance between (z) and (x)) leads to a computational complexity exponential in d. Instead, we measure the distance on set of marginal densities: given an index set J. for every index set with size 2s that contains J, denoted by K, we measure the squared & distance between pr (Z f (z)) and p. (2 fz)),, and then aggregate over k. This design leads to the following instantiation of Problem 4.1
I. Σ 9=1
min (2.)(
Σ
Σ
(Tpr( 1)
(wy(E,,)) 1.. (T.. J, s.1.111 = m (4.2)
2m f= Ih. Exx
where Pe (T., f) and pa (T fr f) are short for pT( = In. f(x) f) and (T Ix. f(x)
f.y = 7). respectively.
Compared to the naive approach, the above formulation is much more computationally efficient: the number of parameters in the above objective only polynomial in the feature dimension d. For fixed J, the problem is simply linear regression over the weight w.y (z,, y) and thus can be efficiently solved. In practice, one can estimate Pr (*x, f) and ps f. g) via labeled source and unlabeled target samples, and then solve the empirical version of the above problem. Compared to using -iveeieee,, solving the empirical version produces the correct shifted index set and a weight function close to the true weight w² (z, y) (under mild conditions). This is formally stated as follows.
Suppose (i) the source and targel are under exact m-SJS, (ii) for any set I C [d], [ S m and any & E X, the marginal probability density (or mass) functions (p=((()) ( = Tuut y = i)}²_ are linearly independent, and (ii) w(x, y) is bounded by a constant M. Then there exists some constant e (indepen-
Theorem Consider when all features are discrete, i.e., for each I, E 21.2..... v).
dent of d, n, and nz), such that if v 21, + / 2n < c log + m log u + 2og L + log 1/8.
then with probability 1- 8, (i) the index set j. learned by Problem 4.2 matches the true shift index set I, and (ii) the produced weights w wj(# ( j,y) satisfies [ (z w (x.w)) 5
O ( (v2m log dl + m loggg + 2 2og L + log 1/8
(V 2m,
LM (121))

---

Roughly speaking, this statement ensures that, when source and target sample sizes are large enough, with high probability, the true shift index set can be identified with finite samples, and the error rate of the learned weight function is approximately the inverse of sample sizes' square root.
Comparisons of SEES-c and SEES-d. SEES-d enjoys mathematical guarantees, but SEES-C can be computationally more efficient. In practice, we can discretize continuous features to use SEES-d.
4.2 Empirical gap calculator and shifted feature selector
Now we explain how the other two components of SEES work. The empirical gap calcula- tor computes the performance shift Á via three steps. First, it estimates the source perfor- mance by Ei=1 f(x"'' (1(x""","") """"). Next, it estimates the performance on the induced target dis- tribution. Note that the performance on the induced target domain is f fr (zz f) dxdd = I Úz v)p= (z, y)((I, y)dady = (() [10( ()((z, v)]. Thus, we use the weighted loss on the source samples En w(x i y ))())(("))))) y*)) as the estimation. Finally, their difference, i.c., Å = 1 Ei= (wixx* (*) 1)((=" z/ah)) is returned as the estimated performance shift.
The shifted feature selector picks a set of features as the shift explanation. For discrete data, the weight function úz 2) learned by the density matcher's instantiation is parameterised as a shifted index J and the corresponding weight w ا Thus, a natural choice is to return = J as the explanation. For continuous data, the weight function is (( y) = > 0N ( y). where ās.y is learned by the corresponding instantiation. Recall that e; denotes all basis functions that
depends on feature Then B, - Exee. Σn= a2.y can be viewed as the total contribution of feature i. Thus, simple choice is to pick features with the m largest contributions. Formally, we use
i = {ilB ((d-m)) ) where B (d)-m) is the d - smallest value in B.
5 Experiments
In this section, we study the performance SEES on several real world datasets with synthetic and natural distribution shifts. Our goal is four-fold: (i) understand when and how SEES estimates the performance shift, (ii) evaluate the trade-offs between the estimation performance reached by SEES and the required dataset sizes, (iii) explore the effects of shift sparsity on SEES' performance, and (iv) validate the effectiveness of SEES on datasets with real world distribution shifts.
Datasets, ML models and baselines. Six datasets are used for evaluation purposes. we first simulate various SJS on BANKCHURN M1. COVID-19 (2) and CREDIT [39 to systematically understand the performance of SEES. Next, we apply SEES on EMPLOY, INCOME, and INSUR- ANCE IT] with real world distribution shifts and perform an in-depth analysis. We use a gradient boosting tree model as the ML model, and results for more models can be found in the Appendix. For comparison, we adopt two state-of-the-un methods for comparison: BBSE (22) for label shift and KLIEP [32 for covariate shift. More details the experiments can be found in the Appendix.
Case study. We start with case study on the dataset COVID-19. The task is to predict whether person tests positive for COVID-19. We simulate a joint shift of the feature aged and label. Specifically, both source and target data contain 5000 young and aged individuals. The positive rate is 40% for both young and aged group from the source. In the target data, we raise the positive rate to 80% for aged group and 50% for young group. This simulates a shift due to a COVID variant more harmful to the elder than its ancestor. We adopt SEES-4 -d as all features are categorical.
Figure 3 summarizes this case study. First note that identifying which feature is shifted is not obvious. As shown in (igure3]a), the marginal distribution of most features except age and gender has changed from the source to the target. The actual joint shift (Figure B(b) is, on the other hand, due age group and the labels. Identifying shifted age is challenging as the label on the target (last red bar in (ggure[(a)) also shifts but cannot be observed. On the other hand, SEES-d correctly identifies the shifted feature age, and produces a weight function close to the true weight (Figure ((b)) and (c)). This is primarily because SEES-d explicitly exploits the joint shift modeled by SJS. In fact, SEES-d's performance is significantly better than existing methods. As shown in Figure 3 d), the
7

---

Data distribution (%)
60
Target
Source
40
20
cough
sore throat
fever
hard breath headache
age>60
male
contact risk
label
(a) Marginal feature label distribution
(age)
<0 200
2.00
0.33
Î = (age)
=60
1.98
0.34
Method
MSE
PCC
Est Gap
True Gap
Age
Age
SEES-d
0.002
0.996
16.7
50
BBSE
0.144
0.857
26.3
15.5
w "xx,))>
1.25
0.83
y)
1.33
0.78
KLIEP
0.573
-0.112
1.58
Positive Negative Label (COVID-19)
Positive Neqative Label (COVID-19)
(b) True joint shift
(c) Est shift by SEES-d
(d) Performance of weight acc gap estimation
Figure 3: A case study on the COVID-19 dataset. (a) The marginal distribution of labels and all features. The label on the target domain (the last red bar) is not observable. (b) The actual joint shift between source and target data. (c) The mean square error (MSE), pearson correlation coefficient (PCC) between learned and true weights, and the the estimated accuracy gap. Overall, SEES-d significantly improves estimation performance over existing methods.
mean square error (MSE) between the true weights and learned weights is only 0.002 when adopting SEES-d -d but 144 and 0.573 when using BBSE and KLIEP, respectively. The Pearson correlation coefficient (PCC) between the weights and weights learned SEES-d is 0.996, indicating strong correlation. The weight estimation performance directly affects how precise the estimated accuracy gap is. The estimated gap Á of SEES-d is 16.7%, which is close to the true gap (15.5%). By contrast, BBSE tends overestimate the gap (26.3$) while KLIEP underestimates it (1.58%).
Trade-offss between estimation performance and sample size. Next, we study the trade-offss between estimation performance and the number of available samples. For simplicity, we simulate various sparse joint shifts with s = 1 via (i) specifying marginal distribution of the shifted feature and labels first, and (ii) then drawing random samples from the original dataset conditional on values of specified labels and shifted feature. Same number of samples are allocated to both source and target datasets. Figure shows the simulated data shift (column 1), the squared &a loss of accuracy estimation (column 2), weight estimation (column 3), and the shifted feature discovery rate (column 4) for three datasets. Overall, we observe that the estimation error of SEES-c and SEES-d diminishes the number of samples increases, while that of BBSE and KLIEP is almost flat.
Table 1: Root mean square error of estimated accuracy gap (%) under real shifts for a gradient boosting model. The numbers averaged over all source-target pairs in each dataset. Results for other models (e.g., a neural network) can be found in the Appendix. For each dataset and ML model, SEES provides significant estimation error reduction baselines.
EMPLOY
INCOME
INSURANCE
SEESS-
SEES-J
BBSB
KLIEP
SEESS-
SEES-d
KLIEP
SELS-c
SEES-d
3.00
5.20
5.20
2.40
3.40
1.70
2.20
Effects of shift sparsity. We have focused on 1-SJS for simplicity, but how the sparsity affects the estimation performance remains unknown. To answer this, we fix the number of samples to be 10,000, and then measure the performance of compared methods for joint distribution of label and different number of features. Overall, we observe that the estimation error often grows more features shift jointly with the labels. This because more shifted features often implies higher complexity in the distribution shifts and thus more parameters to estimate. is worthy noting that

---

0.5
target source
SEES-c
10
1.0
0.4
Squared Iona
SEES-d
BESE
03
0.2
0.2
0.0
FR
ESS
DE
DE
5400
10000
5000
18690
0.0
50.00
10000
Yea
samples
samplex.
exmplex
(a) data shift
(b) estimated accuracy
(c) estimated weights
(d) shift discovery rate
0.4
diecoarry rila
0.3
Squaree & lova
Fraction
*
20
0.6
0.4
0.1
8.2
0.:3
00 Py -4000 401y0
5000 Number
50.00
Number
Number
(e) data shift
(I) estimated accuracy
estimated weights
(h) shift discovery rate
0.20
Squared distance
10
Shit foatur disevoory rate
Foction
0.16
Squared
to"
0.50
10"
0.85
0.2
0.00
mar
mar
oth
5000
19090
50.00 samplex
10000
0.0
5000 samplex
10000
samplax
(1) data shift
estimated accuracy
(k) estimated weights
(1) shift discovery rate
Figure 4: Trade-offs between shift estimation and sample size. vary the shifted features (first column), and measure performance of the estimated accuracy shift (second column), estimated weights (third column), and how often the true shifted features are discovered (last column). The first, second, and third row corresponds to dataset BANKCHURN, COVID-19, 19, and CREDIT, respectively. Overall, both SEES-c and SEES-d consistently outperforms existing estimation approaches.
O-SJS degenerates to label shift, and the performance of SEES-c and SEES-d is slightly worse than BBSE (designed for label shift) under O-SJS. More details can be found in the appendix.
Accuracy gap estimation on real shifts. Finally we validate the effectiveness of SEES on accuracy estimation with real world shifts. EMPLOY and INCOME are partitioned by geography (states) and INSURANCE is partitioned by time (year). For each partition pair, we train gradient boosting model on one and estimate its performance on the other. Table Table[[shows shows the estimation error averaged over all partition pairs for each dataset. SEES consistently outperforms BBSE and KLIEP, and reduces the estimation error by to 66% (1-1.7/5.0), We also evaluate other models (including neural network) and observe similar results. More results can be found the Appendix.
Conclusion
In this paper, we propose Sparse Joint Shift (SJS), new distribution shift model that accounts for both label and covariate shifts. We show how SJS unifies and generalizes existing distribution shift models and remains identifiable under reasonable assumptions. We develop SEES. an algorithmic framework for unsupervised model performance estimation and explanation under SJS. Both theoretical analysis and empirical study validates the effectiveness of SEES. Our work contributes to making ML more reliable when data can change. A natural next step is how to improve estimation performance under stimulate more research on SJS, we wwwm/stantattoooooooooooetwwwt.t staaftaaaooo wowoddduueeaaaa////oooooa
SJS when small number of target labels can be queried. htt.uureeaaaa..ppe...oossiiaiaae also release code in https: : httggttttbb.... com/

---

Acknowledgement
This work was supported in part by Google PhD Fellowship. Sloan Fellowship, NSF CCF 1763191, NSF CAREER AWARD 1651570 and 1942926, NIH P30AG059307, NIH U0IMH098953, grants from the Chan-Zuckerberg Initiative, Sutherland, and affiliate members and other supporters of the Stanford DAWN project, including Meta, Google, and VMware. We also thank anonymous reviewers for helpful discussion and feedback.
10

---

References
The BANKCHURN
/wwp wwwp:/aaggee.com/ooeeee... bank- www.tmuttteeeeeeee churn predictionnnnteebox.. [Accessed 2022].
dataset.
gov. [2] The COVID-19 government dataset. http://////aaaaa.o../aaa////////
[Accessed 2022].
[3] Amr Alexandari, Anshul Kundaje, and Avanti Shrikumar. Maximum likelihood with bias-corrected calibration is hard-to-beat label shift adaptation. In International Conference on Machine Learning. pages 222-232. PMLR, 2020.
Steffen Bickel and Tobias Scheffer. Dirichlet-chhnneed spam filtering based on biased samples. Advances in neural information processing systems, 19, 2006.
[5] Alfred Buck and John J Gart. Comparison of a screening test and a reference test in epidemiologic studies: I. indices of agreement and their relation to prevalence. American Journal of Epidemiology. 83(3):586-53)) 1966.
Jiefeng Chen, Frederick Liu, Besim Avei, Xi Wu, Yingyu Liang, and Somesh Jha. Detecting errors and estimating accuracy on unlabeled data with self-training ensembles. Advances in Neural Information Processing Systems, 34, 2021.
[7] Mayee Chen, Karan Goel, Nimit S Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Ré. Mandoline: Model evaluation under distribution shift. In International Conference Machine Learning, pages 1617-1629 PMLR, 2021.
[8] Ching- Yao Chuang, Antonio Torralba, and Stefanie Jegelka. Estimating generalization under distribution shifts via domain-invariant representations. arXiv preprint urXiv:2007.0351,, 2020.
[9] Weijian Deng, Stephen Gould, and Liang Zheng. What does rotation prediction tell us about classifier accuracy under varying testing environments? In International Conference Machine Learning, pages 2579-2589. PMLR, 2021.
[10] Weijian Deng Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pages 15069- 15078, 2021.
[11] Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, 34, 2021.
[12] Pinar Donmez, Guy Lebanon, and Krishnakumar Balasubramanian. Unsupervised supervised learning i: Estimating classification and regression error without labels. Journal of Machine Learning Research, 11(4), 2010.
[13] George Forman. Quantifying counts and costs via classification. Data Mining and Knowledge Discovery. 17(2):164-206, 2008.
[14] Saurabh Garg, Sivaraman Balakrishnan, Zachary Chase Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict
performance. ICLR, 2022.
[15] Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Schölkopf. Covariate shift by kemnel mean matching. Dataset shift in machine learning, 3(4):5, 2009.
[16] Patrick Grother, Mei Ngan, and Kayee Hanaoka. Face recognition vendor test part 3: Demographic effects, 2019-12-19 2019.
[17] Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1134 1144, 2021.
[18] Hirotaka Hachiya, Masashi Sugiyama, and Naonori Ueda. lmporteaceeweiggheed lcast-squares probabilistic classifier for covariate shift adaptation with application human activity recognition. Neurocomputing, 80:93-101, 2012.
[19] Suwicha Jirayucharoensak, Setha -gnu, and Pasin Israsena. Eeg- Eeg-based emotion recognition using deep learning network with principal component based covariate shift adaptation. The Scientific World Journal, 2014, 2014.

---

[20] Harsurinder Kaur. Husanbir Singh Pannu, and Avleen Kaur Malhi. A systematic review on imbalanced data challenges in machine learning: Applications and solutions. ACM Comput. Surv., 544) 2019.
[21] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang. Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in- the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664, PMLR, 2021.
[22] Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. International conference on machine learning, pages 3122-3130. PMLR, 2018.
[23] Charles F Manski and Steven R Lerman. The estimation of choice probabilities from choice based samples. Econometrica: Journal of the Econometric Society, pages 1977-1988, 1977.
[24] Jose G Moreno-Torres Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V Chawla, and Francisco Herrera. A unifying view on dataset shift in classification. Pattern recognition, 45(1):521-530, 2012.
[25] Sangdon Park, Osbert Bastani, James Weimer, and Insup Lee. Calibrated prediction with covariate shift via unsupervised domain adaptation. International Conference on Artificial Intelligence and Statistics, pages 3219-333330 3229, PMLR, 2020.
[26] Stephan Rabanser, Stephan Günnemann, and Zachary Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. Advances in Neurul Information Processing Systems. 32, 2019.
[27] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389-5400. PMLR, 2019.
[28] Ashkan Rezaci, Anqi Liu, Omid Memarrast, and Brian D Zichart. Robust faimess under covariate shift. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9419-9427, 2021.
[29] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems, 33:1 13311559-11511 2020.
[30] Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihon likellkeliood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
[31] Masashi Sugiyama, Matthias Krauledat, and Klaus-Rober Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8(5), 2007.
[32] Masashi Sugiyama. Shinichi Nakajima, Hisashi Kashima, Paul Buenau, and Motoaki Kawanabe. Direct importance estimation with model selection and its application covariate shift adaptation. Advances in neural information processing systems, 20, 2007.
[33] Masashi Sugiyama, Taiji Suzuki, Shinichi Nakajima, Hisashi Kashima, Paul von Bünau, and Motoaki Kawanabe. Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statistical Mathematics, 60(4):699-744) 2008.
[34] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 13:18583-18599, 2020.
[35] Ryan Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. Advances in neural information processing systems, 32, 2019.
[36] Peter Welinder, Max Welling, and Pietro Perona. A lazy man's 's approach to henchmarking: Semisupervised classifier evaluation and recalibration. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, pages 3262-3269, 2013.
[37] Ruihan Wu, Chuan Guo, Yi Su, and Kilian Q Weinberger. Online adaptation to label distribution shift. Advances in Neural Information Processing Systems, 34, 2021.
[38] Makoto Yamada, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, and Masashi Sugiyama. Relative density-ratio estimation for robust distribution comparison. Advances in neural information processing processin systems, 24, 2011.
12

---

[39] 1-Cheng Yeh and Che-hui Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert systems with applications, 36(2):2473-2480) 2009.
[40] Eric 7hao, Angi Liu, Animashree Anandkumar, and Yisong Yue. Active learning under label shift. In International Conference Artificial Intelligence and Statistics, pages 2412-3420. PMLR, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]. See Appendix A
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix[A] [
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes].
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes].
(b) Did you include complete proofs of all theoretical results? [Yes] .See Appendix
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]. Please see Appendix D
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Please see Appendix D
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes].
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A].
13


Reviews:
[{"review_id": "o6o11tnV0mB", "reviewer": "Reviewer_wUbV", "summary": "The authors propose an interesting model of data shift which only label and a small set of features change between the training and test distributions. They design simple distribution matching algorithms enforcing shift sparsity and show that they work well in sparse shift scenarios. \n\n", "questions": "- Using linear features in SEES-c appear odd as x_k can be negative, making the weight function w(x,y) negative. Should there be extra assumptions? \n- The authors should provide some evaluations when the sparse-shift assumption is not entirely correct, as the sparse shift assumption can be difficult/expensive to verify in practice. For example, what would happen if we assume a shift of 2 features only when there are 4 shifted features? Would the performance of the method degrade gracefully?  \n", "limitations": "- The main limitation of the method is knowing when the sparse-shift assumption is true so that the method can be applied. The discrete version SEES-d can also grow computationally expensive as the size of the shifted features s grows.   \n\n\n==========\n\nAfter revision, I am keeping my recommendation of weak accept. The authors have addressed most of the questions raised by the reviewers and performed additional experiments. \n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The proposed model of shift is realistic and covers target shift and a part of covariate shift. \n- The proposed algorithms SEES-c and SEES-d are natural for continuous and discrete features, and manage to discover the underlying shifts in the scenarios evaluated in the experiments. \n- For weakness, SEES-d does not scale well with size of shifted feature set s, as it requires searching over all subsets J of size at most s. \n- The authors evaluate mostly on artificially created shift scenarios when they know the ground truth has only a few features shifted. It is not clear how the algorithm would perform in more complicated scenarios, such as medical data collected from different hospitals. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_aTdShEtStJ", "reviewer": "Reviewer_9XVT", "summary": "The performance of ML models may be degraded due to distribution shifts. To handle the covariate shift and label shift, prior work assumes the the type of the shit is known. In this paper, authors propose a joint way to measure covariate and label shift and explain model performance due to shifts without knowing wether its covariate or label shift. First, the paper describes when a joint shift is identifiable rigorously, then proposes an algorithmic framework to measure the shift in terms of importance weights (aka density ratio and likelihood ratio) under the assumption on the sparseness of the joint shift, and use the importance weights to estimate the model performance change. The proposed algorithm is evaluated based on six datasets along with two comparing methods, demonstrating the efficacy of the proposed algorithm. The claimed contributions include (1) proposing a new distribution shift model, i.e., a sparse joint shift (SJS), (2) proposing a general framework for performance shift estimation and explanation under SJS, and (3) demonstrating the efficacy of the proposed approach. ", "questions": "* (minor) line 122: “s (|I| < s)” is confusing; s is an integer but looks like it is used as a function. I guess it’s better to write “s (i.e., |I| < s)”.\n\n* Definition 1 is introduced in this paper; then real and motivational examples when SJS happens are needed. In particular, what is the real meaning of the invariance (i.e., the equation in line 122)? Is this invariance actually observable in practice?\n\n* It would be better if the definition of the sparse covariate shift is clearly stated; it is partially appeared in line 153 (and it’s kind of understandable based on the context), but having a full definition would be better (e.g., p_s(x_I) \\neq p_t(x_I) and p_s(y|x) = p_t(y|x). Moreover, what’s the real world example for the sparse covariate shift? I think this paper introduces many interesting concepts without motivational examples. It may be interesting if the sparse covariate shift can be motivated using adversarial patches. \n\n* Is the identifiable (p_s, p_t) in Theorem 1 actually observable in practice?  It would be better if we can frequently observe the identifiable distributions in real datasets. \n\n* The algorithm requires to know the sparsity of SJS, which cannot be known in advance. In this case, the sensitivity of the algorithm performance would be useful to justify the algorithm. Figure 5 embeds partial results on this, but it’s better to see when 0<=s<=d. If the algorithm is quite sensitive to s, what’s the rule of thumb on choosing s?\n\n* The paper claims KLIEP [30] is the state of the art method for estimating the covariate shift; but KLIEP is a quite old paper (i.e, published in 2007), and it works under some assumptions (e.g., a basis (or kernel) function is properly chosen, which is always difficult). In this case, comparing other approaches is required. In particular, there are many papers on “Density Ratio Estimation” [R1], and especially a probabilistic classifier based density ratio estimation is quite simple and still used (e.g., [33] or [R2]). This could be a good baseline for the covariate shift experiments. \n\n\n* I believe, to justify the efficacy of the proposed approach for SJS, three controlled experiments are required (at least): when the dataset contains (1) only covariate shift, (2) only label shift, and (3) both (sparse) covariate shift and label shift. For (1), the proposed approach needs to be as good as known density ratio estimators for covariate shift. For (2), the proposed approach needs to be as good as known approaches (e.g., [21]), But for (3), the proposed approach should be better than other shift dedicated approaches. Without these results, the proposed approach may not claim to be effective for both shifts when we don’t know the type of shifts. I believe synthetically generating datasets for (1-3) is not difficult (e.g., use exponential tilting in [33] for covariate shift).\n\n[R1] Density Ratio Estimation in Machine Learning by Masashi Sugiyama and others.\n[R2] https://arxiv.org/abs/2003.00343\n", "limitations": "I think the authors adequately addressed the limitations and potential negative societal impact of their work. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### originality\nI found that the attacking problem on estimating and explaining model performance under shifts is interesting and timely. Moreover, jointly considering the label and covariate shift under the umbrella of a sparse joint shift (SJS) looks like a novel attempt though the motivation on SJS is weak currently. The algorithm on estimating w(x,y) well exploits known techniques while adding a new component on the sparseness of SJS; but, comparison results are a bit weak. Measuring the performance shift by using the estimated w(x,y) is quite standard. See Questions.\n\n### quality\nThe paper introduces a new shift, called SJS, which combines label and (sparse) covariate shifts, and proposes an algorithm for SJS, claiming that the proposed approach is good for handling both label and sparse covariate shifts. However, the empirical results do not support this claim well. I expect to see controlled experiments (e.g., experiments on a dataset with only covariate shift, a dataset with only label shift, and a dataset for both) to show the benefit of the proposed approach, but these results are missing. See Questions.\n\n### clarity\nThe paper is overall well written, though I found a few points that would improve the clarity if fixed (e.g., the definition of sparse covariate shift, motivational examples on s-SJS, whether the identifiable SJS cases are practical, etc.); I’ll clarify more in the Questions section.\n \n### significance\nI think the crux of this paper is the introduction on SJS and its analysis on identifiability; it would be great if the paper includes clear justification that SJS can be frequently seen in practice and empirical supports that the proposed approach is useful for SJS. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CXRKpmiTQpz", "reviewer": "Reviewer_8kzf", "summary": "This paper deals with the interesting and important problem of distribution shift. The authors formalize a model (Sparse Joint Shift, SJS) which unifies existing shift models by considering joint shift of both labels and a few features.The authors show conditions for identifiability of models under these shifts, and propose Shift Estimation and Explanation under SJS (SEES), a method to estimate the empirical performance gap and explain it in terms of shifted features. The authors show empirical benefits of SEES compared to BBSE and KLIEP baselines.", "questions": "- Can you expand the empirical results with robustness measures (e.g. variance over multiple experimental runs)?\n\n- How should we consider extensions of this framework (for instance, when we have access to multiple dataset shifts and possible task representations)?", "limitations": "This paper is appropriately presented in the context of related work. Limitations of the SJS framework are briefly discussed in the Conclusion but could be discussed more straightforwardly throughout the paper.\n\n\n\n===Edit after revision===\n\nAfter reading the other reviews and responses, I would keep my recommendation of acceptance.\n\nThank you for reporting the variance in experimental results. These suggest very stable training.\n\nI agree with reviewer 9XVT about comparing the methods across many types of shift and thank the authors for including some of this comparison in the updated appendix. I believe this is an important topic. I am also interested in reviewer 9XVT's question about the sensitivity to sparsity parameter s and it now seems to me that this may limit the model to be used in applications where we can evaluate performance wrt this hyperparameter (eg by cross-validation).", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is written well, with straightforward assumptions and conclusions, and was enjoyable to read. The problem addressed is important and the provided SJS framework appears to be a useful way to approach this problem. I also appreciate the identifiability condition and the discussion of implications.\n\n\nWeaknesses:\n\nThe empirical measurements are rather shallow, although measured over a few different datasets. All empirical measurements should have robustness measures, but are currently presented as only point estimates. Finally, when claiming seeking as a desired characteristic of the method, it would be good to evaluate the explainability.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "54nPj1Q9xg", "reviewer": "Reviewer_erZL", "summary": "This paper introduces Sparse Joint Shift (SJS), a new distribution shift model that detects both label and covariate shifts simultaneously. The authors show how the proposed framework includes existing distribution shifts models and discuss under what assumptions SJS is identifiable. Overall, this paper is very novel and generalize the existing distribution shift approaches in an elegant way.  The authors also introduce a new algorithm able to detect SJS which performs better than the existing distribution shift models. The paper has strong theoretical and empirical discussion and I enjoyed reading it very much.  ", "questions": "To prevent confusion with the source s, I suggest authors use a different letter to denote the size of the set. ", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: 1- Novelty, 2- Theoretical foundation, 3- experiments.\n\nWeakness: 1- If space allowed, it would have been refreshing to see experiments on more complex data sets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept