Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper deals with the interesting and important problem of distribution shift. The authors formalize a model (Sparse Joint Shift, SJS) which unifies existing shift models by considering joint shift of both labels and a few features.The authors show conditions for identifiability of models under these shifts, and propose Shift Estimation and Explanation under SJS (SEES), a method to estimate the empirical performance gap and explain it in terms of shifted features. The authors show empirical benefits of SEES compared to BBSE and KLIEP baselines.
strengths_and_weaknesses: Strengths:

The paper is written well, with straightforward assumptions and conclusions, and was enjoyable to read. The problem addressed is important and the provided SJS framework appears to be a useful way to approach this problem. I also appreciate the identifiability condition and the discussion of implications.


Weaknesses:

The empirical measurements are rather shallow, although measured over a few different datasets. All empirical measurements should have robustness measures, but are currently presented as only point estimates. Finally, when claiming seeking as a desired characteristic of the method, it would be good to evaluate the explainability.

questions: - Can you expand the empirical results with robustness measures (e.g. variance over multiple experimental runs)?

- How should we consider extensions of this framework (for instance, when we have access to multiple dataset shifts and possible task representations)?
limitations: This paper is appropriately presented in the context of related work. Limitations of the SJS framework are briefly discussed in the Conclusion but could be discussed more straightforwardly throughout the paper.



===Edit after revision===

After reading the other reviews and responses, I would keep my recommendation of acceptance.

Thank you for reporting the variance in experimental results. These suggest very stable training.

I agree with reviewer 9XVT about comparing the methods across many types of shift and thank the authors for including some of this comparison in the updated appendix. I believe this is an important topic. I am also interested in reviewer 9XVT's question about the sensitivity to sparsity parameter s and it now seems to me that this may limit the model to be used in applications where we can evaluate performance wrt this hyperparameter (eg by cross-validation).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose an interesting model of data shift which only label and a small set of features change between the training and test distributions. They design simple distribution matching algorithms enforcing shift sparsity and show that they work well in sparse shift scenarios. 


strengths_and_weaknesses: - The proposed model of shift is realistic and covers target shift and a part of covariate shift. 
- The proposed algorithms SEES-c and SEES-d are natural for continuous and discrete features, and manage to discover the underlying shifts in the scenarios evaluated in the experiments. 
- For weakness, SEES-d does not scale well with size of shifted feature set s, as it requires searching over all subsets J of size at most s. 
- The authors evaluate mostly on artificially created shift scenarios when they know the ground truth has only a few features shifted. It is not clear how the algorithm would perform in more complicated scenarios, such as medical data collected from different hospitals. 

questions: - Using linear features in SEES-c appear odd as x_k can be negative, making the weight function w(x,y) negative. Should there be extra assumptions? 
- The authors should provide some evaluations when the sparse-shift assumption is not entirely correct, as the sparse shift assumption can be difficult/expensive to verify in practice. For example, what would happen if we assume a shift of 2 features only when there are 4 shifted features? Would the performance of the method degrade gracefully?  

limitations: - The main limitation of the method is knowing when the sparse-shift assumption is true so that the method can be applied. The discrete version SEES-d can also grow computationally expensive as the size of the shifted features s grows.   


==========

After revision, I am keeping my recommendation of weak accept. The authors have addressed most of the questions raised by the reviewers and performed additional experiments.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces Sparse Joint Shift (SJS), a new distribution shift model that detects both label and covariate shifts simultaneously. The authors show how the proposed framework includes existing distribution shifts models and discuss under what assumptions SJS is identifiable. Overall, this paper is very novel and generalize the existing distribution shift approaches in an elegant way.  The authors also introduce a new algorithm able to detect SJS which performs better than the existing distribution shift models. The paper has strong theoretical and empirical discussion and I enjoyed reading it very much.  
strengths_and_weaknesses: Strengths: 1- Novelty, 2- Theoretical foundation, 3- experiments.

Weakness: 1- If space allowed, it would have been refreshing to see experiments on more complex data sets.
questions: To prevent confusion with the source s, I suggest authors use a different letter to denote the size of the set. 
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The performance of ML models may be degraded due to distribution shifts. To handle the covariate shift and label shift, prior work assumes the the type of the shit is known. In this paper, authors propose a joint way to measure covariate and label shift and explain model performance due to shifts without knowing wether its covariate or label shift. First, the paper describes when a joint shift is identifiable rigorously, then proposes an algorithmic framework to measure the shift in terms of importance weights (aka density ratio and likelihood ratio) under the assumption on the sparseness of the joint shift, and use the importance weights to estimate the model performance change. The proposed algorithm is evaluated based on six datasets along with two comparing methods, demonstrating the efficacy of the proposed algorithm. The claimed contributions include (1) proposing a new distribution shift model, i.e., a sparse joint shift (SJS), (2) proposing a general framework for performance shift estimation and explanation under SJS, and (3) demonstrating the efficacy of the proposed approach. 
strengths_and_weaknesses: ### originality
I found that the attacking problem on estimating and explaining model performance under shifts is interesting and timely. Moreover, jointly considering the label and covariate shift under the umbrella of a sparse joint shift (SJS) looks like a novel attempt though the motivation on SJS is weak currently. The algorithm on estimating w(x,y) well exploits known techniques while adding a new component on the sparseness of SJS; but, comparison results are a bit weak. Measuring the performance shift by using the estimated w(x,y) is quite standard. See Questions.

### quality
The paper introduces a new shift, called SJS, which combines label and (sparse) covariate shifts, and proposes an algorithm for SJS, claiming that the proposed approach is good for handling both label and sparse covariate shifts. However, the empirical results do not support this claim well. I expect to see controlled experiments (e.g., experiments on a dataset with only covariate shift, a dataset with only label shift, and a dataset for both) to show the benefit of the proposed approach, but these results are missing. See Questions.

### clarity
The paper is overall well written, though I found a few points that would improve the clarity if fixed (e.g., the definition of sparse covariate shift, motivational examples on s-SJS, whether the identifiable SJS cases are practical, etc.); I’ll clarify more in the Questions section.
 
### significance
I think the crux of this paper is the introduction on SJS and its analysis on identifiability; it would be great if the paper includes clear justification that SJS can be frequently seen in practice and empirical supports that the proposed approach is useful for SJS. 

questions: * (minor) line 122: “s (|I| < s)” is confusing; s is an integer but looks like it is used as a function. I guess it’s better to write “s (i.e., |I| < s)”.

* Definition 1 is introduced in this paper; then real and motivational examples when SJS happens are needed. In particular, what is the real meaning of the invariance (i.e., the equation in line 122)? Is this invariance actually observable in practice?

* It would be better if the definition of the sparse covariate shift is clearly stated; it is partially appeared in line 153 (and it’s kind of understandable based on the context), but having a full definition would be better (e.g., p_s(x_I) \neq p_t(x_I) and p_s(y|x) = p_t(y|x). Moreover, what’s the real world example for the sparse covariate shift? I think this paper introduces many interesting concepts without motivational examples. It may be interesting if the sparse covariate shift can be motivated using adversarial patches. 

* Is the identifiable (p_s, p_t) in Theorem 1 actually observable in practice?  It would be better if we can frequently observe the identifiable distributions in real datasets. 

* The algorithm requires to know the sparsity of SJS, which cannot be known in advance. In this case, the sensitivity of the algorithm performance would be useful to justify the algorithm. Figure 5 embeds partial results on this, but it’s better to see when 0<=s<=d. If the algorithm is quite sensitive to s, what’s the rule of thumb on choosing s?

* The paper claims KLIEP [30] is the state of the art method for estimating the covariate shift; but KLIEP is a quite old paper (i.e, published in 2007), and it works under some assumptions (e.g., a basis (or kernel) function is properly chosen, which is always difficult). In this case, comparing other approaches is required. In particular, there are many papers on “Density Ratio Estimation” [R1], and especially a probabilistic classifier based density ratio estimation is quite simple and still used (e.g., [33] or [R2]). This could be a good baseline for the covariate shift experiments. 


* I believe, to justify the efficacy of the proposed approach for SJS, three controlled experiments are required (at least): when the dataset contains (1) only covariate shift, (2) only label shift, and (3) both (sparse) covariate shift and label shift. For (1), the proposed approach needs to be as good as known density ratio estimators for covariate shift. For (2), the proposed approach needs to be as good as known approaches (e.g., [21]), But for (3), the proposed approach should be better than other shift dedicated approaches. Without these results, the proposed approach may not claim to be effective for both shifts when we don’t know the type of shifts. I believe synthetically generating datasets for (1-3) is not difficult (e.g., use exponential tilting in [33] for covariate shift).

[R1] Density Ratio Estimation in Machine Learning by Masashi Sugiyama and others.
[R2] https://arxiv.org/abs/2003.00343

limitations: I think the authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: The authors study the important problem of distribution shift under a new SJS model. Identifiability results are proved and empirical experiments illustrate the value of the proposed model. During discussion, some concerns on the experiments were addressed. Overall, there was a weak consensus to accept this paper, which I concur with.
confidence: Less certain
Decision: Accept