Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The work presents a framework for alerting when the risk of a deployed model exceeds pre-defined levels. Observing a sequence of target instances, coming from possibly different distribution than source one, it defines two hypothesis tests (based on additive and multiplicative error) for harmful changes to the risk. It reduces the problem to sequential estimation of bounds for the target risk and uses recent work on time-uniform confidence sequences to design valid hypothesis testing methods. Empirical performance on a controlled synthetic setting and real world data from MNIST, CIFAR show that the methods detects harmful changes to risk in reasonable samples.
main_review: The work highlights an important problem of alerting when model performance differs from that in training, statistically significantly. The framework is broadly applicable to any model and multiple loss functions. Multiple issues in monitoring of models are addressed such as distribution shifts, sequential testing, different loss function. The solution uses some recent work on non-parametric, non-asymptotic testing in a novel way and has good performance under comprehensive set of experiments.

The main weakness is the presentation. The resolution of issues with past work, such as shifts and sequential testing, is not described well. Technical details on computing the confidence bounds are missing. Apart from these details, the framework and the experiments are described well.

---

## Questions to address in the response

1. What is the difference in the method between the two settings i.i.d. and independent test examples?
2. What type of distribution shifts such as covariate, label shift can be addressed?
3. Please comment on the utility of testing the running risk? As the model decisions have already been made, the running risk tells the risk in the (recent) past and not the imminent risk. If the distribution shifts considerably, then the imminent risk can be considerably different. In what scenarios (or some smoothness assumptions on the drift) can such a test for running risk help in giving early warnings? I feel elaborating on this detail gives a better understanding on when to use such methods.
4. What can be included to describe the method in more detail? Please see suggestions below. 
5. Does the alternate methods (EB and Hoeffding based ones) provide valid intervals in the considered setting of shifts and sequential testing? How would we expect such methods to perform, too loose or incorrect confidence intervals?
6. How to choose between betting and conjugate mixture empirical-Bernstein methods for confidence bounds for source and target risk? Both are used in real data experiments, one for source and one for target. My understanding is both provide symmetric intervals and thus are applicable.

---

## Suggestions

The use of time-uniform confidence sequences to resolve issues with past works should be explained in more detail for readers not familiar with advantages of such confidence sequences. 

The description of the exact testing procedures used needs to be self-contained including how to find upper and lower confidence bounds for the risk in terms of a given model, loss, and confidence parameter \delta. 

The context for introducing different loss functions at the start of Section 2 is not clear from the Introduction, which mostly talks about sequential testing without much attention to different loss functions. The discussion on loss functions is not related back to the problem or the method.

Please add the takeaway from Figure 4b. Should we expect other methods to take more samples to detect the harmful shift?

Results for Figure 4a seems to show that the betting based bounds are tighter only for small number of samples and about the same for otherwise. The discussion on the Figure can be made more specific, instead of saying that it is tighter in general.

The contributions of the work can be outlined at the end of Introduction so that reader has some context.

The necessity of time-uniform confidence sequences can be elaborated for readers not familiar with multiple testing under dependent data. Consider highlighting the issue with using traditional tests.

Please briefly describe what is \beta used in Section 3.2 to output prediction sets in the main text. It is fine to refer to the text in the Appendix for details.
 
Clarify if the method works for any loss function or the theoretical results need some conditions on the loss function.

Describe the full-form of PM-EB and the formulae for the bound.

In the expression for running risk before Eq (6), clarify what is the distribution for the expectation.

It will be worth mentioning (if not already) that instead of comparing against the upper confidence bound for source risk, one can use a given risk threshold deemed to be harmful and the method still applies.

The line is disconnected from the neighbouring sentences - If accuracy is a guiding performance metric, one would rarely deploy models performing only sightly better than prediction by chance.
summary_of_the_review: The work addresses an important problem for monitoring models after deployment for harmful changes and derives principled hypothesis tests. Empirical study sufficiently validates the method on real datasets. My main concern is the presentation of the method. It lacks details on related to implementation and guarantees for the methods for readers who are unfamiliar with past work (Waudby-Smith and Ramdas 2021, and Howard et al. 2021). The work provides provides a strong contribution and I feel, subject to the response, the presentation can be improved.

---
Based on the response

The presentation of the methods has been substantially improved. In particular, the contributions paragraph at the end of Introduction gives a good overview of the work. I would encourage authors to more prominently list the confidence bound variants (betting-based, PM-H, PM-EB, CM-EB, Hoeffding) in Sec 2.2 to more clearly layout the options to implement the tests.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper develops a tool for testing online whether the performance of a model on the test data becomes significantly worse than the performance on the training data, which allows differentiating between benign and harmful shifts. The proposed framework is based on sequential testing for a significant risk increase. The theoretical guarantee of the type I error control is provided. The paper also gives simulation and real data experiments to demonstrate the efficacy of the proposed algorithm. 
main_review: Strengths: The paper applies sequential testing to detect harmful distribution shifts. The setting is quite different from existing work. 

Weaknesses: 
Writing. The writing quality and the presentation of the paper can be substantially improved. For example, in section 2, It might be better to move the loss functions to the experimental section, as they are not part of the algorithm. 

Novelty. Overall, I don't see enough originality, and it's not very challenging to adapt the sequential testing to this setting. The main idea is to detect the bad distribution shifts by nonparametric sequential testing based on comparing the risk function on the target data and the risk function on the source data. The monitoring statistics are the risk functions (and their CI). 

Comments. 1. The proposed method requires the labels from the test data, which is a restrictive setting for practical use. The paper claims that the label can be revealed in a delayed fashion, but we could also do batch detection for the past observations (and wait for a period and redo the testing). The paper doesn't provide any expected sample size analysis. I think it'd be better to give some comparison with the batch detection algorithm.  2. Since the algorithm requires to compute the CI at each time step, how efficient is it?
summary_of_the_review: I'd recommend reject at this point.

-----------------------------------------------------
Thank the authors for the response! The authors substantially improved the writing and the presentation in the updated version. They also added new content regarding comparison with other papers/other potential methods, which makes the contribution more clear. I read reviews from other reviewers as well as the corresponding responses. The answer clears most of my concerns. Although I think the theoretical contribution is not significant, adapting sequential estimation to the practical setting would be nice.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a sequential testing schema for identifying malignant distribution shifts that indicate a necessary update such as retraining. The framework presented is designed to be generalizable and applicable to many scenarios (binary, multiclass, set functions, etc.).
main_review: $\textbf{Strengths}$

Detailed and clear with proper explanations and proofs of the presented schema and related propositions.

Builds on related work in the field with (to the best of my knowledge) a novel contribution.

Discussion of relevant distribution shifts and their effect on classifiers is really interesting (see weaknesses).

$\textbf{Weaknesses}$

Minor grammatical/style issues. For example, on page 3 the $\textbf{Multiclass losses}$ sub-header has a period, but other similar ones do not. I suggest adding a period to all of them for consistency. There are also other grammatical issues such as on page 3: "those naturally arise in multilabel classification, where more than a single label can be the correct one, or as a result of post-processing point predictors, which could target...." where "those" is a vague pronoun reference and the use of commas is incorrect.

Although the contribution is novel (to my knowledge), I think that the paper is missing an explicit discussion as to why this contribution is important (e.g. a particular case where this new schema enables users to prevent something that they otherwise could not and being unable to has negative repercussions). 

Although the discussion of relevant distribution shifts and their effect on classifiers is really interesting, it left me wanting more. I think the paper could benefit from expressing how the testing schema presented can be used to identify malignant shifts and how to do so (e.g. humans thing translation wouldn't be an issue but it is, how do we discover more facts like this provable?).
summary_of_the_review: I give the paper a 6 because I find the contribution novel and clear and would be happy to see it accepted in the conference; however, I would not champion pushing to accept given my belief that there is room for significant improvement.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors have proposed a framework for detecting dataset shift (under i.i.d assumption or via drift) such that the model’s performance suffers a significant (defined by the user) performance degrade. The framework accommodates several types of risk functions that can be used to evaluate the model’s performance. The underlying engine of the framework is sequential testing via non-parametric sequential estimation. Such a framework can potentially be useful for raising alarms or indicating when the model needs to be retrained. 
main_review: The proposed framework leverages well-established results in statistics to detect dataset shifts. While the framework addresses the problem in the most practical settings (sequential), the points of comparison with the existing frameworks were not made rigorous enough for the reader. Thus, I have the following questions - 

1. Why is sequential testing necessary in dataset shift under i.i.d assumption? I can imagine it to be useful under the independence assumption. Under the i.i.d setting, how does the performance of the proposed framework compare to Kamulete, 2021? Is the power of the test comparable? Can a similar curve (Figures 2,3,4)  for a number of samples be obtained for Kamulete, 2021? When will the testing framework of Kamulete, 2021 fail or work better? MNIST-C experiments are under the i.i.d dataset shift. Can a comparison with this testing framework be made? 

2. Similarly, when the i.i.d assumption is violated (distribution drift), can the comparison with the testing framework of Vovk et al. 2021 be made? 


Minor comments - 
- Figure 4(a): Horizontal line is yellow, not red
- PM and CM are not spelt out in the main paper
- “the this” often used in the paper
- “ intervene in a data-adaptive way”: Does this refer to model retraining? Are there other examples?

summary_of_the_review: Based on my knowledge and background research conducted while reviewing this work, I think the authors have proposed a novel framework to detect dataset shifts. It can potentially be used to improve several applications fraught with predictive models. The work is well supported with the help of illustrative examples. My only concern is their comparison with the existing works on dataset shift detection is not well explained/not supported.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a sequential testing method for tracking distribution shifts for deployed models. The main idea is to utilize sequential hypothesis testing on a given objective function in a range of time stamps. The authors have studied a variety of different confidence intervals/sequences that certifies benign or harmful distribution drifts and have also demonstrated theoretical support for their claims. Empirical results on both simulated and real data have shown this sequential testing method is effective in detecting severe distribution drifts.
main_review: **Strengths**

1. This paper has studied distribution drift in both i.i.d. and independence settings, where the non- i.i.d. assumption is more realistic and provide stronger results: since in real situation the distribution is more likely to drift gradually instead of a sharp change.

2. The sequential detection method is an intuitive way to continuously monitor model performance. In addition, the authors have also discussed several variants of multiclass losses to be combined with the sequential detection framework, which is commonly used in a variety of tasks.

**Weakness**

1. [Experimental results] All reported results of methods and baselines are proposed in the paper, e.g., Hoeffding, PM-EB, and betting are methods with different confidence bounds. There should be a direct comparison with other baselines and non-sequential methods, or at least show the behavior of a state-of-the-art approach to demonstrate what's the advantage of sequential testing. In addition, it is better to conduct both label and covariate shift experiments in sections 3.1 and 3.2.

2. [Notations] The authors first defined $Z$ in section 2.2 but it first appears in section 2.1; the definition of $R^{(t)} (f)$, what is the expectation over?

3. [Others] In section 1, the authors have mentioned current methods require assumptions on both covariate and label shift. Although the proposed method doesn't need this assumption, it is great to showcase how to overcome this issue when the assumption is violated.

summary_of_the_review: This paper proposes an effective and theoretically sound method for sequentially detecting harmful distribution drifts. Overall, this work is solid but still requires some improvements on the experiment.

## Update after Rebuttal

We thank the authors for their detailed responses and for updating the manuscripts. The authors have addressed my questions, but we tend to keep the current score based on the significance of the improvements made.
review_title: null

Meta-review: This paper is concerned with the problem of distribution shift, and develops techniques for detecting when the risk of a deployed model performs significantly worse on a testing distribution than on the training distribution.

The reviews for this paper were extremely consistent: after the discussion period, all five reviewers unanimously recommended acceptance, and several praised the authors for significantly improving their paper in response to reviewer criticism. Outstanding issues are (i) motivating the importance of the setting, and (ii) comparing with prior work. None of the reviewers seemed to think that these issues should be barriers to acceptance, but please seriously consider them, and all reviewer concerns.
Decision: Accept (Poster)