Title: Sequential Covariate Shift Detection Using Classifier Two-Sample Tests
Authors: Sooyong Jang
Sangdon Park
Insup Lee
Osbert Bastani
Venue: ICLR 2022

Abstract: A standard assumption in supervised learning is that the training data and test data are from the same distribution. However, this assumption often fails to hold in practice, which can cause the learned model to perform poorly. We consider the problem of detecting covariate shift, where the covariate distribution shifts but the conditional distribution of labels given covariates remains the same. This problem can naturally be solved using a two-sample test--- i.e., test whether the current test distribution of covariates equals the training distribution of covariates. Our algorithm builds on classifier tests, which train a discriminator to distinguish train and test covariates, and then use the accuracy of this discriminator as a test statistic. A key challenge is that classifier tests assume given a fixed set of test covariates. In practice, test covariates often arrive sequentially over time---e.g., a self-driving car observes a stream of images while driving. Furthermore, covariate shift can occur multiple times--- i.e., shift and then shift back later or gradually shift over time. To address these challenges, our algorithm trains the discriminator online. Furthermore, it evaluates test accuracy using each new covariate before taking a gradient step; this strategy avoids constructing a held-out test set, which can reduce sample efficiency. We prove that this optimization preserves the correctness---i.e., our algorithm achieves a desired bound on the false positive rate. In our experiments, we show that our algorithm efficiently detects covariate shifts on ImageNet.


Review discussion:
### Pros:
1) This work introduces a novel method to train the classifier and detect covariate shifts in a sequential manner. And the online pipeline is interesting.
2) This work is well formulated with elaborate proofs and theoretically solid.
### Cons:
1) This method evaluates each example before taking a gradient step on that example. It means that every sample can be used only once for training the classifier, i.e., training for one epochs. It seems to be a great limitation of this method. With a held-out test set, this problem doesn’t exist. As a common sense, training for more epochs can improve the performance. If the baselines (H2, H5) are better trained, I think their performances would be much better.
2) Besides, the experiments seem to be a litte unfair. The proposed method and baselines (H2, H5) differ in both training sets and testing sets because of the different sampling strategies. The proposed method use the whole set for both training and testing while baselines sample some examples for testing and use the others for training.
3) I also concern about the problem of cold boot. At the begining of training, the classifier is with poor performance. how can we evaluate the samples in this stage?
NO.
Thank you for the valuable comments and concerns. The following includes our response; we have updated our paper accordingly and highlighted the updates in red.
1. This method evaluates each example before taking a gradient step on that example. It means that every sample can be used only once for training the classifier, i.e., training for one epochs. It seems to be a great limitation of this method. With a held-out test set, this problem doesn’t exist. As a common sense, training for more epochs can improve the performance. If the baselines (H2, H5) are better trained, I think their performances would be much better.
* **Response:** We agree that each example can be used multiple times for the baselines (H2, H5) to train the classifier.
However, this strategy can also be used with our approach.
Instead, the restriction is that we can only use each once in the CP interval, which is the case for both our approach and the baselines. We focused on a single gradient step since we anticipate these algorithms being used in the online setting, where it is infeasible to take multiple passes over the training data (also, this strategy is orthogonal to our contributions). We have added a discussion to our paper (In Appendix C.1).
2.
Besides, the experiments seem to be a litte unfair. The proposed method and baselines (H2, H5) differ in both training sets and testing sets because of the different sampling strategies. The proposed method use the whole set for both training and testing while baselines sample some examples for testing and use the others for training.
* **Response:** The ability to use all the data is our key contribution, and our experiments are designed to highlight this advantage of our approach.
Unlike the baselines, which correspond to the classical Classifier Two-Sample Tests (C2STs), our approach does not require a separate held-out set to construct Clopper-Pearson intervals, so it can use the entire set of examples.
In particular, Lemma 1 guarantees it is valid to use the held-out set in this way.
3. I also concern about the problem of cold boot. At the begining of training, the classifier is with poor performance. how can we evaluate the samples in this stage?
* **Response:** If the classifier performs poorly, our approach will conservatively report “no shift” (in particular, the Clopper-Pearson interval we construct will be large). Our experiments show that we can detect shifts with very few examples, which suggests that cold boot is not a major problem. In general, it is very challenging to detect shifts with too few examples, since a small number of examples might simply be outliers in the source distribution.
Thanks for the response. Most of my concerns are addressed.  And additional explanations and experiments make this work more credible.  I’d like to update my rating from 5 to 6 but will not argue strongly for acceptance.
Strong points:
- the method is simple and allows to use any classification model that can be trained in an online manner, which makes it of practical interest
- the paper is clearly written
- there is strong theoretical grounding coming from classical statistics
Weak points:
- there is some obscurity in the proof of the crucial theorem 3, which could hide a flaw
- an empirical comparison with existing two-sample test would be interesting to better justify the need for this method and as an additional baseline
- in the experiments, the false positive rate is above the specified level:
- the given reason (multiple hypothesis testing) is not convincing
- it makes the detection power comparison less meaningful
Detailed review:
### COMPARISON TO SEQUENTIAL TWO-SAMPLE TESTS:
It is argued (sections 1 and 2) that sequential tests are not appropriate in the scenarios proposed since they assumed that all the data is shifted. This assumption is only relevant with respect to False Negative Rate guarantees (e.g. consistency).
Nevertheless, guarantees on FPR are obtained under the assumption that the distributions are equal.
Therefore, sequential two-sample test are applicable to the scenario of this paper.
Providing comparisons with some of the cited sequential two-sample test would definitely make the paper stronger, justifying better the need for the proposed method and its advantages. You could run a sequential two-sample test by drawing samples from the training set (without replacement---this requires the training set to be roughly as large as the testing horizon) and from the test set in the order they come, until a rejection occurs and then you could reset everything and start testing again.
### LEMMA 1:
The notation $x\_{i..j}$ should be introduced and $i,j$ should be specified.
Does $\hat{g}\_{i,j-1}$ refer to the classifier itself (in the case of a NN a set of weights and biases?) ?  Or is it the accuracy on the range $i..j-1$?
In the proof, $\hat{g}_{i}$ appears as a random variable, so I guess it represents all the variables defining the model.
This should be formalized.
### PROOF OF THEOREM 3
It is not clear how Lemma 1 is used:  more precisely, **why can you say that  $s_{w,t}$ is a binomial variable with parameter 1/2?** Before Lemma 1, you say:
*the key to have valid bounds is proving the independence on predictions $\hat{y}\_1, \dots, \hat{y}\_t$ (and $\hat{y}'\_1, \dots, \hat{y}'\_t$) to have a valid Clopper-Pearson interval, since they are seemingly
dependent through online learned classifier $\hat{g}_t$.
First, our key result shows that our estimate of the
accuracy of $\hat{g}\_t$ valid—i.e., the labels $\hat{y}\_{i:j}$ are conditionally independent.*
What do you mean exactly by a valid estimate of the accuracy? This should be formalized.
Also, where does $t$ lie with respect to $i..j$ ?
### LACK OF CONTROL OF FPR IN EXPERIMENTS:
It is suggested that the lack of control of the FPR (which is often around 3% and sometimes above, while the specified level is 1%) is due to multiple hypothesis testing.
However, I can't see which multiplicity you refer to. These rates are obtained sample-wise over the $R$ repetitions, and therefore they should be empirical estimates of the theoretical rate considered in Thm 3. A multiple hypothesis testing problem would occur if we use multiple statistics to get to one decision which is not the case here.
Three possible explanations come to my mind:
- a possible leak between the data used for training the Neural network and the data used to compute the statistic (i.e. some data points used to train a model $\hat{g}$ are also used as data points to compute the statistic with $\hat{g}$)
- the practical assumption that $\mathcal{S}$ is a uniform distribution over the training set : even if the training and test set come from the same distribution, this practical assumption will make $\mathcal{S}$ different from $\mathcal{T}$.
- A flaw in theorem 3?
Therefore I suggest to :
- Clarify the proof of Theorem 3
-  fix/give a better explanation of the uncontrolled FPR observed in the experiments
-  provide an empirical comparison to sequential two-sample tests: this would show the need for your approach and make the paper stronger
### MINOR ISSUES/COMMENTS:
1) Bayes Ball algorithm and reference to Bishop 2006:
Citation should be more precise since it is a text book .
In fact, I couldn't find "Bayes Ball" in that book.
It seems to be called d-separation there.
2) "$\hat{g}\_t$ necessarily achieves a trivial accuracy of 1/2" : I guess you refer to theoretical accuracy. You should make it precise since there could a confusion with the achieved empirical accuracy which is not necessarily exactly 1/2.
3) The description of the algorithm should be improved: in particular, $w$ should be included as input and the formula for $\hat{\mu}_{w,t}$ should appear there.
The title says "Sequential Calibrated ...": the word "calibrated" appears nowhere else
4) THEOREM 5: it would be interesting to discuss the consequences of the assumptions $a(w,\alpha)\leq w$ and $b(w,\alpha)\geq 0$.
5) First inequality using the VC dimension: It would help the reader to give some justification/precise reference for this inequality.
6) Table 2b is misleading : higher FPRs are observed in the plots, when going beyond 200 samples
7) Proof of Thm 5: $\mu_{\hat{g}}^*$ seems to correspond to $\mu_{w,t}^*$ , why this new notation?
8) Equalities (8) and (9) should be inequalities.
9) Definitions should use a different equal sign ($\equiv$ or $:=$)
TYPOS:
"through online learned classifier" -> "through the online learned classifier"
covarate, covarite-> covariate
To checking -> for checking
Our estimate of the accuracy of $\hat{g}\_t$ **is** valid
Statement of theorem 5: "where $\epsilon\in(0,1/2]$ is the accuracy"  Maybe a comma is missing before "is"
Tables: G-shift -> GI-shift
NO.
Thank you for the valuable comments and concerns. The following includes our response; we have updated our paper accordingly and highlighted the updates in red.
1. COMPARISON TO SEQUENTIAL TWO-SAMPLE TESTS
* **Response:** We have implemented the Sequential likelihood ratio test (Wald Test), and added the results to our paper. We described how we implemented the Wald test in Section 5.1, and included the testing result in Section 5.2 and 5.3.
In summary, the test satisfies FPR bound, but is less effective at detecting shifts in all scenarios. Please see Figures 2 & 3 and Tables 2 & 3.
We will also update the Wald test results for the additional experimental results in Appendix D as the experiments finish.
2. LEMMA 1:
* **Response:** Here, $x_{i:j}$ means $x_i, …, x_j$, $\hat{g}\_i$  refers to a classifier itself (e.g., weights and biases of a neural network),
and $ \hat{g}_{i:j-1}$ means the classifiers between time step $i$ and $j-1$.
Also, $\hat{g}_i$ is a random variable since it depends on the previous inputs $x_i$ (as random variables) for training samples, but $\hat{g}_i$ itself has no randomness. We will more clearly describe this.
3. PROOF OF THEOREM 3
* **Response:** $s_{w, t}$ is the sum of Bernoulli random variables and these random variables are independent due to the conditional independence of predictions $\hat{y}\_1, \dots \hat{y}\_t$ and $\hat{y} ’\_1, \dots \hat{y}’\_t$;
thus $s_{w, t}$ is a Binomial random variable. Without this conditional independence we cannot say $s_{w, t}$ is a Binomial by the definition of the Binomial random variables (i.e., the sum of independent Bernoulli variables). For the “valid estimate of the accuracy”, we model the accuracy using the Binomial random variables, which is the true model, and estimating the accuracy is the same as the estimating the parameter of the Binomial distribution.
Finally, $s_{w_t}$ will have parameter ½ only if there is no shift. We will clarify this.
4. LACK OF CONTROL OF FPR IN EXPERIMENTS:
* **Response:** What we mean is that we are estimating FPR, so the FPR bound violations are the result of small experiment repetition (denoted by R), and this issue is exacerbated since we are evaluating FPR for many scenarios. If we used an infinite number of samples to estimate the FPR, then we would always meet the bound. In our previous version, we used R=1,000 repetitions to estimate FPR analysis, which resulted in violations. We have increased R to 20,000, and show the updated FPR results in Tables 2(b) and 3(b) in our paper (one table is for natural shift and the other one is for synthetic shift). As can be seen, with this higher choice of R, we have no FPR bound violations.
MINOR ISSUES/COMMENTS:
5. Bayes Ball algorithm and reference to Bishop 2006: Citation should be more precise since it is a text book .
In fact, I couldn't find "Bayes Ball" in that book. It seems to be called d-separation there.
* **Response:** Thank you for pointing this out, we have fixed it.
6. "$\hat{g}_t$  necessarily achieves a trivial accuracy of 1/2" : I guess you refer to theoretical accuracy. You should make it precise since there could a confusion with the achieved empirical accuracy which is not necessarily exactly 1/2.
* **Response:** Yes, this is theoretical accuracy. We will call it “expected accuracy”.
7. THEOREM 5: it would be interesting to discuss the consequences of the assumptions $a(w,α) \le w$ and $b(w,α) \ge 0$.
* **Response:**  This condition is related to the valid range of w to be the bound meaningful (i.e., for the valid domain of the CDF F); we have updated the condition in a more concrete form. The condition means that the bound is valid if $w \ge 201$ given $\alpha = 0.01$. We will update the paper.
8. Table 2b is misleading : higher FPRs are observed in the plots, when going beyond 200 samples
* **Response:** As pointed out, to compute the true FPRs we need an infinite number of random experiments;
previously, we only used 1000 repetitions (i.e., $R=1000$), but we have increased the number of repetitions to 20000 (i.e., $R = 20000$); the updated Table 2b shows that our approach satisfies the desired FPRs empirically.
9. Proof of Thm 5: $\mu_{\hat{g}}^\*$  seems to correspond to $\mu\_{w,t}^\*$, why this new notation?
* **Response:** This is a mistake. Thank you for pointing this out; we have fixed it.
10.  Definitions should use a different equal sign (≡ or :=)
* **Response:** Thank you for pointing this out; we have fixed it.
11. Typos
* **Response:** We appreciate pointing out these typos, and we fixed them.
Thank you for your response updates. The followings are the answers to your additional comments.
- We tried several values of $\epsilon$ (in particular, $\epsilon \in [0.01, 0.02, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4]$). Our proposed approach clearly outperforms Wald tests with any of these choices of \epsilon (see here https://ibb.co/TM0FWck). We chose $\epsilon=0.2$ since it provided good results for both the "shift" and "no shift" segments of the back-and-forth scenario. Since Wald's test has memory, smaller values of $\epsilon$ tend to perform poorly when shifting from "shift" to "no shift". We will include results for all choices of $\epsilon$ in an update to our paper.
- We have previously compared to non-classifier-based tests, which perform very poorly due to the high dimensionality of the data.
For instance, in our updated paper, we show that our approach significantly outperforms the Deep Kernel Two-Sample Test, a state-of-the-art non-classifier-based two-sample test. We expect non-classifier-based sequential tests to suffer similar shortcomings, especially given that sequential tests are not suited to our setting.
positives:
The authors investigate an essential problem in real-world applications. This draft is well-organized and easy to follow.
negatives:
- The proposed method attempts to adapt the offline covariate shift detecter to the sequential detection and learning scenario.
However, due to the statistical nature of the two-sample test, the proposed approach still need a relatively large number of data to work. As shown in section 4, the error bound depends on the sample size in the order of $O(1/\sqrt{m})$. In the online learning scenario, the size of sampled (target) data is usually limited, and thus the effectiveness of this basic guarantee is limited.
- The experiments are conducted on the modified ImageNet dataset to simulate various covariate shifts. It is suggested to test the performance of the proposed algorithm on a realistic public benchmark dataset.
NO.
Thank you for the valuable comments and concerns. The following includes our response; we have updated our paper accordingly and highlighted the updates in red.
1. The proposed method attempts to adapt the offline covariate shift detecter to the sequential detection and learning scenario.
However, due to the statistical nature of the two-sample test, the proposed approach still need a relatively large number of data to work. As shown in section 4, the error bound depends on the sample size in the order of $O(1/\sqrt{m})$. In the online learning scenario, the size of sampled (target) data is usually limited, and thus the effectiveness of this basic guarantee is limited.
* **Response:**  In general, $O(1/\sqrt{m})$ is the best sample complexity we can hope to achieve. Our experiments show that our algorithm is very effective in practice even with just a few examples: as demonstrated in Tables 2 (a) & 3 (a), our algorithm only requires a small number of examples to detect covariate shift in all scenarios.
2. The experiments are conducted on the modified ImageNet dataset to simulate various covariate shifts. It is suggested to test the performance of the proposed algorithm on a realistic public benchmark dataset.
* **Response:** Section 5.2 is about a natural shift experiment which emulates a realistic shift scenario. In the experiment, we split 120 dog classes of ImageNet to two sets, and consider that the two subsets of the dog classes are source and target sets, respectively. In this setting, our approach is sample-efficient, and detects this shift well compared to the two baselines.
Pros:
1.  This research topic is valuable since the community cares about the AI reliability and trustworthy ML.
Although deep networks can address many real-world problems, it is not reliable (e.g., OOD) and can be attacked (e.g., adversarial attacks).
Thus, detecting such shifts is meaningful in many fields.
2. This paper proposed a test to dynamically detect the distribution shift, which seems novel and interesting.
3. Experiments are conducted in many situations, which supports the effectivenss of the proposed method.
Cons:
1. Some key discussions are missing. As reviewed in this paper, kernel MMD is another type of two-sample tests. If we use SGD or Adam to online update the kernel parameters, can kernel MMD be used to address your problem? It would be great to provide some empirical evidence, but detailed discussions are also ok.
2. In (1), what does \sP stand for?  It is a probability measure without proper explanations/definitions. For a probability measure P, we often use P(A) to represent the measure on a set A.
3. Some key references are missing. In the literature, concept drift detection and adaptation are very close to your problem setting. Please conclude some discussions between your problem and them.
4. How to generate the data used in this paper? What kind of shifts you considered? Mean shift? Covariance shift?
NO.
Thank you for the valuable comments and concerns. The following includes our response; we have updated our paper accordingly and highlighted the updates in red.
1. Some key discussions are missing. As reviewed in this paper, kernel MMD is another type of two-sample tests. If we use SGD or Adam to online update the kernel parameters, can kernel MMD be used to address your problem? It would be great to provide some empirical evidence, but detailed discussions are also ok.
* **Response:** We have added experimental results to Appendix B; as can be seen, our approach significantly outperforms the adapted version of Kernel MMD. Due to time constraints (kernel MMD is slow), we only have results for Natural Shift; we will include complete results in the main paper once we have run them for all our scenarios.
2. In (1), what does \sP stand for? It is a probability measure without proper explanations/definitions. For a probability measure P, we often use P(A) to represent the measure on a set A.
* **Response:** It is a probability measure on the event when the random variable s satisfies the specified condition. We will clarify this.
3. Some key references are missing. In the literature, concept drift detection and adaptation are very close to your problem setting. Please conclude some discussions between your problem and them.
* **Response:** We thank you for suggesting additional references. Concept drift detectors typically assume that ground truth labels are provided for test examples, whereas our approach only requires unlabeled test examples. The former is substantially easier, since it suffices to check for drift in the distribution of errors made by the classifier, which is usually very simple (e.g., a Bernoulli distribution for the 0-1 loss), making it easy to exactly check for concept drift.
In contrast, we are checking for drift in a high-dimensional covariate distribution. We have  added a discussion of concept drift detection in Section 2 on Related Work.
4. How to generate the data used in this paper? What kind of shifts you considered? Mean shift? Covariance shift?
* **Response:** We are considering covariate shift (i.e., shift in p(x)). In our experiments, we consider two such shifts on ImageNet: natural shift and synthetic shift. For natural shift, we split 120 different dog breeds of ImageNet into two groups, and we consider the images from one subset of dog breeds to be the source distribution and ones from the other subset to be the target distribution. For synthetic shift, we have perturbed images by five different image transformations from ImageNet-C (Contrast, Defocus blur, Elastic transform, Gaussian blur, and Gaussian noise) with five different severities like Hendrycks and Dietterich (2019).
Reference
- Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations.Proceedings of the International Conference on Learning Representations, 2019.
Thanks for the responses. Most of my concerns are addressed well.
Thus, I would like to increase my score from 5 to 6.
However, I still have a concern regarding the type I error of DK. In DK, they use a permutation test to calculate the p-value. It should be around the alpha you chose. Could the authors explain this situation? It raises my concerns about if the generated data contain some dependency among data. If you set alpha to 0.05, the type I error of DK is still abnormal?

Meta-review:
role: PC
data: comment: This paper proposes to repeatedly apply the classifier two-sample tests (proposed by Kim, Ramdas, Singh, Wasserman, in 2016, and developed further by Lopez-Paz, Oquab, in 2017) for the purpose of detecting covariate shift. The authors propose methods to extend the aforementioned tests to a sequential setting. Overall, the reviewers do not lean towards acceptance, and neither do I. Several constructive suggestions are provided by reviewers, some are summarized below.

The authors claim that sequential tests are not desirable in such a setting, and thus choose to pay a multiple testing price by repeatedly applying a batch test. However, sequential tests are in fact applicable (they will control type-1 error) but may have a worse power if the alternative is not true at the very start --- but these were entirely dropped from the simulations; in fact, comparing the increased type-1 error of the authors' approach to the increased type-2 error of sequential approaches may be worth clarifying. 

Perhaps the "right" solution that the authors are looking for could be gotten by converting a sequential test into a sequential changepoint detection algorithm (via repeated application of a sequential test, each started at a new time). Also see "Conformal test martingales for change-point detection" and "Inductive Conformal Martingales for Change-Point Detection" by Vovk et al., which are currently not cited.

Decision: Reject