Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a learning-based testing method to identify distribution shifts (a new domain) with access to finite samples of the test domain. The authors define harmful covariate shift (HCS) that may weaken the generalization of a predictive model, and constrained disagreement classifiers (CDC) that performs consistently on ID data and inconsistently on OOD data. Then they detect HCS by the inconsistency of the outputs of the constrained disagreement classifiers. Theoretical analysis shows that the disagreement rate of CDCs can be used to detect covariate shifts. The proposed algorithm uses the empirical distribution of the CDCs on the training data to determine the reject region.
strength_and_weaknesses: **Strength:**
1. The authors propose disagreement cross-entropy to improve the optimization procedure of the PQ-learning problem.
2. This work develops a hypothesis testing method to detect covariate shifts and the detection principle is consistent with the common theoretical understanding that extrapolation increases the variance of outputs.
3. Experiments show that the proposed method is sample efficient.

**Weakness:**

1. The motivation is unclear. In Section 3, the generalization set $\mathcal{R}$ is a set of distributions. Is it possible that $\mathcal{P}$'s support set overlaps with $\mathcal{Q}$'s support set? If $\mathcal{Q}$'s support set is a subset of $\mathcal{P}$'s support set, how to understand the harmful covariate shift? Can you disentangle the effect of correlation shift ( the relation between the inputs and outputs $P(Y|X)$ ) and the effect of harmful covariate shift? 
2. Need more analysis on the utility of disagreement-cross-entropy. In Eq. (3), if we remove the disagreement-cross-entropy loss, how does the performance of Detectron change? Does the learning procedure of $g$ change the correlation between $Y$ and the features in $f(X)$? 
3. What are the requirements for the pre-trained model $f$? Is $f$ required to achieve good prediction accuracy? How does the proposed detection method perform if the ID data is ImageNet?
4. If the pre-trained model $f$ is over-parameterized, can we force the disagreement rate on ID data to be 0%? Then how to implement Algorithm 1?
5. Theorem 1: "… If the rate … greater than … there must be covariate shift." This is already a hard threshold. Why do you still consider a hypothesis test approach?
6. Can we consider the task in this paper as an OOD detection task with auxiliary data? If so, a comparison with recent OOD Detection methods is required, e.g. POEM.



clarity,_quality,_novelty_and_reproducibility: The novelty and Quality are fair. This work contributes some new ideas. It has minor technical flaws and some typos. The notations should be improved. For example, the notation $[ \cdot ]$ in Eq. (2) should be an indicator function and there is no definition of $\hat y_c$. 
The clarity is poor. The content should be carefully reorganized.

summary_of_the_review: I recommend marginal rejection. Overall, this is a potentially promising work. But I do not think the current version is ready for publication.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work develops a new hypothesis testing procedure for identifying harmful covariate shifts. To identify harmful covariate shifts, the authors propose to use disagreement cross entropy for multiple models and present Detectron (Disagreement) based on a permutation test. 
strength_and_weaknesses: - Strength
  - The idea of using an ensemble of models that maximize out-of-domain disagreement makes sense. 
  - Related work section fairly covers literature in the field.
  - Extensive experiments are performed.
 
- Weakness
  - The paper critically lacks clarity and is not mathematically rigorous. It really causes lots of confusion. Please see Clarity, Quality, Novelty, And Reproducibility.
  - Implication of Theorem 1 is not clear. As far as I understand, it is almost impossible to know when the following condition holds: the rate which $g$ disagrees with $f$ on $n$ unseen samples from $\mathcal{Q}$ is greater than that from $n$ unseen samples from $\mathcal{P}$ w.p. greater than $p^{*} = (1-4^{-n} \binom{2n}{n} )/2$. Given that the probability is derived from the expectation of such random experiments, we might need multiple sets of $n$ unseen samples, which is not realistic. 

- Question:
  - Theorem 1: do you have an insight into why $p^*$ is independent of $N$? Does it imply that this theorem holds even $N=1$? It is not intuitive... 
 
clarity,_quality,_novelty_and_reproducibility: Clarity: One of the main problems of this paper is clarity.
 - In Problem Setup of Section 3, $N$ is defined as the number of classes. In Theorem 1 and Algorithm 1, it becomes the sample size of $\mathbf{P}$ and $\mathbf{Q}$, respectively.
 - In Problem Setup of Section 3, $\mathbf{P}$ is defined as a set of $n$ labeled samples, but in equation (3), it is a set of input data.
 - In the Constrained Disagreement Classifier, $\mathcal{P}$ is not defined. 
 - PQ learning part (page 4): $\epsilon$ and $\delta$ are not defined, and it is not clear the definition of $rej_h (x)$ and $err_h (\tilde{x})$. How are they expressed as mathematical equations? What types of out-of-distributions are considered?

Reproducibility:
 - Code is submitted.


summary_of_the_review: The submitted paper considers an interesting problem, and the proposed approach makes sense at a high level. However, it lacks clarity which causes much confusion, so I think it is not ready for publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of detecting covariate shifts where a trained model would perform differently. In particular, the authors propose a framework that prepares new models that disagree with the base model and then uses the disagreement rates (or prediction entropy) to build the statistical test.
strength_and_weaknesses: (+) The problem of detecting performance change with an unlabelled dataset is a practical question for many ML applications.

(+) The proposed framework of training disagreeing models and using the model predictions to build test statistics provides an interesting development to the research problem.  

(-) While this paper has good coverage of related work on drift (covariate shift) detection, the coverage of methods that estimate the model performance on unlabelled datasets is limited. This limited coverage could be crucial as some methods use similar ideas of modelling the disagreement among multiple models and can be used directly to solve the targeted research problem. (See below for details.)

(-) Some details of the paper might need additional justification, such as the objective learning function for model disagreements, the proposed test statistics, and baseline approaches. (See below for details). 
clarity,_quality,_novelty_and_reproducibility: Clarity (4/5): This paper is mostly easy to follow, and readers with similar backgrounds should get the central concept without significant issues. Some mathematical notations might require additional marks to avoid confusion. For instance, $N$ is first used as the number of classes and then as the number of samples. $n$ denotes the dataset size at first but is then used as the number of unseen samples.

Quality (3/5): While the paper provides an interesting solution and good coverage of drift detection approaches, some technical details might require further justifications. Also, the paper doesn't cover some related work that aims to estimate the performance with an unlabelled test set.

On the technical details: (1) regarding the proposed disagreement-cross-entropy, while the authors provide some intuition, it should be aware that the cross-entropy loss and associated entropy measure both come with theoretical properties (i.e. proper scoring rules). I wonder if the authors can provide any properties of the proposed loss, as it is helpful to know whether the loss can lead to a biased disagreement (other than a pre-defined and consistent one). (2) At the beginning, I was surprised to see that the learned deep-MMD performed poorly in detecting drifts that even affect model performance. Then I realised that the experiments are performed with quite a limited sample size. In this case, I might further consider reducing the complexity of the projection net in methods like deep kernel MMD and classifier two-sample test, as it is just not practical to train a large model with such a limited sample size. (3) Following the discussion on sample size,  it is clear that the proposed method requires repeated training on both P and Q for multiple models. I wonder if this could be a potential reason for experimenting on a small dataset. It might be helpful if the authors could further comment on the running costs of the method for larger datasets. 

On the related work side, there have recently been many attempts to directly estimate the model performance with a (drifted) unlabelled test set, which is undoubtedly closely associated with this paper. (https://arxiv.org/abs/2202.05834) provides a nice list of some of these methods. In particular, (https://proceedings.mlr.press/v48/platanios16.html) also make use of multiple models' predictions to infer the actual error rate of the test set, which shares a strong link to the proposed method in this paper. It is therefore suggested to justify further the proposed method compared to the aforementioned related work.

Novelty (3/5) The novelty is reasonable as the authors proposed an interesting statistical test on changing performance with the test set. As mentioned above, the problem has many existing attempts, and the idea of using model disagreements can be shared by existing work.

Reproducibility (5/5): This paper provides an excellent example of reproducible research. Both the methods and experiments are described in detail. The source code is also available for additional in-depth investigation.
summary_of_the_review: While the proposed framework provides interesting solutions for training different models and building the statistical tests, this paper might require further work on involving and discussing critical related work before getting published as high-impact research.


=======After author feedback============

Thank the authors again for the extensive feedback. The added explanation and experiments regarding performance estimation are a nice addition to the paper. I have decided to change the score to 6 as a result, but I think some systematic refinement on the paper would make it much stronger (i.e. the disagreement loss, and connections to performance estimators).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method for detecting covariate shift by training two classifiers to agree on a distribution P and disagree on another distribution Q, and then build a hypothesis test on top of these two classifiers. Their proposed method is able to detect covariate shifts better than several competing approaches. However, there seems to be potential failure cases for the method that the authors do not discuss sufficiently. 

strength_and_weaknesses: Strengths: 
- The authors provide very detailed theoretical and empirical analysis of their method, in both the main paper and the appendix. 
- The proposed method works well, beating several other methods for detecting covariate shifts in the experiments. 


Weaknesses: 
- While the approach provides good empirical performance, the core idea of the method is still unclear to me, in terms of how the assumptions might fail. The main idea of the method is to detect harmful covariate shifts by training two classifiers, forcing them to agree on P while disagreeing on Q. There are several possible scenarios: 
  1. If it's possible to train such a pair of classifiers f and g, it does not necessarily mean the shift from P to Q is harmful. Consider linear model on disjoint supports P and Q. It just means P and Q are different. If we just want to detect shifts between P and Q, why do we not use non-parametric tests such as the KS test, or train a logistic regression model distinguishing P and Q as commonly used in covariate shift correction? 
  2. If it's not possible to train such a pair of classifiers f and g, it might just be that the models are underfitting. Consider fitting linear models to quadratic target y = x^2 between -1 and 1. P is positive region, Q is negative region. It is not possible to fit linear models that agree on P but disagree on Q, but P and Q are distinct.  

  Although these are specific boundary cases on potential failure modes, the fact that the current method does not seem to cover them properly make me uncertain about the the method. 

clarity,_quality,_novelty_and_reproducibility: - The paper is mostly clearly written, and well-supported by the Appendix. 

summary_of_the_review: This paper propose an interesting approach for detecting covariate shifts, and the authors spend a lot of effort in its theoretical and empirical analysis. However, there seems to be potential cases that the method can fail which the authors do not cover sufficiently. A better discussions on these, even listing them as limitations or assumptions, can help improve the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a method to detect covariate shift by training two classifiers to agree on a distribution P and disagree on another distribution Q. The disagreement rate of the two classifiers is used to detect covariate shift as part of a statistical test.

The discussion phase was fruitful with meaningful exchanges from all reviewers that result in better understanding on the reviewers’ side and better submission compared to the initial submission. Some important negative points raised and **resolved** were

* Lack of analysis of the disagreement cross entropy (DJjN); -> The authors added an analysis to the appendix
* Lack of clarity and justification on some technical parts (wkvQ, En7A) -> revised and the reviewers’ concerns were resolved
* Corner cases not discussed (e.g., models underfitting and disagree) -> resolved by the rebuttal.

The reviewers agreed that the proposed approach that relies on disagreement of two models is practical and novel. The proposal is theoretically grounded (DJjN), and interesting (En7A). The submission covers related work very well (En7A, wkvQ), and has extensive experiments (wkvQ). Concerns were addressed satisfactorily after the rebuttal.

Recommendation: accept. 

justification_for_why_not_higher_score: This paper has both theoretical and experimental components. I feel that theoretical contributions are fair but not extensive.
justification_for_why_not_lower_score: The reviewers agreed that the proposed approach that relies on disagreement of two models is practical and novel. The reviewers find this aspect unique. It is worth publishing.
Decision: Accept: poster