Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method to compute the distance between two distributions. 
The authors show how this can be used for novelty detection, out-of-distribution classification, and the like. The authors present results primarily focusing on these tasks, and compare them with traditional novelty detection methods.
strength_and_weaknesses: Strengths
- The paper is nicely written

Weakness
- The results on novelty class detection are not convincing. Traditional methods perform almost similarly to the proposed method
- The performance for out-of-distribution detection is also not convincing. There is no clear winner, and the proposed method is only good on half of the datasets. Similar comment for Table 3.
- The authors do not compare with Deep One-Class Classification, which is a relevant work.
- Figure 3 is not well presented. The captions should be self-explanatory.
- Overall it is not clear what is the utility of this work compared to other works in the literature, as it hardly introduces a new concept, findings, or better performance. 
 



clarity,_quality,_novelty_and_reproducibility: At a high level, the algorithm for Compute Bound is very similar to other methods of distribution distance which uses statistics of distribution to compute the distance between them. I would suggest the authors compare it theoretically with other methods, where it can work, and where it will fail.

While the authors talk about how their method can be useful for distribution shifts, there are no experiments in the paper regarding that. How shall we validate that Theorem 2 actually holds in practice?

When we say the distance between two distributions, isn't it supposed to be at the input space itself? when we compute the distance on the extracted feature space, does that really allow for a good distance metric? because then the distance is dependent on the model. 

summary_of_the_review: The paper tries to attack an important problem of measuring the distance between two distributions. However, the technique that the paper proposes is quite similar to the statistics-based method for distribution distance computation. The results section of the paper is missing a comparison with some commonly used techniques, and the proposed method does not stand out as the clear winner in any of the settings.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors define an upper bound for the overlap index and then by converting it into confidence score they can train a one-class classifier. Since the bound is estimated on unknown distribution then the classifier is also distribution free. 
strength_and_weaknesses: Strength: 
Interesting idea
Weakness:
Minor improvement in results AUPR is not better than others or not that much increased. 
clarity,_quality,_novelty_and_reproducibility: The paper is well written and well structured
summary_of_the_review: Given the comments, I vote for borderline reject.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed an upper bound to overlap index to measure the distributional distance between two distributions in a distribution-free setting, i.e. when we are only given access to finite samples from the underlying distributions. The authors first present a theoretical result to approximate the accuracy of a model with their upper bound on the overlap index.  Second, the authors employ this bound to study the distribution membership of given examples. Experimental results on out-of-distribution and novelty detection show that the proposed bound outperform existing heuristics on average in the AUROC metric. 
strength_and_weaknesses: **Strengths** 
-  The paper presents an interesting yet simple upper bound of the overlap index. The upper bound is distribution-free and can be computed only with samples. 
-  An interesting application of the bound on one-class classification is presented in the paper.  Experimental evidence for novelty detection, out-of-distribution detection, and backdoor detection shows the efficacy of the method. Moreover, the proposed one-way classifier does not need to train any parameters. 



**Weaknesses** 
- In Theorem 2, authors present an approximation to the accuracy of a model under distribution shift. However, it is unclear if this bound is practically useful. Moreover, there are no experiments highlighting the efficacy or practicality of the bound. Specifically, this may happen because we can not compute the term q in the bound. 
- Abstract and introduction of the paper should be toned down with respect to the above contribution. In particular, it should be made clear that the application to accuracy estimation is only of theoretical interest.  
- Several crucial experimental details and design choices are missing in the paper. Please refer to the response below for details. 

clarity,_quality,_novelty_and_reproducibility: In addition to several weaknesses highlighted above, it would be useful for a reader for reproduciblity and clarity if the following points could be clearly discussed in the paper: 

- More discussion should be added on choices of the conditional function g. Currently, authors use indication function $I(||x|| < r)$. Can authors present some motivation behind this choice and other choices that can be worth exploring in the future? 
- In the abstract authors mention that their one-way classifier "requires only a small number of in-class samples to be accurate." However, this argument is not substantiated with any theoretical or concrete empirical justification. In my opinion, this statement should be toned down to precisely reflect the experimental finding. Moreover, in general, empirical experiments can not show that the proposed bound requires only a small number of samples to be accurate. 
- More details about the baseline methods compared in Section 4.4 should be included in the paper. Are there hyperparameter choices in the baseline methods that impact the final results? If yes, how did the authors choose those hyperparameters in their comparison?
- For high-dimensional benchmark datasets, authors use WideResNets to extract features that I assume to act as the input $x$. Can authors provide more details on how this Resnet is trained? Moreover, the experimental setup for Section 4.4.2 is missing. More details should be added to justify how the comparisons made in Tables 2 and 3 are fair with other baseline methods. 

**Other Writing suggestion**:
- Language in Theorem 2 should be significantly improved. In its current form, it is very confusing. While it is not hard to infer the real meaning using parenthesis like in "If a model is trained on D with p accuracy on D (including the overlap area between D and D∗)" doesn't make it clear that p is the accuracy on D which is restricted to the overlap region. It would be great if authors can formally define these quantities. Moreover, the last phrase "is upper bounded due to (14)" is unclear. What is this quantity upper bound on? Is it the overlap index?
- (Very minor) For Definitions 1, 2, and 3, authors may consider defining P and Q distribution in the text before these definitions to avoid redundancy


I believe that all the suggestions made above can be taken into account during the rebuttal phase, and I look forward to the author's response and will be happy to change my score if these questions are answered satisfactorily. 
summary_of_the_review: Overall, the paper presents an interesting yet simple upper bound of the overlap index. The upper bound is distribution-free and can be computed only with samples. An interesting application for one-class classification is discussed in the paper. However, the section with the application of accuracy estimation under distribution shift is a bit skim, and currently, it is unclear if the proposed upper bound on the overlap index can be leveraged in any non-trivial manner. Moreover, the paper lacks some crucial experimental and baseline details.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes to consider a distance between distributions by focusing on the overlap index between the two distributions. It proposes a method to compute an upper bound on this index which is distribution-free. The bound then converted into a confidence score function which is efficient to compute. 

Strengths:
-Method simple and efficient
-interesting applications in 1-class classification, novelty/anomaly detection
  
Weaknesses:   
-Experimental evaluation of the work is not sufficient are requires additional study
-Position of the work with respect to other methods of the literature is unclear and requires better justification.
-Global writing can be improved.
  
After the rebuttal phase, authors have provided answers and precisions to the remarks of the reviewers. This has been appreciated and taken into consideration. During discussion, reviewers agreed that the presentation of work still requires some work (discussion remains superficial) and the significance with respect to other methods of the literature has to be better justified both methodologically and experimentally. Despite the interestingness of the paper, there was a consensus on a weak reject.  

Overall the paper has an interesting potential which is not justified in a convincing manner yet. I propose then rejection.  
justification_for_why_not_higher_score: The 3 reviewers agreed on a consensus of weak reject, there are still some elements that have to be justified.
Decision: Reject