Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper shows that binary discrimination between in- and out- of distributions is equivalent to several generative model based OOD detection approaches such as likelihood ratios. Moreover, the paper shows that when the binary classifier between in- and out- trained in a shared fashion with a standard classifier for in-distribution classes, and using a score which integrates information from p(i|x) and p(y|x, i), the OOD detection performance achieves the state-of-the-art.
main_review: The proposed method is not completely new, but the purpose of this paper is to identify common objectives as well as the identification of the implicit scoring functions of different OOD detection methods. Also the proposed method is simple and efficient. The connection to the previous work is comprehensively discussed.

I found the table results are a bit too busy to follow, with many different methods and scores. It would be great if you can highlight the best values in each column. Also, the FPR@95%TPR can be very sensitive. I would recommend to report AUROC in the main text. 

Here is a relevant paper to the idea of training the binary discriminator between in- and out-distribution together with a standard classifier on the in-distribution in a shared fashion. It may be worth noting that. 

Roy, Abhijit Guha, et al. "Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditions." arXiv preprint arXiv:2104.03829 (2021).
https://arxiv.org/abs/2104.03829
summary_of_the_review: The paper provides helpful insights to connect methods for OOD detection tasks. It has both comprehensive theoretical and empirical analysis. The paper can be further polished to make it easy to follow.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors show that 1) a simple baseline,  binary discriminator between in- and out-of- distribution data, is competitive with state-of-the-art OOD detection techniques and 2) the Bayes optimal scoring functions of several proposed methods for out-of-distribution detection are equivalent to the scoring functions of simple binary discriminators. The authors also aim to provide a better understanding of key components of various OOD detection methods such as Outlier Exposure.

main_review: Strengths:

The paper is trying to demonstrate that a simple baseline can out-perform other more complicated methods. I appreciate such efforts and I think they provide a good check for the field in general.

Weaknesses:

1) The paper is poorly written and hard to follow. For example, the intro discusses why OOD detection is an important problem and then delves into related work, but fails to properly motivate the contributions by, say, discussing why the rankings induced by the Bayes optimal classifier are relevant, why we may want to revisit and better understand the differences between various OOD methods, or why we may want to consider a binary discriminator method. As another example, Section 2 presents theories and definitions related to when scoring functions are equivalent without explaining to the reader why it is necessary to understand or follow the math.

2) Table 1 actually shows that Outlier Exposure outperforms a Binary Classifier on average on the CIFAR-10 dataset? So only on one of the two datasets do we see the claimed result?

3) Experiments are not that thorough. Only CIFAR-10 and CIFAR-100 are used as in-distribution datasets, and OpenImages is the only out-of-distribution dataset.

3) A binary discriminator requires knowing which data is OOD at training time, and having a large amount of this OOD data. In practice, we may not have access to samples from the OOD dataset to train on, or we may have only a small number of samples. It's not clear to me whether other OOD detection methods avoid this requirement.  I would appreciate if there was more discussion about this topic.
summary_of_the_review: The paper was poorly written and difficult to follow. I recommend that the authors try to make their presentation more clear and concise.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper analyzes different OOD detection methods and show that even if the formulation for many OOD methods were different, the binary discrimination is equivalent to those different types of methods when the rankings induced by Bayes optimal classifier are analyzed. They also derive implicit scoring functions for confidence loss of OE and for BGC and compare them with “optimal” scoring functions. They also claim that training binary discriminator (in-dist vs out-dist) in a shared fashion along with standard classifier reaches state-of-the-art OOD performance. 
main_review: This uses new dataset i.e. OpenImages as $D^{ood}_{train}$ instead of TinyImages because it was retracted for ethical concerns. How was this decision to choose this specific dataset motivated? Are there any specific characteristics that you searched for? Have you explored how this train OOD dataset should be chosen?
The method addresses many OOD methods in the literature from the past, but refrained from including other/more recent ones that uses self supervised[5][6], contrastive learning[7][8][9], adversarial resampling[11], energy based methods[9] etc. It would have been great if authors could discuss more on this and present how their method/derivation fits on any(not all) of these more recent frameworks/methods. 

## Minor Issue: 
I think there is a typo in lower table 2 heading, Did you mean C-10 instead of C-100 between SMOOTH and 80M FPR? 

## Strength
This paper provides insights and findings of properties of different OOD detection approaches. This will help in getting more understanding of OOD methods and building novel ones further.  Most of the previous work were mainly empirical and lacked mathematical foundation
The finding of uncertainty based methods being equivalent under the hood with binary discriminators is interesting. It explains and provides some groundings to why those different methods might be performing similar ways.
It includes well-motivated experiments, their clean code implementation and results as well as a comprehensive appendix to support the claims. 
## Weakness
The empirical significance and novelty of third contribution seems limited to me since the claim resembles the ones made by Thulasidasan et al. [2021] and Mohseni et al. [2020]. I’d request the authors to discuss more on the effect of their advancements(using unlabelled data($x_{r}^{IN}$) with BGC) and empirically discuss/show the significance of the gains.  



summary_of_the_review: The paper provides a good survey and insights of existing OOD approaches discussing their similarities and differences. While I commend the mathematical derivation  and analysis of multiple existing OOD methods, the novelty of empirical contributions seem marginally significant to me. I am open to increasing my score  if authors can address some of my concerns above, provide some judgments on more recent methods and/or further insights on the statistical significance (like standard deviation) of the top performing models i.e.  OE, BGC and SHARED_COMBI and provide evidence to significance of the  superiority of SHARED_COMBI over others. Especially since the scores in tables for OE and BGC seem slightly different from the ones reported in the original paper (comparing Appendix F since a different dataset was used in main table). 

### References:
[1] Neal, Lawrence, et al. "Open set learning with counterfactual images." in  ECCV 2018.\
[2] Thulasidasan, Sushil, et. al. (2021) "An Effective Baseline for Robustness to Distributional Shift"\
[3] Mohseni, et al. "Self-Supervised Learning for Generalizable Out-of-Distribution Detection." AAAI. 2020.\
[4] Zhang et. al. "Universum prescription: Regularization using unlabeled data." in AAAI 2017.\
[5] Hendrycks et. al. Using self-supervised learning can improve model robustness and uncertainty. In Neurips 2019.\
[6] Golan et. al. "Deep anomaly detection using geometric transformations." in Neurips 2018.\
[7] Winkens, Bunel, Roy et. al. (2020). Contrastive training for improved out-of-distribution detection.\
[8] Tack, Mo et. al "CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances" in Neurips 2020.\
[9] Liu et. al (2020) "Hybrid discriminative-generative training via contrastive learning."\
[10] Liu et al. "Energy-based Out-of-distribution Detection, in Neurips 2020."\
[11] Li et. al. "Background Data Resampling for Outlier-Aware Classification", in CVPR 2020.\
[12] Hendrycks et al. "Deep anomaly detection with outlier exposure". In ICLR, 2019.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: In this paper, the authors bring together recent work in the OOD detection problem and provide to the reader a sound mathematical framework to understand similarities and differences among these. The framework is based on the equivalence class of scoring function under the AUC / FPR@qTPR metrics and bayes optimality. The tools introduced in the paper allow the authors to explain why different methods perform largely similarly, when one scoring function should be preferred to others and draw conclusions regarding training with / without label data of the in-distribution set.
main_review: + (+) The paper is clearly written, has a good story line, is self contained and appendix A is useful/sufficient to grasp all mathematical parts.
+ (+) The main contribution of the paper is to unify under the same mathematical language different methods that were previously related less clearly/effectively.
+ (+) In the theoretical part of the paper - especially Sec. 3.1 and 3.2 -, mathematical steps are always interwined with intuitive but sound explanations (that I personally found eye-opening at times).

+ (-) The analysis framework is based on bayes optimality which is asymptotic - as the authors state in the ethics and reproducibility statement section at the end of the manuscript. But this is not an ethic nor a reproducibility matter: it is a peculiarity / design choice of this work that should be crystal clear to the reader, who will decide if this is a limitation she cares about or less. I would have liked to read it in Sec. 2 or Sec. 3 at most.
summary_of_the_review: Theoretical insights that unify different methods under the same light and allow new analysis are so valuable to the community, especially when they are so clearly exposed. The authors also show these theoretical insights are actionable, by drawing new conclusions later supported by empirical evidence. I think people working in the field will benefit from such a clear and encompassing narrative.

My review could be positively biased: Since I am not an active researcher in the field it was difficult for me to provide absolute meaning to the numbers in the experimental section, although on a relative scale I was able to follow the authors' conclusions. Because of my lack of expertise in this specific field I cannot provide positive nor negative feedback on this and on the related work reporting.

For these reasons I will take into great consideration all other reviews and discussion for the final score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors aim to to recognize common objectives in OOD detection as well as to identify the implicit scoring functions of different OOD detection methods. They show that binary discrimination between in- and (different) out distributions is equivalent to several different formulations of the OOD detection problem. They find that, when trained in a shared fashion with a standard classifier, this binary
discriminator reaches an OOD detection performance similar to that of Outlier Exposure.
main_review: Strengths:
* The paper is well-written and relatively easy to follow.
* All three of the author's contributions seem interesting and well-supported.
* I feel that the authors achieve notable progress towards their stated goal (a "better understanding of the key components of different OOD detection methods and to identify the key properties which lead to SOTA OOD detection performance"), particularly with their first and third contributions.

Weaknesses:
* In my view, the authors do not make particularly clear what the problem their work is aiming to solve is or why it needs to be solved. What's missing from the existing literature, and why is it important to fill in that gap? Their entire introduction is essentially a "related work" section, and then suddenly shifts to a list of three contributions. But the introduction does not explain the need for these contributions. Their stated goal (which I quoted above) comes across as quite vague. It is therefore also difficult for me as a reviewer (with limited expertise in this area) to make a judgement call as to the importance of this paper.
summary_of_the_review: The authors present well-supported contributions that seem interesting, but do little to explain what problem their contributions solve or why their contributions are needed.
review_title: null

Meta-review: The paper contributes to the understanding of out-of-distribution detection by showing that binary discrimination between in- and out-distribution examples 'is equivalent to several different formulations of the out-of-distribution detection problem'. The paper shows this in an asymptotic setup based on studying likelihood ratios for distinguishing in-distribution examples from out-of-distribution examples. The paper also provides numerical results showing that a simple baseline based on binary classification works well.

The paper got very mixed responses ranging from strong accept to reject:
- Reviewer YhZ7 (recommending 3: reject) raises several important concerns, specifically that the paper doesn't explain the significance of its contributions adequately, that experiments are not thorough enough (for example that only one out-of-distribution dataset is considered), and that to train a binary classifier one needs to have sufficiently many out-of-distribution examples. 
The authors argued in response that the purpose of the paper is to provide an understanding of existing methods that are often empirically driven, made revisions to the exposition, and point out that they actually evaluate on six/seven out-of-distribution test sets. 
After discussion, the reviewer is still concerned that the paper states 'We show that when training the binary discriminator between in- and out-distribution together with a standard classifier on the in-distribution in a shared fashion, the binary discriminator reaches state-of-the-art OOD detection performance' as a contribution and that this claim is not supported by the results in the paper. The authors say they are happy to drop this particular statement and emphasize that their contribution is that that a binary classifier can be a useful tool for OOD detection. The reviewer is not satisfied by this response, as the reviewer feels that this makes the contribution much less impactful. 

- Reviewer iH61 (recommending 6: marginally above, initially reject) pointed out that the significance of one of the contributions is limited, since the claims resemble the ones by Thulasidasan et al. [2021] and Mohseni et al. [2020], and initially recommends to reject. The authors respond that those two papers only aim at good performance, but do not unify existing approaches, as the paper under review does. The reviewer slightly raised their score, but again points out that the previous works already show that a binary discriminator performs well. 

- Reviewer Lwwq (recommending 10: strong accept) appreciates the unification of different methods and votes for strong acceptance. The reviewer also points out that he/she is not an expert in the field, and thus this reviewer's rating should be taken with care. 

- Reviewer YRfA (recommending 8: accept) points out that the authors make notable progress towards a better understanding of OOD methods, but is concerned about what problem the authors are trying to solve and its significance, and states that he/she cannot judge the importance of the paper.

- Reviewer vYWv (recommending 6: marginally above, initially recommending reject) finds that the paper provides helpful insights to connect methods for OOD detection tasks, and weakly recommends acceptance.  

The reviewer's opinions on this paper vary significantly. Initially, a major selling point of the paper was that 'the binary discriminator reaches state-of-the-art OOD detection performance', but after discussion, the authors and reviewers agree that this statement is not supported by experiments, and the idea of using a binary discriminator is also not new, and thus everyone agrees that this statement should be removed. 
This leaves as the major contribution an improved understanding of a variety of methods, and casting them as versions of a binary classifier. 
This by itself would be sufficient to carry a paper, however the stated equivalence is rather weak as it is based on an asymptotic analysis, and in the asymptotic regime, out-of-distribution detection is rather trivial because the distributions are given. This also explains why in the paper's experiments all the methods that are asymptotically related behave quite differently in experiments. 

I do not recommend this paper for acceptance. I've read the paper and I've thought quite a while it and its reviews. I have also discussed the paper with a colleague who works actively on out-of-distribution detection, since I'm not an expert on this topic myself. While in general I find it very valuable to unify and to understand existing out-of-distribution algorithms better, I don't see how the particular interpretation provided by the paper is impactful, since it is unclear how the connection drawn in an asymptotic setup for Bayes classifiers actually extend to concrete OOD detection algorithms, which operate in the finite sample regime.
Decision: Reject