Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a new approach to detect out-of-distribution (OOD) examples in the transductive setting. The idea is to train an ensemble of models that fit the in-distribution (ID) data well, but disagree with each other on OOD examples.

Pros
+ Extensive experiments are conducted to compare the proposed alogrithm with a number of baselines.  The proposed algorithm seems to significantly out-perform the baselines.
+ The paper is generally well-written and easy to follow.

Cons
- The concept of the F-detectable OOD set OOD(P, F) is somewhat misleading: the name suggests that it is a set of OOD examples, while in fact this set is not necessarily a subset of OOD examples, because when there are multiple empirical minimizer for P, they may disagree on some ID examples.
- A key assumption (the empirical minimizers of F generalises well) is justified by a miscitation: the paper states that Theisen et al. 2020 shows that the assumption seems to hold true for trained deep neural networks, while that paper only discusses overparametrized linear classifiers.
- Another key assumption (Assumption 2.1) seems rather strong and unlikely to hold in practice, and its validity seems to be overstated. Specifically, the assumption assumes an empirical minimizer of the training set augmented with ID test examples labelled using an arbitrary label is an empirical minimizer on the training set. This will imply that the ID test examples are not important in the underlying data distribution. While the papers states that "In Section 3 we provide empirical evidence which shows that Assumption 2.1 holds in many practical scenarios", there is no direct support for the assumption in Sec 3.
- The paper states that OOD(P, \tilde{F}) \subset OOD(P, F), because \tilde{F} \subset F. This is not necessarily true, because the empirical minimizers in \tilde{F} can disagree on more ID examples than the empirical minimizers in F.
- Fig. 4 is used to justify that early stopping is an effective regularization method. However, the figure shows that at around epoch 35, there is a sharp drop of the validation accuracy, and early stopping may stop training here, making the network underfit.
- The paper doesn't seem to describe how the test statistics are converted into predictions of whether the examples are OOD. This is important for reproducibility and is needed for justifying the soundness of the algorithm.

Minor comments
- Write down ID in full when it's used the first time
- Fig. 2 is not mentioned in the main text. In addition, the caption doesn't seem to match the figure.
- Fig. 3: what does zoro bias mean here?
- Alg. 1: y_{1}, ..., y_{K} have already been used a the training set labels.
- The "two sample test" doesn't really use two samples.

**Post-rebuttal**

The revised version removed questionable or invalid notations/assumptions/justifications, and there are some defences for previous formal justifications in the rebuttal. IMHO, the rebuttal raises further concerns about the technical quality, and the paper still requires stronger justification for acceptance. I'll lower my score instead.

Specifically, I find the rebuttal generally confusing, and I disagree with various points in the rebuttal/revised version.

- Condition 3.1 doesn't seem right: by definition, a classifier in 
 has a misclassification probability of at most  on the ID data, but Condition 3.1 states that the misclassification rate is ? Importantly, I don't see why this condition is needed as there is no justification on how it is connected to the disagreement test. In addition, "As a consequence, with very high probability 1 − (1 − δ)s we cannot fit a set of s random i.i.d. in-distribution points with the wrong label" is quite vauge and doesn't seem right either.
- "According to the definition, there exists a class of functions F that is complex enough such that OOD(P, F) is the complement of the support of the training distribution.": I don't think the definition implies this. In addition, if OOD(P, F) is the complement of the training distribution, then ID examples not in the training distribution are included in OOD(P, F).
- "If we defined OOD(P_n, F) with P_n the empirical distribution, then the reviewer would be correct and indeed this set could contain ID samples.": in theory, OOD(P, F) CAN contain ID samples, unless additional assumptions are made.
- "two-sample test": while I think this is a minor issue to call the test a two-sample test, I still don't think it agrees with the standard usage of the term (which means two random samples are used).
- "one may simply pick the time point with the highest validation accuracy, after training for a fixed number of epochs": I doubt that this can be called "early stopping". So I don't think the response addresses my question regarding Fig. 5 (previous Fig. 4). In addition, since "early stopping" is used for regularization, this makes it questionable whether "regularization" is indeed needed.
- The test statistic is included in the main paper now, but how about the threshold? In rigorous statistical testing, the threshold can be rigorously calculated, while in this case, it is not clear how the threshold is set.
review_title: strong experimental results, but more rigorous justification and further details are required

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a technique to detect OOD data using disagreement among an ensemble of classifiers. It assumes a transductive setup where OOD data is available during training time.


1. The application setting for OOD as stated in the paper is incorrect:

Abstract: "For settings where the test data is available at training time.."

Introduction: "..Most prior work approaches OOD detection inductively, trying to infer OOD samples after only observing the (ID) training data,.."

The main property defining OOD data is that it is never seen during training. The problem fundamentally is that of detecting unknown-unknowns. This is what makes the problem hard and sets it apart. The proposed approach violates this and hence is not justified as an OOD detection algorithm. The paper text (including title) should be changed.

The proposed technique should instead be positioned as one that helps in discriminating between two known distributions. In that respect, the baseline/competitor algorithms should be selected appropriately. The current choices (except Malalanobis-T) are biased towards the proposed algorithm.

2. Section 2, ID data.: It is assumed for simplicity that a deterministic function f* maps ID data to correct labels. However, this assumption looses generalizability. Considering that real world data has a lot of labeling noise, most such functions as f* would end up memorizing the train data and would have bad performance on ID data in the test set. The paper is not clear what would happen if the train data is noisy.

3. Section 2, Transductive OOD Detection: Here the assumption is that there is lot more correctly labeled train data than test data. The basis for this assumption is not clear and clearly limits the applicability. In most transductive cases, the opposite is true: there is more unlabeled data than labeled, and semi-supervised techniques are used to reduce human labeling effort.

4. Section 2.2, Regularized ensembles: "..points in T for which at least two models in the..." -- Does that not make the model fragile? In case the ensemble has 20 members and only one member disagrees with 19 others, would it be justified to label the data as outlier/OOD?

5. Section 2.2: "...our method indicates that using the regularized function class suffices for many hard OOD detection scenarios." -- This is an overstatement considering that results were presented on only a couple of datasets (CIFAR variants, SVHN)

6. Section 3.2, Statistical test for OOD detection: "The null hypothesis is accepted for high values of Tavg-TV." -- I am missing something here: if there is a lot of disagreement among ensembles, then d_TV would be high and as a result for such data Tavg-TV would be also high. Hence, high value of Tavg-TV should probably result in rejecting of null hypothesis instead of accepting.


============================
Update after going through the updated paper and discussions
---------------------------------------------
The paper adds better illustrations in the revised paper to explain the technique and the approach seems to be effective even though it is simple. Given the approach, 'transductive' in the title looks okay.

Figure 1, Figure 3, and Figure 4 in the current version suggest that the one-class SVM/SVDD techniques [1, 2] would be important to compare against. Popular anomaly/outlier detection algorithms Isolation Forest [3], LOF [4] would also be relevant here. The paper can be strengthened with these additional baselines. RETO has at least one short-coming over the outlier detector techniques: it needs to wait some time until it can collect enough test data.

[1] Lukas Ruff, Robert A Vandermeulen, Nico Görnitz, Lucas Deecke, Shoaib A Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In ICML, volume 80, pp. 4390–4399, 2018.

[2] Lukas Ruff, Robert A Vandermeulen, et. al., Deep Semi-Supervised Anomaly Detection, ICLR 2020.

[3] Liu, Fei Tony; Ting, Kai Ming; Zhou, Zhi-Hua (December 2008). "Isolation-based Anomaly Detection". ACM Transactions on Knowledge Discovery from Data

[4] Breunig, M. M.; Kriegel, H.-P.; Ng, R. T.; Sander, J. (2000). LOF: Identifying Density-based Local Outliers. Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data. SIGMOD.
review_title: A technique for discriminating between two known distributions, incorrectly claimed to be OOD detector.

final_score: rating: 6: Marginally above acceptance threshold
review_content: I think the authors did a great job with the overall structure of the manuscript. It is great to see so many figures that illustrate the different settings, also it really helps to have the main research questions exposed right in the corresponding paragraph. 

One superficial thing that could be improved is that often times I found the mathematical notation to be not very ‘convenient’ in that it wasn’t really necessary for a derivation or the like, but it made it really difficult to read the text, as it required jumping back and forth between the occurrences of a term and its definition, while the actual meaning of a term wouldn’t have taken much more space than its mathematical counterpart. 

A minor thing: the authors use the abbreviation RETO in the abstract without mentioning the full name. 

Figure 1 is great, I really appreciate these types of figures and I believe these comparisons are a great contribution to the community. Yet without details on the experimental settings that led to these results it is rather difficult to assess the value of this contribution. In particular as the figure is a central part of the motivation of the proposed approach, it would be great to at least reference the experimental details in Appendix A.1 - but even after having found this information (which shouldn’t be hidden from the reader for such essential parts), it remains a bit unclear how exactly the parameters were chosen and which experiments were performed exactly. 

One of the main strengths of the paper is the extensive experimental validation. The authors did a great job at comparing with a variety of different methods. 

The authors state that “For simplicity, we consider a noiseless setting, i.e. we assume that there exists a deterministic function f*”. It is perfectly fair to make such an assumption, but it is also important to keep in mind that in many real world application scenarios, especially when ground truth training data was acquired in an automated fashion or in crowdsourced annotation, label noise is not unlikely to violate that assumption. So it could help, for the assessment of the manuscript, to mention when this assumption would be violated in real world settings and how the proposed approach would be affected by such an assumption or a violation thereof.

The authors also state that “Empirically, this assumption [generalizability] seems to hold true for trained deep neural networks” and refer to Theisen 2020 - I might be missing something, but that reference is focussing on defining the “typical” rather than “worst” case, in particular for linear methods. In light of other findings like those that highlight that Neural networks will learn even noise labels perfectly fine, see e.g. Zhang et al https://arxiv.org/abs/1611.03530 it would be great to substantiate that claim a bit more. 

The authors state that “Yu & Aizawa (2019) consider a similar transductive setting” and argue that that approach has the disadvantage that "the method produces indistinguishable outputs on ID and OOD samples”. I’m not sure I understand why that distinction is an advantage, after all the ultimate goal would be to perform well on target test data, independent of whether that data is ID or OOD. 

A major limitation of the experimental validation is that bayesian methods are excluded from the evaluation. I have to admit I am not the most bayesian person I know, and I really enjoyed reading the Ovadia et al. 2019 NIPS paper, referenced by the authors, and I agree that this is a fair point, but after all this sort of transductive setting could be considered as one of the main strengths of bayesian/probabilistic networks. I would have expected to see at least one method from that field in the comparisons. 

Overall the experimental results are very impressive. Yet there are some methodological details that could be clarified, next to the points listed above. Most importantly it felt that the hyperparameters of all methods were chosen in a bit of an ad hoc fashion. I think it would substantially improve the results to employ standard practices for the hyperparameter optimization for all methods, even just grid search or random search would be fine.

 
review_title: Simple method achieving impressive empirical results with some potential for improvement w.r.t. clarity of experimental procedures

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The problem of good predictive uncertainty-based out of distribution (OOD) detection is essential for classification systems to be deployed in safety-critical environments. The authors present a method RETO that achieves state-of-the-art performance in a transductive OOD detection setting. Like other predictive uncertainty-based approaches RETO can ultimately be used downstream on problems like active learning or abstaining on OOD samples in combination with selective classification.

Benchmark data such as CIFAR, SVHN, and MNIST are used to compare conventional and proposed baseline methods, such as k-NN, Vanilla Ensembles OE, Mahal, Mahal-T, and MCD. Experimental results, including those of supplemental materials, show that the proposed method provides good accuracy while reflecting the hardness of the task.

However, there is not enough discussion on how the proposed method can achieve such a high level of accuracy compared to the conventional methods; early stopping is used in RETO, but is it promised to reproduce the same level of performance in other tasks?


review_title: Good performance for out of distribution (OOD) detection.

Meta-review: Although the rebuttal helped clarify the reviewers' confusion on notational confusion and the motivation of problem setup, all reviewers are still in a position of unable to championing the paper:
- the technical concerns by Reviewer 4 need to addressed
- the paper would have been stronger if baselines such as one-class classification / outlier detection have been compared
- the algorithm also has at least one short-coming over other techniques that it needs to wait some time until it can collect enough test data

We hope the reviews can help the authors strengthen the paper in the next revision. 
Decision: Reject