Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents an ensemble-based semi-supervised learning method for novelty detection. 
The goal of their training scheme is to create an ensemble of models that has a high disagreement on the out-of-distribution (OOD) samples in the unlabeled set. The training resembles a self-training algorithm that labels the unlabeled pool and uses implicit regularization via early-stopping to find a "sweet spot" in terms of disagreement on OOD samples between the models in the ensemble.
The final decision of whether an input is considered as out- or in-distribution is based on a hypothesis test using the average disagreement between the softmax outputs in the ensemble. In-distribution samples should be accurately classified with high confidence and show little disagreement in the ensemble while out-distribution samples should be detectable as the different models will produce different outputs. 
main_review: -The setting of the paper assumes that a mixture of in- and out-distribution samples is available during training and the authors argue for the validity of this assumption in the conclusion section of the paper. While I agree that this specific setting can come up in real-world applications, I think it is still important to evaluate unseen out-distributions. For example, if I train a model on medical images using their procedure, can the model detect "garbage samples", for example uniform noise, that might result from a technical failure in the image capturing pipeline. Another interesting setting might be to only train a CIFAR10 model on the first 50 CIFAR100 classes in the unlabeled pool available during training and evaluate OOD-detection performance on the last 50 classes from CIFAR100 and other standard benchmarks such as AUROC against SVHN or Uniform and Gaussian Noise. The paper clearly demonstrates that the model is able to beat baseline methods such as nnPU and MCD if the distribution of out-of-distribution samples stays the same between train and test time, however, I think it important to also evaluate robustness with respect to unseen out-of-distributions, especially as methods such as Outlier Exposure have shown that they can generalize to various unseen out-distributions. 
It might also be interesting to replace the unlabeled set U with a general image dataset with enough variance, such as a subset of 80 million tiny images, OpenImages or YFCC-100M which all contain some samples of the standard CIFAR10/100 classes but also a vast number of unseen classes. Using such a general dataset might also lead to a better generalization to unseen out-distributions, similar to Outlier Exposure and it would allow to computer AUROC of a CIFAR10 model against all CIFAR100 classes, which is a challenging benchmark that is often looked at in related papers. 

-The paper shows strong improvements over a wide variety of baseline methods in terms of AUROC and TNR for several tasks based on SVHN, FMNIST, CIFAR10, CIFAR100 and a medical benchmark. 

-The success of the method largely relies on the models in the ensemble having sufficiently different outputs on OOD samples. The authors argue that during training, the model will first first the correctly labeled samples in U_ID before fitting the wrongly labeled ones. 
In the appendix, the authors give further theoretical insight and state that under some assumptions, there exists a checkpoint after T iterations where the model fits the correctly labeled samples in U_ID but not the incorrectly labeled ones. In practice, this T value can not be calculated so the final checkpoint is selected based on accuracy on a validation set that only contains in-distribution samples. However, I am not sure how this relates to disagreement on actual out-of-distribution samples, which is arguably the goal of the training scheme to achieve best OOD-performance. In detail, what happens if all models in the ensemble predict the same label c on an out-distribution sample, wouldn't all models still learn this output before we end training? This systematic bias might be a problem if the OOD classes are similar to the in-distribution classes, for example train or busses might be similar to trucks in CIFAR10 and as the method does not seem to contain an explicit way to generate diverse initialization for their fine-tuning scheme. 
Figure 9 in the appendix gives hope that one has sufficient disagreement,, however, I am not totally convinced that the first 5 CIFAR10 classes are sufficiently close to the last 5 classes to completely eliminate this concern. 

-I liked that the authors show that the method works with varying proportions of OOD in the unlabeled dataset unless U consists of 95% in-distribution samples. 

-The authors also propose to use a different disagreement score and from Table 3 it seems like at least on some tasks, especially CIFAR100 vs SVHN it greatly increases results. While I value this as a contribution from the authors, it would be interesting to see (if applicable) how ensemble-based baseline methods (such as MCD) perform with this disagreement score. This could further verify that both the new disagreement score and the training scheme are important parts of ERD and the improvements are not mostly caused by the disagreement score. 

-The analysis in section 3.3 assumes that the label is a deterministic function of the datapoint. Typically, one assumes that p(y|x) is not deterministic and even in practice this might not always be true, for example if an image contains both a cat and a dog. Could your theory be applied to this more general setting or would your training algorithm pick up those samples as having high disagreement? 
summary_of_the_review: Overall, while there are relatively few technical novelties in the paper, the given results demonstrate that this simple scheme is able to improve performance over baseline methods in their evaluation.
For me, generalization to unseen OOD-sets is still a very important part of any novelty detection method and should be evaluated. 
As the method heavily relies on implicit regularization of disagreement between models in the ensemble, I think there could be more experiments in the paper that show that there is sufficient disagreement even on close OOD tasks and how this changes during fine-tuning. 
Also, it would be nice to clearly demonstrate that the improvements over the baselines methods are caused by both the new disagreement score and the early-stopping-based training.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Even though conventional OOD detection algorithms can distinguish far OOD samples, current methods that can identify near OOD samples require training with labeled data that is very similar to these near OOD samples. In turn, the authors develop a new ensemble-based procedure for semi-supervised novelty detection (SSND) that only utilizes a mixture of unlabeled ID and OOD samples to achieve good detection performance on near OOD data.
main_review: The paper is generally well-written, however, there is some confusion listed as follows:

1.	How is the ensemble number selected? What is the validation dataset for choosing it?
2.	Some claims are vague in the submission, for example, in section 3.3, the authors claimed that the Maximum discrepancy method (MCD) tends to result in ensembles that do not disagree enough on OOD data and thus subpar performance. Is there any proof or empirical evidence behind such claims?
3.	In figure 5, is it possible that the decrease of validation accuracy is due to the fitting on the labels of the OOD data rather than the noisy in-distribution labels? One quick experiment to verify this is using clean labels for ID data and then fitting on a new OOD dataset with artificial labels to see the accuracy change on the validation dataset.
4.	In figure 5, how is the early stopping tolerance scalar 7 selected? Is there any sensitivity analysis on it?
5.	It is improper to claim that “However, running a grid search to select the right hyperparameters can be more computationally expensive than simply using one run of the training process to select the optimal stopping time”. Since early stopping requires you to do inference per training epoch and it may be more expensive than tuning the hyperparameters using grid search.
6.	How is the proposed method compared to self-supervised-based approaches, such as CSI[1] and SSD[2]? Since these methods do not require ID labels at all, it has fewer requirements than the proposed approach. How is the method compared to different uncertainty scores, such as energy scores [3]?


[1]Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, pages 11839–11852, 2020

[2]Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. In Proceedings of the 9th International Conference on Learning Representations, pages 1–17,2021

[3] Weitang Liu, Xiaoyun Wang, John Owens and Yixuan Li, Energy-based Out-of-distribution Detection, in NeurIPS 2020.


summary_of_the_review: Some confusion exists both empirically and theoretically. I recommend weak-reject at this stage.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a semi-supervised ensemble approach to novelty detection. The main idea consists in generating base classifiers that disagree on the out-of-distribution data (ODD). An early-stop criterion is used to achieve the wanted level of disagreement among the component of the ensemble.

The main assumption is the availability, for training, of clean labeled in-distribution (ID) data, and of unlabeled data which include both ID data and out-of-distribution (OOD) data. The unlabeled data follows the same distribution of the data the ensemble is tested on.

Each base classifier is first trained on the labeled in-distribution (ID) data. For a given base classifier, the unlabeled data is randomly assigned to one of the labels (a different label is chosen for each base classifier), and the classifier is tuned on the resulting labeled data. The tuning is stopped when the optimal tradeoff between high validation accuracy (on inliers) and low training error are achieved. This regularization process is aimed at avoiding fitting the incorrect assigned labels in the unsupervised portion of the training data and to achieve disagreement among the base classifiers on the OOD data.
main_review: Strengths:

- simple idea; easy to understand
- a theoretical justification of the approach is provided (in the appendix) 
- good results

Weaknesses:

- The semi-supervised requirement of the approach limits its applicability. The availability of clean labeled data is rare in practical scenarios.

- The main set of experiments assume that the unlabeled data used during training follows the same distribution of the data used for testing.

- The paper does not clearly identify the limitations of the proposed approach and scenarios in which the approach would not work. 

The scenarios depicted in Figures 2 and 3, while useful to convey the overall idea behind the methodology, show an ideal case only. What happens if the ID classes present a non negligible overlap, and this overlap is also reflected in the unlabeled ID data? Would the early stopping criterion still be effective to achieve the desired disagreement in the ensemble? When does the proposed technique break? Under which conditions? What happens when the OOD do not belong to specific classes with a well-defined support? What happens when there are fewer OOD data. This latter point is somewhat discussed in the appendix, but it should be covered in the main paper as well. 

The data used for Figure 1 should be specified.

The authors need to make it clear in Algorithm 1 that the validation set V is made up of in-distribution data.

How was the ensemble size K=5 chosen?

It's unclear whether the values in Table 2 are averages of multiple runs, as they should be. Statistical significance should also be reported.

In Table 2, for each dataset half of the classes were used to construct the ID data, and half for the OOD data. How was the desired ratio between ID and OOD achieved? Were the classes for OOD data sub-sampled? This is not discussed in the paper.


summary_of_the_review: A reasonable, simple, and interesting approach for anomaly detection with ensembles. The main weaknesses are the reliability on clean ID data and the lack of a discussion and analysis of the approach limitations.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors develop an ensemble-based procedure for semi-supervised novelty detection (SSND). It utilizes a mixture of unlabeled ID and OOD samples to perform on near OOD data. A regularization technique is further used  to promote diversity on the OOD data while preserving agreement on ID data.
main_review: Strength:
1. Strong empirical Results on Hard/Near OOD settings.
2.Theoretical insights on the benefits of training with early stopping and the usage of artificial labels 
Weakness:
1. The setting of this framework is too restrictive and thus makes the proposed method less practical for real life applications.  For instance, the proposed method requires a carefully selected validation set with ID samples and an unlabeled test set where half of the samples are ID and the other half are OODs. In practice, it is difficult, and somes impossible, to predefine the OOD samples and thus it is not likely to prepare such a high-quality validation set.  Also, it is necessary to investigate how the distributions of the validation set will affect the final performance. Moreover,  as the OOD samples are relatively rare to obtain, the resulting validation set could be highly imbalanced. Hence, using the validation accuracy to guide the model learning is definitely not an optimal solution. 
2. The experimental comparisons are not fair. The proposed methods are based on esnemble of 5 networks and this will linearly increase the computational complexity. Thus, it is necessary to have a more fair ``Vanilla Ensembles" by training all the baseline method 5 times with some randmization techniques, such as random sampling on the training set (bagging).
3. The training pipeline of the proposed method is complicated and it is not clear how will each part affect the final performance. For instance, the proposed method finetune K models, where each model will choose one different artificial label on the unlabled set. How will the choice of this artificial label affect the final performance?
summary_of_the_review: The proposed method have some empirical results on the Hard/Near OOD. However, this relies heavily on the carefully selected  and high-quality validation sets.  Also, the experimental comparison is not fair.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a novel transductive novelty detection method using the disagreements of the ensemble models.
More specifically, the authors directly utilize the unlabeled test set samples, providing different labels for those unlabeled samples, and training multiple models with fine-tuning. Then, this framework tries to identify the OOD samples using the disagreement of those samples.
The authors provide some promising experimental results to discover the OOD samples in transductive settings.
main_review: 1. Problem setting
- In the introduction, the authors mentioned that we have access to the unlabeled test data that includes ID and OOD samples (and using this for training). 
- It means that we already have access to the OOD and this may not be applicable to discovering new classes in the inference time that we have never seen before.
- OOD means that the samples from the out of the training set distribution. However, if we use some OOD samples during training, those samples are no longer OOD.
- If we use this for a testing batch, we should retrain the model with a new testing batch. This may be not practical in some online settings. 
- It would be good to mention the limitations clearly. Like changing the title as "transductive novelty detection instead of just novelty detection)

2. Number of unlabeled samples
- It seems like the number of unlabeled samples are also important for fine-tuning.
- This is because if we have too small a number of unlabeled samples, it will not provide enough randomness to the ensemble.
- It would be great if we have this additional analysis with different numbers of unlabeled data. (including very small number of unlabeled samples)

3. Number of label categories
- How about if the problem is binary classification?
- In that case, K should be 2. (We can increase K but it does not have an impact).
- In that case, I think the randomness that the ensemble can get is not enough.

4. Validation set
- Based on Section 3, it seems like the validation set is the subset of ID.
- In that case, the best validation performance would be "before" fine-tuning. Because, before fine-tuning, we only use the ID samples to train the model.
- With newly labeled unlabeled data, if the number of labeled categories is large, most of them are incorrectly labeled. Therefore, I think the validation error should be decreased from the beginning.
- The authors said that they wait for 1 epoch. However, it is not sure whether this rule of thumbs can be applicable to the general datasets.

5. Ablation studies
- It would be also good if the methods can work with different K. Especially, if the label categories are small (e.g., binary classification).
- It would be good if the methods can work with small unlabeled sets like 10, 20. 
- It would be good if the methods can work with different OOD data ratios in the unlabeled data. (Like 1%, 2% OOD and 99%, 98% ID samples in the test set)
summary_of_the_review: Strengths
- The proposed method is intuitively making sense. 
- The main idea "Introducing the randomness to multiple models using different label assignments to the unlabeled datasets" would be promising to discover the samples which are not presented in training set.
- The experimental results are clear and concrete. The baselines are somewhat complete and the comparisons seem fair.

Weakness
- We need another training procedure for different testing set which is less practical. 
- Experimental settings should be more diverse. More ablation studies are needed. 
- Robustness of the model training is one of the biggest concerns about this paper. Relying on the validation error for early stopping to avoid overfitting to the ID unlabeled samples seem somewhat heuristic.

--------------------------------------------------

I am appreciate to get the detailed responses from the authors in this rebuttal.
I carefully read all the reviews from other reviewers and authors response.
However, there are still remaining concerns.

1. Practicality of the settings
- Not only me, other reviewers also concerned on the setting of this paper.
- Also, the authors did not answer about re-training problem with new test set. Retraining again and again for new test set would be very painful.

2. Fundamental limitations of the methods
- The robustness of the proposed method is highly related to the diversity of the submodels for the ensemble.
- However, this diversity is significantly limited if the number of unlabeled data is small or the problem is the binary classification.
- Even though the authors provide some interesting results with this setting, the fundamental limitations of the model itself remain concerns.

3. Early stopping
- As I mentioned above, the best validation performance would be the model "before the fine-tuning" because there is no distribution mismatch between training and validation. 
- It is unclear how fundamentally address this problem even after reading the rebuttals.

Therefore, even though the authors provide some additional experiments (thank you), I am going to stay on my score (6) and cannot provide more than the score (6).
review_title: null

Meta-review: The authors propose a semi-supervised novelty detection method which tries to identify out-of-distribution samples in the unlabeled data (consisting of in- and out-distribution samples) using a disagreement score of an ensemble. The ensemble is generated by fine-tuning the trained classiifer on the labeled training data plus the unlabeled data which all get a fixed label (which is repeated several times to generate the ensemble). The main idea is that one uses early stopping based on an in-distribution validation set in order to avoid overfitting on the unlabeled points which allows then identification of the out-distribution points via the disagreement score.

The reviewers appreciated the simplicity of the approach and the extensive experimental results. The authors did a good job in trying to answer all questions and concerns of the reviewers. 

However, some concerns remained:
- the setting assumes that the OOD data is fixed which was considered as partially unrealistic and thus evaluation of the OOD detection performance on unseen OOD distributions was requested in order to understand the limitations of the method (this was only partially done by the authors). 
- the theoretical result is for a two-layer network and completely based on previous work. As the authors use much deeper networks later on in the experiments, this result cannot be used to theoretically justify the approach. 
- there remained concerns about the necessary diversity of the ensemble and the early stopping procedure

While I think that the paper has its merits, it is not yet ready for publication. I encourage the authors to to take into account the above points and other remaining concerns of the reviewers in a revised version.
Decision: Reject