Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper attempts to extend the existing OOD detection problem to a real world scenario where the in-distribution data is only partially labeled (semi-supervised learning), unlike existing OOD problem formulation where they assume that in-distribution data is fully labeled. The proposed approach, termed as STEP in the paper, extends the Mahalanobis distance based OOD approach into the given semi-supervised OOD detection problem formulation. The proposed approach is evaluated on OOD benchmark with the most recent methods proposed for addressing traditional OOD formulation. Extensive experiment shows the effectiveness of proposed method for the proposed semi-supervised OOD detection problem as compared to existing methods.
main_review: + The paper is very well written, has good structure, and is easy to follow. It does a good job in proposing the problem formulation and clearly explaining how the proposed OOD detection problem formulation defers from the typically used OOD detection problem formulation.


+ The paper focuses on Mahanalobis distance based OOD detection. Sec.2.2 points out some of the difficulties in adapting existing Mahanalobis distance based OOD detection methods to the proposed semi-supervised OOD setting. The arguments made by the paper are reasonable and the formulation for semi-supervised OOD that follows also makes sense.


+ Based on the objective formulated to overcome the estimation inaccuracies in computing covariance matrix under semi-supervised OOD formulation. To create a robust Mahanalobis distance based OOD detector, a novel training strategy is derived from the updated formulation which utilizes a projection matrix to overcome on the limited availability of labeled data.


+ Experiments are conducted on standard benchmark datasets used for supervised OOD formulation. The comparison with most recent OOD methods is also provided and across multiple metrics, the proposed method is able to achieve better performance. Specifically, over the regular Mahanalobis method. Furthermore, ablation analysis shows different configuration of the proposed method providing key insights into the various aspects of the proposed idea.




limitations_and_societal_impact: - Though the problem formulation is novel and clearly explained, there is still an inherent assumption over the data which is not given as much attention. There is an inherent assumption in the proposed semi-supervised OOD formulation that all the unlabeled in-distribution data belongs to only same K category as that of partially labeled data. However, if we were to consider that majority of training data is unlabeled in a true sense, there is no way of knowing what categories are present. If we consider this case, the proposed formulation should be modified to include known-unknown-distribution in unlabeled data as well, in addition to unknown-unknown-distribution used only during testing.


- A possible way to change this formulation in my suggestion is borrowing few aspects of open-set semi supervised learning problem formulation as proposed here [1]. Though, it only deals with known-unknown-distribution in unlabeled data, in my opinion, it is more realistic scenario and stays true to the practical definition of unlabeled data.


- The paper does an excellent job in comparing with existing most recent methods proposed to solve supervised OOD detection. However, the need of the proposed method is not properly established through experiments. To fully validate why we need to follow the proposed method for semi-supervised OOD, a baseline having SSL method as first step and OOD detector as second step should be created, SSL+OOD. For example, the model should be trained with a popular SSL method first like MixMatch [2] and later the model should utilize confident predictions to create Mahanalobis based [3] OOD detector. This provides a strong baseline for proposed problem formulation. Since, it the method compared in the paper were created with the supervised OOD detection in mind, directly comparing with them is not entirely fair to those methods. The improvement over suggested baseline would provide a more accurate comparison as it is trained by considering both SSL and OOD nature of the problem individually. In my opinion, the effectiveness of the method should come from the consideration of SSL and OOD nature jointly while developing the method.



[1] Yu, Qing, et al. "Multi-task curriculum framework for open-set semi-supervised learning." European Conference on Computer Vision. Springer, Cham, 2020.


[2] Berthelot, David, et al. "MixMatch: A Holistic Approach to Semi-Supervised Learning." Advances in Neural Information Processing Systems 32 (2019).


[3] Lee, Kimin, et al. "A simple unified framework for detecting out-of-distribution samples and adversarial attacks." Advances in neural information processing systems 31 (2018).
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The paper introduces a semi-supervised OOD detection setting and proposes STEP as an approach for this situation. This presented methods combines a Mahalanobis distance with a semi-supervisedly trained backbone model using topological structure preserving optimized projections.

main_review: The description of the methodology is not clear enough for me to see that it is sound and that the motivating principles are fulfilled.
The presented evaluation is insufficient to judge the efficacy of the presented method. The results are only shown for four OOD datasets, with two pairs of them being the same data only downscaled in different ways.
Availability of validation data from the tested out-distribution can not be assumed in an OOD detection setting, as is mentitoned in l.200. However, the evaluation (Fig. 2) for this most relevant setting of 'new OOD datasets' is extremely lacking: only two evaluations are shown, and for each the OOD dataset used for training only differs from the test dataset in in image rescaling methods. A sufficient evaluation would need to include several new OOD datasets, for example as in https://arxiv.org/abs/1812.04606 (Hendrycks et al.'s Outlier Exposure) Table 7.
Overall, problems of language and structure make the paper very difficult to read and particularly the methodological section hard to assess. The results section is not comprehensive enough to appraise the practical benefits of the method. 


Questions and points that remained unclear to me:
 - l. 96 How is the covariance matrix calculated? It is taken over all classes, and estimated on all ID samples, right? If so, how do missing labels make it less accurate?
 - Eq. 3: Should the first line be a sum instead of a quantifier over all i,j?
 - The discussion in 2.2 works with Mahalanobis distance only in the feature space (=input image space). Previous approaches [27] use latent representations of a classifier. Could this not be applied here with a classifier trained on the limited labeled data? In 2.3.2. it is still not clear to me where intermediate representations are actually used, since a backbone is mentioned for the calculation of Sigma-hat, but it is still a matrix in image space.
 - l. 142 & Eq. 7: What are the "detection-specific space" and N?
limitations_and_societal_impact: Yes, the authors have included a very adequate Broader Impact section.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposed semi-supervised OOD detection and a novel objective formulation that learns a new representation space in which OOD samples could be separated well.
main_review: Pros: The setting is reasonable and the solution seems valid.

Cons:
1.	From my part, this work seems to heavily rely on pre-trained networks by contrastive learning. I am curious to see how is the method perform compared to the pure contrastive learning-based method, such as CSI [1] and SSD[2]. Unfortunately, Table 2 contains no information about this comparison.
2.	The algorithm seems to have a O(n^2) complexity, I am excited to know what is the typical training time on benchmark datasets.
3.	It looks like the hyperparameters are tuned on the test set in Figure 3 and the best result is presented in the main table. Besides, I wonder how is the weight determined for the two losses? Is the weight 1 optimal one?
4.	It would be better to present the results under some other detection metrics in the experiment section, such as the Mahalanobis distance, energy score.
5.	I am wondering how is your method compared with those baselines based on likelihood ratios under deep generative models? Moreover, some popular baselines are important to compare with from my side, such as MSP, Gram matrices[3].
6.	It is a bit unclear that how is the ensemble of representations from each backbone network’s layer used in experiments? Does each baseline have such a design?
7.	It is interesting to look at the in-distribution classification accuracy of your proposed method on benchmark datasets.

Minor points:
1.	Maybe it will give the proposed work a more comprehensive evaluation if some results on different network architectures are presented, say Wideresnet?
2.	It is interesting to observe whether semi-supervised methods, such Mixmatch[4] perform well for OOD detection?

[1]Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, pages 11839–11852, 2020

[2]Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. In Proceedings of the 9th International Conference on Learning Representations, pages 1–17,2021

[3]Chandramouli Shama Sastry, Sageev Oore, Detecting Out-of-Distribution Examples with In-distribution Examples and Gram Matrices, ICML2020

[4]David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, Colin Raffel, MixMatch: A Holistic Approach to Semi-Supervised Learning, NeurIPS 2019


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: This paper studies semi-supervised learning with limited label data and  unlabeled data with OOD samples. The covariance in the Mahalanobis distance cannot be estimated in a robust way when only limited labeled data is available. The main idea of the paper is to find a projection  space that will preserve the local topology of the data points. More specifically, the projection matrix should maximize the distance between a given sample and all other  samples subject to the condition that the Mahalanobis distances between this point and points within its local neighborhood are preserved. 
main_review: If Mahalanobis distance is not a reliable measure due to limited data size then it is not clear why the proposed approach is trying to preserve this distance in the objective function. 

The Mahalanobis distance in the original space becomes the Euclidean distance in the projected space if the projection matrix is selected as the lower triangular matrix of the precision matrix. OK but how is this going to help estimate a more robust precision matrix? In order for Cholesky to work the precision matrix has to be well-conditioned, which means for every class at least d+1 samples are needed.

Not sure why the OOD datasets are selected from different datasets when a subset of the classes in CIFAR-10  or CIFAR-100 could have been used as OOD. Using scene dataset as OOD in CIFAR-10/100 makes the problem unrealistically easier.

Writing of the paper needs improvement. There were many grammar issues and broken sentences. 

---------------
Update:

Regrettably, the author rebuttal does not address my main concerns about the paper. I am not convinced that the main idea of the paper that aims to leverage ill-conditioned Mahalanobis distance to learn useful structural information about the data so as to more effectively perform OOD detection will be very effective in OOD detection.  I stand with my original rating of "clear reject".

limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The paper attempts to extend the existing OOD detection problem to a real world scenario where the in-distribution data is only partially labeled (semi-supervised learning), unlike existing OOD problem formulation where they assume that in-distribution data is fully labeled. The proposed approach, termed as STEP in the paper, extends the Mahalanobis distance based OOD approach into the given semi-supervised OOD detection problem formulation. The proposed approach is evaluated on OOD benchmark with the most recent methods proposed for addressing traditional OOD formulation. Extensive experiment shows the effectiveness of proposed method for the proposed semi-supervised OOD detection problem as compared to existing methods.

However, there exists some limitations as follows. 

1) Theory: the theory part is not thoroughly evaluated. 

2) Experiments: 2.1) Insufficient OOD experiments: the authors only consider CIFAR-10/100, while most OOD papers also experiment with SVHN and TinyImageNet dataset. 2.2) Baselines: how is your method compared with those baselines based on likelihood ratios under deep generative models? Moreover, some popular baselines are important to compare with, such as MSP, Gram matrices.

In the end, although the idea and problem formulation presented in the paper has sufficient contribution to make to the field, I think that this paper may not be ready for publication at NeurIPS, but the next version must be a strong paper if above limitations can be well addressed.
Decision: Accept (Poster)