Pseudo-Labels are All You Need for Out-Of-Distribution Detection

Authors: Atik Garg, Yu-Shuen Wang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Pseudo-Labels, Unsupervised Out-of-Distribution Detection

Abstract:
Detecting out-of-distribution (OOD) samples is a significant challenge in real-world deep-learning applications, such as medical imaging and autonomous driving. Traditional machine learning models, primarily trained on in-distribution (ID) data, often struggle when encountering OOD instances, resulting in unreliable predictions. While supervised OOD detection methods generally outperform unsupervised approaches due to the availability of labeled data, our research uncovers a crucial insight: their success is not necessarily due to recognizing the actual object categories in the images; instead, these methods rely on a specific classification strategy that may not correspond to real-world understanding. Essentially, supervised methods detect OOD samples by identifying the difficulties in classifying unfamiliar data. This challenge is similar to what unsupervised OOD detection methods face, as they also depend on the failure to reconstruct OOD data due to the lack of prior exposure. In this study, we bridge the gap between supervised and unsupervised OOD detection by introducing a novel approach that trains models to classify data into pseudo-categories. We employ self-supervised learning (SSL) to convert raw data into representations, which are then clustered to generate pseudo-labels. These pseudo-labels are subsequently used to train a classifier, enabling its OOD detection capabilities. Experimental results show that our approach surpasses state-of-the-art techniques. Furthermore, by training models on different sets of pseudo-labels derived from the dataset, we enhance the robustness and reliability of our OOD detection method.

Decision: reject
Reviewer scores: 3, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1729503865975

thread_type:
official_review

<thread_object>
creator: reviewer_rnuz

<content>
summary:
This paper proposed an unsupervised OOD detection method using pseudo-categories generation and self-supervised learning, which achieved quite good performance in the OOD benchmark for CIFAR10. The theory is interesting and point out a critical problem: "Does a specific classification strategy necessarily align with real-world understanding? "

strengths:
The authors propose the use of pseudo-categories generation and self-supervised learning to enhance the unsupervised OOD detection.

The idea is well reasoned with supporting experiments.

The authors propose framework with different backbone to improve performance through combination of  self-supervised training and supervised pseudo-classification learning.

The paper is well written and easy to follow.

weaknesses:
I reckon there are several critical weaknesses in this paper.

1. The paper does not correctly point out the weaknesses of supervised and unsupervised OOD detection. In line 43-44, the reliance on extensive human-labeled data is not the main weakness for all supervised methods, but only for methods based on outliers exposure. Similarly, in line 47-48, "slow" and "complexity of models" are not the main limitations for unsupervised OOD detection. The objective of this paper is not clear.

2. A deeper investigation of why and how self-supervised learning happen must be provided. The main manuscript does not provide details and evaluations for self-supervised learning, which seems to be important in the proposed framework (correct me if I am wrong). Moreover, the explanation preceding the method is very difficult to understand, although the method itself is straightforward.

3. The ablation study is not sufficient. The ablations on self-supervised learning task and other clustering methods should be provided.

4. The results are not valid enough. The main results are only based on the OOD detection performance on CIFAR-10 (in-distribution dataset). At least, the experiment should be applied on more in-distribution datasets like CIFAR -100, minst, and imagenet. See :
 J. Yang, P. Wang, D. Zou, Z. Zhou, K. Ding, W. Peng, H. Wang, G. Chen, B. Li, Y. Sun, et al. Openood: Benchmarking generalized out-of-distribution detection. Advances in Neural Information Processing Systems, 35:32598–32611, 2022.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729804394289

thread_type:
official_review

<thread_object>
creator: reviewer_ccfa

<content>
summary:
The paper presents a method to bridge the gap between supervised and unsupervised OOD detection methods by training models to classify data into pseudo categories. They employ a self supervised learning (SSL) technique for feature extraction, and used clustering to generate pseudo labels for training to realize robust OOD detection framework. Overall, the paper is written well and is well motivated by the fact that pseudo labels from high-level representations can improve the classification performance. The authors have also conducted significant amount of experiments and provided discussions on various test scenarios to validate their performance.

strengths:
The paper is written well and easy to follow up. The motivation for detection OOD samples using a SSL technique is quite interesting. There are different types of experiments done to validate the proposed method such as comparison with different backbone networks, and using different types of OOD and ID datasets.

weaknesses:
The novelty of the paper is not that convincing because they have integrated existing methods such as BT loss and mixup regularization into the training objective. In the introduction section, the main contribution is not clear. Although the paper provides motivation on how pseudo labels from high-level representation are necessary for effective OOD classification but this hypothesis is not well proved in the pilot study. The paper seems to use different methods such as a specific siamese network Barlow Twins architecture, but why is this used and why is this preferred over other architecture is not discussed. Also, the integration of BT loss with mixup regularization is not clear. The explanation saying "for the regularization" is not convincing. There is not enough evidence to support this, or the authors have not done any ablation on this to the least. Finally, the paper lacks the information about reproducibility. Not enough information regarding the algorithms and source codes are provide.

questions:
1. What is cluster-based labels? It should be first described as the reader will not have knowledge about it in prior while reading the paper.
2. In Table. 2 for random label assignment, why the auroc is significantly higher than its test accuracy counterpart.? While for raw image clustering and GT labels, the difference between auroc and test accuarcy is not that significant?
2. In line 183, findings and analysis, what does certain methodology mean? There should be clear explanation of this method.
3. How is the method reliant on low-level pixel colors? There is no enough evidence to support this statement. Also not enough prior information is provided on this matter. 
4. Based on the pilot study, it is not concluded that pseudo labels need to be derived from high-level representations. Instead it is concluded that, because existing classification networks uses the representations of input from a layer it performs better. Its not that, if pseudo labels are derived from these representations, the classification accuracy will increase. This explanation is naive and enough proof is not provided to support this hypothesis. There is not enough proof provided using the pseudo labels and the high-label representations.
5. In section, 3.3, it is said that Y^A and Y^B are passed through encoder and a projector. What is a projector? Is it a decoder?
6. What us Y^M here and where does it come from?
7. The schematics of the figure 1 is quite irrelevant to the explanation provided in section 3.3. For instance, what does dotted vertical line represent between two encoders. And Where is Y^M coming from ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655576233

thread_type:
official_review

<thread_object>
creator: reviewer_chkx

<content>
summary:
This paper seeks to bridge the gap between supervised and unsupervised OOD detection by proposing a new approach that trains models to categorize data into pseudo-classes. Through self-supervised learning (SSL), raw data is transformed into representations, which are then clustered to create pseudo-labels. These pseudo-labels are used to train a classifier capable of OOD detection. The method is evaluated against several supervised and unsupervised OOD detection techniques, and an in-depth discussion of its various aspects is provided.

strengths:
1. The writing is straightforward and clear, and the relevant works in introduction and related works part are also clear. The logic of the method section is coherent, and the experimental setup is thoroughly described.
2. This paper conducts extensive experiments to validate the effectiveness of the proposed approach across various supervised and unsupervised OOD detection methods, as well as ID and OOD datasets and model architectures.
3. The experiments presented in the discussion are engaging and provide meaningful insights.

weaknesses:
1. The novelty of this work is somewhat limited, as SSL and pseudo-labeling have been previously introduced in the OOD detection domain [1], albeit with a different approach. This reduces the contribution’s novelty primarily to the method of generating pseudo-labels. Additionally, [1] should be included in your experiments given the strong similarities to your work.
2. Comparison to supervised methods – the authors demonstrate that their method outperforms supervised methods, but they do not clarify the experimental settings used for this comparison (e.g., how do they compare ID performance). See further questions below.
3. Computational effort comparison – training the SSL network introduces additional computational requirements, which were not addressed in comparison to other unsupervised approaches. Furthermore, the networks being compared are not of the same size. The authors should, at minimum, include a column in the table indicating the network sizes to ensure a fair comparison
4. Experiment statistical significance – the reported results are not averaged across multiple seeds, which is crucial in the OOD domain, especially when results are close (e.g., SVHN as ID in Table 3).


Minor comments:

5. The method section would benefit from a formulaic explanation of the OOD detection decision function (often related to as $G(x,\tau)$), including details on how the OOD cut-off threshold is determined.

6. Adding the names of the OOD detection methods to the tables would improve readability.

[1] Mohseni, S., Pitale, M., Yadawa, J. B. S., & Wang, Z. (2020, April). Self-supervised learning for generalizable out-of-distribution detection. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 04, pp. 5216-5223).‏

questions:
1. Learned feature space comparison – to confirm that the SSL embeddings are central to the method’s success, an experiment comparing them to the embedding spaces of off-the-shelf networks is needed. For example, the feature spaces of ResNet or CLIP could be used for comparison. This would also partially address weakness 3 above.
2. It would be interesting to assess whether pseudo labels are even necessary by automatically labeling the ID datasets using state-of-the-art open-set classification approaches like Grounded SAM or Grounding DINO [2].
3. In the supervised methods experiment (Section 4.2), what are the ID accuracy results for the different backbones? Additionally, what pseudo-labels are used in this scenario? If ground-truth labels are employed, then the approach effectively reduces to the vanilla FeatureNorm (Yu et al., 2023). Alternatively, if only OOD detection performance is measured, the task becomes a classic anomaly detection problem.
4. Is there a specific reason for choosing FeatureNorm (Yu et al., 2023) over other state-of-the-art OOD detection methods (e.g., Mahalanobis or Energy)? Since you train a standard classification network in the final step, it would be beneficial to evaluate how other OOD detection methods perform with the trained network.
5. Is there a reason why FPR results were not shown in Table 3?

[2] Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., ... & Zhang, L. (2023). Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499.‏
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717341761

thread_type:
official_review

<thread_object>
creator: reviewer_s5mm

<content>
summary:
This paper proposes to detect out-of-distribution samples with unlabeled training data, by assigning pseudo-labels from self-supervised features clustering. Empirical results demonstrate the superiority of the method over state-of-the-art supervised and unsupervised approaches.

strengths:
1. Detecting out-of-distribution samples with unlabeled training data is meaningful because the detector from supervised learning usually comes with high labeling costs. This paper explores the possibility of an unsupervised OOD detector with comparable or even better performance than the supervised ones.
2. This paper observes that pseudo-labels from self-supervised representation clustering can replace and outperform the ground-truth labels, which is surprising.
3. The benchmarks are comprehensive and presented clearly.

weaknesses:
1. The motivation is not clear. Specifically, what is the meaning of the insight that "the success of supervised methods relies on a classification strategy that may not align with real-world understanding", and what is the objective and function of Section 3.2?
2. The method is neither novel nor solid. The methodology design, both the self-supervised representation learning and detector, depends on existing works. Additionally, the key mechanism behind the superiority of pseudo-labels from SSL is not well understood.
3. The experiment results are not convincing. On the one hand, the unsupervised benchmark seems carefully designed and not commonly used. For example, vertically and horizontally flipped versions of each ID dataset act as extra OOD datasets. On the other hand, the details about tuning of pseudo-categories number is missing. How does the author evaluate the classification accuracy with more categories than ID training datasets?

questions:
1. Given that SSL can extract features with semantic information, is it necessary to train the model SSL pseudo-labels?
2. Why do the pseudo-labels generated by SSL offer advantages over GT labels? The author should give a deeper understanding.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732499752268

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript. Your thoughtful comments and suggestions have provided invaluable guidance in improving the clarity and quality of our work.

In our rebuttal, we have thoroughly addressed all of your concerns and supplemented the paper with several new experiments to strengthen our findings. Specifically:

1. We have added detailed experiments and justifications to demonstrate why our method outperforms FeatureNorm.
2. We conducted ablation studies to explain why we selected mix-Barlow Twin over other architectures and how it contributes to the success of our approach.
3. We provided additional analysis to clarify how clustering methods and the number of pseudo-categories are chosen and their impact on model performance.
4. We included experiments with ImageNet as the in-distribution dataset to expand the scope of our evaluations and further validate our approach.

We believe these additional results and explanations address your concerns and demonstrate the robustness and generalizability of our method. If you find that our responses and new results have adequately resolved your questions, we would be grateful if you could consider revising and increasing your evaluation score.

We have made every effort to address your comments thoroughly, and we are confident that, if our paper is accepted, we will be able to revise it to its best possible form within the allocated time. Thank you again for your valuable feedback and for helping us improve our work.

Sincerely,

The authors.
</content>
</thread_object>

</thread>


Metareview:
metareview:
However, while using SSL and related methods to create pseudo-labels for enhancing OOD detection is an interesting idea, this approach raises certain concerns that the authors have not adequately addressed in the current version. Additionally, I have reservations about the novelty of the proposed method. Therefore, I am inclined to recommend rejecting this paper in its present form.