Look Around and Find Out: OOD Detection with Relative Angles

Authors: Berker Demirel, Marco Fumero, Francesco Locatello
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: out-of-distribution, out-of-distribution detection, decision boundaries

Abstract:
Deep learning systems deployed in real-world applications often encounter data that is different from their in-distribution (ID). A reliable system should ideally abstain from making decisions in this out-of-distribution (OOD) setting. Existing state-of-the-art methods primarily focus on feature distances, such as k-th nearest neighbors and distances to decision boundaries, either overlooking or ineffectively using in-distribution statistics. In this work, we propose a novel angle-based metric for OOD detection that is computed relative to the in-distribution structure. We demonstrate that the angles between feature representations and decision boundaries, viewed from the mean of in-distribution features, serve as an effective discriminative factor between ID and OOD data. Our method achieves state-of-the-art performance on CIFAR-10 and ImageNet benchmarks, reducing FPR95 by 0.88% and 7.74% respectively. Our scoring function is compatible with existing feature space regularization techniques, enhancing performance. Additionally, its scale-invariance property enables creating an ensemble of models for OOD detection via simple score summation.

Decision: reject
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730367338380

thread_type:
official_review

<thread_object>
creator: reviewer_vyey

<content>
summary:
This work proposed a novel angle-based metric for OOD detection that is computed relative to the in-distribution structure. They demonstrate that the angles between feature representations and decision boundaries, viewed from the mean of in-distribution features, serve as an effective discriminative factor between ID and OOD data. Experiments on CIFAR10 and ImageNet shows SOTA performance compared to other detection methods.

strengths:
The performance is good and the analysis is easy to understand.
The metric is scale-invariant, allowing ensemble for better performance.
The experiment is comprehensive.

weaknesses:
1. From Figure 1, the angle \alpha is helpful for better distinguishing ID and OOD data. And there lacks a comparison between the sine of \alpha, the sine of \theta, and the division of them. I think only the sine of \alpha in Figure 2 is not convincing to demonstrate that the angle \alpha is not very informative for ID and OOD separation.

2. For experiments, I think the author should report detection results on a vanilla trained model, which is a more common and practical setting for post-hoc detection methods. The current results are all based on supervised contrastive training models.

3. For experiments, it should compare with ReAct method on ImageNet OOD benchmark (in Table 2), since my empirical experience tells that ReAct always shows remarkable performance on ImageNet dataset.

questions:
no more question
</content>
</thread_object>

</thread>
<thread>
creation date:
1730426485617

thread_type:
official_review

<thread_object>
creator: reviewer_mwrp

<content>
summary:
This paper presents Look Around and Find Out (LAFO), a novel approach for out-of-distribution (OOD) detection using angle-based metrics. By calculating angles between feature representations and decision boundaries in relation to the mean of in-distribution (ID) features, LAFO improves OOD detection performance by leveraging the geometric relationships within feature space. The proposed method demonstrates robust performance across multiple benchmarks (CIFAR-10, ImageNet), significantly reducing false positive rates (FPR95) compared to state-of-the-art methods. Additionally, LAFO is hyperparameter-free, scale-invariant, and compatible with ensemble models, which enhances its practical utility.

strengths:
- The angle-based approach relative to ID mean is novel in differentiating ID and OOD samples.
- LAFO’s lack of hyperparameters simplifies its use in practical scenarios and avoids overfitting issues associated with tuning.
- The model achieves impressive results on CIFAR-10 and ImageNet, showcasing its scalability from smaller to larger datasets.
- LAFO can be combined with other activation shaping methods, demonstrating flexibility in enhancing model confidence scores.

weaknesses:
- While effective, using only the ID mean for centering may limit adaptability across highly variable datasets. Incorporating other statistics could improve robustness.
- The experiments focus on ResNet architectures. Additional comparisons with transformer-based or CLIP-based architectures could provide more insights.
- The paper does not fully explore scenarios where LAFO may struggle, such as in cases with minimal separability between ID and OOD distributions.
- Although LAFO is efficient, the paper could address its performance in real-time or resource-constrained settings to provide a more comprehensive view.

questions:
The effectiveness of LAFO in scenarios with severe ID class overlapping?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691044707

thread_type:
official_review

<thread_object>
creator: reviewer_qz8h

<content>
summary:
This paper presents a novel method for out-of-distribution (OOD) detection based on feature representations in neural networks. The proposed approach, LAFO (Look Around and Find Out), introduces an angle-based metric that measures the angle between feature representations and decision boundaries relative to the mean in-distribution (ID) feature. This approach leverages the relationship between feature vectors and decision boundaries to differentiate between ID and OOD samples effectively.

strengths:
1. The idea sounds novel. The paper introduces a novel angle-based metric for OOD detection, which measures the angle between feature representations and decision boundaries relative to the mean of in-distribution (ID) data.

2. This paper conducts extensive experiments to validate the proposed approach, including the standard benchmarks, and demonstrates its flexibility by incorporating it into ensemble methods and combining it with activation shaping algorithms.

3. This paper also explained the connection between LAFO and the similar approach fDBD.

weaknesses:
1. The exploration of ID statistics beyond the mean is limited. As OOD detection can benefit from a richer representation of ID statistics, does the "ID mean" refer to the mean across all classes? How about class-specific means or other statistical summaries? If the author includes these experiments or analyses, the paper will be strengthened. 

2. The experiments do not sufficiently address why and how LAFO enhances ensemble performance compared to other methods. It would be beneficial to see a more detailed analysis of how the angle-based scores behave in various ensemble settings, such as different architectures or training losses, to better understand when and why LAFO performs optimally.

questions:
1. Why is the CSI baseline used for CIFAR10 OOD benchmark, but not used for imageNet benchmark?

2. Did you consider the application of LAFO on multi-modal foundation models, such as CLIP?

3. The feature is evaluated through the composed function $f_1\circ...\circ f_{L-1} \circ g $, can you explain the reason why using this way or show some references for this? Did you consider other ways to represent features?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730760291755

thread_type:
official_review

<thread_object>
creator: reviewer_dg6s

<content>
summary:
The paper proposes to calculate the angle between the feature representation and the decision boundary, viewing from the mean of ID representations, to compute a score for identifying OOD examples. The method is evaluated on two popular benchmarks: CIFAR100 and Imagenet for OOD detection.

strengths:
1. The paper is well-written and easy to follow.

2. Relying on the angle between feature representations and the decision boundary seems to be novel.

3. The geometric interpretation of the presented method is convincing.

4. The presented method can easily be integrated into existing frameworks.

weaknesses:
1. I am somewhat skeptical about the performance gain. Although the paper claim performance gains across both benchmarks, the improvement is marginal for CIFAR100 (0.8% FPR95) and only evident on average. Looking at Table 1 and Table 2, the method lags behind other methods on an individual basis. It’s important to discuss why the method does not generalize well on an individual basis.

2. The method is only compared on ResNet architectures. How does it perform on other recent architectures, such as Vision Transformers? Given that the method relies heavily on feature and decision boundaries, validating it on diverse architectures is essential to confirm its architecture-agnostic and plug-in characteristics.

Minor Fixes: Please review the references. Some include only the publication year without the publication venue.

questions:
Mostly, my concerns are on performance gain and the experiments on different architectures. If authors can convincingly address my concerns, I am willing to change my rating.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732105516702

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to all reviewers and the AC

comment:
We would like to thank all reviewers for their comments, suggestions, and valuable feedback. We very much appreciate that they consider our submission “ *novel* ” (all reviewers), with “*comprehensive experiments*” (Qz8h, Vyey) “, “*impressive and scalable results*” (MwRp) and “*flexible*”  (Qz8h, Dg6s, MwRp).

Below we address common questions and concerns among reviewers. For individual questions and concerns, we are going to reply to each reviewer separately. We remain available for any further questions or concerns during the rebuttal period.

We have included a section titled 'Rebuttal Experiments' in the Appendix and are working on integrating it into the main paper.  
   
**Performance on different architectures (Reviewer Dg6s, Reviewer MwRp):** In the model ensemble experiment (Figure 3), we demonstrated that LAFO generalizes well to multiple architectures, including ResNet and Vision Transformers (ViT). In this rebuttal, we provide additional detailed results for more architectures:

- ResNet, without supervised contrastive loss (Tables 7 and 8\) \[1\]  
- ViT (Table 9\) without ensembling \[2\]  
- CLIP (Table 15\) \[3\]  
- MobileNet v2 (Table 14\) \[4\]

LAFO consistently achieves the best performance compared to baseline methods (e.g., Energy \[5\], MSP \[6\], kNN \[7\], fDBD \[8\]) across all these architectures, demonstrating its architecture-agnostic and plug-in characteristics.

**Performance analysis on different ID statistics (Reviewer Qz8h, Reviewer MwRp):** We thank the reviewers for highlighting this valuable direction. In the ablation in Table 4 in the paper we considered using the predicted and the target class mean statistics to show the performance of dynamic reference view. Following the suggestions, we included additional ablations in this rebuttal analyzing alternative sources of ID statistics beyond the ID mean, including:

- Individual class means  
- Sum aggregation over the scores of individual class means  
- Feature-wise minimum, maximum, and median of ID features  
- Sum aggregation of minimum, maximum, and median


Detailed results (Tables 12 and 13\)  show LAFO with ID mean statistics consistently performs better than or on par with these alternative statistics on CIFAR \[9\] and ImageNet \[10\] OOD benchmarks.

**Future direction:** Exploring further alternatives, such as n-th-order moments of the ID statistics, remains a promising direction, as incorporating them in the formulation of the score is not trivial.

**Experiment with CLIP (Reviewer Qz8h, Reviewer MwRp):** We applied LAFO to the CLIP \[3\] architecture, specifically CLIP-ViT-H14. 

To access decision boundaries, we trained a linear probe on CLIP features. This aligns with the premise of OOD detection, as most score functions (except kNN \[7\]) inherently rely on logits or a classification head. 

Table 15 demonstrates that LAFO significantly outperforms strong baselines such as MSP \[6\], Energy \[5\], kNN \[7\], and fDBD \[8\], achieving the best performance on both FPR95 and AUROC.

To conclude, we would like to remark that for this experiment, we cannot guarantee that the classes in our OOD benchmark datasets are completely distinct from those in CLIP’s training data and similar concepts may be present in the open vocabulary textual descriptions used to learn the features.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The LAFO (Look Around and Find Out) paper proposes an angle-based metric for out-of-distribution (OOD) detection. It calculates the angle between feature representations and their projection onto decision boundaries, relative to the mean of in-distribution features. Experiments on CIFAR-10 and ImageNet benchmarks demonstrate improved OOD detection performance compared to existing methods, including fDBD, showcasing LAFO's practicality and effectiveness. 

However, while LAFO improves upon fDBD, the advancements could be seen as incremental rather than groundbreaking. Using angles instead of distances and mean-centering are relatively straightforward modifications, not fundamentally new paradigms. More importantly, this paper does not deeply investigate why angles work better than distances in OOD detection. The argument for using angles feels somewhat empirical and less grounded in theory, which weakens the perceived depth of the contribution.