Hypercone Assisted Contour Generation for Out-of-Distribution Detection

Authors: Annita Vapsi, Andrés Muñoz Garza, Nancy Thomas, Keshav Ramani, Daniel Borrajo
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: OOD detection, Out-of-distribution detection, Computer Vision, Deep Learning, Representation Learning

Abstract:
Recent advances in the field of out-of-distribution (OOD) detection have placed great emphasis on learning better representations suited to this task. While there have been distance-based approaches, distributional awareness has seldom been exploited for better performance. We present HACk-OOD, a novel OOD detection method that makes no distributional assumption about the data, but automatically adapts to its distribution. Specifically, HACk-OOD constructs a set of hypercones by maximizing the angular distance to neighbors in a given data-point's vicinity, to approximate the contour within which in-distribution (ID) data-points lie. Experimental results show state-of-the-art FPR@95 and AUROC performance on Near-OOD detection and on Far-OOD detection on the challenging CIFAR-100 benchmark without explicitly training for OOD performance.

Decision: reject
Reviewer scores: 5, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730561663924

thread_type:
official_review

<thread_object>
creator: reviewer_tivv

<content>
summary:
The paper presents a post-training strategy for Out-Of-Distribution (OOD) detection for image classification. The proposed method assumes no access to the OOD samples and employs a set of hypercones with varying cutoff distances in feature space to define the class boundaries of in-distribution data. The work evaluates this method and a combination with a previous OOD technique on the Far-OOD and near-OOD detection benchmarks, with comparisons to a set of baselines.

strengths:
1. The paper addresses an important problem and proposes a new distance-based model based on hypercones. 
2. The evaluation of the method seems comprehensive and it achieves strong results in some cases.

weaknesses:
1. The paper is not clearly motivated. The discussion on training-based and distance-based post-training methods are insufficient. While the introduction section listed many previous methods, it is unclear what OOD modeling challenges this method aims to address, in particular for the distance-based approach.
2. The assumption of this method is very restrictive. As stated in Line 064, it requires "that ID and OOD data are separable in the space", which is unrealistic for real-world data. 
3. The novelty of this method is limited. The proposed hypercone representation is similar to a mixture of Gaussian kernels for the ID data distribution.     
4. The presentation of this work lacks clarity and the technical details are difficult to follow. Several parts of Sec 4.3 are confusing: 1) Line 260: Why do the hypercone representations rely on the test data feature Z_{test}, which should not be used during model construction? 2) Line 299: How is the score function defined and what threshold is used during the inference (in Sec 4.4)?    
5. The experimental evaluation is lacking in three aspects: 1) The experimental setup is limited, which only considers three ResNet-based backbones. More modern architectures, such as ViT, should be included to validate its generalization. 2) The ablative study is lacking. What contributions are from the hypercones? What if it is replaced by Gaussians? 3) The performance of the original version HAC_k-OOD is mixed in both Table 1 and 2, and in most of cases, it is worse than the SOTA methods. It is unclear whether the proposed representation is truly effective.

questions:
Please address the questions in the weaknesses part as above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655332566

thread_type:
official_review

<thread_object>
creator: reviewer_fqvd

<content>
summary:
The paper introduces a post-training method for out-of-distribution (OOD) detection. The method approximates the contour of each class with a set of hypercones and defines per-hypercone decision boundaries. 
More specifically, the hypercones are drawn as follows: 
1. compute per-class centroids, which will be the apex of the class hypercones 
2. take a class sample and set the hypercone axis to be the vector that points its (penultimate) representation 
3. set the opening angle to be the angle between the hypercone axis and the k-th nearest neighbor from the sample representation
4. set the decision boundary using the distribution of representations within the hypercone's angular boundary

The method is an extension of another technique, SSD+, which assumes the decision boundary could be modeled with a unique hypersphere (or multidimensional ellipsoid) per class. It's training does not require OOD data. 

Experimentally, the authors follow common practice and evaluate the OOD performance with the CIFAR-100 dataset as an in-distribution dataset and many different datasets as out-of-distribution datasets. Two types of pre-trained classification models are tested: models trained with a softmax cross-entropy loss and with a supervised contrastive loss. 
The method achieves SOTA results in the supervised contrastive learning setup and is competitive in the cross-entropy setting.

strengths:
1. The paper is well-written and easy to follow. The authors provide a good summary of the different approaches to OOD, a background section on hypercones, and a clear and precise method description. 
2. Relevance and novelty of the method: the algorithm doesn't require assumptions about the data distribution and can model complex embedding spaces since it draws multiple hypercones per class and since it defines per-hypercone decision boundaries. 
3. The authors discuss some limitations of the method (e.g., it works less well with smaller models ).

weaknesses:
1. Limited evaluation: the method is only evaluated on models pre-trained on the *quite simple* CIFAR datasets and not on more complex datasets such as the ImageNet-200 or ImageNet-1k OOD benchmarks.

questions:
- (see Weakness 1): how does the method perform on the ImageNet (200 or 1k) OOD benchmarks? Its evaluation on benchmarks beyond *quite simple* datasets (i.e., CIFAR-10/CIFAR-100) would strengthen the claims. 
- What is the intuition behind the use of hypercones? Isn't the embedding space of a class more "dense" close to its centroid?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730725938231

thread_type:
official_review

<thread_object>
creator: reviewer_kcxv

<content>
summary:
The paper introduces a post-training out-of-distribution (OOD) detection method, HAC_k-OOD, which models the training data distribution through a set of hypercones and assesses OOD status based on whether a test sample falls within any hypercone. Specifically, for each class, the method first computes the class centroid and defines an angular boundary using the k-th nearest neighbors for each training point. Additionally, a radial boundary is set based on the mean and variance of the sample norms within the angular boundaries. During inference, a sample is classified as OOD if it lies outside either the angular or radial boundaries. Experiments were conducted using ResNet-18, ResNet-34, and ResNet-50, with both supervised contrastive learning and cross-entropy loss. The method was evaluated on CIFAR-10/100 as the in-distribution dataset and tested on various OOD datasets, covering both near and far OOD scenarios.

strengths:
1. The method takes an interesting approach to distance-based OOD detection by relaxing the distributional assumptions and, unlike naive KNN, still leveraging nearby training data statistics to construct class contours. To the best of my knowledge, the use of hypercones for this purpose is novel and appears well-motivated.

2. The authors present their method clearly, making the paper easy to follow and understand.

3. Although the method involves hyperparameter k, the authors provide a practical approach to estimating it without requiring additional OOD data.

4. The experiments investigate various model sizes and training losses, demonstrating their impact on distance-based methods. Overall, the method benefits more from larger models trained with contrastive loss, as these models produce more distinguishable features.

weaknesses:
1. The computational complexity is a concern for this method. Since the computation appears to increase with the size of the training dataset, it’s unclear if this approach would be feasible for large-scale, real-world applications. Although the authors state that the method is computationally efficient and support this with inference time per sample, I encourage them to provide a more detailed discussion on this aspect. For instance, what is the time required to construct the hypercones? A comparison of inference times with other methods would also be valuable.

2. How would the method perform on a large-scale, real-world dataset like ImageNet? Many recent OOD detection methods use ImageNet-1k as the in-distribution (ID) dataset. I encourage the authors to consider experiments on this dataset to evaluate the general applicability of the method in more realistic scenarios.

3. Recent work has explored OOD detection using CLIP as a backbone model (eg. [a1]), as CLIP may offer a more robust feature space. It would be interesting to see how this method performs when applied to a CLIP-based model.

4. Could the authors elaborate on why this method outperforms a naive KNN approach? One advantage seems to be that the method leverages the nearest neighbors within the training set (as opposed to KNN’s i.i.d. approach) to construct hypercones, which may capture more robust information about class boundaries. Additionally, an ablation study using either angular or radial boundaries separately for OOD detection could provide valuable insights into the method’s effectiveness and support future research.

5. While the paper is generally well-written, a few sections could be clearer. For example, in lines 80-81, P_in is referenced without being introduced in a previous formula. Additionally, in Section 5.2, only ResNet-34 is mentioned as the backbone model, though ResNet-18 and ResNet-50 are also used.

[a1] Ming, Yifei, et al. "Delving into out-of-distribution detection with vision-language representations." Advances in neural information processing systems 35 (2022).

questions:
Please see the weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730739355727

thread_type:
official_review

<thread_object>
creator: reviewer_cvgv

<content>
summary:
HACk-OOD is a post-training OOD detection method using hypercone projections to construct class-specific contours in embedding space. The approach achieves SOTA performance on CIFAR-100 and improves with larger networks. Including Imagenet experiments could further demonstrate scalability on large datasets.

strengths:
HACk-OOD introduces a unique method using hypercone projections to delineate class contours, avoiding traditional Gaussian distribution assumptions and offering greater flexibility in complex feature spaces. The method achieves competitive, often superior, results on challenging datasets like CIFAR-100, demonstrating strong performance in both near and far OOD detection scenarios.

weaknesses:
1. Experiments are limited to CIFAR-based datasets, testing on a large-scale dataset like Imagenet would better validate the method’s scalability. Also evaluating HACk-OOD on the OpenOOD benchmark would provide a clearer comparison to recent methods. I would consider rating this paper higher if Imagenet results were provided.

2. Missing comparisons with some of the latest post-hoc OOD methods, such as ASH and SCALE. Including these would offer a more comprehensive assessment of its relative performance.

Djurisic, Andrija, et al. "Extremely simple activation shaping for out-of-distribution detection." ICLR 2022
Xu, Kai, et al. "Scaling for training time and post-hoc out-of-distribution detection enhancement." ICLR 2023

questions:
See Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732547057028

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to thank the reviewers for taking the time to read our manuscript and for their insightful questions. We responded to each individual question to the best of our ability. We hope that our responses are comprehensive and encourage the reviewers to ask follow-up questions if anything seems unclear. 

Below we would like to tackle some of the repeating themes in questions raised by reviewers.

ImageNet Results:
We considered using ImageNet as an in-distribution (ID) dataset. However, due to legal considerations related to the indemnity provisions outlined in its terms of access, we are unable to do so. The terms state:
"Researcher accepts full responsibility for his or her use of the Database and shall defend and indemnify the ImageNet team, Princeton University, and Stanford University, including their employees, Trustees, officers, and agents, against any and all claims arising from Researcher's use of the Database, including but not limited to Researcher's use of any copies of copyrighted images that he or she may create from the Database."
Specifically, the lack of copyright information for images in the ImageNet dataset makes it incompatible with the data use policies to which we adhere, preventing us from accepting the terms of service. For this reason, we also might need to remove Imagenet-F from our Near-OOD experiments as well. We are still in discussions regarding our use of Imagenet-F and will provide an update as soon as we have one.
While we acknowledge that the absence of ImageNet results may be seen as a limitation, we believe the novelty and robustness demonstrated by our approach in the, harder, CIFAR-100 benchmark shows its potential effectively. We hope that our contributions be evaluated on the strength of our methodology and the results obtained using alternative datasets.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel post-hoc OOD detection approach that leverages hypercones to define class-specific boundaries in the embedding space. The method demonstrates a good performance on CIFAR-100 in both near-OOD and far-OOD scenarios. While the use of hypercones offers a unique perspective for OOD metric design and visualizations are provided in the supplementary material, there is no clear explanation or theoretical justification provided as to why the hypercones are superior for OOD detection. Additionally, the notation could be clearer. Key limitations include the lack of scalability testing on larger datasets like ImageNet, the absence of evaluations using modern architectures such as ViTs, and insufficient ablation studies to isolate the contributions of individual hypercone components as mentioned by multiple reviewers. Although the authors addressed many reviewer concerns with additional experiments and clarifications, the limited evaluation scope and unresolved questions resulted in a mixed reception, leaving the paper marginally below the threshold for acceptance.