Title: What If the Input is Expanded in OOD Detection?
Venue: NeurIPS 2024
Accepted: yes
Keywords: Out-of-distribution Detection

Trustworthy Machine Learning
Abstract: Out-of-distribution (OOD) detection aims to identify OOD inputs from unknown classes, which is important for the reliable deployment of machine learning models in the open world. Various scoring functions are proposed to distinguish it from in-distribution (ID) data. However, existing methods generally focus on excavating the discriminative information from a single input, which implicitly limits its representation dimension. In this work, we introduce a novel perspective, i.e., employing different common corruptions on the input space, to expand that. We reveal an interesting phenomenon termed *confidence mutation*, where the confidence of OOD data can decrease significantly under the corruptions, while the ID data shows a higher confidence expectation considering the resistance of semantic features. Based on that, we formalize a new scoring method, namely, *Confidence aVerage* (CoVer), which can capture the dynamic differences by simply averaging the scores obtained from different corrupted inputs and the original ones, making the OOD and ID distributions more separable in detection tasks. Extensive experiments and analyses have been conducted to understand and verify the effectiveness of CoVer.

What If the Input is Expanded in OOD Detection?

Abstract

Out-of-distribution (OOD) detection aims to identify OOD inputs from unknown classes, which is important for the reliable deployment of machine learning models in the open world. Various scoring functions are proposed to distinguish it from in-distribution (ID) data. However, existing methods generally focus on excavating the discriminative information from a single input, which implicitly limits its representation dimension. In this work, we introduce a novel perspective, i.e., employing different common corruptions on the input space, to expand that. We reveal an interesting phenomenon termed confidence mutation , where the confidence of OOD data can decrease significantly under the corruptions, while the ID data shows a higher confidence expectation considering the resistance of semantic features. Based on that, we formalize a new scoring method, namely, Co nfidence a Ver age (CoVer), which can capture the dynamic differences by simply averaging the scores obtained from different corrupted inputs and the original ones, making the OOD and ID distributions more separable in detection tasks. Extensive experiments and analyses have been conducted to understand and verify the effectiveness of CoVer. The code is publicly available at: https://github.com/tmlr-group/CoVer .

1 Introduction

Out-of-distribution (OOD) detection [23, 28, 44] is important for reliable machine learning model deployment in open-world scenarios, where various samples from unknown classes, i.e., OOD data, are constantly emerging [4]. Deep neural networks [20] (DNNs) are demonstrated to be overconfident about these OOD data, which may result in disasters for some safety-critical applications [5, 21]. Traditional OOD detection methods [23, 29, 28, 30, 45, 46, 1, 61] design various scoring functions based on the outputs or representations extracted from well-trained models. Recently, some research also extended it into a zero-shot setting [31], which leverages the multi-modal information based on vision-language models (VLMs) and requires no further training on in-distribution (ID) data. A series of methods [48, 36, 52, 26] are proposed for improving OOD detection based on such advances.

Although promising progress has been achieved, existing methods mainly focus on excavating the discriminative information of a single input. For instance, ReAct [45], DICE [46], and ASH [1] integrates the activation regularization or reshaping to the forward path of a single input in singlemodal DNNs; MCM [31] characterizes the confidence of a single input by the similarity between visual features and text representation of ID classes in VLMs. However, specializing in a single input

Multiple Inputs (CoVer)

AUROC: 94.63    FPR95: 30.70

AUROC: 91.30    FPR95: 44.85

AUROC: 96.37

↑

FPR95: 20.10

↓

may implicitly constrain the representation dimension for detection, leaving some hard-to-distinguish OODsamples with features similar to ID samples fail to be identified (refer to the distribution overlap in the left panel of Figure 1). Therefore, it naturally motivates the following critical research question: Can we expand the dimension of the input space to explore OOD discriminative representations?

In this work, we introduce a novel perspective to investigate that, i.e., employing the common corruptions [22] in the input space. Through a systematical comparison, we reveal an interesting phenomenon termed confidence mutation , where the confidence of OOD data can decrease significantly under the corruptions, while ID data shows higher confidence expectation considering different input dimensions. Specifically, as shown in Figure 1, corrupted inputs result in lower confidence in both OOD and ID data. However, one critical dynamic discovery is that its confidence about overconfident OODdata is changed more than the unconfident ID data under the same corruptions (refer to Figure 2), indicating a natural difference in feature-level resistance of the originally overlapped parts (refer to Figure 3). With the original inputs, we can find that the model is overall more confident in ID data.

Based on the above, we propose a new scoring framework, namely, Co nfidence a Ver age (CoVer), as illustrated in Figure 4. At the high level, we expand the original single representation dimension into multiple ones to excavate discriminative information. In detail, we introduce a simple but effective method for identifying OOD data with confidence mutation, which can be formalized as an average of OOD scores (e.g., Eq. (6)) obtained by different corrupted inputs and the original one. With the expectation among multiple input dimensions, CoVer can effectively reflect the knowledge of invariant semantic features that are discriminative from ID data to OOD data. It also matches an intuition that ID data can be more likely recognized as high confidence by models considering different input views, especially with the corruptions affecting the non-semantic high-frequency parts.

We conducted extensive experiments to verify the effectiveness of our proposed method. Since CoVer is an input-side design compatible with single-modal and multi-modal networks, we adpot various benchmarks for DNN-based and VLM-based OOD detection tasks. Under extensive evaluations, our CoVer can achieve the superior performance compared with different baselines. Moreover, CoVer exhibits excellent compatibility, as evidenced by the better performance of some methods combined with CoVer. Finally, a range of ablation studies of the scoring framework and further discussions from different perspectives are provided. In summary, our main contributions can be listed as follows,

Conceptually, we introduce a novel perspective for identifying OOD inputs by considering the common corruptions to expand the representation dimensions. (in Section 3.1)

We reveal an interesting phenomenon termed confidence mutation , where the confidences of OOD data can vary to significantly lower than ID data under corruptions (in Section 3.2).

Technically, we formalize a novel scoring method, namely, Confidence aVerage (CoVer), a simple average of the confidence estimated from extended corrupted inputs and the original one. The corresponding empirical analysis is presented to understand it (in Section 3.3).

ID Score Distribution

Original

Corrupted

CoVer

OOD Score Distribution

Original

Corrupted

CoVer

!!!

ID Original Score Distribution

Index

!!!

!!!

0.00100

0.00125

7058

4491

1019

Index

22862

10176

9715

!!!

7058

4491

1019

22862

10176

9715

ID Corrupted Score Distribution

0.00100

0.00125

Score

0.00110

Score

0.00110

!!!

!!!

Confident ID

Unconfident ID

!!!

!!!

OOD Original Score Distribution

Index

!!!

!!!

0.00100

0.00125

5826

6871

7433

Index

5565

47

581

!!!

5565

47

581

5826

6871

7433

OOD Corrupted Score Distribution

0.00100

0.00125

Score

0.00115

Score

0.00115

!!!

!!!

Overconfident OOD

Unconfident OOD

!!!

!!!

!!!

Empirically, extensive experiments on both traditional and zero-shot OOD detection benchmarks have verified the effectiveness and compatibility of our CoVer, and we conduct various ablations and further discussions to provide a comprehensive analysis (in Section 4).

2 Preliminaries

In this section, we briefly introduce the preliminaries of OOD detection on basic setups and the advanced zero-shot setting on VLMs. For related works, we leave detailed discussions in Appendix B.

Problem setups. Let X be the input space and Y = { y 1 , ..., y K } be the label space, where K is the number of ID classes. Given the ID random input x i ∈ X and OOD random input x o ∈ X , we consider the ID marginal distribution D ID and those with the OOD marginal distribution D OOD , where D OOD is defined as an irrelevant distribution of which the label set has no intersection with Y . The goal of OOD detection is to figure out inputs with the OOD distribution D ID from those with the ID distribution D OOD , which can be considered as a binary classification problem. For traditional OOD detection, given a model f trained on ID data with logit outputs, a score function S ( · ) and a threshold λ , the detection model g ( · ) can be defined as,

where x is detected as ID data if and only if S ( x ) ≥ λ ; otherwise, it is rejected as OOD data that should not be predicted by the model f . Designing a practical S ( x ; f ) is crucial for OOD detection.

CLIP-based vision-language models CLIP [42] has shown impressive performance in the zeroshot classification task by profiting from massive amounts of training data and large-size models. Here we briefly review the mechanism of CLIP-based VLMs. A CLIP-based model f usually contains an image encoder f image and a text encoder f text . Given a random input x ∼ D ID and a label y ∼ Y , we use f image and f text to extract the image features h ∈ R d and the text features e j ∈ R d as follows:

where p ( · ) refers to the prompt template for the input label, d is the embedding dimension. The predictions are formulated as the consine similarity between the image features h and text features e j ,

Original

0.00109289

Confidence Score

0.00107441

Original

Corrupted

0.00109536

Confidence Score

0.00108758

Original

Corrupted

0.00109389

Confidence Score

0.00105135

Original

Corrupted

0.00111933

Confidence Score

0.00107411

Original

Corrupted

Corrupted

Original

Corrupted

Original

Corrupted

Original

Corrupted

Raw

FFT

High Freq

Raw

FFT

High Freq

Unconfident ID

Unconfident ID

Overconfident OOD

Overconfident OOD

Zero-shot OOD detection Different from traditional OOD detection methods based on a classifier f well-trained on single-modal, recent zero-shot OOD detection studies [31, 26] leverage a pre-trained VLM-based model (e.g. CLIP [42]) without any fine-tuning on ID training data. The text features from given ID label names (i.e. ID classes) as the class-wise weights functionally play the same role as the classifier. With guaranteed ID classification accuracy, the primary goal of zero-shot OOD detection in this paper is to distinguish OOD samples that do not belong to any known ID classes.

3 CoVer: Confidence Average

In this section, we formally present our proposed new scoring framework, i.e., Confidence aVerage (CoVer). First, we introduce the motivation of representation dimension expansion and present the notable discovery (Section 3.1). Second, we conduct the exploration for the confidence mutation of overconfident OOD data considering the corrupted inputs (Section 3.2). Lastly, we provide the detailed implementation and analysis of our formalized CoVer score (Section 3.3).

3.1 Representation Dimension Expansion

DNNs are demonstrated to be overconfident on those OOD samples, and a series of works [23, 45, 1, 54, 31, 26] are dedicated to eliminating the effects through the perspective of feature representation. However, achieving that is demonstrated to be hard as it generally requires careful optimization [54], or additional prior knowledge [36] on the single input. As illustrated in the middle panel of Figure 1, adopting some agnostic corruptions on the single input may result in worse separability between the ID and OOD distribution. Specializing in a single input seems to implicitly constrain the representation dimension for detection. In this work, we naturally raise the following question,

What if we expand the dimension of representation for the original inputs to enhance OOD discriminative representations?

Using the same corruption adopted in Figure 1, we can conduct the dimension expansion by simultaneously considering both the corrupted variant and the original input. One notable discovery is that considering multiple inputs can achieve better performance on OOD detection, even though the single corruption transformation brings negative effects on identifying OOD samples. The surprising comparison results attract us to further explore the underlying mechanism of expanding the representation dimension for the original input, especially the dynamics before and after adopting the corruptions.

3.2 Confidence Mutation under Corruptions

Although employing corruption on the single inputs leads to worse ID-OOD separability, we can find obvious shifts toward less confidence in both OOD and ID distributions. It is expected for OOD data that corruption can help the model mitigate overconfidence, while the ID data are also affected severely and enlarge the overlap on the single dimension. In contrast, considering the multiple inputs by averaging the confidence scores shows variance reduction for ID distribution, which indicates the distinguishable dynamics between ID and OOD data. To elicit the underlying mechanism, we provide a definition to characterize the change of model confidence on the inputs under corruption.

Definition 3.1 (Confidence Difference) . Given a well-trained model f and a score function S ( · ) measuring the confidence of f on an input x , we have a basic static to characterize the differences between the original input and that under a corruption c ( · ) : MU c ( x, S, f ) ≜ ( S ( x ; f ) -S ( c ( x ); f )) .

Based on the comparison in Figure 1, we divide the ID and OOD data into four groups according to the model confidence on their original inputs, and present an overall comparison of the confidence differences in Figure 2. We reveal the critical dynamic differences in the corrupted variants of ID and OOD data, where both large MU( x, s, f ) exist in the data whose natural inputs own higher confidence in each part, demonstrating the model confidence on the overconfident OOD data decrease more than the unconfident ID data under the same corruption. We can get the empirical observation,

Observation 3.2 (Confidence Mutation) . Given the overconfidence OOD inputs x o ∈ D OOD, we can observe more significant differences in the change of confidences under the same corruption c ( · ) than the ID samples x i ∈ D ID with similar model confidence (constrained by ϵ ) on the natural inputs,

In Figure 3, we further visualize the samples of unconfident ID data and overconfident OOD data. Under the comparison of saliency maps and the Fast Fourier Transformation, we find the confidence mutation reflects the feature level vulnerability of OOD data compared with ID data. Intuitively, the former can be severely affected by the common corruption to eliminate the non-semantic features, which generally exist at the high-frequency level. In contrast, the semantic feature of unconfident ID data can maintain confidence as the limited effects of corruption on the low-frequency part.

Observation 3.3 (Resistance of ID features in frequency views) . Assuming that ID data owns the ID semantic features existing at the low-frequency level (extract by Γ ξ ) while the OOD data has some non-semantic features at the high-frequency level for activating the high confidence of the model on its prediction, we can observe the following empirical relation on adopting the same corruptions,

where Γ indicates the Fourier transformation. We suggest that common corruptions [22] might act as perturbations of high-frequency features within the input representation. For OOD samples, which inherently lack ID semantic features, altering high-frequency features could potentially lead to notable changes in model confidence, while the ID data shows relatively better resistance on it (see the left panel of Figure 3). This observation tentatively supports the notion that ID data maintains an overall higher confidence expectation under conditions of expanded representation dimension. To validate its generality, additional results involving various common corruptions are presented in Appendix C.2.2.

3.3 Scoring Function Implementation and Analysis

Based on the previous understanding of confidence mutation, we formalize our CoVer, a new scoring framework for OOD detection. The procedure of CoVer mainly contains the following four parts as illustrated in Figure 4, and the final averaged multi-dimensional scores can be provided as follows,

where E x ∼ d ( X , ˜ X ) is the confidence expecation over all scores dimensions, K is the number of ID classes, τ is the temperature coefficient of the softmax function. In the following, we detailedly introduce the specific operations to obtain the final S CoVer and the corresponding notations.

To enlarge the dimension of the original single input for confidence average, we introduce various corrupted inputs. In this work, we employ the corruption functions defined in [22], which consists

Brightness

Contrast

Fog

Snow

Spatter

JPEG

+

···

Original

Corrupted

1 ~ N Types

Saturate

Elastic

Pixelate

Maximum Softmax Score

Free Energy Score

(((

Score Functions

Pretrained Model

𝑺(𝒙

𝑶

)

+

𝑺(𝒙

𝑨

𝟏

)

+··· +

𝑺(𝒙

𝑨

𝑵

)

𝑵+𝟏

𝑺 𝒙 =

𝑶

𝑨

𝑵

𝑨

𝟏

···

𝑺(𝒙

𝑶

)

𝑺(𝒙

𝑨

𝑵

)

𝑺(𝒙

𝑨

𝟏

)

···

···

···

Score Distribution

CoVer

Classifier

Text

Encoder

Image

Encoder

Multi Modal

𝑶

𝑨

𝑵

𝑨

𝟏

···

···

CLIPN Score

NegLabel Score

total of 90 distinct corruptions. We provide the visualization of these different corruptions in Appendix C.4.1. Given the input space X and a set of corruption functions C , the corrupted inputs can be formulated as { c ( x ) | x ∈ X , c ∈ C } → ˜ X , resulting in the multi-dimensional input spaces d ( X , ˜ X ) .

Given an input image x ∼ d ( X , ˜ X ) , we adopt an image encoder with fixed parameters to extract the feature of the original dimension h O and features of corrupted dimensions h 1 , ...h N . Then we predict the logit output s ( x ) for each dimensional feature h d , ∀ d = O, 1 , ..., N . For the DNN-based model f , the outputs of these features are denoted as s ( x ) = f ( x ) = logits d . For the VLM-based model, the outputs are label-wise matching scores based on the cosine similarity: s j ( x ) = h d · e j ∥ h d ∥·∥ e j ∥ .

For the logit output s ( x ) predicted from a specific input dimension, we assign it with an OOD score to implement one dimension of the CoVer score. As shown in the right-middle panel of Figure 1, the OODscore can be formalized by some traditional scoring functions, like the softmax scoring function [23] (refer to Eq. (6)) and the free energy scoring function [30]. In addition, the OOD score can also be formulated by variants of some novel scoring functions, like those in CLIPN [52] and NegLabel [26]. The detailed implementations for alternative scoring functions can be found in Appendix C.3.

4 Experiments

In this section, we present the comprehensive verification of the proposed CoVer in the OOD detection benchmarks. First, we introduce several critical parts of experimental setups (in Section 4.1). Second, we provide the performance comparison and compatibility experiments of our CoVer with various DNN-based and VLM-based OOD detection methods (in Section 4.2). Third, we conduct extensive ablation studies and further discussions to understand the properties of our CoVer (in Section 4.3).

4.1 Experimental Setups

In this part, we present the critical parts of experimental setups and leave more details in Appendix C.

Datasets. Following previous work [1, 31], we adopt the ImageNet-1K OOD benchmark [24], which uses the ImageNet-1K [14] as ID data and iNaturalist [49], SUN [55], Places [60], and Textures [7] as OOD data. For each of the OOD datasets, the classes do not overlap with the ID dataset. As the same as MCM [31], we also use subsets of ImageNet-1K for fine-grained analysis, like ImageNet-10 that mimics the class distribution of CIFAR-10 but with high-resolution images. For hard OOD evaluation, we exploit ImageNet-20 with 20 categories similar to ImageNet-10 in the semantic space

(e.g., dog (ID) vs. wolf (OOD)). To have more experimental comparison, we also reproduce one setting from spurious OOD detection [35], whose hard OOD inputs are created to share the same background (i.e., water) as ID data but have different object labels (e.g., a boat rather than a bird). To select the most effective corruption types for each method, we use SVHN [37] as the validation set.

Model Setup. In this paper, we implement CoVer on various architectures, including DNN-like ResNet50, and VLM-like CLIP [42], AltCLIP [6], MetaCLIP [56], GroupViT [57]. Unless otherwise instructed, for VLM-based zero-shot OOD detection, we use CLIP-B/16 which consists of an image encoder based on ViT-B/16 Transformer [15] and a text encoder built with the masked self-attention Transformer [50]. We use the algorithmically generated corruptions defined in [22]. Each type of corruption has a severity level ϵ from 1 to 5, with ϵ = 1 being the least severe and increasing up to ϵ = 5 . By default, we use the CoVer score in the max-softmax form and set τ = 1 as the temperature.

Baseline Methods and Evaluation Metrics. We conpare the proposed method with various competitive methods. Specifically, we adopt Maximum Softmax Probability (MSP) [23], ODIN [29], Mahalanobis [28], Energy [30], ReAct [45], DICE [46] and ASH [1] as traditional OOD detection baseline methods. The VLM-based OOD detection methods we compared with include MCM, a method specifically designed for zero-shot OOD detection, as well as some traditional methods including MSP, ODIN, Energy, Mahalanobis, GradNorm [24], ViM [51], KNN [47], ZOC [18] that were re-implemented using a finetuned CLIP ViT-B/16 on the ImageNet-1K, see Appendix A for more details. For a fair comparison, we keep the original hyperparameter setups of the comparative methods and adopt the following metrics to evaluate the OOD detection performance: (1) the false positive rate (FPR95) of the OOD samples when the true positive rate (TPR) [29] of the in-distribution samples is at 95%, (2) the area under the receiver operating characteristic curve (AUROC) [13].

4.2 Main Results

Overall results of OOD detection performance comparison with different baselines. To evaluate the effectiveness of CoVer, we compare it with existing baseline OOD detection methods on the ImageNet-1K benchmark in two aspects. In Table 1, we present the performance comparison with traditional OOD detection methods using ResNet-50 as the backbone. Our CoVer combined with ASH-S can achieve better OOD detection performance, which verifies the effectiveness of our method with an average multi-dimensional estimated confidence score. In Table 2, we provide the results compared with VLM-based OOD detection methods, which are classified into two groups: fine-tuning methods that require extra data for fine-tuning CLIP and zero-shot methods that require no training. Our method CoVer consistently achieves better performance across the four OOD datasets, aligning with the analysis that OOD and ID data are better distinguished under the expanded dimensions.

Compatibility experiments of CoVer combined with different OOD detection methods. Since it is a simple design for representation dimension expansion, CoVer can be easily integrated into previous OOD detection methods and achieve performance improvements. In Table 3, we first consider some methods with minor modifications to the architecture of ResNet50, e.g., ReAct, DICE and ASH. While fixing the detection models, we replace each of their OOD scores with our CoVer score. Then we integrate the proposed CoVer into several VLM-based OOD detection methods. For LoCoOp [36], a CLIP-fine-tuning method with the MCM score, we follow its fine-tuning strategy and report the results with the CoVer's scoring mode. For CLIPN [52] and NegLabel [26] that both introduce a new OOD score, we redesigned them based on the critical idea of confidence average, as detailed in the Appendix C.3. Our CoVer can consistently help these methods gain better performance without specific modality limitations, which shows the algorithmic robustness of our proposed method. More results with other methods and further discussions and analyses are provided in Appendix C.2.

Zero-shot OOD detection performance comparison on hard OOD detection. Following the settings in [31, 26], we explore the superiority of our proposed CoVer compared with MCM [31] on hard OOD detection tasks, as shown in Table 4. Specifically, we alternately use ImageNet-10 and ImageNet-20, ImageNet-10 and ImageNet-100 as ID and OOD data to simulate the setups presented in [19]. The results demonstrate CoVer has a better distinguished ability than MCM for semantically hard OOD data. We also report the experimental data on zero-shot spurious OOD detection task in the last row of Table 4, which also shows the better performance of the proposed CoVer than MCM.

4.3 Ablation and Future Discussions

In this part, we conduct extensive ablation studies and provide a thorough understanding of our CoVer. The extra results and discussions (e.g., impact and limitations) are provided in Appendix C.2.

Ablation on diverse VLM architectures. We compare our CoVer score with the MCM score on different VLM architectures and the results are reported in Table 5. The first part shows the superiority

Brightness

Fog

Contrast

Saturate

Corruption Type

0

20

40

60

80

100

FPR95

Origin

Uni-Dim

Multi-Dim

(a) Multi-Dim Strength

1

2

3

4

5

6

36

38

40

42

44

46

FPR95

Origin

Severity 1

Severity 2

(b) Expanded Dimensions

Fog

JPEG

Corruption Type

36

38

40

42

44

46

48

50

FPR95

Severity 1

Severity 2

Severity 3

Severity 4

Severity 5

(c) Severity Level

DICE

ASH-B

MCM

NegLabel

Methods

20

25

30

35

40

45

50

55

FPR95

ResNet-50

CLIP-B/16

w.o. CoVer

of our CoVer on different backbones of CLIP, and the larger backbone boosts the performance of OODdetection. The second part shows that CoVer can generalize better to various VLM architectures compared to MCM. More fine-grained results on different OOD sets can be found in Appendix C.2.1.

Superiority of multi-dimensional scoring framework. To verify the superiority of our CoVer with a multi-dimensional scoring framework compared with the uni-dimensional one, we report the performance comparison using different corruption types in Figure 5(a). Here, the uni-dimensional framework is denoted as using a single corrupted input for confidence estimation. The results indicate that extended dimensions provide valuable clues for enhancing OOD discriminative representations, thus enlarging the separability between ID and OOD data. We leave the fine-grained results and further discussions in Appendix C.2.2, which provide a better understanding of the improvement.

Significance of the number of expanded measuring dimensions. As a critical aspect of our CoVer, the number of the extended confidence measuring dimensions will control the performance enhancement for OOD detection. In Figure 5(b), we present results for varying the number of expanded dimensions using various corruptions with the same severity level. The results demonstrate that an increasing number of expanded representation dimensions gradually improves the performance then probably declines, while consistently outperforming the baseline. This indicates that the addition of measuring dimensions prioritizes enhancing the distinction of OOD data with mutated confidence while ID data shows resistance. More detailed results and analyses are provided in Appendix C.2.3.

Comparison of the variation trends in different corruption severity levels. In Figure 5(c), we show the performance by varying the severity level ϵ for each specific corruption style. We can observe that three types of trends emerge with an increasing level of severity, i.e., up, down, and up then down. This phenomenon indicates that an appropriate level of corruption is critical for the optimization of CoVer. One possible reason may be that the threshold maximizing the distinction between ID and OOD data varies from different types of corruption. To further explain this observation, we provide more results on the ϵ among various corruption styles and more discussions in Appendix C.2.4.

Generality of integrating with different OOD detection schemes. Since the proposed CoVer is a general scoring framework with an average of confidence scores measured from multiple dimensions, the specific realization for each dimensional score have multiple choices. In Figure 5(d), we present the comparison with different realizations integrated w./w.o. CoVer (e.g., ResNet-50 based DICE [46] and ASH-B [1], CLIP-B/16 based MCM [31] and NegLabel [26]), where they have different performance improvements compared with the original baseline without constraints for the modality.

5 Discussion

5.1 Broader Impact

OOD detection is crucial for deploying reliable deep learning systems in real-world applications [21], ensuring models can effectively identify data that differ significantly from the training distribution. This ability is vital in safety-critical areas [5], where incorrect predictions due to unexpected inputs can lead to severe consequences. For instance, in the field of autonomous driving, OOD detection helps the system recognize and react appropriately to novel scenarios not covered during training, such as new road signs or altered traffic conditions due to construction. This is imperative as it prevents autonomous vehicles from making potentially hazardous decisions based on learned but now irrelevant data, thereby enhancing their safety and robustness in dynamic environments.

Our research highlights a fundamental yet overlooked challenge in existing OOD detection methods, which often specialize in a single input type. This specialization may inadvertently limit the representational dimensions for detection, complicating the identification of subtle OOD samples. For effective OOD detection, it is crucial to not only improve empirical performance through enhanced OOD discriminative representations but also to address this pervasive issue within the general scoring framework. Our new scoring framework leverages expanded input dimensions and utilizes a confidence score expectation to address these concerns, which also shares similar intuitions with some related work [43] in adversarial defense via random transformation. Comprehensive experiments demonstrate its effectiveness and compatibility, suggesting that our method is potentially a new generalized framework and provides new insights into OOD detection from a different perspective.

5.2 Limitation

While our method introduces a promising framework for OOD detection and provides unique insights through the use of corrupted images to enhance representational dimensions, it does face certain challenges. First, our method indeed faces several failure cases. When CoVer utilizes certain severe corruption types (e.g., Spatter, Elastic transform), its performance is worse than with single input. This is because these types are more severe compared to others, leading to excessive damages to semantic features. Effective corruption types are those only perturb non-semantic features, which generally exist at the high-frequency level, resulting in different confidence variations between ID and OOD data. Notably, except for leveraging the validation set, our approach lacks a standardized criterion for selecting the types and intensities of corruptions, which is essential for uniform effectiveness across various scenarios. Additionally, the expansion of input dimensions, though beneficial for detection accuracy, may lead to increased evaluation times. These limitations highlight areas for potential improvement, particularly in balancing detection capabilities with computational efficiency. Despite these challenges, the enhanced detection capabilities our method introduces mark a significant step forward in the reliability of machine learning models against OOD inputs.

6 Conclusion

In this paper, we introduce a novel perspective for OOD detection, i.e., expanding the representation dimensions. With the different common corruptions, we reveal an interesting phenomenon termed confidence mutation , where the confidence values of some overconfident OOD samples can vary significantly compared with the original inputs. To this end, we propose a new scoring framework, namely, Confidence aVerage (CoVer), which simultaneously considers the original and expanded input dimensions. Adopting a simple but effective average operation, CoVer can capture the dynamical discrimination of OOD samples and better enhance the separability of ID and OOD distributions. We have conducted extensive experiments to present its effectiveness and compatibility with different methods. We hope our work can draw new insights from a different view on OOD detection.

References

Djurisic Andrija, Bozanic Nebojsa, Ashok Arjun, and Liu Rosanne. Extremely simple activation shaping for out-of-distribution detection. In ICLR , 2023.

Haoyue Bai, Gregory Canal, Xuefeng Du, Jeongyeol Kwon, Robert D Nowak, and Yixuan Li. Feed two birds with one scone: Exploiting wild data for both out-of-distribution generalization and detection. In ICML , 2023.

Julian Bitterwolf, Alexander Meinke, Maximilian Augustin, and Matthias Hein. Breaking down out-of-distribution detection: Many methods based on OOD training data estimate a combination of the same core quantities. In ICML , 2022.

Julian Bitterwolf, Maximilian Mueller, and Matthias Hein. In or out? fixing imagenet out-ofdistribution detection evaluation. In ICML , 2023.

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. In arXiv , 2021.

Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Fulong Ye, Qinghong Yang, and Ledell Wu. Altclip: Altering the language encoder in clip for extended language capabilities. In arXiv , 2022.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR , 2014.

Pierre Colombo, Eduardo Dadalto, Guillaume Staerman, Nathan Noiry, and Pablo Piantanida. Beyond mahalanobis distance for textual ood detection. In NeurIPS , 2022.

Pierre Colombo, Marine Picot, Federica Granese, Marco Romanelli, Francisco Messina, and Pablo Piant. A halfspace-mass depth-based method for adversarial attack detection. TMLR , 2023.

Pierre Colombo, Marine Picot, Nathan Noiry, Guillaume Staerman, and Pablo Piantanida. Toward stronger textual attack detectors. In arXiv , 2023.

Maxime Darrin, Pablo Piantanida, and Pierre Colombo. Rainproof: An umbrella to shield text generators from out-of-distribution data. In arXiv , 2022.

Maxime Darrin, Guillaume Staerman, Eduardo Dadalto Câmara Gomes, Jackie CK Cheung, Pablo Piantanida, and Pierre Colombo. Unsupervised layer-wise score aggregation for textual ood detection. In AAAI , 2024.

Jesse Davis and Mark Goadrich. The relationship between precision-recall and ROC curves. In ICML , 2006.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In arXiv , 2020.

Xuefeng Du, Yiyou Sun, Jerry Zhu, and Yixuan Li. Dream the impossible: Outlier imagination with diffusion models. In NeurIPS , 2023.

Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. VOS: learning what you don't know by virtual outlier synthesis. In ICLR , 2022.

Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of-distribution detection based on the pre-trained model clip. In AAAI , 2022.

Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection. In NeurIPS , 2021.

Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning . MIT Press, 2016.

Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for realworld settings. In ICML , 2022.

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR , 2019.

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR , 2017.

Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In NeurIPS , 2021.

Wenyu Jiang, Hao Cheng, Mingcai Chen, Chongjun Wang, and Hongxin Wei. Dos: Diverse outlier sampling for out-of-distribution detection. In arXiv , 2023.

Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Negative label guided ood detection with pretrained vision-language models. In ICLR , 2024.

Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. In Predicting structured data , 2006.

Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS , 2018.

Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR , 2018.

Weitang Liu, Xiaoyun Wang, John D Owens, and Yixuan Li. Energy-based out-of-distribution detection. In NeurIPS , 2020.

Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of-distribution detection with vision-language representations. In NeurIPS , 2022.

Yifei Ming, Ying Fan, and Yixuan Li. Poem: Out-of-distribution detection with posterior sampling. In ICML , 2022.

Yifei Ming and Yixuan Li. How does fine-tuning impact out-of-distribution detection for vision-language models? IJCV , 132(2):596-609, 2024.

Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embeddings for out-of-distribution detection? In arXiv , 2022.

Yifei Ming, Hang Yin, and Yixuan Li. On the impact of spurious correlation for out-ofdistribution detection. In AAAI , 2022.

Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Locoop: Few-shot out-of-distribution detection via prompt learning. In NeurIPS , 2023.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop , 2011.

Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In CVPR , 2015.

Jaewoo Park, Yoon Gyo Jung, and Andrew Beng Jin Teoh. Nearest neighbor guidance for out-of-distribution detection. In CVPR , 2023.

Marine Picot, Nathan Noiry, Pablo Piantanida, and Pierre Colombo. Adversarial attack detection under realistic constraints. 2022.

Marine Picot, Guillaume Staerman, Federica Granese, Nathan Noiry, Francisco Messina, Pablo Piantanida, and Pierre Colombo. A simple unsupervised data depth-based method to detect adversarial images. 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Edward Raff, Jared Sylvester, Steven Forsyth, and Mark McLean. Barrage of random transforms for adversarially robust defense. In CVPR , 2019.

Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In NeurIPS , 2019.

Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In NeurIPS , 2021.

Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In ECCV , 2022.

Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In ICML , 2022.

Leitian Tao, Xuefeng Du, Xiaojin Zhu, and Yixuan Li. Non-parametric outlier synthesis. In ICLR , 2023.

Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR , 2018.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit matching. In CVPR , 2022.

Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say no. In ICCV , 2023.

Qizhou Wang, Zhen Fang, Yonggang Zhang, Feng Liu, Yixuan Li, and Bo Han. Learning to augment distributions for out-of-distribution detection. In NeurIPS , 2023.

Qizhou Wang, Feng Liu, Yonggang Zhang, Jing Zhang, Chen Gong, Tongliang Liu, and Bo Han. Watermarking for out-of-distribution detection. In NeurIPS , 2022.

Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR , 2010.

Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, ShangWen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In arXiv , 2023.

Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR , 2022.

Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, et al. Openood: Benchmarking generalized out-of-distribution detection. In NeurIPS Datasets and Benchmarks Track , 2022.

Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. In arXiv , 2021.

Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. TPAMI , 40(6):1452-1464, 2017.

Jianing Zhu, Yu Geng, Jiangchao Yao, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Bo Han. Diversified outlier exposure for out-of-distribution detection via informative extrapolation. In NeurIPS , 2023.

Jianing Zhu, Hengzhuang Li, Jiangchao Yao, Tongliang Liu, Jianliang Xu, and Bo Han. Unleashing mask: Explore the intrinsic out-of-distribution detection capability. In ICML , 2023.

Appendix for CoVer

The whole Appendix is organized as follows. In Appendix A, we present the detailed definitions and implementation of baseline methods that are considered in our exploration. In Appendix B, we provide detailed discussions about related works. In Appendix C, we provide extra experimental details and more comprehensive results with further discussion on the underlying implications. In Appendix D, we provide some preliminary statistical analysis about CoVer. Finally, in Appendix E, we provide the further analysis for a better understanding of our work.

Reproducibility Statement

To ensure the reproducibility of experimental results, we provide the source code at https:// github.com/tmlr-group/CoVer . Below we summarize several important aspects to facilitate reproducible results:

Datasets. The datasets we used are all publicly accessible, which is introduced in Section 4. Following MCM [31], we also use subsets of ImageNet-1K for fine-grained analysis, like ImageNet-10. For hard OOD evaluation, we exploit ImageNet-20 with 20 categories similar to ImageNet-10. We also reproduce the spurious OOD detection [35] with r = 0 . 9 , which determines relative size of majority vs. minority groups.

Assumption. We set our main experiments to a zero-shot scenario where a well-trained CLIP-like model on the original classification task is available [42]. Under this assumption, CoVer detects OOD samples in parallel with the zero-shot classification task and has no impact on ID classification performance.

Open source. The source code is available at https://github.com/tmlr-group/CoVer . We provide a backbone for our experiments as well as several auxiliary components, such as score estimation.

Environment. All experiments are conducted on NVIDIA GeForce RTX 3090 GPUs with Python 3.10 and PyTorch 2.2.

A Details about Considered Baselines and Metrics

In this section, we provide details about the baselines for the scoring functions, as well as the corresponding hyper-parameters and other related metrics that are considered in our work.

Maximum Softmax Probability (MSP). [23] proposes to use maximum softmax probability to discriminate ID and OOD samples. The score is defined as follows,

where f represents the given well-trained model and c is one of the classes Y = { 1 , . . . , C } . The larger softmax score indicates the larger probability for a sample to be ID data, reflecting the model's confidence on the sample.

ODIN. [29] designed the ODIN score, leveraging the temperature scaling and tiny perturbations to widen the gap between the distributions of ID and OOD samples. The ODIN score is defined as follows,

where ˜ x represents the perturbed samples (controled by ϵ ), T represents the temperature. For fair comparison, we adopt the suggested hyperparameters [29]: ϵ = 1 . 4 × 10 -3 , T = 1 . 0 × 10 4 .

Mahalanobis. [28] introduces a Mahalanobis distance-based confidence score, exploiting the feature space of the neural networks by inspecting the class conditional Gaussian distributions. The Mahalanobis distance score is defined as follows,

where ˆ µ c represents the estimated mean of multivariate Gaussian distribution of class c , ˆ Σ represents the estimated tied covariance of the C class-conditional Gaussian distributions.

Energy. [30] proposes to use the Energy of the predicted logits to distinguish the ID and OOD samples. The Energy score is defined as follows,

where T represents the temperature parameter. As theoretically illustrated in [30], a lower Energy score indicates a higher probability for a sample to be ID. Following [30], we fix the T to 1 . 0 throughout all experiments.

ASH. [1] designs a extremely simple, post-hoc method called Activation SHaping for OOD detection. It removes a large portion of an input's activation at a late layer and adjusts the rest of the activation values by scaling them up or assigning them a constant value. The simplified representation is then passed throughout the rest of the network. The logit output is used to classify ID samples and calculate scores for OOD detection as usual. For ASH-B version, we adopt the MSP score and implement it with the hyperparameter p = 65 ; For ASH-S version, we apply it with energy score and the hyperparameter p = 90 . Both settings are suggested by [1].

MCM. [31] proposes Maximum Concept Matching (MCM), a simple yet effective zero-shot OOD detection method based on aligning visual features with textual concepts. Formally, the MCM score can be defined as:

CLIPN. [52] proposes a novel CLIP architecture, which equips CLIP with a 'no' logic via the learnable 'no' prompts and a 'no' text encoder. Specifically, CLIPN proposes two novel inference algorithms to perform OOD detection via using negation semantics, where the algorithm named agreeing-to-differ (ATD) is more effective in experimental results. The ATD form of the CLIPN score can be formulated as follows,

where C is the number of classes, s i,j ( x ) and s no i,j ( x ) are denoted as the inner product of the image feature and the corresponding text feature

where f text no is the "no" text encoder and p ( y no j ) the text with 'no' logic.

NegLabel. [26] proposes a novel post hoc OOD detection method, called NegLabel, which takes a vast number of negative labels from extensive corpus databases and designs a novel scheme for the OOD score collaborated with negative labels. NegLabel score can be formulated as

where S ∗ ( · , · ) represents a fusion function that combines the similarity of a sample with ID labels sim( x , Y ) and the similarity of the sample with negative labels sim ( x , Y -) . The sum-softmax form of NegLabel score is defined as follows,

where K is the number of ID labels, τ is the temperature coefficient of the softmax function and M is the number of negative labels, s i ( x ) and s neg j ( x ) are formulated as the cosine similarity, defined as follows,

B Detailed Discussion with Related Works

In this section, we provide detailed discussions about related works.

Traditional OOD Detection. There has been an increasing interest in OOD detection since the phenomenon of overconfidence in OOD data was first discovered in [38]. As a formal benchmark for OOD detection, [23] proposed using softmax prediction probability as a conventional baseline method. Afterward, numerous approaches have been developed to address visual OOD detection, which can be classified into two categories, i.e., post hoc scoring mechanism and training-time regularization [59, 58]. Post hoc methods are dedicated to exploring a better OOD score by freezing the model's parameters. ODIN [29] improves the previous MSP [23] by scaling with the temperature and slightly perturbing the inputs. Mahalanobis introduces the Mahalanobis distance in the feature space to measure the confidence score. Energy [30] exploits the energy function [27] to distinguish ID and OOD data. Both ReAct [45] and DICE [46] are improved from Energy, ReAct by feature clipping, and DICE by discarding the most prominent weights in the fully connected layer. ASH [1] designs an extremely simple method that removes a large portion of an input's activation and adjusts the rest. On the other hand, training-time regularization methods exploit the potential access to partial OOD data during model training. MOS [24] groups all classes and introduces an extra class to each group to reformulate the loss function during training. VOS [17] enhances the quality of the energy score by creating synthetic virtual anomalies. CIDER [34] exploits KNN [47] to boost OOD detection performance through the optimization of contrastive loss. DAOL [53] alleviates the OOD distribution discrepancy by crafting an OOD distribution set that contains all distributions in a Wasserstein ball centered on the auxiliary OOD distribution. The presence of outliers leads to superior performance compared to training without outliers, as evidenced by numerous previous studies [3, 25, 2, 16, 62].

OOD Detection with vision-language representations. With the rapid development of multimodal large language models (MLLMs), such as CLIP [42], much attention has been paid to OOD detection with vision-language representations [33]. MCM [31] proposed the first zero-shot OOD detection framework that combines the temperature scaling strategy and maximum softmax probability as the OOD score. Following MCM, some works fine-tuned CLIP's image encoder for visual OOD detection [48, 36]. NPOS [48] utilized generated OOD data to optimize the ID-OOD decision boundary. LoCoOp exploited the portions of CLIP's local features as OOD features to realize OOD regularization. Some latest methods [52, 26] boosted OOD detection by adding extra clues obtained from negative textual information. CLIPN [52] equipped CLIP with a "no" logic via a text encoder that can understand negative prompts. NegLabel [26] introduced numerous negative labels and distinguished OOD samples by examining their affinities between ID and negative labels.

Data depths and information projections. Computing OOD scores on the embedding output of the last layer of the encoder is not the best choice for textual OOD detection. To address this, [12] proposed aggregating OOD scores across all layers and introduced an extended text OOD classification benchmark, MILTOOD-C. In a similar vein, RainProof [11] introduced a relative information projection framework and a new benchmark called LOFTER on text generators, considering both OOD performance and task-specific metrics. Building on the idea of information projection, REFEREE [40] leveraged I-projection to extract relevant information from the softmax outputs of a network for black-box adversarial attack detection. On the other hand, APPROVED [41] proposed to compute a similarity score between an input sample and the training distribution using the statistical notion of data depth at the logit layer. HAMPER [9] introduced a method to detect adversarial examples by utilizing the concept of data depths, particularly the halfspace-mass (HM) depth, known for its attractive properties and non-differentiability. Furthermore, TRUSTED [8] relied on the information available across all hidden layers of a network, leveraging a novel similarity score based on the Integrate Rank-Weighted depth for textual OOD detection. LAROUSSE [10] employed a new anomaly score built on the HM depth to detect textual adversarial attacks in an unsupervised manner.

C Additional Experimental Results and Further Discussion

In this section, we provide additional experimental results from different perspectives to verify the effectiveness our proposed CoVer. First, we introduce the additional experimental setups for the empirical verification in previous figures and implement details about our method. Second, we

provide comprehensive results with further discussions of ablations. Finally, extensive visualization analyses are provided for a better understanding of CoVer.

C.1 Additional Experiment Setups

Implement Details. Unless otherwise specified (e.g. Table 1 and Table 3), we conduct the major experiments based on pre-trained CLIP-B/16 for zero-shot OOD detection, following the previous research work [31, 26]. Furthermore, the primary form of CoVer is based on the maximum-softmax score function, as defined in Section 3.3. For the extended corrupted inputs, We utilize the SVHN dataset as the validation set to determine the most effective corruption types for each method in all experiments. The detailed adopted corruption types for each method are provided in Table 6, where Corruption Type (X) denote the corruption at X severity level.

Figure 1. In Figure 1, we compare the score distributions and detection results with different input modes. Specifically, we use the corruption type of Contrast (4) as an example and report the results on the iNaturalist dataset of the ImageNet-1K benchmark. In the right panel of 1, we realize our CoVer by averaging the confidence scores obtained by the original and corrupted( Contrast (4)) inputs.

Figure 2. In Figure 2, we conduct experiments to demonstrate the detailed explanations for the discovery illustrated in Figure 1. For the fair comparison, all samples here, including confident ID, unconfident ID, overconfident OOD, and unconfident OOD samples, are randomly sampled from the ID and OOD distributions in the left and middle panels of Figure 1.

Figure 3. In Figure 3, we visualize several samples for further understanding of the confidence mutation. Specifically, we use the Fast Fourier transform (FFT) to obtain the low-frequency and high-frequency portions of the image with the radius of circular filter r = 0 . 6 . Same as Figure 1 and Figure 2, unconfident ID data and overconfident OOD data are randomly sampled from the corresponding part of ImageNet and iNaturalist datasets, respectively. We continue to adopt Contrast (4) for corruption.

Figure 5. For Figure 5(a) to Figure 5(d), we provide discussions about the detailed experimental settings and their fine-grained results in Appendix C.2.

C.2 Full Results of Ablations

C.2.1 Ablation on VLM Architectures.

The detailed results are shown in Table 7. It is evidenced that in addition to CLIP Vit-B/16, our CoVer can boost the OOD detection performance on various kinds of VLM architectures compared to MCM, including the CLIP architecture based on different backbone networks (e.g., ResNet50 and Vit-L/14) and other types of VLM architectures (e.g., AltCLIP, MetaCLIP, and GroupViT).

C.2.2 Superiority of Multi-Dimensional Scoring Framework.

In Figure 5(a), we present some representative results to verify the superiority of the multi-dimensional scoring framework. Here, we report the fine-grained results using different types of corruptions with a severity level of 5, as shown in Table 8. We can find that there is constantly a bad OOD detection

performance under the single-dimensional scoring framework with a single corrupted image as the input sample. However, considering the multiple inputs, i.e., both original and corrupted inputs, averaging their confidence scores shows a huge performance enhancement. For example, comparing the uni-dimensional and multi-dimensional results when the corruption type is Contrast , the incorporation of the extended dimension provides +17.57% AUROC (from 74.20% to 91.77% ) and -52.02% FPR95 (from 88.61% to 36.95% ), indicating that corrupted inputs require the combination of original inputs to reveal its ability to enhance the distinguishability of OOD samples at the feature level. For more detailed experiments regarding CoVer expanded by corrupted inputs at other severity levels, please refer to Appendix C.2.4.

To have a more intuitive understanding on the superiority of multi-dimensional inputs, we provide a comparison with score distributions and detection results w/w.o. an expanded input dimension transformed by Brightness , Fog , Motion blur , and Speckle noise in Figure 6. From the middle column, we notice that the OOD samples are more difficult to detect by the model confidence from single corrupted inputs. This is mainly because the confidence of the ID sample, which was originally high, drops drastically when it is corrupted thereby interfering with the model's judgment of the OOD sample. In contrast, through a simple but critical average operation, CoVer generally achieves better ID-OOD separability. This phenomenon can be attributed to two main reasons. First, the ID samples have an overall higher confidence expectation, eliminating the originally confident ID interference present under the single corrupted input. Secondly, as illustrated in Figure 2 and Figure 3, the data lies in the originally overlapped part of the ID and OOD distributions, i.e., unconfident ID data and overconfident OOD data, demonstrate significant differences in the variation of confidences under the same corruption. Specifically, the ID data shows resistance while the OOD data shows vulnerability, thus better exposing the OOD samples to be rejected.

C.2.3 Imapact of the Number of Expanded Measuring Dimensions.

In Figure 5(b), we report the OOD detection performance variations evaluated on different numbers of expanded measuring dimensions. However, we have not specifically analyzed the impact of employing different types of corruptions under each number of extended representation dimensions. In Table 9, we further present the detailed results for different numbers of extended dimensions under corruption severity levels 1 and 2, with each number we enumerate two different kinds of combinations of corruption types. The experimental results demonstrate that considering confidence estimation on both original input and expanded variant dimension constantly enhances the OOD detection performance across four OOD datasets.

However, as the expanded dimension gives priority to the phenomenon of confidence mutation that is more discriminative in OOD data than that is in ID data, the newly added representation dimension sometimes leads to a slight decline in performance. For instance, the best performance is three expanded dimensions that grouped by Defocus blur , Motion blur and Fog inputs under the corruption severity level 1; the combination of five types of corruptions including Defocus blur , Motion blur , Gaussion blur , Fog , and Saturate achieve a better OOD detection performance when the variants are generated by corruptions at severity level 2. This phenomenon suggests that, while

extended dimensions preferentially provide additional clues that make the OOD samples more salient, excessive extended dimensions or the inclusion of input dimensions transformed from some specific uncommon corruptions may result in a greater degree of interference in the ID samples, leading to a slight declination of the performance.

C.2.4 Full Results of CoVer with Variants Corrupted at Different Severity Levels.

In Figure 5(c), we report three representative variation trends of performance with an increasing level of the corruption severity. In Table 10 and Table 11, we present the full results of the proposed CoVer with 18 different types of corruptions at 5 severity levels. It is worth noting that the experimental results here are all based on the average of the model confidence scores measured on one original input and one extended corrupted input. Furthermore, we present the performance changing trends of our CoVer based on all 18 corruption styles in Figure 7. It can be seen that the performance of CoVer is more sensitive to the severity levels of corruptions compared to the number of extended representation dimensions analyzed in Appendix C.2.3. Specifically, as shown in Figure 7, some common corruption types from categories including weather (e.g., Brightness and Fog ) and blur (e.g., Motion blur and Defocus blur ) can achieve lower FPR95 values. We further observe that the commonality of these better-performing corruption types is that their perturbations to image features are more mild compared to other types (e.g., Spatter and Elastic transform from the digital class, Impulse noise from the noise category), and they generally do not excessively corrupt the semantic features. In Appendix C.4, we will more intuitively demonstrate the distinctions between these various types of corruptions through the visualizations of corrupted ID and OOD samples, illustrating the varying degrees of enhancement or attenuation they impart on the performance of CoVer.

Original Input

Multiple Inputs (CoVer)

AUROC: 94.63    FPR95: 30.70

AUROC: 83.54    FPR95: 80.59

AUROC: 96.49

↑

FPR95: 19.30

↓

Original Input

Multiple Inputs (CoVer)

AUROC: 94.63    FPR95: 30.70

AUROC: 82.12    FPR95: 71.66

AUROC: 94.98

↑

FPR95: 26.13

↓

Motion Blur Input

Original Input

Multiple Inputs (CoVer)

AUROC: 94.63    FPR95: 30.70

AUROC: 88.42    FPR95: 60.34

AUROC: 95.61

↑

FPR95: 24.19

↓

Original Input

Brightness Input

Multiple Inputs (CoVer)

AUROC: 94.63    FPR95: 30.70

AUROC: 92.84    FPR95: 41.42

AUROC: 95.11

↑

FPR95: 28.29

↓

14000

16000

16000

C.3 Exploration on the Formulation of CoVer

In Section 3.3, we discuss about the scheme of designing the OOD score with extra corrupted inputs:

The critical mechanism of S CoVer can be summarized as follows: S CoVer is an confidence expectation with respect to the OOD score estimated from natural input and its corrupted variant.

To have a closer look at our proposed CoVer, we further separately explain the implementation mechanism of CoVer at the input level and the output level. At the input level, CoVer is an extension of the input representation dimension, which introduces additional variant feature inputs into the model. On the other hand, CoVer is an extension of the confidence assessment dimension at the output level, which enables the integration of multi-dimensional confidence by averaging the confidence under every representation dimension. Such a design criterion leaves a further exploration on the specific formulation of CoVer. In other words, the realization of the estimated confidence score under each evaluation dimension can have multiple choices.

Table 12 shows the results of different implementation schemes for the CoVer score function. We also provide the original score function formulations of each method and their performance for comparison. All experiments in this table are evaluated on ImageNet-1K benchmark and all methods are based on CLIP-B/16. The results indicate that these scoring methods can achieve more comparable performances under the setting of the CoVer mode, as analyzed in Section 4.2,

Turning the attention to the implementation form of the proposed CoVer, we divide it into two categories: traditional scoring functions (from row 1 to row 4) and newly designed scoring functions based on negative textual logic (from row 5 to row 8). For traditional maximum-softmax and free energy scoring functions, we simply employ the expectation of them on the expanded multiple representation dimensions to realize our CoVer, as demonstrated in Eq. (17). For novel CLIPN and NegLabel scoring functions that exploit extra textual information, we apply the CoVer's mechanism to the calculation of the similarity between the image input and the negative text input. Formally it can be summarized as follows:

where x ∼ d ( X ) is denoted as the input image sampled from original input space, and ˜ x ∼ d ( X , ˜ X ) refers to the input image sampled from expanded input space. x and ˜ x are involved in calculating the similarity to the positive textual concept y and negative textual concept y -, respectively. S ∗ is the realization form of CLIPN and NegLabel, which can be found in row 6 and row 8 of Table 12.

Method

Mode

Score Function

OOD Dataset

iNaturalist

SUN

Places

Textures

Average

AUROC

↑

FPR95

↓

AUROC

↑

FPR95

↓

AUROC

↑

FPR95

↓

AUROC

↑

FPR95

↓

AUROC

↑

FPR95

↓

MSP

Original

i

e

s i

(

x

)

/τ

∑

K

j

=1

e

s j

(

x

)

/τ

94.61

30.95

92.57

37.57

89.77

44.65

86.10

57.77

90.76

42.73

CoVer

E

x

∼

d

(

X

,

˜

X

)

i

e

s i

(

x

)

/τ

∑

K

j

=1

e

s j

(

x

)

/τ

95.98

22.55

93.42

32.85

90.27

40.71

90.14

43.39

92.45

34.88

Energy

Original

log

∑

K

j

=1

e

s

j

(

x

)

/T

85.54

80.49

84.21

78.75

84.81

72.29

66.63

92.89

80.30

81.11

CoVer

E

x

∼

d

(

X

,

˜

X

)

log

∑

K

j

=1

e

s

j

(

x

)

/T

88.28

70.78

86.24

76.39

86.17

70.3

67.87

92.61

82.14

77.52

CLIPN

Original

∑

C

j

=1

e

s no

i,j

(

x

)

/τ

e

s i,j

(

x

)

/τ

+

e

s no

i,j

(

x

)

/τ

·

e

s i,j

(

x

)

/τ

∑

C

k

=1

e

s i,k

(

x

)

/τ

95.63

21.62

94.27

25.18

93.15

30.51

90.34

41.68

93.35

29.66

CoVer

E

x

∼

d

(

X

)

,

˜

x

∼

d

(

X

,

˜

X

)

∑

C

j

=1

e

s no

i,j

(˜

x

)

/τ

e

s i,j

(

x

)

/τ

+

e

s no

i,j

(˜

x

)

/τ

·

e

s i,j

(

x

)

/τ

∑

C

k

=1

e

s i,k

(

x

)

/τ

95.41

23.14

95.72

17.13

94.80

23.05

88.59

40.82

93.63

26.04

NegLabel

Original

∑

K

i

=1

e

s i

(

x

)

/τ

∑

i K

=1

e

s i

(

x

)

/τ

+

∑

j M

=1

e

s

j

(

x

)

/τ

99.49

1.93

95.46

20.95

91.58

36.45

89.89

45.12

94.10

26.11

CoVer

E

x

∼

d

(

X

)

,

˜

x

∼

d

(

X

,

˜

X

)

∑

K

i

=1

e

s i

(

x

)

/τ

∑

i K

=1

e

s i

(

x

)

/τ

+

∑

j M

=1

e

s

j

(˜

x

)

/τ

99.59

1.15

94.56

28.84

95.01

25.65

92.39

40.39

95.39

24.01

C.4 Visualization Analysis

C.4.1 Visualization of Corrupted ID and OOD Samples

In Figure 8, we provide the visualized examples under 18 different types of algorithmically generated corruptions which are officially introduced in [22]. Furthermore, there are 5 severity levels for each corruption style, ranging from inconsequential to devastating corruption of the original clean image as shown in Figure 9. By synthesizing these 18 types and 5 severity levels, we generate a comprehensive and diverse set of 90 distinct forms of corrupted inputs, which can be leveraged to enhance the dimensionality of input representation.

Original Sample

JPEG

Contrast

Spatter

Saturate

Pixelate

Impulse Noise

Speckle Noise

Defocus Blur

Elastic

Snow

Fog

Brightness

Clean

Severity = 1

Severity = 2

Severity = 3

Severity = 4

Severity = 5

C.4.2 Comparison with Salient Maps and Corresponding Confidence Variations.

To better illustrate the effectiveness of CoVer, which primarily stems from the differing confidence variations between ID and OOD samples under identical corruption conditions, we provide some visualization results of ID and OOD images, as shown in Figure 10 and Figure 11. All the images are picked from the datasets in the ImageNet-1K OOD detection benchmark. Each subfigure shows the feature maps of the original image and its corrupted variants, including Contrast (Severity Level 4) and Defocus Blur (Severity Level 2). For comparison, we also provide the corresponding confidence variations between the original (red bar), the corrupted (blue bar), and the averaged (yellow bar) one. The confidence scores are based on the form of maximum-softmax scoring function given by CLIP-B/16. We continue to divide the data into four categories, denoted as confident ID data (refer to row 1 to row 2 in Figure 10), unconfident ID data (refer to row 3 to row 6 in Figure 10), overconfident OOD data (refer to row 1 to row 4 in Figure 11), and unconfident OOD data (refer to row 5 to row 6 in Figure 11). Here we focus on the differences between unconfident ID data and overconfident OOD to verify the analysis claimed in Section 3.2.

In Figure 10, it is obvious that ID images have more significant ID-semantic feature activations (see the foreground salient responses) due to the knowledge of the ID label space. Firstly, for confident ID data, the changes in confidence post-corruption can manifest as either minor or abrupt. For example, in the image in row 1, column 1 (ILSVRC2012_val_00020025), the corruption results in only a negligible loss of the semantics of the ID category present in the foreground. Conversely, for the image in row 2, column 1 (ILSVRC2012_val_00044407), the corruption enhances disturbances from non-semantic areas, leading to a loss of model confidence. Nevertheless, by averaging the original and corrupted confidence scores, the confident ID data remains stable within a higher confidence interval, demonstrating the superiority of CoVer's mechanism. Secondly, for unconfident ID data, due to the presence of ID semantic features, the model exhibits resilience in its confidence when subjected to corruption. For instance, the image in row 4, column 1 (ILSVRC2012_val_00022503) shows the model's resilience, as it continues to focus on the foreground regions belonging to the ID category despite various degrees of corruption, thus maintaining its confidence score unaffected. The same results can also be seen in row 3, column 1 (ILSVRC2012_val_00048997) and row 4, column 2 (ILSVRC2012_val_00020119).

In Figure 11, particularly in the case of overconfident OOD images, there is an excessive reaction to the corruption. This phenomenon is apparent because, unlike ID data, OOD images inherently lack semantic information about ID categories, leading to the disappearance of areas with high feature activation under the same type and severity level of corruption. For instance, the image in row 2, column 1 (f_formal_garden_00003688) demonstrates that regions highly responsive in their original state become irrelevant after corruption, resulting in the confidence mutation. This shift further confirms that the model's overconfidence in them is primarily due to misleading non-semantic features, such as textures and styles. Similar results can also be found in other OOD datasets, such as the image in row 3, column 1 (sun_arsrlxiznzlekfvg), and the image in row 4. column 2 (wrinkled_0070). However, the confidence of unconfident ID data, comparable to that of these overconfident OOD samples, remains stable. Consequently, by averaging the confidence scores before and after corruption, CoVer effectively captures the distinctions between OOD and ID data that initially overlap, thus enhancing the separability of ID and OOD distributions. Furthermore, for unconfident OOD data, due to their overall low relevance to ID semantics, the confidence scores consistently remain in a lower range (like the image in row 5, column 1, named 1b0ac86be7f53fd9058646315ed17269).

Dataset Name: ImageNet

Fig Name: ILSVRC2012_val_00020025

Origin

Contrast

Defocus Blur

Dataset Name: ImageNet

Fig Name: ILSVRC2012_val_00013357

Origin

Contrast

Defocus Blur

Confidence Variation

Confidence Variation

Dataset Name: ImageNet

Fig Name: ILSVRC2012_val_00044407

Origin

Contrast

Defocus Blur

Dataset Name: ImageNet

Fig Name: ILSVRC2012_val_00040000

Origin

Contrast

Defocus Blur

Confidence Variation

Confidence Variation

Dataset Name: ImageNet

Fig Name: ILSVRC2012_val_00048997

Origin

Contrast

Defocus Blur

Dataset Name: ImageNet

Fig Name: ILSVRC2012_val_00018279

Origin

Contrast

Defocus Blur

Confidence Variation

Confidence Variation

Dataset Name: ImageNet

Fig Name: ILSVRC2012_val_00022503

Origin

Contrast

Defocus Blur

Dataset Name: ImageNet

Fig Name: ILSVRC2012_val_00020119

Origin

Contrast

Defocus Blur

Confidence Variation

Confidence Variation

Dataset Name: ImageNet

Fig Name: ILSVRC2012_val_00035061

Origin

Contrast

Defocus Blur

Dataset Name: ImageNet

Fig Name: ILSVRC2012_val_00031446

Origin

Contrast

Defocus Blur

Confidence Variation

Confidence Variation

Dataset Name: ImageNet

Fig Name: ILSVRC2012_val_00020767

Origin

Contrast

Defocus Blur

Dataset Name: ImageNet

Fig Name: ILSVRC2012_val_00045824

Origin

Contrast

Defocus Blur

Confidence Variation

Confidence Variation

ConfidenceScore (1e-3)

9

Score（1e-3）

Confidence

ConfidenceScore(1e-3)

Score（1e-3）

Confidence

ConfidenceScore(1e-3)

Confidence Score (1e-3）

Score(1e-

Confidence

Score（1e-3）

Confidence

ConfidenceScore(1e-3)

Confidence Score(1e-3）

ConfidenceScore(1e-3)

Score(1e-3）

Confidence

Dataset Name: iNaturalist

Fig Name: 740a68e988259f377b319dafb2e7e0dd

Origin

Contrast

Defocus Blur

Dataset Name: iNaturalist

Fig Name: 30bbe31cab9b83df7582a891924ab75e

Origin

Contrast

Defocus Blur

Confidence Variation

Confidence Variation

Dataset Name: Places

Fig Name: f_formal_garden_00003688

Origin

Contrast

Defocus Blur

Dataset Name: Places

Fig Name: h_hayfield_00003239

Origin

Contrast

Defocus Blur

Confidence Variation

Confidence Variation

Dataset Name: SUN

Fig Name: sun_arsrlxiznzlekfvg

Origin

Contrast

Defocus Blur

Dataset Name: SUN

Fig Name: sun_bofnqnjunvcoqtpu

Origin

Contrast

Defocus Blur

Confidence Variation

Confidence Variation

Dataset Name: Textures

Origin

Contrast

Defocus Blur

Dataset Name: Textures

Origin

Contrast

Defocus Blur

Confidence Variation

Confidence Variation

Dataset Name: iNaturalist

Fig Name: 1b0ac86be7f53fd9058646315ed17269

Origin

Contrast

Defocus Blur

Dataset Name: Places

Origin

Contrast

Defocus Blur

Confidence Variation

Confidence Variation

Dataset Name: SUN

Fig Name: sun_asfgylptpdqklqtx

Origin

Contrast

Defocus Blur

Dataset Name: Textures

Origin

Contrast

Defocus Blur

Confidence Variation

Confidence Variation

ConfidenceScore (1e-3)

Score（1e-3）

Confidence

ConfidenceScore (1e-3)

Confidence Score(1e-3）

ConfidenceScore(1e-3)

Score（1e-3）

Confidence

Score（1e-3）

Confidence

ConfidenceScore(1e-3)

Confidence Score(1e-3)

ConfidenceScore(1e-3)

Confidence Score (1e-3)

D Preliminary Statistical Analysis

In this section, we present the statistical implications with detailed definitions and assumptions. The primary objective is to show that CoVer can lead to an increase in the separability of the distributions of ID and OOD by introducing confidence expectations under the extended representation dimension. In the following parts, we first introduce the considering performance metric and some preliminary setups for the analyses.

Metric. The separability between ID and OOD data can be reflected by the FPR λ , which is the performance metric of an OOD detector, defined as follows,

where F in and F out represent the cumulative distribution functions (CDFs) corresponding to the confidence scores obtained by ID and OOD samples, respectively. λ ∈ [0 , 1] is denoted as the true positive rate (TPR), indicating the proportion of samples that are correctly classified as one of the ID categories. The metric FPR λ quantifies the overlap degree between the scores that the OOD detector assigns to ID and OOD samples, with lower values indicative of superior performance.

Preliminary setups. Following previous works [32, 31], owing to the robust representational capabilities of pre-trained DNNs and the consistent alignment between cosine similarity scores and labels observed in CLIP-like models, we assume that the features extracted from DNNs or the cosine similarity scores in CLIP-like models approximately conform to a Gaussian Mixture Model (GMM) with equal class priors: ( 1 2 N ( µ pi , σ pi ) + 1 2 N ( µ po , σ po ) ) , where µ pi and µ po are the means of the ID and OOD distribution, while σ pi and σ po are the corresponding standard deviations.. Specifically, we use D ID = N ( µ pi , σ pi ) and D OOD = N ( µ po , σ po ) denote the ID marginal distribution and the OOD marginal distribution, respectively.

Assumptions. Refer to Figure 1 and Figure 6, the comparisons of score distributions obtained by different input modes, we can derive a series of assumptions about the variation relationships between µ pi and µ po , and between σ pi and σ po . Empirical exploration can be found in Figure 12.

Assumption D.1 (Variation of µ pi and µ po ) . Let ∆ µ pi and ∆ µ po represent the changes in the means of ID and OOD distributions, respectively, after corruption and averaging. We assume that | ∆ µ pi | > | ∆ µ po | , resulting in a narrowing gap between µ pi and µ po .

This assumption is predicated on the observation that the means of ID distributions, µ pi , decrease more significantly under identical corruption levels compared to OOD distributions, µ po . The generally higher initial confidence scores of ID samples make their means more susceptible to substantial decreases (refer to the left panel of Figure 12). This reduction is greater than that experienced by OODsamples, thereby significantly narrowing the gap between µ pi and µ po . For an intuitive example, the gap between the means of confident ID data (left panel of Figure 12) and overconfident OOD data (right-middle panel of Figure 12) is closer. This illustrates the pronounced impact of the averaging operation on ID distributions compared to OOD distributions.

Assumption D.2 (Variation of σ pi and σ po ) . We define the changes in the variances of ID and OOD distributions as ∆ σ pi and ∆ σ po , respectively. We postulate that the reduction in variance for ID distributions, ∆ σ pi , is greater than that for OOD distributions, ∆ σ po : | ∆ σ pi | > | ∆ σ po | .

This assumption is supported by the observation that high-confidence ID samples, due to their higher initial confidence levels, experience larger and more abrupt drops in confidence upon corruption (see the left panel of Figure 12). Consequently, the averaging process post-corruption results in a significantly greater reduction in the variance σ pi in ID distributions compared to σ po in OOD distributions. This marked decrease in variability within ID confidence scores, relative to the OOD ones, underscores the efficacy of the averaging operation in dramatically stabilizing the ID confidence scores more than the adjustments observed in OOD confidence scores.

Given the preliminaries and assumptions above, we can derive the following extended lemma to demonstrate the superior performance of CoVer.

Lemma D.3 (Declination of FPR λ ) . Assuming the variation relationships between µ pi and µ po , and between σ pi and σ po , CoVer enables a lower FPR λ .

Proof of Lemma D.3 . We aim to investigate the relationship between the FPR λ metric and the variations in µ pi , µ po , σ pi , and σ po . The FPR λ metric can be reformulated as follows:

where Φ is he cumulative distribution function of the Gaussian distribution, and Φ -1 is its inverse function. Considering the differences before applying CoVer:

With the assumptions that ∆ µ and ∆ σ are affected by the averaging process in CoVer, we express these changes as:

where µ ′ pi and µ ′ po are the new means post-CoVer, σ ′ pi and σ ′ po are the new variances post-CoVer. Given that ∆ µ new is reduced and ∆ σ new indicates a significant contraction, particularly a reduction in σ ′ pi relative to σ ′ po , the numerator in ( µ pi + σ pi · Φ -1 ( λ ) -µ po ) /σ po , decreases while the denominator basically unchanged. This results in the argument of Φ becoming smaller. Since Φ is monotonically increasing, a decrease in the argument directly translates to a lower value of FPR λ , thereby reducing the probability of falsely classifying OOD samples as ID.

This analysis underscores the benefit of CoVer, particularly through its influence on expanding representation dimensions and optimizing the detection framework. By narrowing the gap between the mean scores and significantly reducing the variance in ID distributions relative to OOD, CoVer enhances the model's discriminative capability and improves its robustness, ultimately leading to more reliable OOD detection.

E Future Analysis

E.1 Discussion about Extra Runtime

Considering the addition of corrupted inputs, CoVer would result in non-negligible additional runtime. If there are N expanded dimensions, it will take N times the duration of a single input to implement CoVer. However, our CoVer is only applied in the inference phase of OOD detection, and it is generally fast, as shown in Table 13. We believe that the performance improvements offered by CoVer are well-worth the extra few minutes of runtime.

E.2 Exploring the Impact of Corruption Types and Severity Levels

Appropriate types of corruptions and corresponding severity levels are crucial for the optimization of our CoVer. In all experiments, we use the SVHN dataset as the validation set to select the most effective corruption types for each method. Specific examples of selections are provided in Table 14. For the types of corrupted inputs and their corresponding severity levels, we have conducted related explorations (e.g., Tables 10 and 11, Figures 6 and 7 of our original submission) for performance references. Some specific corruptions (e.g., Brightness, Fog, Contrast, Motion Blur, Defocus Blur) can generally improve the OOD detection performance, as the corruptions are mainly on the nonsemantic level of the input, instead of damaging the semantic features too much like the other types. Empirically, refer to Table 15, we can use the same type of corruption as the expanded input (e.g., here Brightness with severity 1 used) to perform better than the original version. This provides the verification of the previous intuition about the general guidance for choosing appropriate corruption types, and understanding dimension expansion for OOD detection.

E.3 Exploration on the harder OOD dataset

NINCO [4] has proposed three OOD datasets with no categorical contamination which include NINCO, OOD unit-tests, and NINCO popular OOD datasets subsamples, which are demonstrated to be harder than common OOD detection benchmarks. Here, we evaluate the effectiveness of CoVer on these datasets in 16. The results demonstrate that CoVer, when combined with ASH, consistently achieves better performance across the three NINCO OOD datasets.

E.4 Comparison with other Competitive Methods

Comparison with NNGuide and MaxLogit. To provide a comprehensive comparisons, we have conducted comparison experiments with NNGuide [39] and MaxLogit [21] to enrich our analysis in Table 17. First, our experimental results show that CoVer outperforms these competitive post-hoc methods on the ResNet50 architecture. Second, the performance of these post-doc methods, especially NNGuide, encounter significant drop when conducted on CLIP-B/16 architecture. We believe the reason for the poor performance is the difference in training data. Many pos-hoc methods are designed on ImageNet pre-trained networks, where only ID data are used during training. In contrast, when training CLIP, both ID datas and OOD datas are used. This leads to different activations of OOD data. Another reason is that the pos-hoc method relies heavily on the choice of hyperparameters. The hyperparameters of NNGuide need to be re-selected on different models. Despite these issues, our CoVer can still perform better than these methods. Furthermore, we also combine our CoVer with MaxLogit and NNGuide and report the results in Table 18, which further demonstrates the effectiveness and compatibility of our method.

Comparison with Watermarking. In addition, Watermarking [54] is another competitive methods needed to be considered. We have added the analysis about the Watermarking method with our CoVer in the following two aspects.

Conceptually, we have noticted that Watermarking utilizes a well-trained mask to help the original images be distinguishable from the OOD data. However, Watermarking is still trying to excavate the useful feature representation in a single-input perspective. In contrast, the critical distinguishable point and also the advantage of our CoVer method lies in input dimension expansion with the corrupt variants, which instead provide a extra dimension to explore the confidence mutation to better identify the OOD samples.

Experimentally, we have conducted the comparison and report the results in Table 19. The results show that, on the one hand, training an optimized watermarking for effectively distinguishing between ID and OOD samples is a time-consuming process. On the other hand, CoVer achieves this by introducing corrupted inputs to capture the confidence variations between ID and OOD data during the test phase, which is simpler, faster, and more effective.

Comparison with Data Depth, Information Projection, and Isolation Forest. We have also conducted comparison experiments between our CoVer and baselines methods from data depths, information projections, and isolation forest, as detailed in Table 20.

Due to the large scale of the ImageNet training set, we sampled 50 samples per class to construct a subset from the training data to represent the training distribution, as recommended by the similar work named NNGuide [39]. For data depths, we reimplemented APPROVED [41] for comparison. For information projections, we reproduced REFEREE [40] for comparison. For Isolation Forest, we use logits as the input to detect the anomaly logits in ID and OOD samples.

The results indicate that AD and textual OOD detection methods, such as Data Depth and Information Projection, may not suit for visual OOD detection tasks, a view also mentioned in related surveys [8, 9]. Similarly, classical ML methods for AD, such as Isolation Forest, seem to be failed to excavate discriminative representations when applied to image OOD detection. However, since these methods are insightful in distinguishing the outliers, we believe it is worth further efforts in the future to adopt the critical intuition into the OOD detection problem.

E.5 Compatibility with each DNN-based mehtods

It is worth to note that, in Table 1, we only reported the results of CoVer combined with ASH because it best demonstrates the excellence of CoVer. In Table 3, we also show the results of CoVer combined with DICE and ReAct, and CoVer can also provide performance gains for them. Here in Table 21, we further report the comparison of CoVer combined with each mentioned DNN-based methods (adding MSP, ODIN, and Energy score), which strongly demonstrates its superiority.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have clearly stated the main contributions made in the paper and important assumptions and limitations in both the Abstract and the Introduction (Section 1).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 5.2, we have discussed the limitations of this work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include new theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have presented a reproducibility statement at the beginning of the Appendix. In addition, we have provided the code and data in the supplementary material to reproduce the main experimental results of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided open access to the code and data with sufficient instructions in the supplementary material to faithfully reproduce the main experimental results of the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In Section 4.1, we have briefly introduced the experimental setups about datasets, model setups, and evaluation metrics. Furthermore, we have provided full details of experimental settings in Appendix C.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In the third ablation study about the number of expanded measuring dimensions of our proposed method, the results are accompanied by error bars, and the factors of variability are clearly stated (refer to Section 4.3). In Appendix C.2.3, we have provided the full results of this experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided sufficient information on the computer resources, such as the type, amount, and memory of compute workers GPU required for the experiments in the reproducibility statement at the beginning of the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics, and the authors remain anonymous.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In Section 5.1, we have discussed the potential broader impact of this paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All used assets in the paper are properly credited, with explicit mentions of their licenses and terms of use, in full compliance with the recommended guidelines.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn previous OOD detection methods, extracting discriminative information from OOD data relative to ID data is challenging with the representation of a single input. This paper provides a novel perspective for out-of-distribution (OOD) detection by leveraging multiple types of corruptions to expand the original single input space. Utilizing the interesting phenomenon of confidence mutation, the authors introduce a new scoring method termed CoVer, which averages the confidence scores measured from multiple input spaces, achieving better separability between ID and OOD distributions. Extensive experimentation underscores that the CoVer approach not only outperforms previous methods in both DNN-based and VLM-based OOD detection benchmarks, but also exhibits commendable compatibility when combined with different OOD detection methods. Moreover, the CoVer technique demonstrates exemplary adaptability across diverse VLM architectures.\n\nSTRENGTHS:\n1. Well written and technically sound. This paper is well organized and written. Both the motivation and the main contributions of the proposed method are easy to follow. Sufficient experiments have been conducted to demonstrate the effectiveness of the proposed method.\n\n2.Novelty. The challenges of the previous methods are well summarized in OOD detection, i.e., single input constrains the representation ability for detection. The proposed method expands the input space to aware the ID and OOD distribution effectively. \n\n3. Extensive experiments and ablations. The paper conducts extensive experiments, including OOD detection using different backbones, hard OOD detection, and various aspects of ablation studies. The performance improvements strongly verify the effectiveness of the proposed method.\n\n4. Clear justification and theoretical analysis. The paper provides a clear and intuitive justification for the proposed method by a thorough theoretical analysis. This analysis clearly elucidates the mechanisms beyond CoVer, making the paper convincing and compelling.\n\nWEAKNESSES:\n1.Lack explaination and comparison. Different types of corruptions are used in the method, which is similar to the idea of Watermarking (Wang, et al. 2022), where a generic watermark is trained to distinguish between ID and OOD data. The advantages are not deeply analyzed.\n\n2.Less clarification. The CoVer approach is flexible and compatible and can be combined with different methods to achieve performance gains. The combination is not clearly presented for different methods.\n\nQUESTIONS:\n1.Watermaking[1] is a very popular and SOTA method for OOD detection. Compared with watermarking and so on, what are the advantages of using corrupted images to expand the input space. Since the corrupted images are the important data in the proposed method, it is necessary to present the advantages for OOD with corrupted images more clearly.\n[1]Wang Q, Liu F, Zhang Y, et al. Watermarking for out-of-distribution detection. Advances in Neural Information Processing Systems, 2022, 35.\n\n2. The proposed method can enhance most of the OOD methods and improve their performance. However, these methods usually designed with different techniques, such as fine-tuning and zero-shot methods. Hence, I wonder whether there exsit differences to improve these methods with CoVer and how to guarantee the effectiveness when combine CoVer with these methods.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first identifies the shortcoming of previous out-of-distribution (OOD) detection methods: the single-input paradigm limits the representation dimension for extracting valid information. To address this issue, a novel method CoVer that expands the original input space with multiple types of corruption is proposed. Based on the phenomenon of confidence mutation, CoVer achieves better performance by averaging the confidence scores across different dimensions. The proposed method achieves the best performance in the extensive experiments.\n\nSTRENGTHS:\n1. The motivation of the paper is easy to follow. The idea of using common corrupted images to expand the input space is both novel and reasonable. While previous methods based on single input often struggle with detecting hard-to-distinguish OOD samples, the proposed method offers a practical approach to improving the performance of OOD detection.\n\n2. The extensive experiments demonstrates the solid contributions of the proposed work and significant improvements have ahieved for different methods. Using the ImageNet-1K benchmark, the experiments validate the compabability of the proposed CoVer method effectively.\n\n3. The proposed method is designed with robust theoretical foundation of confidence mutation and the proposed score function. The analysis is presented clearly and accessibly for the novel insights, making it easy to understand the mechanism of the proposed framework.\n\nWEAKNESSES:\n1. Although the proposed method can improve the performance of the SOTA methods, whether they face the same challenges and how does the proposed method improve each method are not clearly explaiend.\n\n2. For the different methods, what types of the inputs can improve them effectively lacks of deeply analysis.\n\nQUESTIONS:\n1. Since the proposed method is a framework to improve the OOD performance, I wonder how does the proposed CoVer method improve each method? Why the proposed method is effective to improve different types of the method should be further explaiend.\n2. For the proposed method, it mainly focuses on expanding the inputs. While for the different method, they usually focus on different knowledge. I wonder how to decide what types of the inputs that can improve the performance of different methods.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors ntroduce a new approach to out-of-distribution (OOD) detection by expanding input representation dimensions using common corruptions. Traditional methods focus on single-input representations, limiting their effectiveness. This work identifies \"confidence mutation,\" where OOD samples' confidence levels drop significantly under corruptions, while in-distribution (ID) data maintains higher confidence due to resistant semantic features.\n\nThe proposed method, Confidence aVerage (CoVer), averages confidence scores from original and corrupted inputs to improve OOD detection. Extensive experiments show that CoVer enhances performance across various benchmarks. Key contributions include:\n\n1. Expanding input representation dimensions for better OOD detection.\n2. Introducing confidence mutation to distinguish OOD data.\n3. Proposing the CoVer scoring method for improved separability.\n4. Validating CoVer's effectiveness through extensive experiments\n\nThis is a nice tricks that are many connection to others fields such a s NLP.\n\nSTRENGTHS:\nNice idea which have not been applied to OOD detection but is well known on other fields such as NLP (e.g . retrieval)\n\nWEAKNESSES:\n**Omission of Data Depths and Information Projections in Related Work**\n\nData depths and information projections have shown significant promise in the OOD detection community due to their ability to provide robust and high-dimensional representations of data. Data depths, in particular, are well-suited for this problem as they project in all possible directions, capturing the entire distribution's structure and providing a natural fit for OOD detection tasks. Despite their relevance, these approaches are notably absent from the related work section of this paper. Including them could provide a more comprehensive overview of the field and strengthen the contextual foundation of the study. Relevant works that should be considered include:\n\nM. Darrin. \"Unsupervised Layer-wise Score Aggregation for Textual OOD Detection.\"\n\nM. Picot. \"A Halfspace-Mass Depth-Based Method for Adversarial Attack Detection.\" TMLR 2023.\n\nP. Colombo. \"Beyond Mahalanobis Distance for Textual OOD Detection.\" NeurIPS 2022.\n\nQUESTIONS:\n1. Why were data depths, which project onto all possible directions and are naturally suited for OOD detection, not explored in your related work?\n2. Why were classical methods like Isolation Forest, which are well-known for their anomaly detection capabilities, not considered?\n3. Can you compare your results against the previously introduced baselines involving data depths, information projections, and classical methods like Isolation Forest?\n\nTo me the paper is incomplete and the positioning as well as the choice of the methods (react, dice) is poor and deserve the paper, whereas classical ML tools are appropriated. \n\nI encourage the authors to revise.\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to identify Out-Of-Distribution (OOD) samples by applying common image corruptions (noise, blur, etc.) to the input. The  phenomenon is referred to as confidence mutation, where original inputs, along with corruptions, increase the confidence of in-distribution (ID) data, while the confidence of OOD data decreases. Also, a new scoring function is proposed, Confidence aVerage (CoVer), which averages the OOD scores of different corrupted inputs and the original one.\n\nExperiments are on 2 benchmarks (1) traditional OOD - ImageNet-1K as In-Distribution (ID) and iNaturalist, SUN, Places, and Textures as OOD (2) zero-shot OOD - mixture of ImageNet-10, ImageNet-20, ImageNet-100, WaterBirds, and Spurious. The metrics used for evaluation are AUROC and FPR95.\n\nMethods exhibit better performance with the addition of CoVer.\n\nSTRENGTHS:\nThe use of corruptions to improve OOD performance is creative. The implementation seems straightforward and does not require extra datasets. Also, CoVer seems to improve every method it's applied to.\n\nThe experiments utilize similar backbones/datasets as in the literature. This is helpful when comparing with other methods.\n\nAll figures in the paper, including Fig. 1-4, are intuitive and easy to follow. The captions are detailed, and the overview (Fig. 4) makes understanding CoVer very straightforward.\n\nThe paper is well-written and clear. It effectively discusses the problem, presents the idea, and describes the experiments in a coherent manner. Additionally, all equations are clearly explained.\n\nWEAKNESSES:\nThe experimentation/idea is somewhat weak and seems more like an application rather than significant knowledge advancement. For example, CoVer is not standalone but applied to an existing method like ASH (see Table 1) or on top of a VLM, thereby leveraging the performance of other methods. Additionally, the claimed contribution of a novel scoring method doesn't seem entirely novel, as averaging is straightforward and an existing scoring function like Softmax/Energy is subsequently employed. \n\nComparisons in Table 1 with different baselines is a little unfair because CoVer makes use of \"extra\" data (corrupted images), while others simply use a single input/image.\n\nEvaluating OOD performance when using VLM/CLIP models is problematic. Such models utilize a vast amount of data which makes it near impossible to know what images/classes have been seen before and are considered \"in-distribution\". While the community continues to use them in various settings, their use with OOD problems should be a clear limitation.\n\nI find it unclear how to determine which corruption types or severity levels to use if I implemented CoVer. The results suggests variability/inconsistency without a clear guideline for decision-making.\n\nThe runtime of CoVer is not discussed. From my understanding, one would need to pass every corrupted input and extract its features, which is time consuming and uses more space.\n\nThere is no discussion of failure cases. When using corrupted inputs, I assume there are instances when CoVer labeled an ID image as OOD.\n\nSince the OOD datasets used in Table 1 differ significantly from ImageNet, it would be intriguing to observe CoVer's performance on more challenging ones such as NINCO [1] (ensures no categorical contamination) without leveraging CLIP.\n\nThe experiments do not utilize or compare with NNGuide [2] or MaxLogit [3], more recent and better performing methods than those in the paper.\n\n[1] Bitterwolf, Julian, Maximilian Müller, and Matthias Hein. \"In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Park, Jaewoo, Yoon Gyo Jung, and Andrew Beng Jin Teoh. \"Nearest Neighbor Guidance for Out-of-Distribution Detection.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[3] Hendrycks, Dan, et al. \"Scaling Out-of-Distribution Detection for Real-World Settings.\" International Conference on Machine Learning. PMLR, 2022.\n\nQUESTIONS:\nIn Table 1, why is CoVer only combined with ASH and not other methods? \n\nTable 6 in the appendix demonstrates that CoVer is integrated with other methods using different types of corruptions and severity levels for each method. This raises questions about the comparisons. Why are the types and levels of corruptions not the same across all methods?\n\nSince the paper has different ablations (various corruption types, number of dimensions/corrupted inputs, severity levels) and the experiments make an explicit choice on each one, was a validation set ever used? I did not see anything in the code or the paper, indicating choices were tuned on the test set.\n\nConsidering that other methods only use one input, how much additional runtime does CoVer add? For example, given the size of ImageNet-1K, using CoVer would necessitate extracting 6 million features (1.2 million * 5 corruptions).\n\nHow effective would CoVer be if it encountered much more difficult OOD data without leveraging VLMs/CLIP?\n\nLIMITATIONS:\nThe authors discussed limitations and broader impact in Section Appendix E.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}